add PyTorch to doc/programming guide #1653
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks. LGTM. |
hhbyyh
approved these changes
Oct 9, 2019
Closed
|
Need to backport to branch 0.6 @helenlly |
Litchilitchy
pushed a commit
that referenced
this pull request
Nov 14, 2019
* fix bn param copy (#1554) * bigdl-old-caffeloader * zoo-new-caffeloader * move convert after layer creation * fix recursion * delete unused converter * fix copy of old bn params * SessionRecommender python api and example (#1465) * session recommender python * fix release bug of AbstractInferenceModel (#1556) * fix release bug of AbstractInferenceModel * fix release bug of AbstractInferenceModel * fix bug * fix session recommender (#1558) * Pytorch training and inference for model with multiple output and multiple input (#1544) * inference with table output * add unit test * multi and unit test * remove duplicate ut * clear caching data * support multiple shape * multi input ut * release * remove empty line * update so * Session recommender document (#1559) * add document for session recommender * Bert inference perf improve (#1555) Split big Dense into several small dense in TransformerLayer Ensure bert is using MKL to do math operations. * modify run-pytests to check version before test ray (#1532) * Create .keep * update .keep path * add rl_pong example * move to rl_pong direction * remove original file * add parameter server example * add license * PEP8 checks * PEP8 checks * Add into integration test * Wrap tests into bash function * Update license * PEP8 checks * Correct syntax of rl_pong * modify run-pytests to check version before test ray * test pyspark version and spark home * add check spark_home's pyspark in case pyspark can't be found * add check version before run ray examples * change spark home * change spark home * install packages which are needed in ray examples * check error * fix error * change execution to spark-submit * change memory * change object memory to test * add atari_py dependency * remove .keep * move ray test to new files * change some ray-pip lines into function * remove rl_pong and fix parameter_server iterations * add iteration * change iterate, print info * add more info * add __init__ files * change ray to rayexample to avoid conflict and change spark-submit to python to submit tasks * renamed foreach_evaluator to foreach_worker because rllib update and rename file rllib to rllibexample * add a dedicated file for the ray test * PEP8 check fix * PEP8 check fix * remove test_split * remove --doctest-modules about ray * add time.sleep * pytorch doc and example update. (#1560) * pytorch doc * update example * remove transfomer * doc and example update * fix sample_input * check none * param update * check trace * Update Dockerfile * update powerby page (#1542) * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Make keras datasets to a package (#1562) * update * add datasets package * add license * support multi input models for nnframes (#1553) * support multi input for nnframes * update ut * add doc and unit test * doc update * scala style * create temp file with minimal permissions (#1565) * create temp file with minimal permissions * fix unit test * fix style check * Add optim method option in TFOptimizer.from_keras (#1574) * Add Polynomial Decay based on epochs (#1575) * poly on epoch * doc and ut * style * Simple TFNet and NNFrames example (#1580) * TFNet example * comments * Add session_recommender into Scala integration-test (#1561) * integration-test * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * support reading tfrecord in tfdataset (#1584) * support reading tfrecord in tfdataset * add doc and test * fix style * fix dependency * fix test * add resource file * refactor test * fix style * address comment * add test back * fix style * fix tests * fix python 3.5 * Flink example (#1586) * Restnet50 example * Resnet50 example * add README * update flink example * update Flink example * update Flink example * update Flink example * update Flink example * Update README.md * Update README.md * Update README.md * OpenVINO resnet_v1_50 python example with VNNI (#1582) * This Python resnet_v1_50 example support both fp32 and int8 OpenVINO model. Replacing fp32 model path with int8 model path will give you better performance. * VNNI: If the running platform has VNNI instruction, then using int8 quantized models can generally give you additional performance boost. * Fix openvino VNNI doc typo (#1590) * Fix ZOO_NUM_MKLTHREADS * Fxi typo * Support TensorFlow metric in TFOptimizer (#1593) * test * support stateless metric * support metric in TFOptimizer * fix tests * fix style * add tests * fix ray and add more test (#1596) * fix ray and add more test Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * modify raycontext and move test file to func Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * modify process and add sc.stop in the end Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * delete one repeat and check PEP8 Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * change file name and remove some useless code Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * rename test yarn reinit file Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * ignore test reinit raycontext Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * Save Keras-like model to pure keras or tensorflow protobuf. (#1600) * checkpoint * some update * refine api * some update * fix build fail * meet code review * style check * fix typo * fix style check * Update visualization.md (#1461) * update snapshot release for bigdl 0.9.1 (#1605) * Add Estimator Python API and Inception Example (#1597) * Using non-freezing graph for TF training (#1488) * support unfreeze training * support two version * fix * fix * fix test * fix test * fix test * fix tests * reduce test * add tests back * fix style * split tests * isolate tfpark tests * change back tfnet * add multi thread sample to minibatch (#1589) * add mt sample to minibatch * revert log level * delete some debug code * add comments * fix style check * update pom * Fix confusing scaler variable name (#1608) * Fix imageset read relative path (#1610) * fix imageset read relative path * get relative path * Update build.sh (#1618) * Fix typo in ChannelScaledNormalizer (#1619) * Fix predict batch size in anomaly_detection.py (#1614) * Fix predict batch size in anomaly_detection.py * Add `input_dir` check to avoid NonType error. * Add usage. * NNFrames: support caching training data on Disk (#1588) * disk cache for NNFrames * update doc and example * update train method * fix style * fix ut * fix ut * Enable submit scala application (#1624) * refactor scripts * fix * change docs * fix image preprocessing (#1632) * Resnet50 example * add README * update Flink example * load various images * update * solve image preprocess * add labels * fix images processing * Update ImageClassificationStreaming.scala * delete the image in resources * fix pytroch on yarn model output is NaN (#1625) * add yarn mode * fix pytroch on yarn Nan * fix * fix predict * delete debug code * move evaluate to mappartition * update resnet-finetune * update resnet-finetune * Update README.md * Update README.md * Update README.md * save TorchNet as Pytorch script module (#1564) * support save pytorch model to script * unit test * use temp folder * add import * correct evaluate * import * style fix * pytorch native code cleanup (#1642) * code clean * remove load * fix style * Support TorchNet save load with NNFrames (#1645) * nnestimator * nnestimator * Save KerasModel to pure keras or tf protobuf (#1606) * checkpoint * add unit test * revert neuralCF * revert neuralCF * some update * some update * some change * update bigdl to 0.9.1 (#1648) * add execute permission (#1651) * restructure tfpark examples (#1654) * restructure examples * fix test scripts * fix license * Fix invalid cross-device and filesystem closed issues in Streaming examples (#1656) * Fix Invalid cross-device link in image_path_writer.py. * Fix HDFS filesystem closed issue in ImagePathWriter and StreamingObjectDetection. * Modify document. * Update doc link for image classification (#1663) * Dockerfile (#1664) * remove py27 in Dockerfile * Update Dockerfile * Expose PyTorch to doc/programming guide (#1653) * Update mkdocs.yml * Update mkdocs.yml * doc for rayonspark (#1669) * update TFOptimizer doc (#1658) * update doc * address comments * add file link * fix style * Clean up dist-mac (#1666) * Clean up dist-mac. * Modify mac openvino bin path. * update Featureset apiguide (#1680) * Update featureset.md * Update featureset.md * Update featureset.md * Update FeatureSet.scala * Fix sentiment numpy issue and docker file issue (#1650) * sentiment * Add lib6 * doc * change to tfnet readme to using bash shell (#1682) * Fix transformer example and update doc (#1676) * fix and update doc * meet review * minor * fix typo * improve TFPark readme (#1684) * improve readme * fix path * change sh to bash * Add Estimator Doc (#1687) * Add Estimator Doc * Add estimator to mkdocs * Replace \ with <br> * Fix * Modify vnni openvino example doc (#1691) * Use full path to avoid calibration error. * change <mainClass> and version in pom.xml (#1667) * change <mainClass> and version in pom.xml * update bigdl version to 0.9.1 * Update release-download.md (#1698) * update doc links in master (#1700) * Update release-docs.md * Update release-docs.md * Autoencoder (#1436) * add anomaly-detection-hd * modified according to comments * Data download sccipt added * visualization optimized * minor fix * typo correction * refactor autoencoder * Update README.md * Update index.md * Update README.md * fix link (#1711) * Reduce openvino debug logs (#1713) * Replace OpenVINO related decompression output with one line log info. * Remove unnecessary `ls`. * bump to 0.7.0-S (#1719) * bump to 0.7.0-SNAPSHOT * TFNet support load saved model (#1714) * add tests * add comments * skip serialization test * fix style * fix bug (#1730) * Update release-download.md (#1721) * Upgrade pip bigdl dependency to 0.9.1 (#1709) * upgrade * update doc * Update RayOnSpark doc (#1732) * update ray doc and pip doc * update * update Spark Summit talks and slides in powerby and presentation (#1734) * Update presentations.md * Update powered-by.md * Update presentations.md (#1737) * Minor fix for detecting sc is local (#1735) * fix sc local * update * Update powered-by.md (#1739) * Add tfpark example test (#1736) * add tfpark example test * fix path * add pip * Update powered-by.md (#1741) * fix script (#1740) * Support iterate a dataset in sequential order when training (#1743) * support iterate a dataset in sequential order when training add unit test fix style * unpersist * fix bug * [Bug Fix] Fix estimator python validation method argument type bug (#1746) * fix estimator validation method argument type bug * fix style * handle null * fix estimator NPE problem (#1750) * add load TF saved model as TFNet inference model (#1745) * Refine OpenVINO VNNI Perf (#1726) * Add pure Scala Perf for Resnet. * Refine document and log. * TFoptimizer use analytics-zoo estimator (#1749) * tfoptimizer use analytics-zoo estimator * fix style * fix bug * fix tests and add documents * fix style * fix test * fix example * fix test * fix tests * fix tests * always repartition * Fix CategoricalCrossEntropy may NaN. (#1705) * clip in categoricalCrossEntropy * add unit test * clean up * remove bigdl engine init (#1753) * Update powered-by.md (#1756) * update bigdl version to 0.10.0 (#1757) * update ut for inference model to test no bigdl properties (#1758) * local estimator and lenet resnet example (#1725) * local estimator and lenet resnet example * add params to examples * add params to examples * fix style issue * fix typo * change to abstract module * fix tfnet bug and add transfer learning example * change val to args * update * update * add scala doc * add scala doc * Add files via upload * Delete boston_housing.py * scala serving v1 * loader * docker of serving; * serving folder structure and config yaml * struture change * deleted pom * pom * Update README.md * Update README.md * mod * yaml parsing and bash rename * final mod of forward * add img cls and obj detect * Update README.md * switch to inference model * naming loader to helper * naming * Inf Model Concur, Dockerfile order, log * params limit * result format and api instructions and dummy label map * multiple model checking * Update README.md * Update README.md * dynamic load foreachBatch * revert pom.xml * exception handling fix * style fix * delete those not used for this version * style fix * license * style * style fix; * style * tensorboard-inference; * inference throughput * inference-tensorboard * fix inference bug * 1244 * pom for serving * blas support
Litchilitchy
pushed a commit
that referenced
this pull request
Nov 22, 2019
* fix bn param copy (#1554) * bigdl-old-caffeloader * zoo-new-caffeloader * move convert after layer creation * fix recursion * delete unused converter * fix copy of old bn params * SessionRecommender python api and example (#1465) * session recommender python * fix release bug of AbstractInferenceModel (#1556) * fix release bug of AbstractInferenceModel * fix release bug of AbstractInferenceModel * fix bug * fix session recommender (#1558) * Pytorch training and inference for model with multiple output and multiple input (#1544) * inference with table output * add unit test * multi and unit test * remove duplicate ut * clear caching data * support multiple shape * multi input ut * release * remove empty line * update so * Session recommender document (#1559) * add document for session recommender * Bert inference perf improve (#1555) Split big Dense into several small dense in TransformerLayer Ensure bert is using MKL to do math operations. * modify run-pytests to check version before test ray (#1532) * Create .keep * update .keep path * add rl_pong example * move to rl_pong direction * remove original file * add parameter server example * add license * PEP8 checks * PEP8 checks * Add into integration test * Wrap tests into bash function * Update license * PEP8 checks * Correct syntax of rl_pong * modify run-pytests to check version before test ray * test pyspark version and spark home * add check spark_home's pyspark in case pyspark can't be found * add check version before run ray examples * change spark home * change spark home * install packages which are needed in ray examples * check error * fix error * change execution to spark-submit * change memory * change object memory to test * add atari_py dependency * remove .keep * move ray test to new files * change some ray-pip lines into function * remove rl_pong and fix parameter_server iterations * add iteration * change iterate, print info * add more info * add __init__ files * change ray to rayexample to avoid conflict and change spark-submit to python to submit tasks * renamed foreach_evaluator to foreach_worker because rllib update and rename file rllib to rllibexample * add a dedicated file for the ray test * PEP8 check fix * PEP8 check fix * remove test_split * remove --doctest-modules about ray * add time.sleep * pytorch doc and example update. (#1560) * pytorch doc * update example * remove transfomer * doc and example update * fix sample_input * check none * param update * check trace * Update Dockerfile * update powerby page (#1542) * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Update powered-by.md * Make keras datasets to a package (#1562) * update * add datasets package * add license * support multi input models for nnframes (#1553) * support multi input for nnframes * update ut * add doc and unit test * doc update * scala style * create temp file with minimal permissions (#1565) * create temp file with minimal permissions * fix unit test * fix style check * Add optim method option in TFOptimizer.from_keras (#1574) * Add Polynomial Decay based on epochs (#1575) * poly on epoch * doc and ut * style * Simple TFNet and NNFrames example (#1580) * TFNet example * comments * Add session_recommender into Scala integration-test (#1561) * integration-test * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * Update integration-test.robot * support reading tfrecord in tfdataset (#1584) * support reading tfrecord in tfdataset * add doc and test * fix style * fix dependency * fix test * add resource file * refactor test * fix style * address comment * add test back * fix style * fix tests * fix python 3.5 * Flink example (#1586) * Restnet50 example * Resnet50 example * add README * update flink example * update Flink example * update Flink example * update Flink example * update Flink example * Update README.md * Update README.md * Update README.md * OpenVINO resnet_v1_50 python example with VNNI (#1582) * This Python resnet_v1_50 example support both fp32 and int8 OpenVINO model. Replacing fp32 model path with int8 model path will give you better performance. * VNNI: If the running platform has VNNI instruction, then using int8 quantized models can generally give you additional performance boost. * Fix openvino VNNI doc typo (#1590) * Fix ZOO_NUM_MKLTHREADS * Fxi typo * Support TensorFlow metric in TFOptimizer (#1593) * test * support stateless metric * support metric in TFOptimizer * fix tests * fix style * add tests * fix ray and add more test (#1596) * fix ray and add more test Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * modify raycontext and move test file to func Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * modify process and add sc.stop in the end Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * delete one repeat and check PEP8 Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * change file name and remove some useless code Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * rename test yarn reinit file Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * ignore test reinit raycontext Signed-off-by: Jieru Hong <hongjieru30@gmail.com> * Save Keras-like model to pure keras or tensorflow protobuf. (#1600) * checkpoint * some update * refine api * some update * fix build fail * meet code review * style check * fix typo * fix style check * Update visualization.md (#1461) * update snapshot release for bigdl 0.9.1 (#1605) * Add Estimator Python API and Inception Example (#1597) * Using non-freezing graph for TF training (#1488) * support unfreeze training * support two version * fix * fix * fix test * fix test * fix test * fix tests * reduce test * add tests back * fix style * split tests * isolate tfpark tests * change back tfnet * add multi thread sample to minibatch (#1589) * add mt sample to minibatch * revert log level * delete some debug code * add comments * fix style check * update pom * Fix confusing scaler variable name (#1608) * Fix imageset read relative path (#1610) * fix imageset read relative path * get relative path * Update build.sh (#1618) * Fix typo in ChannelScaledNormalizer (#1619) * Fix predict batch size in anomaly_detection.py (#1614) * Fix predict batch size in anomaly_detection.py * Add `input_dir` check to avoid NonType error. * Add usage. * NNFrames: support caching training data on Disk (#1588) * disk cache for NNFrames * update doc and example * update train method * fix style * fix ut * fix ut * Enable submit scala application (#1624) * refactor scripts * fix * change docs * fix image preprocessing (#1632) * Resnet50 example * add README * update Flink example * load various images * update * solve image preprocess * add labels * fix images processing * Update ImageClassificationStreaming.scala * delete the image in resources * fix pytroch on yarn model output is NaN (#1625) * add yarn mode * fix pytroch on yarn Nan * fix * fix predict * delete debug code * move evaluate to mappartition * update resnet-finetune * update resnet-finetune * Update README.md * Update README.md * Update README.md * save TorchNet as Pytorch script module (#1564) * support save pytorch model to script * unit test * use temp folder * add import * correct evaluate * import * style fix * pytorch native code cleanup (#1642) * code clean * remove load * fix style * Support TorchNet save load with NNFrames (#1645) * nnestimator * nnestimator * Save KerasModel to pure keras or tf protobuf (#1606) * checkpoint * add unit test * revert neuralCF * revert neuralCF * some update * some update * some change * update bigdl to 0.9.1 (#1648) * add execute permission (#1651) * restructure tfpark examples (#1654) * restructure examples * fix test scripts * fix license * Fix invalid cross-device and filesystem closed issues in Streaming examples (#1656) * Fix Invalid cross-device link in image_path_writer.py. * Fix HDFS filesystem closed issue in ImagePathWriter and StreamingObjectDetection. * Modify document. * Update doc link for image classification (#1663) * Dockerfile (#1664) * remove py27 in Dockerfile * Update Dockerfile * Expose PyTorch to doc/programming guide (#1653) * Update mkdocs.yml * Update mkdocs.yml * doc for rayonspark (#1669) * update TFOptimizer doc (#1658) * update doc * address comments * add file link * fix style * Clean up dist-mac (#1666) * Clean up dist-mac. * Modify mac openvino bin path. * update Featureset apiguide (#1680) * Update featureset.md * Update featureset.md * Update featureset.md * Update FeatureSet.scala * Fix sentiment numpy issue and docker file issue (#1650) * sentiment * Add lib6 * doc * change to tfnet readme to using bash shell (#1682) * Fix transformer example and update doc (#1676) * fix and update doc * meet review * minor * fix typo * improve TFPark readme (#1684) * improve readme * fix path * change sh to bash * Add Estimator Doc (#1687) * Add Estimator Doc * Add estimator to mkdocs * Replace \ with <br> * Fix * Modify vnni openvino example doc (#1691) * Use full path to avoid calibration error. * change <mainClass> and version in pom.xml (#1667) * change <mainClass> and version in pom.xml * update bigdl version to 0.9.1 * Update release-download.md (#1698) * update doc links in master (#1700) * Update release-docs.md * Update release-docs.md * Autoencoder (#1436) * add anomaly-detection-hd * modified according to comments * Data download sccipt added * visualization optimized * minor fix * typo correction * refactor autoencoder * Update README.md * Update index.md * Update README.md * fix link (#1711) * Reduce openvino debug logs (#1713) * Replace OpenVINO related decompression output with one line log info. * Remove unnecessary `ls`. * bump to 0.7.0-S (#1719) * bump to 0.7.0-SNAPSHOT * TFNet support load saved model (#1714) * add tests * add comments * skip serialization test * fix style * fix bug (#1730) * Update release-download.md (#1721) * Upgrade pip bigdl dependency to 0.9.1 (#1709) * upgrade * update doc * Update RayOnSpark doc (#1732) * update ray doc and pip doc * update * update Spark Summit talks and slides in powerby and presentation (#1734) * Update presentations.md * Update powered-by.md * Update presentations.md (#1737) * Minor fix for detecting sc is local (#1735) * fix sc local * update * Update powered-by.md (#1739) * Add tfpark example test (#1736) * add tfpark example test * fix path * add pip * Update powered-by.md (#1741) * fix script (#1740) * Support iterate a dataset in sequential order when training (#1743) * support iterate a dataset in sequential order when training add unit test fix style * unpersist * fix bug * [Bug Fix] Fix estimator python validation method argument type bug (#1746) * fix estimator validation method argument type bug * fix style * handle null * fix estimator NPE problem (#1750) * add load TF saved model as TFNet inference model (#1745) * Refine OpenVINO VNNI Perf (#1726) * Add pure Scala Perf for Resnet. * Refine document and log. * TFoptimizer use analytics-zoo estimator (#1749) * tfoptimizer use analytics-zoo estimator * fix style * fix bug * fix tests and add documents * fix style * fix test * fix example * fix test * fix tests * fix tests * always repartition * Fix CategoricalCrossEntropy may NaN. (#1705) * clip in categoricalCrossEntropy * add unit test * clean up * remove bigdl engine init (#1753) * Update powered-by.md (#1756) * update bigdl version to 0.10.0 (#1757) * update ut for inference model to test no bigdl properties (#1758) * local estimator and lenet resnet example (#1725) * local estimator and lenet resnet example * add params to examples * add params to examples * fix style issue * fix typo * change to abstract module * fix tfnet bug and add transfer learning example * change val to args * update * update * add scala doc * add scala doc * inference-tensorboard
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add pytorch to doc/programming guide