Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix stop_orca_context #2878

Merged
merged 1 commit into from
Sep 17, 2020
Merged

Fix stop_orca_context #2878

merged 1 commit into from
Sep 17, 2020

Conversation

hkvision
Copy link
Contributor

Since stop_orca_context is added at exit time, if the user program finishes successfully and they manually call stop_orca_context, stop_orca_context would be called for one more time at exit time. So we shouldn't call sc.getOrCreate() all the time as this would create a new sc.
For yarn mode, it would through the following error:

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.FileNotFoundException: File file:/tmp/tmptoj600uh/python_env.tar.gz#python_env does not exist

As the tmp conda file is removed when sc is created for the first time.

@hkvision
Copy link
Contributor Author

jenkins passed http://10.239.47.210:18888/job/ZOO-PR-Validation/4239/

@hkvision hkvision merged commit c7903ff into intel-analytics:master Sep 17, 2020
@hkvision hkvision deleted the fix-stop-orca branch September 17, 2020 05:29
shanyu-sys pushed a commit to shanyu-sys/analytics-zoo that referenced this pull request Sep 16, 2021
@hkvision
Fix stop_orca_context being called twice (intel-analytics#2878)
b8f8e15
shanyu-sys added a commit that referenced this pull request Sep 22, 2021
@shanyu-sys @cyita
@hkvision @yangw1234 @qiuxin2012 @jenniew @dding3 @Le-Zheng
Migrate orca context (#4768)
8776fa5 
* Orca PyTorch Estimator load data once (#2669)

* load once

* fix

* fix

* remove num_steps

* remove iter

* meet comments

* wrap once

* style fix

* Add Orca Overview and Context Doc (#2748)

* add docs

* minor

* Add init_orca_context (#2774)

* initial imple

* update

* meet review

* review and style

* remove stopped

* add doc

* minor

* move import

* fix mxnet

* remove

* Update UTs and examples with init_orca_context (#2787)

* update unit tests

* minor

* update

* update mxnet

* move barrier

* fix mxnet

* update

* bug fix

* update

* update test

* update mxnet example

* update mxnet

* minor

* minor

* minor

* update examples

* move ray import dependencies

* readme

* minor

* bug fix

* remove default

* Add website doc for init_orca_context (#2822)

* Support numa binding in init_spark_standalone (#2847)

* support numa binding in init_spark_standalone

* add doc and add to orca context

* address comments

* address comments

* update scripts

* hyperthreading

* fix

* shutdown hook (#2853)

* Support RayOnSpark for k8s and add docs (#2836)

* support ray on k8s

* add to init orca context

* style

* minor

* minor

* ut

* Fix stop_orca_context being called twice (#2878)

* Update website doc of init orca context (#2879)

* update doc

* update

* update Torch example (#3022)

* update example

* Update README.md

* update resnet_finetune.py

* delete some file

* Create README.md

* Update README.md

* Update README.md

* Update README.md

* add detect conda env name

* some change

* update run example scripte

* update init orca context

* attempt to fix ray memory (#3205)

* attempt to fix ray memory

* exclude webui

* Update doc for orca context (#3287)

* update

* add back

* update

* update

* minor

* meet review

* fix typo

* Add memory type support for Orca tf estimator (#3280)

* add mem type for dataframe dataset

* add ZooTrigger support

* update mem type

* add test

* update

* update mem type and orca context

* update orca context docs

* update orca context with comments

* fix style

* Add get_spark_session in OrcaContext (#3520)

* add and modify

* style

* add shard size to dataframe_to_xshards (#3491)

* add shard size to dataframe_to_xshards

* add ut and change default shard_size as None

* add shard size in orca context and fix style

* add ut in pytorch estimator and tf estimator

* move shard_size to internal use

* fix

* Add support for non-barrier mode to launch ray (#4014)

* add support for non-barrier mode

* fix style

* meet review

* meet review

* move barrier mode to zoocontext

* bug fix

* modify

* update

* remove driver cores (#4169)

* Add OrcaContext and make spark default read file backend (#2593)

* orca context

* handle error

* enrich error msg

* meet review

* move log output

* style

* meet review

* add zoocontext

* fix ut

* change import

Co-authored-by: Yina Chen <33650826+cyita@users.noreply.github.com>
Co-authored-by: Kai Huang <huangkaivision@gmail.com>
Co-authored-by: Yang Wang <yang3.wang@intel.com>
Co-authored-by: Xin Qiu <qiuxin2012@users.noreply.github.com>
Co-authored-by: jenniew <jenniewang123@gmail.com>
Co-authored-by: dding3 <ding.ding@intel.com>
Co-authored-by: Le-Zheng <30695225+Le-Zheng@users.noreply.github.com>
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 22, 2021
@hkvision
Fix stop_orca_context being called twice (intel-analytics#2878)
5bdbc8c
Le-Zheng added a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 22, 2021
@shanyu-sys @cyita
@hkvision @yangw1234 @qiuxin2012 @jenniew @dding3 @Le-Zheng
Migrate orca context (intel-analytics#4768)
a76d02c 
* Orca PyTorch Estimator load data once (intel-analytics#2669)

* load once

* fix

* fix

* remove num_steps

* remove iter

* meet comments

* wrap once

* style fix

* Add Orca Overview and Context Doc (intel-analytics#2748)

* add docs

* minor

* Add init_orca_context (intel-analytics#2774)

* initial imple

* update

* meet review

* review and style

* remove stopped

* add doc

* minor

* move import

* fix mxnet

* remove

* Update UTs and examples with init_orca_context (intel-analytics#2787)

* update unit tests

* minor

* update

* update mxnet

* move barrier

* fix mxnet

* update

* bug fix

* update

* update test

* update mxnet example

* update mxnet

* minor

* minor

* minor

* update examples

* move ray import dependencies

* readme

* minor

* bug fix

* remove default

* Add website doc for init_orca_context (intel-analytics#2822)

* Support numa binding in init_spark_standalone (intel-analytics#2847)

* support numa binding in init_spark_standalone

* add doc and add to orca context

* address comments

* address comments

* update scripts

* hyperthreading

* fix

* shutdown hook (intel-analytics#2853)

* Support RayOnSpark for k8s and add docs (intel-analytics#2836)

* support ray on k8s

* add to init orca context

* style

* minor

* minor

* ut

* Fix stop_orca_context being called twice (intel-analytics#2878)

* Update website doc of init orca context (intel-analytics#2879)

* update doc

* update

* update Torch example (intel-analytics#3022)

* update example

* Update README.md

* update resnet_finetune.py

* delete some file

* Create README.md

* Update README.md

* Update README.md

* Update README.md

* add detect conda env name

* some change

* update run example scripte

* update init orca context

* attempt to fix ray memory (intel-analytics#3205)

* attempt to fix ray memory

* exclude webui

* Update doc for orca context (intel-analytics#3287)

* update

* add back

* update

* update

* minor

* meet review

* fix typo

* Add memory type support for Orca tf estimator (intel-analytics#3280)

* add mem type for dataframe dataset

* add ZooTrigger support

* update mem type

* add test

* update

* update mem type and orca context

* update orca context docs

* update orca context with comments

* fix style

* Add get_spark_session in OrcaContext (intel-analytics#3520)

* add and modify

* style

* add shard size to dataframe_to_xshards (intel-analytics#3491)

* add shard size to dataframe_to_xshards

* add ut and change default shard_size as None

* add shard size in orca context and fix style

* add ut in pytorch estimator and tf estimator

* move shard_size to internal use

* fix

* Add support for non-barrier mode to launch ray (intel-analytics#4014)

* add support for non-barrier mode

* fix style

* meet review

* meet review

* move barrier mode to zoocontext

* bug fix

* modify

* update

* remove driver cores (intel-analytics#4169)

* Add OrcaContext and make spark default read file backend (intel-analytics#2593)

* orca context

* handle error

* enrich error msg

* meet review

* move log output

* style

* meet review

* add zoocontext

* fix ut

* change import

Co-authored-by: Yina Chen <33650826+cyita@users.noreply.github.com>
Co-authored-by: Kai Huang <huangkaivision@gmail.com>
Co-authored-by: Yang Wang <yang3.wang@intel.com>
Co-authored-by: Xin Qiu <qiuxin2012@users.noreply.github.com>
Co-authored-by: jenniew <jenniewang123@gmail.com>
Co-authored-by: dding3 <ding.ding@intel.com>
Co-authored-by: Le-Zheng <30695225+Le-Zheng@users.noreply.github.com>
dding3 added a commit to dding3/analytics-zoo that referenced this pull request Oct 4, 2021
@shanyu-sys @cyita
@hkvision @yangw1234 @qiuxin2012 @jenniew @dding3 @Le-Zheng
Migrate orca context (intel-analytics#4768)
a3a6099 
* Orca PyTorch Estimator load data once (intel-analytics#2669)

* load once

* fix

* fix

* remove num_steps

* remove iter

* meet comments

* wrap once

* style fix

* Add Orca Overview and Context Doc (intel-analytics#2748)

* add docs

* minor

* Add init_orca_context (intel-analytics#2774)

* initial imple

* update

* meet review

* review and style

* remove stopped

* add doc

* minor

* move import

* fix mxnet

* remove

* Update UTs and examples with init_orca_context (intel-analytics#2787)

* update unit tests

* minor

* update

* update mxnet

* move barrier

* fix mxnet

* update

* bug fix

* update

* update test

* update mxnet example

* update mxnet

* minor

* minor

* minor

* update examples

* move ray import dependencies

* readme

* minor

* bug fix

* remove default

* Add website doc for init_orca_context (intel-analytics#2822)

* Support numa binding in init_spark_standalone (intel-analytics#2847)

* support numa binding in init_spark_standalone

* add doc and add to orca context

* address comments

* address comments

* update scripts

* hyperthreading

* fix

* shutdown hook (intel-analytics#2853)

* Support RayOnSpark for k8s and add docs (intel-analytics#2836)

* support ray on k8s

* add to init orca context

* style

* minor

* minor

* ut

* Fix stop_orca_context being called twice (intel-analytics#2878)

* Update website doc of init orca context (intel-analytics#2879)

* update doc

* update

* update Torch example (intel-analytics#3022)

* update example

* Update README.md

* update resnet_finetune.py

* delete some file

* Create README.md

* Update README.md

* Update README.md

* Update README.md

* add detect conda env name

* some change

* update run example scripte

* update init orca context

* attempt to fix ray memory (intel-analytics#3205)

* attempt to fix ray memory

* exclude webui

* Update doc for orca context (intel-analytics#3287)

* update

* add back

* update

* update

* minor

* meet review

* fix typo

* Add memory type support for Orca tf estimator (intel-analytics#3280)

* add mem type for dataframe dataset

* add ZooTrigger support

* update mem type

* add test

* update

* update mem type and orca context

* update orca context docs

* update orca context with comments

* fix style

* Add get_spark_session in OrcaContext (intel-analytics#3520)

* add and modify

* style

* add shard size to dataframe_to_xshards (intel-analytics#3491)

* add shard size to dataframe_to_xshards

* add ut and change default shard_size as None

* add shard size in orca context and fix style

* add ut in pytorch estimator and tf estimator

* move shard_size to internal use

* fix

* Add support for non-barrier mode to launch ray (intel-analytics#4014)

* add support for non-barrier mode

* fix style

* meet review

* meet review

* move barrier mode to zoocontext

* bug fix

* modify

* update

* remove driver cores (intel-analytics#4169)

* Add OrcaContext and make spark default read file backend (intel-analytics#2593)

* orca context

* handle error

* enrich error msg

* meet review

* move log output

* style

* meet review

* add zoocontext

* fix ut

* change import

Co-authored-by: Yina Chen <33650826+cyita@users.noreply.github.com>
Co-authored-by: Kai Huang <huangkaivision@gmail.com>
Co-authored-by: Yang Wang <yang3.wang@intel.com>
Co-authored-by: Xin Qiu <qiuxin2012@users.noreply.github.com>
Co-authored-by: jenniew <jenniewang123@gmail.com>
Co-authored-by: dding3 <ding.ding@intel.com>
Co-authored-by: Le-Zheng <30695225+Le-Zheng@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yangw1234 yangw1234 Awaiting requested review from yangw1234

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@hkvision