Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Bug: Load default Jars for job submission CLI #367

Merged
merged 3 commits into from
Feb 7, 2018
Merged

Bug: Load default Jars for job submission CLI #367

merged 3 commits into from
Feb 7, 2018

Conversation

jafreck
Copy link
Member

@jafreck jafreck commented Feb 3, 2018

Fix #366

@jafreck jafreck added this to the v0.5.2 milestone Feb 3, 2018
paselem
paselem reviewed Feb 7, 2018
View reviewed changes
cli/config.py Outdated
spark_defaults_conf = get_file_if_exists('spark-defaults.conf', False)
spark_env_sh = get_file_if_exists('spark-env.sh', False)
core_site_xml = get_file_if_exists('core-site.xml', False)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand why you're making the same 3 calls with False, then True - seems kind of strange to me. Is this mean to override the defaults?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed this is poorly written, but what it's doing is getting the global configuration file, and then overwriting it with the local configuration file if it exists. I will rewrite this before I merge -- too clunky as is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sounds good.

paselem
paselem reviewed Feb 7, 2018
View reviewed changes
Copy link
Contributor

@paselem paselem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, but has one strange bit of code in there I don't fully understand.

@jafreck jafreck changed the title Load default Jars for job submission CLI Bug: Load default Jars for job submission CLI Feb 7, 2018
@jafreck jafreck merged commit af8d037 into Azure:master Feb 7, 2018
jafreck added a commit that referenced this pull request Feb 23, 2018
@jafreck
Update stable to v0.6.0 (#418)
9489301 
* Feature: on node user creation (#303)

* client side on node user creation

* start create user on node implementation

* fix on node user creation

* remove debug statements

* remove commented code

* line too long

* fix spinner password prompt ui bug

* set wait to false by default, formatting

* encrypt password on client, decrypt on node

* update docs, log warning if password used

* Fix list-apps crash (#364)

* Allow submitting jobs into a VNET (#365)

* Add subnet_id to job submission cluster config

* add some docs

* Feature: Spark mixed mode support (#350)

* add support for aad creds for storage on node

* add mixed mode support

* add docs

* switch error order

* add dedicated to get_cluster

* remove mixed mode in print_cluster_conf

* Feature: spark init docker repo customization (#358)

* customize docker_repo based on init args

* whitespace

* add some docs

* r-base to r

* case insensitive r flag, typo fix

* Bug: Load default Jars for job submission CLI (#367)

* load jars in .aztk/ by default

* rewrite loading config files

* Feature: Cluster Run and Copy (#304)

* start implementation of cluster run

* fix cluster_run

* start debug sequential user add and delete

* parallelize user creation and deletion, start implementation of cluster scp

* continue cluster_scp implementation

* debug statements, disconnect error: permission denied

* untesteed parakimo implementation of clus_run

* continue debugging user creation bug

* fix bug with pool user creation, start concurrent implementation

* start fix of paramiko cluster_run and cluster_copy

* working paramiko cluster_run implementation, start cluster_scp

* fix cluster_scp command

* update requirements, rename cluster_run function

* remove unused shell functions

* parallelize run and scp, add container_name, create logs wrapper

* change scp to copy, clean up

* sort imports

* remove asyncssh from node requirements

* remove old import

* remove bad error handling

* make cluster user management methods private

* remove comment

* remove accidental commit

* fix merge, move delete to finally clause

* add docs

* formatting

* Feature: Refactor cluster config to use ClusterConfiguration model (#343)

* Bug: fix core-site.xml typo (#378)

* fix typo

* crlf->lf

* Bug: fix regex for is_gpu_enabled (#380)

* fix regex for is_gpu_enabled

* crlf->lf

* Bug: spark SDK example fix (#383)

* start fix sdk

* fix sdk example

* crlf->lf

* Fix: Custom scripts not read from cluster.yaml (#388)

* Feature: spark shuffle service (#374)

* start shuffle service by default

* whitespace, delete misplaced file

* crlf->lf

* crlf->lf

* move spark scratch space off os drive

* Feature: enable dynamic allocation by default (#386)

* Bug: stop using mutable default parameters (#392)

* Bug: always upload spark job logs errors (#395)

* Bug: spark submit upload error log type error (#397)

* Bug: Spark Job list apps exit code 0 (#396)

* Bug: fix spark-submit cores args (#399)

* Fix: Trying to add user before master is ready show better error (#402)

* Bug: move spark.local.dir to location usable by rstudioserver (#407)

* Feature: SDK support for file-like configuration objects (#373)

* add support for filelike objects for conifguration files

* fix custom scripts

* remove os.pathlike

* merge error

* Feature: Basic Cluster and Job Submission SDK Tests (#344)

* add initial cluster tests

* add cluster tests, add simple job submission test scenario

* sort imports

* fix job tests

* fix job tests

* remove pytest from travis build

* cluster per test, parallel pytest plugin

* delete cluster after tests, wait until deleted

* fix bugs

* catch right error, change cluster_id to base_cluster_id

* fix test name

* fixes

*  move tests to intregration_tests dir

* update travis to run non-integration tests

* directory structure, decoupled job tests

* fix job tests, issue with submit_job

* fix bug

* add test docs

* add cluster and job delete to finally clause

* Feature: Spark add worker on master option (#415)

* Add worker_on_master to ClusterConfiguration

* add worker_on_master to JobConfiguration

* Feature: task affinity to master node (#413)

* Release: v0.6.0 (#416)

* update changelog and version

* underscores to stars
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@paselem paselem paselem approved these changes

@timotheeguerin timotheeguerin Awaiting requested review from timotheeguerin

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jafreck @paselem