Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Feature: Spark mixed mode support #350

Merged
merged 10 commits into from
Feb 7, 2018
Merged

Feature: Spark mixed mode support #350

merged 10 commits into from
Feb 7, 2018

Conversation

jafreck
Copy link
Member

@jafreck jafreck commented Jan 24, 2018

Fix #294
Fix #307
Fix #275

@jafreck jafreck mentioned this pull request Jan 24, 2018
Closed
@jafreck jafreck added this to the v0.5.2 milestone Feb 1, 2018
@jafreck jafreck requested a review from paselem February 6, 2018 21:08
paselem
paselem reviewed Feb 7, 2018
View reviewed changes
cli/config.py Outdated
"You must configure a VNET to use AZTK in mixed mode (dedicated and low priority nodes). Set the VNET's subnet_id in your cluster.yaml.")

# ensure spark_client is built with AAD if using mixed mode
if not spark_client.secrets_config.service_principal.tenant_id and self.mixed_mode:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, since AAD is a requirement for subnets, then I would raise this error first.

paselem
paselem reviewed Feb 7, 2018
View reviewed changes
cli/spark/endpoints/cluster/cluster_create.py Outdated
@@ -128,6 +125,7 @@ def print_cluster_conf(cluster_conf):
cluster_conf.size + cluster_conf.size_low_pri)
log.info("> dedicated: %s", cluster_conf.size)
log.info("> low priority: %s", cluster_conf.size_low_pri)
log.info("mixed mode: %s", cluster_conf.mixed_mode)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I would expose this. I don't think it really adds any value to the user.

paselem
paselem suggested changes Feb 7, 2018
View reviewed changes
Copy link
Contributor

@paselem paselem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see it here, but we should also expose the type of VM when a user does 'aztk spark cluster get --id cluster'. Each dedicated VM should be labelled as such (same with low pri).

@jafreck
Copy link
Member Author

jafreck commented Feb 7, 2018

Looks like this now, what do you think @paselem?

Cluster         sst7
------------------------------------------
State:          active
Node Size:      standard_f2
Nodes:          2
| Dedicated:    2
| Low priority: 0

|               Nodes                |        State        |        IP:Port       | Dedicated  |  Master  |
|------------------------------------|---------------------|----------------------|------------|----------|
| tvm-2576887678_1-20180207t010308z  |        idle         |   40.79.57.90:50000  |     *      |          |
| tvm-2576887678_2-20180207t010308z  |        idle         |   40.79.57.90:50001  |     *      |    *     |

paselem
paselem approved these changes Feb 7, 2018
View reviewed changes
Copy link
Contributor

@paselem paselem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@jafreck jafreck merged commit 748a126 into Azure:master Feb 7, 2018
jafreck added a commit that referenced this pull request Feb 23, 2018
@jafreck
Update stable to v0.6.0 (#418)
9489301 
* Feature: on node user creation (#303)

* client side on node user creation

* start create user on node implementation

* fix on node user creation

* remove debug statements

* remove commented code

* line too long

* fix spinner password prompt ui bug

* set wait to false by default, formatting

* encrypt password on client, decrypt on node

* update docs, log warning if password used

* Fix list-apps crash (#364)

* Allow submitting jobs into a VNET (#365)

* Add subnet_id to job submission cluster config

* add some docs

* Feature: Spark mixed mode support (#350)

* add support for aad creds for storage on node

* add mixed mode support

* add docs

* switch error order

* add dedicated to get_cluster

* remove mixed mode in print_cluster_conf

* Feature: spark init docker repo customization (#358)

* customize docker_repo based on init args

* whitespace

* add some docs

* r-base to r

* case insensitive r flag, typo fix

* Bug: Load default Jars for job submission CLI (#367)

* load jars in .aztk/ by default

* rewrite loading config files

* Feature: Cluster Run and Copy (#304)

* start implementation of cluster run

* fix cluster_run

* start debug sequential user add and delete

* parallelize user creation and deletion, start implementation of cluster scp

* continue cluster_scp implementation

* debug statements, disconnect error: permission denied

* untesteed parakimo implementation of clus_run

* continue debugging user creation bug

* fix bug with pool user creation, start concurrent implementation

* start fix of paramiko cluster_run and cluster_copy

* working paramiko cluster_run implementation, start cluster_scp

* fix cluster_scp command

* update requirements, rename cluster_run function

* remove unused shell functions

* parallelize run and scp, add container_name, create logs wrapper

* change scp to copy, clean up

* sort imports

* remove asyncssh from node requirements

* remove old import

* remove bad error handling

* make cluster user management methods private

* remove comment

* remove accidental commit

* fix merge, move delete to finally clause

* add docs

* formatting

* Feature: Refactor cluster config to use ClusterConfiguration model (#343)

* Bug: fix core-site.xml typo (#378)

* fix typo

* crlf->lf

* Bug: fix regex for is_gpu_enabled (#380)

* fix regex for is_gpu_enabled

* crlf->lf

* Bug: spark SDK example fix (#383)

* start fix sdk

* fix sdk example

* crlf->lf

* Fix: Custom scripts not read from cluster.yaml (#388)

* Feature: spark shuffle service (#374)

* start shuffle service by default

* whitespace, delete misplaced file

* crlf->lf

* crlf->lf

* move spark scratch space off os drive

* Feature: enable dynamic allocation by default (#386)

* Bug: stop using mutable default parameters (#392)

* Bug: always upload spark job logs errors (#395)

* Bug: spark submit upload error log type error (#397)

* Bug: Spark Job list apps exit code 0 (#396)

* Bug: fix spark-submit cores args (#399)

* Fix: Trying to add user before master is ready show better error (#402)

* Bug: move spark.local.dir to location usable by rstudioserver (#407)

* Feature: SDK support for file-like configuration objects (#373)

* add support for filelike objects for conifguration files

* fix custom scripts

* remove os.pathlike

* merge error

* Feature: Basic Cluster and Job Submission SDK Tests (#344)

* add initial cluster tests

* add cluster tests, add simple job submission test scenario

* sort imports

* fix job tests

* fix job tests

* remove pytest from travis build

* cluster per test, parallel pytest plugin

* delete cluster after tests, wait until deleted

* fix bugs

* catch right error, change cluster_id to base_cluster_id

* fix test name

* fixes

*  move tests to intregration_tests dir

* update travis to run non-integration tests

* directory structure, decoupled job tests

* fix job tests, issue with submit_job

* fix bug

* add test docs

* add cluster and job delete to finally clause

* Feature: Spark add worker on master option (#415)

* Add worker_on_master to ClusterConfiguration

* add worker_on_master to JobConfiguration

* Feature: task affinity to master node (#413)

* Release: v0.6.0 (#416)

* update changelog and version

* underscores to stars
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@paselem paselem paselem approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jafreck @paselem