Feature: on node user creation #303

jafreck · 2017-12-23T01:48:57Z

paselem · 2017-12-24T13:15:23Z

aztk/spark/client.py

-            zip_resource_files = upload_node_scripts.zip_scripts(self.blob_client, cluster_conf.custom_scripts, cluster_conf.spark_configuration)
+            if cluster_conf.user_configuration:
+                user_conf = yaml.dump({'username': cluster_conf.user_configuration.username,
+                                       'password': cluster_conf.user_configuration.password,


We may want to omit password until we support encryption.

If we disable password here that means that we would no longer support adding a user with password at cluster creation time. That's a potentially breaking change, so maybe this should PR should wait for or include the encryption feature.

If the goal is just to not have the password in cleartext in user.yaml, we could also hash it, and create the user using the hash.

That would break ssh'ing outside of AZTK, though.

paselem · 2017-12-24T13:15:39Z

aztk/spark/client.py

+                                       'cluster_id': cluster_conf.cluster_id})
+            else:
+                user_conf = None
+            zip_resource_files = upload_node_scripts.zip_scripts(self.blob_client, cluster_conf.custom_scripts, cluster_conf.spark_configuration, user_conf)


nitpick: line too long

paselem · 2017-12-24T13:17:09Z

aztk/spark/utils/upload_node_scripts.py

@@ -114,5 +114,9 @@ def zip_scripts(blob_client, custom_scripts, spark_conf):
            for jar in spark_conf.jars:
                zipf = __add_file_to_zip(zipf, jar, 'jars', binary=True)

+    if user_conf:
+        zipf = __add_str_to_zip(zipf, user_conf, 'user.yaml')


why _add_str_to_zip? Isn't this a file?

The user_conf is a byte string that is written to a file in node_scripts called user.yaml. user.yaml is not a file on the client.

__add_str_to_zip() is probably not the best name for what this method does, though.

…freck/aztk into feature/on-node-user-creation

paselem

Looks like you're missing docs for this. When writing them, please note that using an ssh key-pair is the preferred method of adding a user to a node.

paselem · 2018-01-26T23:00:01Z

aztk/models.py

@@ -23,6 +23,13 @@ def __init__(self, name: str = None, script: str = None, run_on=None):
        self.run_on = run_on


+class UserConfiguration:
+    def __init__(self, username: str, ssh_key: str = None, password: str = None):


Will this get auto-populated from the users' secrets.yaml?

On the CLI, yeah -- that happens here:
https://github.com/jafreck/aztk/blob/06f85c2538db98e168fed8c38fe9b88524f95e5f/cli/spark/endpoints/cluster/cluster_create.py#L87

For the SDK, no the secrets.yaml file is not used at all.

* Feature: on node user creation (#303) * client side on node user creation * start create user on node implementation * fix on node user creation * remove debug statements * remove commented code * line too long * fix spinner password prompt ui bug * set wait to false by default, formatting * encrypt password on client, decrypt on node * update docs, log warning if password used * Fix list-apps crash (#364) * Allow submitting jobs into a VNET (#365) * Add subnet_id to job submission cluster config * add some docs * Feature: Spark mixed mode support (#350) * add support for aad creds for storage on node * add mixed mode support * add docs * switch error order * add dedicated to get_cluster * remove mixed mode in print_cluster_conf * Feature: spark init docker repo customization (#358) * customize docker_repo based on init args * whitespace * add some docs * r-base to r * case insensitive r flag, typo fix * Bug: Load default Jars for job submission CLI (#367) * load jars in .aztk/ by default * rewrite loading config files * Feature: Cluster Run and Copy (#304) * start implementation of cluster run * fix cluster_run * start debug sequential user add and delete * parallelize user creation and deletion, start implementation of cluster scp * continue cluster_scp implementation * debug statements, disconnect error: permission denied * untesteed parakimo implementation of clus_run * continue debugging user creation bug * fix bug with pool user creation, start concurrent implementation * start fix of paramiko cluster_run and cluster_copy * working paramiko cluster_run implementation, start cluster_scp * fix cluster_scp command * update requirements, rename cluster_run function * remove unused shell functions * parallelize run and scp, add container_name, create logs wrapper * change scp to copy, clean up * sort imports * remove asyncssh from node requirements * remove old import * remove bad error handling * make cluster user management methods private * remove comment * remove accidental commit * fix merge, move delete to finally clause * add docs * formatting * Feature: Refactor cluster config to use ClusterConfiguration model (#343) * Bug: fix core-site.xml typo (#378) * fix typo * crlf->lf * Bug: fix regex for is_gpu_enabled (#380) * fix regex for is_gpu_enabled * crlf->lf * Bug: spark SDK example fix (#383) * start fix sdk * fix sdk example * crlf->lf * Fix: Custom scripts not read from cluster.yaml (#388) * Feature: spark shuffle service (#374) * start shuffle service by default * whitespace, delete misplaced file * crlf->lf * crlf->lf * move spark scratch space off os drive * Feature: enable dynamic allocation by default (#386) * Bug: stop using mutable default parameters (#392) * Bug: always upload spark job logs errors (#395) * Bug: spark submit upload error log type error (#397) * Bug: Spark Job list apps exit code 0 (#396) * Bug: fix spark-submit cores args (#399) * Fix: Trying to add user before master is ready show better error (#402) * Bug: move spark.local.dir to location usable by rstudioserver (#407) * Feature: SDK support for file-like configuration objects (#373) * add support for filelike objects for conifguration files * fix custom scripts * remove os.pathlike * merge error * Feature: Basic Cluster and Job Submission SDK Tests (#344) * add initial cluster tests * add cluster tests, add simple job submission test scenario * sort imports * fix job tests * fix job tests * remove pytest from travis build * cluster per test, parallel pytest plugin * delete cluster after tests, wait until deleted * fix bugs * catch right error, change cluster_id to base_cluster_id * fix test name * fixes * move tests to intregration_tests dir * update travis to run non-integration tests * directory structure, decoupled job tests * fix job tests, issue with submit_job * fix bug * add test docs * add cluster and job delete to finally clause * Feature: Spark add worker on master option (#415) * Add worker_on_master to ClusterConfiguration * add worker_on_master to JobConfiguration * Feature: task affinity to master node (#413) * Release: v0.6.0 (#416) * update changelog and version * underscores to stars

jafreck added 5 commits December 21, 2017 16:07

client side on node user creation

9c7af6d

start create user on node implementation

6e0885b

fix on node user creation

8cd247b

remove debug statements

38ada54

remove commented code

dc12ed7

jafreck added the in progress label Dec 23, 2017

paselem reviewed Dec 24, 2017

View reviewed changes

jafreck added 7 commits December 24, 2017 13:06

line too long

569d917

fix spinner password prompt ui bug

e3a2806

Merge branch 'feature/on-node-user-creation' of https://github.com/ja…

7119193

…freck/aztk into feature/on-node-user-creation

Merge branch 'master' into feature/on-node-user-creation

2f5cc87

Merge branch 'master' into feature/on-node-user-creation

527c17f

Merge branch 'feature/on-node-user-creation' of https://github.com/ja…

2a03e0b

…freck/aztk into feature/on-node-user-creation

set wait to false by default, formatting

06f85c2

paselem suggested changes Jan 26, 2018

View reviewed changes

paselem reviewed Jan 26, 2018

View reviewed changes

jafreck added 2 commits January 26, 2018 15:50

encrypt password on client, decrypt on node

9186827

update docs, log warning if password used

b1a0b07

paselem approved these changes Jan 27, 2018

View reviewed changes

jafreck merged commit 85a472c into Azure:master Jan 27, 2018

jafreck removed the in progress label Jan 27, 2018

jafreck mentioned this pull request Feb 1, 2018

Encrypt environment variables stored on pool #338

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: on node user creation #303

Feature: on node user creation #303

jafreck commented Dec 23, 2017 •

edited

Loading

paselem Dec 24, 2017

jafreck Dec 24, 2017

jafreck Dec 24, 2017 •

edited

Loading

paselem Dec 24, 2017

paselem Dec 24, 2017

jafreck Dec 24, 2017

paselem left a comment

paselem Jan 26, 2018

jafreck Jan 26, 2018

Feature: on node user creation #303

Feature: on node user creation #303

Conversation

jafreck commented Dec 23, 2017 • edited Loading

paselem Dec 24, 2017

Choose a reason for hiding this comment

jafreck Dec 24, 2017

Choose a reason for hiding this comment

jafreck Dec 24, 2017 • edited Loading

Choose a reason for hiding this comment

paselem Dec 24, 2017

Choose a reason for hiding this comment

paselem Dec 24, 2017

Choose a reason for hiding this comment

jafreck Dec 24, 2017

Choose a reason for hiding this comment

paselem left a comment

Choose a reason for hiding this comment

paselem Jan 26, 2018

Choose a reason for hiding this comment

jafreck Jan 26, 2018

Choose a reason for hiding this comment

jafreck commented Dec 23, 2017 •

edited

Loading

jafreck Dec 24, 2017 •

edited

Loading