This repository has been archived by the owner on Feb 3, 2021. It is now read-only.
0.7.0
AZTK is now published on pip! Documentation has migrated to readthedocs
This release includes a number of breaking changes. Please follow the migration for upgrading from 0.6.0..
Breaking Changes
- Moved
docker_repounder a newtoolkitkey.docker_repois now only used for custom Docker images. Use toolkit for supported images. - Docker images have been refactored and moved to a different Dockerhub repository. The new supported images are not backwards compatible. See the documentation on configuration files.
Deprecated Features
- Custom scripts have been removed in favor of Plugins, which are more robust. See, the documenation on Plugins.
Added Features
- add internal flag to node commands (#482) (1eaa1b6), closes #482
- Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) (#517 (c98df7d), closes #517
- disable msrestazure keyring log (#509) (3cc43c3), closes #509
- enable mixed mode for jobs (#442) (8d00a2c), closes #442
- getting started script (#475) (7ef721f), closes #475
- JupyterLab plugin (#459) (da61337), closes #459
- managed storage for clusters and jobs (#443) (8aa1843), closes #443
- match cluster submit exit code in cli (#478) (8889059), closes #478
- Plugin V2: Running plugin on host (#461) (de78983), closes #461
- Plugins (#387) (c724d94), closes #387
- Pypi auto deployement (#428) (c237501), closes #428
- Readthedocs support (#497) (e361c3b), closes #497
- refactor docker images (#510) (779bffb), closes #510
- Spark add output logs flag (#468) (32de752), closes #468
- spark debug tool (#455) (44a0765), closes #455
- spark ui proxy plugin (#467) (2e995b4), closes #467
- Spark vnet custom dns hostname fix (#490) (61e7c59), closes #490
- New Toolkit configuration (#507) (7a7e63c), closes #507
Bug Fixes
- add gitattributes file (#470) (82ad029), closes #470
- add plugins to cluster_install_cmd call (#423) (216f63d), closes #423
- add spark.history.fs.logDirectory to required keys (#456) (4ef3dd0), closes #456
- add support for jars, pyfiles, files in Jobs (#408) (2dd7891), closes #408
- add timeout handling to cluster_run and copy (#524) (47000a5), closes #524
- azure file share not being shared with container (#521) (07ac9b7), closes #521
- Dependency issue with keyring not having good dependencies (#504) (5e79a2c), closes #504
- filter job submission clusters out of cluster list (#409) (1c31335), closes #409
- fix aztk cluster submit paths, imports (#464) (c1f43c7), closes #464
- fix broken spark init command (#486) (a33bdbc), closes #486
- fix job submission cluster data issues (#533) (9ccc1c6), closes #533
- fix spark job submit path (#474) (ee1e61b), closes #474
- make node scripts upload in memory (#519) (0015e22), closes #519
- pypi long description (#450) (db7a2ef), closes #450
- remove unnecessary example (#417) (f1e3f7a), closes #417
- Remove unused ssh plugin flags (#488) (be8cd2a), closes #488
- set explicit file open encoding (#448) (5761a36), closes #448
- Spark shuffle service worker registration fail (#492) (013f6e4), closes #492
- throw error if submitting before master elected (#479) (a59fe8b), closes #479
- hdfs using wrong conditions (#515) (a00dbb7), closes #515
- AZTK_IS_MASTER not set on worker and failing (#506) (b8a3fcc), closes #506
- VNet required error now showing if using mixed mode without it (#440) (9253aac), closes #440
- Worker on master flag ignored and standardize boolean environment (#514) (5579d95), closes #514
- Fix job configuration option for
aztk spark job submitcommand (#435) (4be5ac2), closes #435 - Fix keyring (#505) (12450fb), closes #505
- Fix the endpoint (#437) (bcefca3), closes #437
- Fix typo in command_builder 'expecity' -> 'explicitly' (#447) (27822f4), closes #447
- Fix typo load_aztk_screts -> load_aztk_secrets (#421) (6827181), closes #421
- Update file to point at master branch (#501) (4ba3c9d), closes #501
- Update storage sdk from 0.33.0 to 1.1.0 (#439) (f2eb1a4), closes #439
Internal Changes