aztk spark cluster submit exits with code 0 even if job fails

[shtratos]

When I run aztk spark cluster submit ... --wait it exits with code 0 even if the Spark job fails.
This makes it unsuitable for using in scripts as it's very hard to determine if the job succeeded of failed.

e.g.

$ aztk spark cluster submit --id testpy --name pipy1 ~/aztk/examples/src/main/python/pi.py abc --wait && echo "GOOD" || echo "BAD"
-------------------------------------------
Spark cluster id:        testpy
Spark app name:          pipy1
Wait for app completion: True
Application:             /Users/uuu/aztk/examples/src/main/python/pi.py
Application arguments:   ['abc']
-------------------------------------------
...
Traceback (most recent call last):
  File "/mnt/batch/tasks/workitems/testpy2/job-1/pipy4/wd/pi.py", line 36, in <module>
    partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
ValueError: invalid literal for int() with base 10: 'abc'
...
18/04/04 11:55:07 INFO ShutdownHookManager: Deleting directory /mnt/batch/tasks/spark-8e101a49-8fe8-4e4b-9578-d47cc89073b8
18/04/04 11:55:07 INFO ShutdownHookManager: Deleting directory /mnt/batch/tasks/spark-8e101a49-8fe8-4e4b-9578-d47cc89073b8/pyspark-bf8d2022-6aec-4404-8544-b5264f288d56
GOOD

[jafreck]

Thanks for pointing this out, you're absolutely right that this makes scripting difficult. The exit code is already available in the SDK, it is contained in the ApplicationLog model (returned when you call get_application_log). So plumbing this into the CLI is an easy fix.