Fix verify build after integration tests are run

[revans2]

Not perfect but tries to deal with fixing some issues with hive metastore files in the integration tests.

This fixes #656

[revans2]

build

[jlowe]

Removing these directories would be a nasty surprised to anyone who ran spark from the top of the plugin repo. Granted may not be a common thing, but still destruction of data the tests didn't necessarily create.

It would be preferable to have the integration tests run from a directory somewhere underneath integration_tests/target/ so Maven's clean will clean it up by default and the user doesn't expect things there to be unaffected during builds. Would it be better to update run_pyspark_from_build.sh to do something like that?

[revans2]

It would be preferable to have the integration tests run from a directory somewhere underneath integration_tests/target/

This script only works being run from the main test directory. So having someone try to run this script from another directory already will run into issues with not being able to find dependencies. Which because of the set -x should cause the script to fail when spark-submit fails so the files would not be removed.

I can try to play around with some things in pytest, but generally pytest assumes that you are running from the base directory with how it tries to play games with importing things.

[revans2]

build

[revans2]

@jlowe the tests now run in a sub-directory of target by default.