Shim layer support for Spark 3.0.0 Databricks

[tgravescs]

Add in Support for Spark 3.0.0 databricks. This requires to be built on a databricks host, so the default build doesn't build it, but it will try to pull from sonatype. That will obviously be broken until we push once. I did add in an option to skip pulling it in via the -Pexclude-databricks option.

Unfortunately our dev-ci seems to be messed up again so I have not been able to test full end to end with deployment to urm. if it fails I will have to do a followup or work with team to get the dev environment back working.

For databricks, it does require joins and then FileSourcescanExec, so I ended up copying those files again. Hopefully we can do the follow on to combine those somehow - issue #412

[tgravescs]

build

[tgravescs]

I'm going to see if I can get ci-dev env going, otherwise this build won't pass. Or I could change to exclude databricks by default until we have it working, but let me ping the devops from ngc to see if I can get dev env going in reasonable amount of time.

[tgravescs]

build

[tgravescs]

ci dev build passed and pushed one version of jar to urm

[revans2]

The changes look good to me, I just wanted to be sure I understood everything that you wanted to happen.

So is this the final version? Do we intend to push a version of the databricks shim to sonatype and have the final build pull that in for shadeing?

[tgravescs]

good question, ideally yes we push it to sonatype. I'll turn the exclude-databricks on by default for now and then followup with a different issue for pushing to sontaype.

[tgravescs]

note, still updating jenkins files

[tgravescs]

build

[tgravescs]

build