Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Databricks CI improvements and support runtime env parameter to xfail certain tests #297

Merged
merged 73 commits into from
Jun 29, 2020
Merged

Databricks CI improvements and support runtime env parameter to xfail certain tests #297

merged 73 commits into from
Jun 29, 2020

Conversation

tgravescs
Copy link
Collaborator

closes #223 #224

Make the CI Support for Databricks more robust.

  • Add xfail support based on runtime env parameter. xfail the tests that fail on databricks that we have issues for. User can pass in --runtime_env [apache | databricks]
  • Make sure failure handling working properly.
  • make sure if build stopped Databricks cluster shutdown
  • Better parameterization to pass things around then hardcoded
  • use new docker image that was pushed
  • upmerged to latest 0.2 branch and updated the databricks patch
  • Since cudf 0.15 is only available in URM, download that and copy it to the databricks node, where we mvn install it into m2.

@tgravescs
Add a profile to build for Databricks
4f961b2
@tgravescs
move dependency to management section for databricks annoation
3874aa5
@tgravescs
Databricks build and deploy scripts
4fb2cda
@tgravescs
Change the way we match DeserializeToObjectExec because some Spark
d3f73a1 
releases have an extra parameter to it
@tgravescs
Update description on jenkins file
e6e14e3
@tgravescs
cleanup copyrights
320ff12
@tgravescs
remove extra databricks exclude
b171ce5
@tgravescs
more cleanup
2db2618
@tgravescs
disable slack
888c7f8
@tgravescs
remove empty post
9fce430
@tgravescs
Try moving agent
0e8f41f
@tgravescs
remove agent from post
986393f
@tgravescs
change to use docker for entire
d1fae38
@tgravescs
right file
985cbb6
@tgravescs
remove docker file from in post
6e03e85
@tgravescs
try global docker file with post
8fce933
@tgravescs
remove docker agent
bb1cd1d
@tgravescs
try using image
127f5ab
@tgravescs
use docker instead of dockerfile
28e8ae5
@tgravescs
fix build_version
c97cf8d
@tgravescs
parameterize builds and try subprocess
4b86e8e
@tgravescs
add back deploy
fe15d39
@tgravescs
update comment
e4fb805
@tgravescs
add quotes
64d1df7
@tgravescs
use os.system
b2b3a61
@tgravescs
move python to bash script
45a2166
@tgravescs
pass in more variables
277a214
@tgravescs
fix syntax
2f8e2c7
@tgravescs
fix version
d9d44e8
@tgravescs
fix spark version
094b9eb
@tgravescs
Copy link
Collaborator Author

build

revans2
revans2 previously approved these changes Jun 27, 2020
View reviewed changes
integration_tests/src/main/python/parquet_test.py Outdated Show resolved Hide resolved
@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 096ea70 into NVIDIA:branch-0.2 Jun 29, 2020
@tgravescs tgravescs deleted the dbfollowup branch June 29, 2020 14:07
@sameerz sameerz added this to the Jun 22 - Jul 2 milestone Jul 2, 2020
@sameerz sameerz added the build Related to CI / CD or cleanly building label Jul 6, 2020
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@tgravescs
Databricks CI improvements and support runtime env parameter to xfail…
6aeb15a 
… certain tests (NVIDIA#297)

* Add a profile to build for Databricks

* Change xfail to be annotation

Co-authored-by: Thomas Graves <tgraves@nvidia.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@tgravescs
Databricks CI improvements and support runtime env parameter to xfail…
744fe70 
… certain tests (NVIDIA#297)

* Add a profile to build for Databricks

* Change xfail to be annotation

Co-authored-by: Thomas Graves <tgraves@nvidia.com>
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
@nvauto
Update submodule cudf to 6eca42f (NVIDIA#297)
0d38fc2 
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees
No one assigned
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
Jun 22 - Jul 2
Development

Successfully merging this pull request may close these issues.

[FEA] Add pytest option for environment
3 participants
@tgravescs @revans2 @sameerz