Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnionExec array and nested array support #3359

Merged
merged 2 commits into from
Sep 3, 2021

Conversation

rwlee
Copy link
Collaborator

@rwlee rwlee commented Aug 31, 2021

Resolves #1459

Depends on rapidsai/cudf#9130 for unionByName list of struct support. Approved but not merged yet.

Adds type support for arrays and nested arrays for union, unionAll, unionByName. Also includes better UnionExec testing for nested types.

Signed-off-by: Ryan Lee <ryanlee@nvidia.com>
razajafri
razajafri previously approved these changes Sep 1, 2021
@razajafri
Copy link
Collaborator

build

@jlowe
Copy link
Member

jlowe commented Sep 1, 2021

Curious, this depends on a cudf feature that hasn't been merged but the tests pass. This makes me think the tests aren't sufficiently exercising the code, otherwise a test should have failed without the requisite cudf feature.

@ttnghia
Copy link
Collaborator

ttnghia commented Sep 1, 2021

Curious, this depends on a cudf feature that hasn't been merged but the tests pass. This makes me think the tests aren't sufficiently exercising the code, otherwise a test should have failed without the requisite cudf feature.

Yeah, this looks dangerous to me too 😃

@jlowe jlowe marked this pull request as draft September 1, 2021 18:09
@jlowe
Copy link
Member

jlowe commented Sep 1, 2021

Converting this to a draft so this doesn't get accidentally merged by a passerby until this mystery is solved.

@sameerz sameerz added the feature request New feature or request label Sep 2, 2021
@rwlee
Copy link
Collaborator Author

rwlee commented Sep 2, 2021

build

@rwlee
Copy link
Collaborator Author

rwlee commented Sep 3, 2021

I can turn this test error on and off with a very clear change within cudf, but for whatever reason the followup build still passed. The followup build was started before the PR was merged, but it was pretty close timing. If the blossom CI was delayed pulling the snapshot it may have picked up a cudf snapshot with the fix in it. Looking into it tomorrow, but with locally tests demonstrating cause and effect I think this is good to reopen/merge...

@rwlee
Copy link
Collaborator Author

rwlee commented Sep 3, 2021

Confirmed while testing with different versions of the snapshot jars, the new test correctly hits the dependent interleave column functionality.

@rwlee rwlee marked this pull request as ready for review September 3, 2021 19:06
Copy link
Collaborator

@razajafri razajafri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve the plugin side changes assuming it has been verified that this patch fails with older versions of cudf

@rwlee
Copy link
Collaborator Author

rwlee commented Sep 3, 2021

Approve the plugin side changes assuming it has been verified that this patch fails with older versions of cudf

Verified offline with different snapshot builds. Fails with cudf-21.10.0-20210902.122036-35-cuda11.jar and cudf-21.10.0-20210902.063732-34-cuda11.jar, but passes with cudf-21.10.0-20210903.122002-36-cuda11.jar

@revans2 revans2 merged commit 13bb3a3 into NVIDIA:branch-21.10 Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Support union for nested types
6 participants