Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add udf compiler unit tests #583

Merged
merged 9 commits into from
Aug 29, 2020
Merged

Add udf compiler unit tests #583

merged 9 commits into from
Aug 29, 2020

Conversation

wjxiz1992
Copy link
Collaborator

@wjxiz1992 wjxiz1992 commented Aug 19, 2020 •
edited
Loading

  • This change includes unit tests for implemented udf opcodes

@sameerz sameerz added the test Only impacts tests label Aug 19, 2020
@seanprime7

This comment has been minimized.

Sign in to view
@wjxiz1992 wjxiz1992 force-pushed the udf-tests branch 2 times, most recently from 06e1522 to 6bc645f Compare August 20, 2020 06:59
@wjxiz1992 wjxiz1992 marked this pull request as ready for review August 20, 2020 06:59
@wjxiz1992
Copy link
Collaborator Author

build

@wjxiz1992 wjxiz1992 self-assigned this Aug 20, 2020
seanprime7
seanprime7 previously approved these changes Aug 20, 2020
View reviewed changes
@wjxiz1992 @seanprime7
@nvnedelman @abellina
Add udf compiler unit tests
b39a4ef 
- This change includes unit tests for implemented udf opcodes

Signed-off-by: Allen Xu <allxu@nvidia.com>

Co-authored-by: Sean Lee <selee@nvidia.com>
Co-authored-by: Nicholas Edelman <nedelman@nvidia.com>
Co-authored-by: Alessandro Bellina <abellina@nvidia.com>
@abellina
Copy link
Collaborator

build

@wjxiz1992
Remove extra space lines in tests
d113d30 
Signed-off-by: Allen Xu <allxu@nvidia.com>
@seanprime7 seanprime7 mentioned this pull request Aug 25, 2020
Merged
@seanprime7
Remove irrelevant comment
3cfe555 
Signed-off-by: Sean Lee <selee@nvidia.com>
@seanprime7
Get the store index correctly
0abb51a 
Signed-off-by: Sean Lee <selee@nvidia.com>
@seanprime7
Fix the ldc test
30c4fd6 
Signed-off-by: Sean Lee <selee@nvidia.com>
@wjxiz1992
Mark un-compiled test cases
a550df2 
Signed-off-by: Allen Xu <allxu@nvidia.com>
@seanprime7
Copy link
Contributor

cc @nvnedelman

abellina
abellina requested changes Aug 28, 2020
View reviewed changes
Copy link
Collaborator

@abellina abellina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one small nit @wjxiz1992 @seanprime7.

Note I'll need to check the build times, I think it was somewhat slower with the udf compiler tests, but I'll run the CI again and compare where we are at.

udf-compiler/src/test/scala/com/nvidia/spark/OpcodeSuite.scala Outdated Show resolved Hide resolved
@abellina
Copy link
Collaborator

build

@wjxiz1992
Rename fall-back test cases
e0ec22a 
Signed-off-by: Allen Xu <allxu@nvidia.com>
@seanprime7
Fix column comparison
d10d1a5 
x.except(y) doesn't actually perform element-wise comparison
between x and y.  It returns a new dataframe with the rows from
x that do not appear in y.
It was producing false-positives with some unit tests.

Signed-off-by: Sean Lee <selee@nvidia.com>
@seanprime7
Improve the column comparison performance further
652d3d7 
Signed-off-by: Sean Lee <selee@nvidia.com>
@abellina
Copy link
Collaborator

build

@abellina abellina merged commit d9c94ba into NVIDIA:branch-0.2 Aug 29, 2020
@wjxiz1992 wjxiz1992 deleted the udf-tests branch December 18, 2020 03:48
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@wjxiz1992 @seanprime7
@nvnedelman @abellina
Add udf compiler unit tests (NVIDIA#583)
6552ff6 
* Add udf compiler unit tests

- This change includes unit tests for implemented udf opcodes

Signed-off-by: Allen Xu <allxu@nvidia.com>

Co-authored-by: Sean Lee <selee@nvidia.com>
Co-authored-by: Nicholas Edelman <nedelman@nvidia.com>
Co-authored-by: Alessandro Bellina <abellina@nvidia.com>

* Remove extra space lines in tests

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Remove irrelevant comment

Signed-off-by: Sean Lee <selee@nvidia.com>

* Get the store index correctly

Signed-off-by: Sean Lee <selee@nvidia.com>

* Fix the ldc test

Signed-off-by: Sean Lee <selee@nvidia.com>

* Mark un-compiled test cases

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Rename fall-back test cases

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Fix column comparison

x.except(y) doesn't actually perform element-wise comparison
between x and y.  It returns a new dataframe with the rows from
x that do not appear in y.
It was producing false-positives with some unit tests.

Signed-off-by: Sean Lee <selee@nvidia.com>

* Improve the column comparison performance further

Signed-off-by: Sean Lee <selee@nvidia.com>

Co-authored-by: Allen Xu <allxu@nvidia.com>
Co-authored-by: Sean Lee <selee@nvidia.com>
Co-authored-by: Nicholas Edelman <nedelman@nvidia.com>
Co-authored-by: Alessandro Bellina <abellina@nvidia.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@wjxiz1992 @seanprime7
@nvnedelman @abellina
Add udf compiler unit tests (NVIDIA#583)
3b86857 
* Add udf compiler unit tests

- This change includes unit tests for implemented udf opcodes

Signed-off-by: Allen Xu <allxu@nvidia.com>

Co-authored-by: Sean Lee <selee@nvidia.com>
Co-authored-by: Nicholas Edelman <nedelman@nvidia.com>
Co-authored-by: Alessandro Bellina <abellina@nvidia.com>

* Remove extra space lines in tests

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Remove irrelevant comment

Signed-off-by: Sean Lee <selee@nvidia.com>

* Get the store index correctly

Signed-off-by: Sean Lee <selee@nvidia.com>

* Fix the ldc test

Signed-off-by: Sean Lee <selee@nvidia.com>

* Mark un-compiled test cases

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Rename fall-back test cases

Signed-off-by: Allen Xu <allxu@nvidia.com>

* Fix column comparison

x.except(y) doesn't actually perform element-wise comparison
between x and y.  It returns a new dataframe with the rows from
x that do not appear in y.
It was producing false-positives with some unit tests.

Signed-off-by: Sean Lee <selee@nvidia.com>

* Improve the column comparison performance further

Signed-off-by: Sean Lee <selee@nvidia.com>

Co-authored-by: Allen Xu <allxu@nvidia.com>
Co-authored-by: Sean Lee <selee@nvidia.com>
Co-authored-by: Nicholas Edelman <nedelman@nvidia.com>
Co-authored-by: Alessandro Bellina <abellina@nvidia.com>
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
@nvauto
Update submodule cudf to 5a4afecad5ff2769fc7f32565158e1321c358b71 (NV…
164d40a 
…IDIA#583)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seanprime7 seanprime7 seanprime7 left review comments

@abellina abellina abellina approved these changes

Assignees

@wjxiz1992 wjxiz1992

Labels
test Only impacts tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wjxiz1992 @seanprime7 @abellina @sameerz