Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] row range on window operations can overflow #9672

Closed
revans2 opened this issue Nov 12, 2021 · 0 comments · Fixed by #9674
Closed

[BUG] row range on window operations can overflow #9672

revans2 opened this issue Nov 12, 2021 · 0 comments · Fixed by #9674
Labels
bug Something isn't working Spark Functionality that helps Spark RAPIDS

Comments

@revans2
Copy link
Contributor

revans2 commented Nov 12, 2021

Describe the bug
The row range counts can overflow and produce incorrect results in window operations.

Steps/Code to reproduce bug

When I do a spark query like

spark.range(10L).selectExpr("SUM(1) OVER(ROWS BETWEEN 2147483640 PRECEDING and 2147483642 FOLLOWING) as s").show()

on the GPU I get a result like

+----+
|   s|
+----+
|  10|
|  10|
|  10|
|  10|
|  10|
|null|
|null|
|null|
|null|
|null|
+----+

but on the CPU it is all 10s (no nulls).

It gets translated into a cudf::grouped_rolling_window with no group by keys a single column with 10 "1"s in it. Preceding and Following are both set to Scalar{type=INT32 value=2147483641} and Scalar{type=INT32 value=2147483642} respectively. But I tried it with INT64 and it didn't change the results. I will try to turn this into a C++ test. Just waiting for things to build.

It looks very much like the row number checks are all done as ints and not longs, so things are overflowing and we get incorrect results. If I check the following up or down, more or less rows get a correct answer, which really hints at an overflow.

Expected behavior
We get the right answer and I don't have to think about overflow. Otherwise I am going to have to fall back to the CPU for any window query with rows that are larger than 1/2 of Int.MaxValue

@revans2 revans2 added bug Something isn't working Needs Triage Need team to review and classify Spark Functionality that helps Spark RAPIDS labels Nov 12, 2021
@revans2 revans2 removed the Needs Triage Need team to review and classify label Nov 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Spark Functionality that helps Spark RAPIDS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant