Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid comparing window range canonicalized plans on Spark 3.0.x #2984

Merged
merged 1 commit into from
Jul 22, 2021

Conversation

jlowe
Copy link
Member

@jlowe jlowe commented Jul 21, 2021

Fixes #2964

The logical optimizer in Spark 3.0.x is not deterministic when it comes to the processing order of windows that have the same ranges. Therefore on Spark 3.0.x it doesn't make sense to compare canonicalized physical plans for these cases since the logical plans from which the physical plans were derived are not guaranteed to be the same.

Signed-off-by: Jason Lowe <jlowe@nvidia.com>
@jlowe jlowe added the test Only impacts tests label Jul 21, 2021
@jlowe jlowe self-assigned this Jul 21, 2021
@jlowe
Copy link
Member Author

jlowe commented Jul 21, 2021

@mythrocks I noticed there's a ton of commented-out tests WindowFunctionSuite stating they are waiting on #1039 which is now fixed. However it also mentions that these are mostly redundant with the Python integration tests.

I went ahead and updated the commented code in this PR, but I'm wondering what we should do with these disabled tests going forward (potentially in a separate PR). Should these be enabled, or are they providing no value beyond what the integration tests are already covering?

@sameerz sameerz added this to the July 19 - July 30 milestone Jul 21, 2021
@jlowe
Copy link
Member Author

jlowe commented Jul 21, 2021

build

@pxLi pxLi merged commit ca51ed2 into NVIDIA:branch-21.08 Jul 22, 2021
@mythrocks
Copy link
Collaborator

mythrocks commented Jul 22, 2021

@mythrocks I noticed there's a ton of commented-out tests WindowFunctionSuite stating they are waiting on #1039 which is now fixed. However it also mentions that these are mostly redundant with the Python integration tests.

Apologies for the delayed response. We can deal with those tests in a follow-up issue. My view is that those can be removed, since we're already testing the null-ORDER BY column case in window_function_test.py.

@jlowe jlowe deleted the fix-window-canonical branch September 10, 2021 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Only impacts tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] IGNORE ORDER, WITH DECIMALS: [Window] [MIXED WINDOW SPECS] FAILED in spark 3.0.3+
4 participants