Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] GpuCreateMap needs to set hasSideEffects in some cases #5544

Closed
revans2 opened this issue May 19, 2022 · 0 comments · Fixed by #5546
Closed

[BUG] GpuCreateMap needs to set hasSideEffects in some cases #5544

revans2 opened this issue May 19, 2022 · 0 comments · Fixed by #5546
Assignees
Labels
bug Something isn't working P0 Must have for release

Comments

@revans2
Copy link
Collaborator

revans2 commented May 19, 2022

Describe the bug
#5533 (comment)

if

(SQLConf.get.getConf(SQLConf.MAP_KEY_DEDUP_POLICY) ==
        SQLConf.MapKeyDedupPolicy.EXCEPTION.toString

Then GpuCreateMap can throw an exception, which means it has side effects so we need to set that properly. The above is true by default.

Steps/Code to reproduce bug
just a quick guess but

spark.range(10).selectExpr("id", "if(id == 0, null, map(id, id, id DIV 2, id)) as m").show()

Expected behavior
No exception is thrown when creating a map under an if statement with duplicate keys when the if statement would have filtered out that operation.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify P0 Must have for release labels May 19, 2022
@revans2 revans2 self-assigned this May 19, 2022
@revans2 revans2 removed the ? - Needs Triage Need team to review and classify label May 19, 2022
@revans2 revans2 added this to the May 2 - May 20 milestone May 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P0 Must have for release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant