Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] get_json_object cannot handle ints or boolean values #10700

Closed
revans2 opened this issue Apr 12, 2024 · 2 comments
Closed

[BUG] get_json_object cannot handle ints or boolean values #10700

revans2 opened this issue Apr 12, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Apr 12, 2024

Describe the bug
The legacy version of GetJsonObject works just fine, but it looks like the new version does not handle normalization/validation properly. I am also really concerned about null

Steps/Code to reproduce bug

Seq("{'a': true}", "{'a': false}", "{'a': null}").toDF("s").repartition(1).selectExpr("s", "get_json_object(s, '$.a')").collect.foreach(println)
(0 until 100).map(i => "{'a': 1" + ("0" * i) + "}").toDF("s").repartition(1).selectExpr("s", "get_json_object(s, '$.a')").collect.foreach(println)

return nulls for all of the values instead of matching what Spark does, or the legacy version.

Our tests appear to really be lacking in this area.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Apr 12, 2024
@revans2
Copy link
Collaborator Author

revans2 commented Apr 12, 2024

So it looks like most of this has been fixed in 24.06 after the upmerge went in. I will retest things

@res-life
Copy link
Collaborator

Has been fixed after this upmerge from branch 24.04 to 24.06:
Fix merge from 24.04 to 24.06

Now both branch (24.04 and 24.06) are fixed.

I'll add the test cases in this issue into Spark-Rapids.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Apr 16, 2024
@revans2 revans2 closed this as not planned Won't fix, can't repro, duplicate, stale May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants