Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] String CSV parsing does not respect nullValues #127

Closed
revans2 opened this issue Jun 8, 2020 · 1 comment · Fixed by #1407
Closed

[BUG] String CSV parsing does not respect nullValues #127

revans2 opened this issue Jun 8, 2020 · 1 comment · Fixed by #1407
Assignees
@revans2
Labels
bug Something isn't working P1 Nice to have for release SQL part of the SQL/Dataframe plugin
Milestone
Dec 7 - Dec 18

Comments

@revans2
Copy link
Collaborator

revans2 commented Jun 8, 2020

Describe the bug
Null values, default empty string, are not respected by the CSV parser.

Steps/Code to reproduce bug
A test is being added to the integration tests.
@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify SQL part of the SQL/Dataframe plugin labels Jun 8, 2020
@revans2 revans2 changed the title [BUG] String CSV parseing does not respect nullValues [BUG] String CSV parsing does not respect nullValues Jun 8, 2020
@sameerz sameerz added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Aug 18, 2020
@revans2
Copy link
Collaborator Author

revans2 commented Dec 16, 2020

This appears to have been fixed by some changes to CUDF. The test now pass.

XPASS ../../src/main/python/csv_test.py::test_round_trip[-String][APPROXIMATE_FLOAT] https://github.com/NVIDIA/spark-rapids/issues/127
XPASS ../../src/main/python/csv_test.py::test_round_trip[csv-String][APPROXIMATE_FLOAT] https://github.com/NVIDIA/spark-rapids/issues/127

@revans2 revans2 self-assigned this Dec 16, 2020
@revans2 revans2 added this to the Dec 7 - Dec 18 milestone Dec 16, 2020
@revans2 revans2 mentioned this issue Dec 16, 2020
Merged
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 26c22b6 (NVIDIA#127)
1de305c 
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@revans2 revans2

Labels
bug Something isn't working P1 Nice to have for release SQL part of the SQL/Dataframe plugin
Projects
None yet
Milestone
Dec 7 - Dec 18
Development

Successfully merging a pull request may close this issue.

Update docs and tests for null CSV support revans2/spark-rapids
2 participants
@revans2 @sameerz