Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Add tests for Datetimes with bad combination of year/month/day... #1517

Closed
mythrocks opened this issue Jan 14, 2021 · 5 comments
Closed

[FEA] Add tests for Datetimes with bad combination of year/month/day... #1517

mythrocks opened this issue Jan 14, 2021 · 5 comments
Assignees
@andygrove
Labels
audit_3.1.0 task Work required that improves the product but is not user facing test Only impacts tests
Milestone
Feb 1 - Feb 12

Comments

@mythrocks
Copy link
Collaborator

mythrocks commented Jan 14, 2021 •
edited
Loading

Related to audit of apache/spark@1034815519 and apache/spark@b9f2f78de5.

Failure cases for year/month/day combinations have been modified in ANSI mode. We need to add tests to verify ANSI_ENABLED behaviour. We should also check that the tests added in this commit work correctly with the spark-rapids plugin.
Must also verify that datetime parsing fails with bad pattern strings.
@mythrocks mythrocks added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.1.0 labels Jan 14, 2021
@sameerz
Copy link
Collaborator

sameerz commented Jan 19, 2021

@andygrove is there anything in apache/spark@b9f2f78de5 that is more stringent but that we have on by default?

@andygrove
Copy link
Contributor

I will take a look at this tomorrow. It is also related to #1117

@andygrove andygrove self-assigned this Jan 19, 2021
@andygrove andygrove added this to the Jan 18 - Jan 29 milestone Jan 19, 2021
@andygrove
Copy link
Contributor

@andygrove is there anything in apache/spark@b9f2f78 that is more stringent but that we have on by default?

@sameerz Yes, the plugin currently returns null for the invalid dates instead of throwing an exception when ANSI mode is enabled

@andygrove andygrove mentioned this issue Jan 20, 2021
Merged
@andygrove
Copy link
Contributor

I opened a PR to fix this for CAST and filed #1556 to do the same for the string to timestamp functions since this is a different code path.

#1556

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jan 26, 2021
@sameerz
Copy link
Collaborator

sameerz commented Feb 11, 2021

Closing this and tracking the follow up in #1556

@sameerz sameerz closed this as completed Feb 11, 2021
@sameerz sameerz added test Only impacts tests task Work required that improves the product but is not user facing and removed feature request New feature or request labels Mar 2, 2021
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to e8cf0ebd517a9c81e294771a40c74ff5c4fb42da (NV…
3fd71ac 
…IDIA#1517)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
audit_3.1.0 task Work required that improves the product but is not user facing test Only impacts tests
Projects
None yet
Milestone
Feb 1 - Feb 12
Development

No branches or pull requests
3 participants
@andygrove @mythrocks @sameerz