Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Spark3.1.0 test spark.sql.legacy.parquet.int96RebaseModeInRead #1135

Closed
tgravescs opened this issue Nov 17, 2020 · 2 comments · Fixed by #3330 or #3643
Closed

[FEA] Spark3.1.0 test spark.sql.legacy.parquet.int96RebaseModeInRead #1135

tgravescs opened this issue Nov 17, 2020 · 2 comments · Fixed by #3330 or #3643
Assignees
@razajafri
Labels
P1 Nice to have for release Spark 3.1+ Bugs only related to Spark 3.1 or higher task Work required that improves the product but is not user facing
Milestone
Sep 13 - Sep 24

Comments

@tgravescs
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Spark 3.1.0 added some configs for int96 rebase spark.sql.legacy.parquet.int96RebaseModeInRead and spark.sql.legacy.parquet.int96RebaseModeInWrite. Make sure we support and test accordingly
@tgravescs tgravescs added feature request New feature or request P1 Nice to have for release Spark 3.1+ Bugs only related to Spark 3.1 or higher labels Nov 17, 2020
@jlowe
Copy link
Member

jlowe commented Aug 11, 2021

I think we need to support this ASAP, since we try to honor spark.sql.legacy.parquet.datetimeRebaseModeInRead and spark.sql.legacy.parquet.datetimeRebaseModeInWrite and these new INT96 configs override the old datetime versions when using INT96 timestamps.

@sameerz sameerz added task Work required that improves the product but is not user facing and removed feature request New feature or request labels Aug 13, 2021
@sameerz sameerz added this to the Aug 16 - Aug 27 milestone Aug 26, 2021
@sameerz sameerz linked a pull request Aug 28, 2021 that will close this issue
Support int96RebaseModeInWrite and int96RebaseModeInRead #3330
Merged
@tgravescs tgravescs reopened this Sep 23, 2021
@tgravescs
Copy link
Collaborator Author

reopen because reverted due to build failures and not up to date with latest shim changes:
#3627

@razajafri razajafri mentioned this issue Sep 24, 2021
Merged
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 03f08f5be0c71683e2f889f9e00f866256314ed2 (NV…
260ce1c 
…IDIA#1135)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
P1 Nice to have for release Spark 3.1+ Bugs only related to Spark 3.1 or higher task Work required that improves the product but is not user facing
Projects
None yet
Milestone
Sep 13 - Sep 24
4 participants
@jlowe @tgravescs @sameerz @razajafri