Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] In Pluggable Cache Support CalendarInterval while creating CachedBatches #913

Closed
razajafri opened this issue Oct 7, 2020 · 5 comments
Closed

[FEA] In Pluggable Cache Support CalendarInterval while creating CachedBatches #913

razajafri opened this issue Oct 7, 2020 · 5 comments
Assignees
@razajafri
Labels
feature request New feature or request P1 Nice to have for release Spark 3.1+ Bugs only related to Spark 3.1 or higher
Milestone
Nov 23 - Dec 4

Comments

@razajafri
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
There are some types that spark parquet just does not support, and we are not testing those at all, i.e. calendar interval. Do we care? Is there a follow on issue to support them?

Describe the solution you'd like
Look into this more and suggest a solution
@razajafri razajafri added feature request New feature or request ? - Needs Triage Need team to review and classify labels Oct 7, 2020
@razajafri razajafri mentioned this issue Oct 7, 2020
Merged
@tgravescs
Copy link
Collaborator

what happens when you try to read those types?

@revans2
Copy link
Collaborator

revans2 commented Oct 7, 2020

Currently if you configure the experimental cache manager it should fail, but I don't know how gracefully it does that.

@sameerz
Copy link
Collaborator

sameerz commented Oct 9, 2020

Can we create tests to make sure we can handle reading those unsupported types (like CalendarIntervalType)? Please file an issue to track if needed.

@razajafri
Copy link
Collaborator Author

Yeah, waiting for this to be triaged and prioritized. Will see if I can get to this before that.

@sameerz sameerz added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Oct 13, 2020
@sameerz sameerz changed the title [FEA] Support may be needed for DataTypes not supported by Spark Parquet while creating CachedBatches [FEA] Support needed for DataTypes not supported by Spark Parquet while creating CachedBatches Oct 13, 2020
@sameerz sameerz changed the title [FEA] Support needed for DataTypes not supported by Spark Parquet while creating CachedBatches [FEA] Support CalendarInterval while creating CachedBatches Oct 13, 2020
@razajafri razajafri self-assigned this Oct 29, 2020
@razajafri
Copy link
Collaborator Author

Currently the cache serializer throws an exception and crashes. I am in the middle of working a solution by exploding the CalendarIntervalType to ints.

@sameerz sameerz modified the milestones: Oct 26 - Nov 6, Nov 9 - Nov 20 Nov 6, 2020
@sameerz sameerz added the Spark 3.1+ Bugs only related to Spark 3.1 or higher label Nov 17, 2020
@razajafri razajafri mentioned this issue Nov 23, 2020
Closed
12 tasks
@revans2 revans2 changed the title [FEA] Support CalendarInterval while creating CachedBatches [FEA] In Pluggable Cache Support CalendarInterval while creating CachedBatches Dec 11, 2020
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 6a67e8fe49661f7f2f6d8da119996f683a21803a (NV…
5bc475d 
…IDIA#913)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
feature request New feature or request P1 Nice to have for release Spark 3.1+ Bugs only related to Spark 3.1 or higher
Projects
None yet
Milestone
Nov 23 - Dec 4
Development

No branches or pull requests
4 participants
@revans2 @tgravescs @sameerz @razajafri