Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write int96 to parquet #1068

Merged
merged 6 commits into from
Dec 1, 2020
Merged

Conversation

razajafri
Copy link
Collaborator

@razajafri razajafri commented Nov 5, 2020

This is an early WIP. I think it's pretty much complete but still needs to be tested after the chunked writer accepts INT96 option

This is now ready for review.

@sameerz sameerz added the bug Something isn't working label Nov 6, 2020
@sameerz sameerz added this to the Nov 9 - Nov 20 milestone Nov 6, 2020
@jlowe jlowe added feature request New feature or request and removed bug Something isn't working labels Nov 6, 2020
@sameerz
Copy link
Collaborator

sameerz commented Nov 23, 2020

Related cudf PR rapidsai/cudf#6625

@revans2
Copy link
Collaborator

revans2 commented Nov 23, 2020

This also depends on rapidsai/cudf#6679

@razajafri razajafri changed the title [WIP] Write int96 to parquet Write int96 to parquet Nov 30, 2020
@razajafri
Copy link
Collaborator Author

build

@razajafri
Copy link
Collaborator Author

build

@razajafri
Copy link
Collaborator Author

I apologize for the inconvenience but I am using the CI to run the tests as my host is not running the tests

@razajafri
Copy link
Collaborator Author

build

@razajafri
Copy link
Collaborator Author

build

@jlowe jlowe merged commit 955e831 into NVIDIA:branch-0.3 Dec 1, 2020
@razajafri razajafri deleted the parquet_writer_int96 branch December 1, 2020 18:33
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* write int96 to parquet

* Removed unnecessary test for fallback

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

* fixed ParquetWriterSuite

* addressed review comments

* added tests for Spark 3.1.0

Co-authored-by: Raza Jafri <rjafri@nvidia.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* write int96 to parquet

* Removed unnecessary test for fallback

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

* fixed ParquetWriterSuite

* addressed review comments

* added tests for Spark 3.1.0

Co-authored-by: Raza Jafri <rjafri@nvidia.com>
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
[auto-merge] bot-auto-merge-branch-23.04 to branch-23.06 [skip ci] [bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants