Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG][Audit] [SPARK-38314] - Fail to read parquet files after writing the hidden file metadata #4911

Closed
nartal1 opened this issue Mar 8, 2022 · 1 comment · Fixed by #4954
Closed

[BUG][Audit] [SPARK-38314] - Fail to read parquet files after writing the hidden file metadata #4911

nartal1 opened this issue Mar 8, 2022 · 1 comment · Fixed by #4954
Assignees
@firestarman
Labels
audit_3.3.0 Audit related tasks for 3.3.0 bug Something isn't working P0 Must have for release

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Mar 8, 2022

Is your feature request related to a problem? Please describe.
We have copied FileFormatWriter code in GpuFileFormatWriter.scala so I suspect the bug exists in our code as well. Need to update the file to keep it in sync with Spark's code.

PR: apache/spark@50520fe3eb
@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.3.0 Audit related tasks for 3.3.0 labels Mar 8, 2022
@nartal1 nartal1 changed the title [FEA][Audit] [SPARK-38314] - Fix for failing to read parquet files after writing the hidden file metadata [FEA][Audit] [SPARK-38314] - Fail to read parquet files after writing the hidden file metadata Mar 8, 2022
@sameerz sameerz added bug Something isn't working P0 Must have for release and removed feature request New feature or request ? - Needs Triage Need team to review and classify labels Mar 8, 2022
@firestarman
Copy link
Collaborator

GPU does not support writing hidden file meta, and will fall back to CPU writing. Could we have the case to reproduce this failure on GPU ?

@firestarman firestarman mentioned this issue Mar 15, 2022
Merged
@sameerz sameerz changed the title [FEA][Audit] [SPARK-38314] - Fail to read parquet files after writing the hidden file metadata [BUG][Audit] [SPARK-38314] - Fail to read parquet files after writing the hidden file metadata Apr 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@firestarman firestarman

Labels
audit_3.3.0 Audit related tasks for 3.3.0 bug Something isn't working P0 Must have for release
Projects
None yet
Milestone
No milestone
3 participants
@sameerz @firestarman @nartal1