Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet 1.14.2-RC1 breaks compat with Avro 1.7, 1.8, 1.9 #2992

Closed
clairemcginty opened this issue Aug 19, 2024 · 2 comments · Fixed by #2993
Closed

Parquet 1.14.2-RC1 breaks compat with Avro 1.7, 1.8, 1.9 #2992

clairemcginty opened this issue Aug 19, 2024 · 2 comments · Fixed by #2993
Labels
Type: bug
Milestone
1.14.2

Comments

@clairemcginty
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

AvroSchemaConverter references Avro logical type converters that aren't available in Avro versions below 1.10: See more here: https://github.com/apache/parquet-java/pull/1115/files#r1721966099

For reference, these are the local TIMESTAMP converters available in Avro 1.10+: https://github.com/apache/avro/blob/release-1.10.0/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L172-L194

Whereas in 1.9.2, only timestampMillis/timestampMicros area available: https://github.com/apache/avro/blob/release-1.9.2/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L158-L168

Component(s)

Avro
@clairemcginty clairemcginty mentioned this issue Aug 19, 2024
Merged
@wgtmac
Copy link
Member

wgtmac commented Aug 20, 2024

@Fokko Should we include this in the 1.14.2?

@Fokko
Copy link
Contributor

Fokko commented Aug 20, 2024

@wgtmac Yes, looks like it 👍

@Fokko Fokko added this to the 1.14.2 milestone Aug 20, 2024
clairemcginty added a commit to clairemcginty/parquet-mr that referenced this issue Aug 20, 2024
@clairemcginty
apacheGH-2992: Test logical type conversion for different Avro versions
a5380cd
Fokko pushed a commit that referenced this issue Aug 21, 2024
@clairemcginty
GH-2992: Gate LocalTimestamp references in AvroSchemaConverter (#2993)
312a15f 
* PARQUET-2992: Gate LocalTimestamp references in AvroSchemaConverter

* GH-2992: Test logical type conversion for different Avro versions
Fokko pushed a commit to Fokko/parquet-mr that referenced this issue Aug 21, 2024
@clairemcginty @Fokko
apacheGH-2992: Gate LocalTimestamp references in AvroSchemaConverter (a…
d04986f 
…pache#2993)

* PARQUET-2992: Gate LocalTimestamp references in AvroSchemaConverter

* apacheGH-2992: Test logical type conversion for different Avro versions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: bug
Projects
None yet
Milestone
1.14.2
Development

Successfully merging a pull request may close this issue.

GH-2992: Gate LocalTimestamp references in AvroSchemaConverter clairemcginty/parquet-mr
3 participants
@Fokko @clairemcginty @wgtmac