Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Go][Parquet] PlainFixedLenByteArrayEncoder behaves differently from DictFixedLenByteArrayEncoder with null values where schema has Nullable: false #71

Open
asfimport opened this issue Jul 20, 2022 · 0 comments
Labels
Component: Parquet good-first-issue Good for newcomers Type: bug Something isn't working

Comments

@asfimport
Copy link

I have created a small repro to illustrate this bug: https://gist.github.com/phillipleblanc/5e3e2d0e6914d276cf9fd79e019581de

When writing a Decimal128 array to a Parquet file the pqarrow package will prefer to use DictFixedLenByteArrayEncoder. If the size of the array goes over some threshold, it will switch to using PlainFixedLenByteArrayEncoder.

The DictFixedLenByteArrayEncoder tolerates null values in a Decimal128 array with the arrow schema set to Nullable: false, however the PlainFixedLenByteArrayEncoder will not tolerate null values and will panic.

Having null values in an array marked as non-nullable is an issue in the user code - however, it was surprising that my buggy code was working some times and not working other times. I would expect the PlainFixedLen encoder to handle nulls the same way as the DictFixedLen encoder or for the DictFixedLen encoder to panic.

An observation is that most other array types handle nulls with the schema marked as non-nullable when writing to Parquet; this was the first instance I found in the pqarrow package where having the Arrow schema marked as Nullable was necessary for Parquet writing arrays with null values. Again, debatable if this is desirable or not.

Reporter: Phillip LeBlanc

Note: This issue was originally created as ARROW-17133. Please see the migration documentation for further details.

@zeroshade zeroshade added the good-first-issue Good for newcomers label Apr 25, 2023
@assignUser assignUser transferred this issue from apache/arrow Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Parquet good-first-issue Good for newcomers Type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants