[Spec][Upstream] Mapping from DecimalType to Parquet physical type not aligned with spec

[HonahX]

Apache Iceberg version

main (development)

Please describe the bug 🐞

According to the parquet data type mappings spec. DecimalType should map to INT32 when precision <= 9, INT64 when precision <= 18, and fixed otherwise.

However, currently arrow write all decimal type as fixed in parquet. This may not be a big issue since the logical type is correct and may require upstream support:

• [Python][Parquet] Support wiriting decimals as integers arrow#38882

Updated: Thanks @syun64 for providing the link of upstream PR that fix this

• GH-42168: [Python][Parquet] Pyarrow store decimal as integer arrow#42169

Simple test:

from pyiceberg.catalog import load_catalog
from pyiceberg.types import *
from pyiceberg.schema import *
import pyarrow as pa

rest_catalog = load_catalog(
    "rest",
    **{
        ...
    },
)


decimal_schema = Schema(NestedField(1, "decimal", DecimalType(7, 0)))
decimal_arrow_schema = pa.schema(
    [
        ("decimal", pa.decimal128(7, 0)),
    ]
)

decimal_arrow_table = pa.Table.from_pylist(
    [
        {
            "decimal": 123,
        }
    ],
    schema=decimal_arrow_schema,
)

tbl = rest_catalog.create_table(
    "pyiceberg_test.test_decimal_type", schema=decimal_arrow_schema
)

tbl.append(decimal_arrow_table)

> parquet-tools inspect 00000-0-bff20a80-0e80-4b53-ba35-2c94498fa507.parquet

############ file meta data ############
created_by: parquet-cpp-arrow version 16.1.0
num_columns: 1
num_rows: 1
num_row_groups: 1
format_version: 2.6
serialized_size: 465


############ Columns ############
decimal

############ Column(decimal) ############
name: decimal
path: decimal
max_definition_level: 1
max_repetition_level: 0
physical_type: FIXED_LEN_BYTE_ARRAY
logical_type: Decimal(precision=7, scale=0)
converted_type (legacy): DECIMAL
compression: ZSTD (space_saved: -25%)

[sungwy]

Hi @HonahX thank you for raising this issue! Having worked on type casting PRs recently, this one piqued my interest...

It looks like there was a PR merged recently that exposed the feature from C++ Arrow to the Python bindings through a flag store_decimal_as_integer, which was released in version 17.0.0:

>>> import pyarrow as pa
>>> schema = pa.schema([pa.field('decimal', pa.decimal128(precision=2))])
>>> table = pa.Table.from_pydict({"decimal": [1.1, 2.2]})
>>> table.cast(schema)
pyarrow.Table
decimal: decimal128(2, 0)
----
decimal: [[1,2]]
>>> import pyarrow.parquet as pq
>>> pq.write_table(table.cast(schema), "test.parquet", store_decimal_as_integer=True)
>>> pq.read_metadata("test.parquet")
<pyarrow._parquet.FileMetaData object at 0x730658de3600>
  created_by: parquet-cpp-arrow version 17.0.0
  num_columns: 1
  num_rows: 2
  num_row_groups: 1
  format_version: 2.6
  serialized_size: 377
>>> pq.read_metadata("test.parquet").schema
<pyarrow._parquet.ParquetSchema object at 0x730656702200>
required group field_id=-1 schema {
  optional int32 field_id=-1 decimal (Decimal(precision=2, scale=0));

I just checked using the latest release 17.0.0 and I've confirmed that the parquet phyiscal types are being written as Integers