PARQUET-2417: Add support for geometry logical type

[zhangfengcdt]

This PR is to provide a POC to support the proposed changes to the parquet-format to add geometry type to parquet.

Here is the proposal: apache/parquet-format#240

Jira

• My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-2471

Tests

• My PR adds the following unit tests: TestGeometryTypeRoundTrip

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[zhangfengcdt]

CC: @jiayuasu @Kontinuation @wgtmac

[wgtmac]

Thanks for the update! I have left some comments. I think we are reaching the finish line!

[wgtmac]

Thanks for adding these tests!

I think we are missing tests in following cases:

• verify geometry type metadata is well preserved.
• verify all kinds of geometry stats are preserved, including bbox, covering and geometry types.
• verify geo stats in the column index have been generated.

I can do these later.

[jiayuasu]

What is the difference between EnvelopeCovering and BBox stats when the edge == planar? I think they are the same. I thought we agreed that: when the edge = planar, we only generate BBox stats; when the edge = spherical, we generate both BBox and covering stats (but we omit the covering stats since we don't have the tool to calculate it).

[wgtmac]

Make sense. We could also omit Covering stats since there is no downstream consumer at the moment.

[paleolimbot]

Probably the only reason to write a Covering would be to test its serialization/deserialization to thrift and/or make sure it is implemented in at least two places for the spec vote? At least in C++ it is not difficult to accumulate the bounding box and only when it is about to be serialized generate the WKB portion of the covering.

[wgtmac]

Sorry for the delay. I have left a few comments regarding to the statistics. Please take a look. Thanks!

[wgtmac]

This looks weird to me since it includes more information than kind. From the specs, crs and edges should be deduced from the geometry logical type metadata.

[zhangfengcdt]

The issue is that the spec only persist the following metadata for each covering to the parquet meta.

struct Covering {
  /**
   * A type of covering. Currently accepted values: "WKB".
   */
  1: required string kind;
  /**
   * A payload specific to kind. Below are the supported values:
   * - WKB: well-known binary of a POLYGON or MULTI-POLYGON that completely
   *   covers the contents. This will be interpreted according to the same CRS
   *   and edges defined by the logical type.
   */
  2: required binary value;
}

The crs and edge are unique to each covering, so we have to save them with each covering metadata as a part of the kind.

[paleolimbot]

According to that text, a WKB covering with a different CRS and/or edges is not currently permitted. If we need this, we should reword the spec!

[wgtmac]

I agree with @paleolimbot that this should be consistent with the spec.

[wgtmac]

See my comment below this one. This seems to break the contract from base class.