Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Parquet readers [databricks] #9631

Merged
merged 32 commits into from
Nov 7, 2023
Merged

Refactor Parquet readers [databricks] #9631

merged 32 commits into from
Nov 7, 2023

Commits on Nov 2, 2023

  1. Add check for nested types

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    e368aa6 View commit details
    Browse the repository at this point in the history

  2. Recursively check for rebasing

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7da416b View commit details
    Browse the repository at this point in the history

  3. Extract common code

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    df8f861 View commit details
    Browse the repository at this point in the history

  4. Allow nested type in rebase check

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    95d19ee View commit details
    Browse the repository at this point in the history

  5. Enable nested timestamp in roundtrip test

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    b426610 View commit details
    Browse the repository at this point in the history

  6. Fix another test 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7343b17 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'check_rebase_nested' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    0d48f57 View commit details
    Browse the repository at this point in the history

  8. Enable LEGACY rebase in read

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    024e6c9 View commit details
    Browse the repository at this point in the history

  9. Remove comment

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    9a39628 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Change function/class signatures

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    e686bb0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-23.12' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b49963e View commit details
    Browse the repository at this point in the history

  3. Complete modification

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    2c232f8 View commit details
    Browse the repository at this point in the history

  4. Misc 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    ac0f3e4 View commit details
    Browse the repository at this point in the history

  5. Add explicit type 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    c773794 View commit details
    Browse the repository at this point in the history

  6. Rename file and add some stuff in DateTimeRebaseHelpers.scala

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    29df7cd View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2023

  1. Move file and rename class

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    1b5112d View commit details
    Browse the repository at this point in the history

  2. Adopt new enum type 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    63342a9 View commit details
    Browse the repository at this point in the history

  3. Add name for the enum classes

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    6b2d795 View commit details
    Browse the repository at this point in the history

  4. Change exception messages

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    37aa40b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.12' into refactor_parquet_scan

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    d4cdc1b View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2023

  1. Does not yet support legacy rebase in read 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    03f681e View commit details
    Browse the repository at this point in the history

  2. Change legacy to corrected mode 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    14f230f View commit details
    Browse the repository at this point in the history

  3. Extract common code 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    1b464ec View commit details
    Browse the repository at this point in the history

  4. Rename functions 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    0d26d97 View commit details
    Browse the repository at this point in the history

  5. Reformat 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    c2504fd View commit details
    Browse the repository at this point in the history

  6. Make classes serializable 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    edb6c81 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. Merge branch 'branch-23.12' into refactor_parquet_scan 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    54e959f View commit details
    Browse the repository at this point in the history

  2. Change comment 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    3f01690 View commit details
    Browse the repository at this point in the history

  3. Various changes 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    74fe84a View commit details
    Browse the repository at this point in the history

  4. Fix compile errors 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    a455a90 View commit details
    Browse the repository at this point in the history

  5. Fix comments 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    b87493c View commit details
    Browse the repository at this point in the history

  6. Fix indentations 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    321e516 View commit details
    Browse the repository at this point in the history