Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Odd even test spj uneven buckets #3

Open
wants to merge 8 commits into
base: spj-uneven-buckets
Choose a base branch
from
Open

Odd even test spj uneven buckets #3

wants to merge 8 commits into from

Commits on Feb 27, 2024

  1. [SPARK-47094][SQL] SPJ : Dynamically rebalance number of buckets when… 

    … they are not equal

  ### What changes were proposed in this pull request?
-- Allow SPJ between 'compatible' bucket funtions
-- Add a mechanism to define 'reducible' functions, one function whose output can be 'reduced' to another for all inputs.

  ### Why are the changes needed?
-- SPJ currently applies only if the partition transform expressions on both sides are identifical.

  ### Does this PR introduce _any_ user-facing change?
No

  ### How was this patch tested?
Added new tests in KeyGroupedPartitioningSuite

  ### Was this patch authored or co-authored using generative AI tooling?
No
    @szehon-ho
    szehon-ho committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    c81c039 View commit details
    Browse the repository at this point in the history

  2. Fix scalastyle and address review comment

    @szehon-ho
    szehon-ho committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    c5120fa View commit details
    Browse the repository at this point in the history

  3. Fix scalastyle and address review comment

    @szehon-ho
    szehon-ho committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    0754a23 View commit details
    Browse the repository at this point in the history

  4. Fix bug of using un-projected expression in sorting

    @szehon-ho
    szehon-ho committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    0016169 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Merge branch 'spj-uneven-buckets' of github.com:szehon-ho/spark into … 

    …spj-uneven-buckets
    hpal committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    5067852 View commit details
    Browse the repository at this point in the history

  2. test case for non-divisible bucket number & odd divisible number

    hpal committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    97eed0a View commit details
    Browse the repository at this point in the history

  3. revert to previous partition spec

    hpal committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    a1b875e View commit details
    Browse the repository at this point in the history

  4. revert to previous partition spec

    hpal committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3931a58 View commit details
    Browse the repository at this point in the history