Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add descriptive NamedTransformations to Spark UI #1223

Merged
merged 5 commits into from
Aug 20, 2024

Commits on May 22, 2024

  1. Initial commit

    gueniai committed May 22, 2024
    Configuration menu
    Copy the full SHA
    da0c55a View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Add descriptive job group IDs and named transformations

    This makes the Spark UI more developer-friendly when analyzing
    Overwatch runs.
    
    Job group IDs have the form <workspace name>:<OW module name>
    
    Any use of `.transform( df => df)` may be replaced with
    `.transformWithDescription( nt)` after instantiating a `val nt =
    NamedTransformation( df => df)` as its argument.
    
    This commit contains one such application of the new extension method.
    (See `val jobRunsAppendClusterName` in `WorkflowsTransforms.scala`.)
    
    Some logic in `GoldTransforms` falls through to elements of the
    special job-run-action form of Job Group IDs emitted by the platform
    but the impact is minimal relative to the benefit to Overwatch
    development and troubleshooting.  Even so this form of Job Group ID is
    still present in initial Spark events before OW ETL modules begin to
    execute.
    neilbest-db committed May 23, 2024
    Configuration menu
    Copy the full SHA
    1f145aa View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a6a13fe View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. flip transformation names to beginning of label

    for greater visibility in Spark UI. `NamedTransformation` type name
    now appears in labels' second position.
    
    (cherry picked from commit 2ead752)
    neilbest-db committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    7346785 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. revert modified Spark UI Job Group labels

    TODO: enumerate the regressions this would introduce when the labels set by then platform are replaced this way.
    neilbest-db committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    9a7fddb View commit details
    Browse the repository at this point in the history