commit a6a13fe
Author: Neil Best <neil.best@databricks.com>
Date:   Thu May 23 16:39:58 2024 -0500

    improve TransformationDescriberTest

commit 1f145aa
Author: Neil Best <neil.best@databricks.com>
Date:   Thu May 23 15:25:29 2024 -0500

    Add descriptive job group IDs and named transformations

    This makes the Spark UI more developer-friendly when analyzing
    Overwatch runs.

    Job group IDs have the form <workspace name>:<OW module name>

    Any use of `.transform( df => df)` may be replaced with
    `.transformWithDescription( nt)` after instantiating a `val nt =
    NamedTransformation( df => df)` as its argument.

    This commit contains one such application of the new extension method.
    (See `val jobRunsAppendClusterName` in `WorkflowsTransforms.scala`.)

    Some logic in `GoldTransforms` falls through to elements of the
    special job-run-action form of Job Group IDs emitted by the platform
    but the impact is minimal relative to the benefit to Overwatch
    development and troubleshooting.  Even so this form of Job Group ID is
    still present in initial Spark events before OW ETL modules begin to
    execute.

commit 7158765
Author: Neil Best <neil.best@databricks.com>
Date:   Thu May 23 17:20:20 2024 -0500

    Add extension method to show `DataFrame` records in the log

- bump `scalaVersion`, `sparkVersion` & `delta-core`
- adjust test suite, esp. where DDL no longer includes backticks
- set log level to `WARN` for all tests using `BeforeAndAfterEach`

…and-improve-silver-job-runs-performance

Removed a level of indirection and unnecessary conditional branching in definition of chained `lookupWhen` transformations.

Moved defintions to have references to `PipelineTable` objects in scope rather than passing them by argument.

commit aeef7ff
Author: sourav.banerjee <sourav.banerjee@databricks.com>
Date:   Tue Jun 11 17:59:51 2024 +0530

    Dropped Spec column from snapshot

commit a5c8b54
Author: sourav.banerjee <sourav.banerjee@databricks.com>
Date:   Tue Jun 11 17:52:52 2024 +0530

    Comvert all the struct field inside 'spec' column for cluster_snapshot_bronze to mapType

…runs-performance