Add descriptive NamedTransformations to Spark UI

[neilbest-db]

This makes the Spark UI more developer-friendly when analyzing
Overwatch runs.

Job group IDs have the form :

Any use of .transform( df => df) may be replaced with
.transformWithDescription( nt) after instantiating a val nt = NamedTransformation( df => df) as its argument.

This commit contains one such application of the new extension method.
(See val jobRunsAppendClusterName in WorkflowsTransforms.scala.)

Some logic in GoldTransforms falls through to elements of the
special job-run-action form of Job Group IDs emitted by the platform
but the impact is minimal relative to the benefit to Overwatch
development and troubleshooting. Even so this form of Job Group ID is
still present in initial Spark events before OW ETL modules begin to
execute.

[sonarcloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[neilbest-db]

@souravbaner-da, @gueniai, is there anything I can do to help get this merged in?

[neilbest-db]

closes #1226

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[souravbaner-da]

@neilbest-db This is not a licensed module. So we can't use it as per databrickslab standard.

[neilbest-db]

@souravbaner-da, I ust noticed your comment from 2024-06-05. I discussed this with @sriram251-code here in Slack back on 2024-06-04. I believe we have discussed it on >= 1 team calls since then also. If it's still a concern I would like to learn more about the standard you mention. Is that in a document somewhere?

[gueniai]

@neilbest-db if we're no longer overriding the JobGroupIDs, what will be the benefit of this change?

[neilbest-db]

@gueniai, if your previous comment refers to the following code in this diff in src/main/scala/com/databricks/labs/overwatch/pipeline/ETLDefinition.scala:

overwatch/src/main/scala/com/databricks/labs/overwatch/pipeline/ETLDefinition.scala

Lines 28 to 42 in 9a7fddb

	val transformedDF = transforms.foldLeft(verifiedSourceDF) {
	case (df, transform) =>
	/*
	* reverting Spark UI Job Group labels for now
	*
	* TODO: enumerate the regressions this would introduce
	* when the labels set by then platform are replaced
	* this way.
	* df.sparkSession.sparkContext.setJobGroup(
	* s"${module.pipeline.config.workspaceName}:${module.moduleName}",
	* transform.toString)
	*/
	df.transform( transform)
	}

. . . then you already know that the manipulation of the JobGroupID is commented out in this PR. I left that for future reference; see details below. The rest of this PR implements and demonstrates how to manipulate the job description and call site fields in the Spark UI in a convenient way. The core of that abstraction is here:

overwatch/src/main/scala/com/databricks/labs/overwatch/utils/TransformationDescriber.scala

Lines 49 to 50 in 9a7fddb

	sc.setJobDescription( namedTransformation.toString)
	sc.setCallSite( callSite)

I am currently investigating whether there might be an alternative to the current dependence on the Job Group ID, which is not always helpful in linking Spark events to Databricks Workflow job/task runs, especially in cases of:

• Legacy workflows spawned by dbutils.notebook.run()
• Workflow tasks run on all-purpose clusters
• (possibly other cases)

In those known scenarios, the Job Group ID either has a value that refers to the job_id and run_id of the spawned ephemeral run or some other value set by the platform that is NOT LIKE '%job-%-run-%'.

In any case, because SparkContext.setJobGroup() is a member of Spark's public API (Scala|Python), any workflow could override the value that Overwatch relies on to make certain associations. I intend to show that there is a more robust solution if possible. This is the main reason that I am taking such care in my current ongoing investigation that was initiated by a user report of null Spark Task durations in jobruncostpotentialfact_gold ("JRCP").