Skip to content

Commit

Permalink
revert Spark UI Job Group labels
Browse files Browse the repository at this point in the history
prevent certain regressions when the Job Group labels set by the platform are no longer available for parsing.  Labels set by the platform contain tokens that are necessary to preserve referential integrity under certain conditions.  (Which conditions?)
  • Loading branch information
neilbest-db committed Jun 29, 2024
1 parent 2ead752 commit 9fe9f8c
Showing 1 changed file with 12 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,18 @@ class ETLDefinition(

val transformedDF = transforms.foldLeft(verifiedSourceDF) {
case (df, transform) =>
df.sparkSession.sparkContext.setJobGroup(
s"${module.pipeline.config.workspaceName}:${module.moduleName}",
transform.toString)

/*
* reverting Spark UI Job Group labels for now
*
* TODO: enumerate the regressions this would introduce
* when the labels set by then platform are replaced
* this way.
* df.sparkSession.sparkContext.setJobGroup(
* s"${module.pipeline.config.workspaceName}:${module.moduleName}",
* transform.toString)
*/

df.transform( transform)
}
write(transformedDF, module)
Expand Down

0 comments on commit 9fe9f8c

Please sign in to comment.