[BUG] - SparkEventsBronze - Schema Exception

[GeekSheikh]

In certain circumstances where the sparkEventLogs have both executorAdded/Removed events and one of the following events with root level fields "Executor ID" and "executorId" respectively. This is not handled by the schemaScrubber and therefore needs to be manually handled.

Error is seen by the user in the pipReport as
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found duplicate column(s) in the data to save: executorid

Additional Event types that can cause this issue

org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage
org.apache.spark.scheduler.SparkListenerExecutorExcludedForStage
org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
org.apache.spark.scheduler.SparkListenerExecutorUnexcluded