[BUG] No telemetry exported form a Scala notebook when using AzureMonitorExporterBuilder

[greatvovan]

Describe the bug
When running a Scala notebook in a Databricks cluster, instrumentation using AzureMonitorExporterBuilder does not produce any output in the App Insights backend.

Exception or Stack Trace
No errors.

To Reproduce

1. Have a Databricks workspace
2. Have an App Insights instance
3. Create a smallest compute cluster with default configuration (I used "Personal Compute" policy).
4. Install com.azure:azure-monitor-opentelemetry-exporter:1.0.0-beta.28 in the cluster.
5. Create a Scala notebook
6. Run the below code in the notebook (copy in a cell and run)

Code Snippet

  val aiConnStr = "..."

  import org.apache.logging.log4j.LogManager
  import org.apache.logging.log4j.Level
  import org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilderFactory
  import org.apache.logging.log4j.core.config.Configurator
  import io.opentelemetry.instrumentation.log4j.appender.v2_17.OpenTelemetryAppender
  import io.opentelemetry.api.OpenTelemetry
  import io.opentelemetry.api.common.Attributes
  import io.opentelemetry.api.common.AttributeKey
  import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk
  import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder
  import com.azure.monitor.opentelemetry.exporter.AzureMonitorExporterBuilder
          
  val configBuilder = ConfigurationBuilderFactory.newConfigurationBuilder()
  val configuration = configBuilder
    .add(
      configBuilder
        .newAppender("Console", "CONSOLE")
    )
    .add(
      configBuilder
        .newAppender("Otel", "OpenTelemetry")
    )
    .add(
      configBuilder 
        .newRootLogger(Level.INFO)
        .add(configBuilder.newAppenderRef("Console"))
        .add(configBuilder.newAppenderRef("Otel"))
    )
    .build(false)
  Configurator.initialize(configuration)

  val sdkBuilder = AutoConfiguredOpenTelemetrySdk.builder()

  new AzureMonitorExporterBuilder()
      .connectionString(aiConnStr)
      .install(sdkBuilder)

  val openTelemetry = sdkBuilder.build().getOpenTelemetrySdk()
  OpenTelemetryAppender.install(openTelemetry)
  println("Auto-configured")

  val logger = LogManager.getLogger()

  logger.info("ScalaEA Log outside span")
  println("log sent")

  val attributes = Attributes.of[java.lang.String, java.lang.Long](
    AttributeKey.stringKey("foo"), "bar",
    AttributeKey.longKey("code"), 42L
  )

  val tracer = openTelemetry.getTracer("my-notebook")
  val span = tracer.spanBuilder("My-span-ScalaEA").startSpan()
  val scope = span.makeCurrent()
  try {
    span.addEvent("ScalaEA span event", attributes)
    logger.info("ScalaEA Log from span")
    Thread.sleep(1000L)
  }
  finally {
    scope.close()
    span.end()
  }
  println("trace sent")

  val meter = openTelemetry.getMeter("my-notebook")
  val gauge = meter.gaugeBuilder("my-gauge-ScalaEA").build()

  gauge.set(111, attributes)
  println("metric sent")

  openTelemetry.shutdown()

  println("Done")

Important: run the configuration code (till sdkBuilder.build().getOpenTelemetrySdk()) only once (see #41859). If you ran it and got ConfigurationException, restart the compute cluster (Run -> Restart compute resource).

The above code as a gist: https://gist.github.com/greatvovan/a2148bccf2c0e0c2305e8a35e0779dc3

Alternative (with manual configuration): https://gist.github.com/greatvovan/5e9e3d8ecaf4e210ade619c5b55455b3

Expected behavior
3 entries in traces table (2 logs, 1 event), 1 entry in customMetrics, 1 entry in dependencies table (span) created in App Insights

Screenshots
N/A

Additional context

• Sending telemetry from a Python notebook on the same cluster works fine, which I am guessing enough to rule out a connectivity problem.
• Documentation used.
• Based on examples.

Setup (please complete the following information):

• OS: Linux-like (unknown)
• IDE: Databricks Notebook
• Library/Libraries: com.azure:azure-monitor-opentelemetry-exporter:1.0.0-beta.28
• Java version: 1.8.0.412
• App Server/Environment: Databricks runtime 15.4 LTS ML (no GPU)
• Frameworks: No

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

EDIT: Updated the gists to look more like linear Scala programs.

[joshfree]

@lmolkova could you please take a look?

[lmolkova]

Thank you for reporting this @greatvovan!

I wonder if the problem could be around Scala notebook having different shutdown hooks and exiting before anything is exported.

Could you please try calling openTelemetry.shutdown() and awaiting it to complete to eliminate this? Thank you!

Otherwise, @jeanbisutti @heyams or @trask should be able to investigate further.

[greatvovan]

@lmolkova I tried openTelemetry.shutdown() as well as Thread.sleep(...) at the end, but it did not change anything.