W3C Distributed tracing not working under java:3.0.1-BETA.2

[petergphillips]

We've just tried switching from the 2.6.2 agent to the 3.0 agent.

In the 2.6.2 AI-Agent.xml file we used to specify

<HTTP enabled="true" W3C="true"/>

in order to enable w3c distributed tracing.

In the 3.0 applicationinsights.json I can't find the equivalent configuration, but https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent suggests that distributed trace propagation is automatically supported.

However looking at application insights I can see that the distributed tracing isn't working. Our node app calls to our kotlin spring boot app and would expect that the operation_Id is then kept the same e.g.

requests
| where operation_Id == "43c31c9ed3244d5db8068a0838a148c9"

would show all the requests across all our applications for that initial user request.

This works using the 2.6.2 agent, but has now stopped working in 3.0 and the requests to our kotlin application are given a new operation_Id each time. Note that this affects both incoming traffic and also outgoing traffic.

• SDK Version: 2.6.2
• Java Agent Version: 3.0.1-BETA.2
• OS type and version: Debian 14
• Application Server type and version (if applicable): Spring Boot 2.4.1
• Using spring-boot? yes

[petergphillips]

Looking at https://docs.microsoft.com/en-us/azure/azure-monitor/app/correlation#java-30-agent it states

Java 3.0 agent supports W3C out of the box and no additional configuration is needed.

but that doesn't appear to be the case for our applications.
Our node app sets

    .setDistributedTracingMode(appInsights.DistributedTracingModes.AI_AND_W3C)

and it all works successfully under the 2.6.2 agent.

[trask]

Hi @petergphillips! This should be working. Can you post a repro that I can use to debug the issue?

[petergphillips]

Sure, I'll have a look at replicating and creating a repo. Did notice just now that we are getting

2021-01-08 11:38:17.013Z WARN  muzzleMatcher - Instrumentation skipped, mismatched references were found: netty -- io.opentelemetry.javaagent.instrumentation.netty.v4_0.NettyInstrumentationModule on jdk.internal.loader.ClassLoaders$AppClassLoader@75b84c92
2021-01-08 11:38:17.039Z WARN  muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.netty.v4_0.AttributeKeys:53 Missing method io.netty.util.AttributeKey#<init>(Ljava/lang/String;)V

on startup. We're using netty 4.1.55 at present which has that deprecated constructor removed so presumably that will be the reason the instrumentation isn't working?

[trask]

ah, you can ignore that warning, it just means that the netty-4.0 instrumentation was not applied, which is good, since you are using netty-4.1, and we have separate instrumentation for netty-4.1 (I just fixed that warning message a couple days ago in #1427)

[petergphillips]

Project is https://github.com/ministryofjustice/prison-estate.git. Need Java 11. Replace with your key for app insights.

Demonstrating working under agent 2.6.2.

./gradlew clean assemble
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=<your instrumentation key> java -javaagent:build/libs/applicationinsights-agent-2.6.2.jar -Dspring.profiles.active=dev -jar build/libs/prison-estate-*.jar
curl http://localhost:8080/prisons/id/MDI -H 'traceparent:00-39041612baf14e769ee5ae91b39a75ff-58c08397197c4397-01'

Similarly not working under 3.0.1:

git checkout pgp-DT-1482-insights-v3
./gradlew clean assemble
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=<your instrumentation key> java -javaagent:build/libs/applicationinsights-agent-3.0.1-BETA.2.jar -Dspring.profiles.active=dev -jar build/libs/prison-estate-*.jar
curl http://localhost:8080/prisons/id/MDI -H 'traceparent:00-39041612baf14e769ee5ae91b39a75ff-58c08397197c4397-01'

In app insights

requests
| where operation_Id == "39041612baf14e769ee5ae91b39a75ff"

should then show the curl requests, but only 2.6.2 requests are shown - 3.0.1 requests have generated their own operation id.

[trask]

Thanks for the repro!!

It looks like I broke propagation when using APPLICATIONINSIGHTS_CONNECTION_STRING env var in 3.0.1-BETA.

We are planning to make 3.0.1 release this week, so very grateful for you finding this, we will fix before releasing.

In the meantime, you can either use 3.0.0, or specify connectionString in the json config file.

[petergphillips]

Thanks @trask. I've had a go specifying the connection string in the json file as

  "connectionString": "${APPLICATIONINSIGHTS_CONNECTION_STRING}"

and I can confirm that I can now see the requests come in with the same operation_Id. However outgoing requests using webclient don't get the http headers added to our requests so we can't therefore correlate the requests with downstream systems. I can see the outgoing requests in dependencies though with the correct operation_Id, which is an improvement over 2.6.2, but was hoping to get the requests traced all the way through. I'll try and create another repo demonstrating the issue.

[petergphillips]

Just to let you know that my initial effort to reproduce has failed, in that the requests are being correlated correctly. I'll investigate further to see how I can reproduce.