Custom RequestTelemetry & RemoteDependencyTelemetry is no longer working with SDK when using Java 3.0 agent

[ajaythakran]

Expected behavior

Application Insights Java SDK v2.6.2 must work with Agent 3.0 as it used to work with Agent 2.6.2:

• to produce custom telemetry events: RequestTelemetry & RemoteDependencyTelemetry
• to propagate W3C correlation headers

Actual behavior

The telemetry events work fine without Agent v3.0.
When Agent v3.0.1 is used, following errors are seen in log:

• Error when trying to generate dependency traceparent using TraceContextCorrelation.generateChildDependencyTraceparent()
  AI: ERROR 18-01-2021 01:17:01.697+0000, 85(XNIO-2 task-1): Failed to generate child ID. Exception information: java.lang.RuntimeException: ThreadContext.getRequestTelemetryContext().getRequestTelemetry().getContext().getOperation() is not supported by the Application Insights for Java 3.0 agent
• Error when trying to get parent spanId
  java.lang.RuntimeException: ThreadContext.getRequestTelemetryContext().getRequestTelemetry().getId() is not supported by the Application Insights for Java 3.0 agent at com.microsoft.applicationinsights.telemetry.RequestTelemetry.getId(RequestTelemetry.java:161) ~[applicationinsights-web-2.6.2.jar:na]

Requesting Fix or Updated Guide on SDK usage w.r.t. Agent v3.0

As per new documentation at https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent, I noticed that the support for above mentioned operations is removed while using Agent 3.0.

But SDK is not useful if there is no way to Set current span in thread context or Access current parent span information. This is required specifically to start new span based on propagated correlation on consumption of message from system like Azure Service Bus Message or to propagate correlation headers (completes asynchronously) for dependency like Azure Service Bus Message. This will also include collection of telemetry request (on consuming) or dependency (on producing) respectively.

Could you please enable support for mentioned operations with Agent 3.0 or provide us guidance on how to use given SDK 2.6.2 for sending custom telemetry events RequestTelemetry & RemoteDependencyTelemetry

System information

Please provide the following information:

• SDK Version: 2.6.2
• OS type and version: Windows 10
• Application Server type and version (if applicable): Spring Boot v2.3 with Undertow
• Using spring-boot? Yes
• Additional relevant libraries (with version, if applicable): Agent v3.0.1

[trask]

Hi @ajaythakran, thanks for reporting this!

There are a couple of issues here:

• The first issue is that we are throwing exceptions, which is a regression between 3.0.0 and 3.0.1
• The second issue is that we didn't have interop coverage for those methods above

Both of these issues should be resolved by #1448, and we are planning to put out 3.0.2 release shortly for this and #1444.

[ajaythakran]

Hi @trask, thanks for quick turnaround.

We use following statement to set the request telemetry context which we assume to be made available for child dependencies span (include the one done with OpenTelemetry):
ThreadContext.setRequestTelemetryContext(requestTelemetryContext)

Will this method continue to work with new Agent 3.0 using interop or There is an alternative?

[trask]

hey @ajaythakran,

are you doing something like this?

RequestTelemetryContext context = new RequestTelemetryContext()
context.getHttpRequestTelemetry().getContext().getOperation().setId(...)
context.getHttpRequestTelemetry().getContext().getOperation().setParentId(...)
ThreadContext.setRequestTelemetryContext(context)
try {
  // auto-collected telemetry inside this block should use the above parent
  ...
} finally {
  ThreadContext.setRequestTelemetryContext(null)
}

[trask]

re-opening as there is still a remaining question/issue

[trask]

@ajaythakran I think your best bet for this use case:

RequestTelemetryContext context = new RequestTelemetryContext()
context.getHttpRequestTelemetry().getContext().getOperation().setId(<traceId>)
context.getHttpRequestTelemetry().getContext().getOperation().setParentId(<parentSpanId>)
ThreadContext.setRequestTelemetryContext(context)
try {
  // auto-collected telemetry inside this block should use the above parent
  ...
} finally {
  ThreadContext.setRequestTelemetryContext(null)
}

Is going to be using the OpenTelemetry API support (which is currently preview): https://github.com/microsoft/ApplicationInsights-Java/wiki/OpenTelemetry-API-support

OpenTelemetry API should be 1.0 in a month or so, at which time we will move this support out of preview (and support OpenTelemetry API 1.0+).

In OpenTelemetry API, the above would look something like:

SpanContext spanContext =
    SpanContext.createFromRemoteParent(
        <traceId>, <parentSpanId>, TraceFlags.getDefault(), TraceState.getDefault());
try (Scope scope = Context.root().with(Span.wrap(spanContext)).makeCurrent()) {
  //
}

Note: if you are parsing a W3C traceparent, it may make sense to use OpenTelemetry API's W3CTraceContextPropagator and implement a TextMapPropagator.Getter to extract the traceparent from your "request" object.

[ajaythakran]

Yes I am doing this way. Besides setting traceId and parentSpanId, I am also setting spanId after creating new Traceparent based on received traceId .

RequestTelemetryContext context = new RequestTelemetryContext();
context.getHttpRequestTelemetry().getContext().getOperation().setId(incomingTraceparent.getTraceId());//traceId
context.getHttpRequestTelemetry().getContext().getOperation().setParentId(incomingTraceparent.getSpanId());//parent spanId
Traceparent processedTraceparent = new Traceparent(0, incomingTraceparent.getTraceId(), null, incomingTraceparent.getTraceFlags());
context.getHttpRequestTelemetry().setId(processedTraceParent.getSpanId()); // spanId
ThreadContext.setRequestTelemetryContext(context)
try {
  // auto-collected telemetry inside this block should use the above parent
  ...
} finally {
  ThreadContext.setRequestTelemetryContext(null)
}

Thanks for notes on OpenTelemetry API. I will be looking forward for its 1.0 version.

[ajaythakran]

Hey @trask, I gave a quick try with new version 3.0.2 and noticed following issues:

1. In case received HTTP request is missing W3C trace headers, the Agent started new trace and collected request telemetry in App Insights but when application tried to create a dependency telemetry using SDK, the following error occurred:
   AI: ERROR 22-01-2021 04:56:59.221+0000, 92(XNIO-2 task-1): Failed to generate child ID. Exception information: java.lang.IllegalArgumentException: invalid traceId
   I am using this API to create dependency traceparent: TraceContextCorrelation.generateChildDependencyTraceparentObj()
2. All custom request telemetry is appearing without Operation Name in application insights. Which causes wrong telemetry aggregation under Performance in Application Insights.
3. cloud role name is missing for Spring Boot application which is used to be injected using SpringBootTelemetryInitializer in CloudContext.
4. Will Agent 3.0 have interop for ThreadContext.getRequestTelemetryContext().getTracestate()? Seen this error when tracestate is missing: No correlation wil happen, Thread context is null

[trask]

hey @ajaythakran, thanks for all the feedback!

(1), (2) and (4) should be fixed by #1454
(3) ya, in 3.0 you need to set cloud role name using env var (or the json config): https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#cloud-role-name

In case you have a chance to try it out, I'm attaching the CI build from PR #1454:
applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

very interested to see what else you find 😄👍

[ajaythakran]

Hey @trask Thanks for beta with fixes. I gave a quick try to beta version 3.0.3 and noticed following issues:

1. When request is instrumented by agent, the error is thrown while accessing tracestate: java.lang.IllegalArgumentException: invalid string in tracestate. I am using this API to retrieve tracestate and noticed Tracestate initialized with empty value: TraceContextCorrelation.retriveTracestate();
2. Using environment variables does not overwrite values set by SDK: APPLICATIONINSIGHTS_ROLE_NAME, APPLICATIONINSIGHTS_ROLE_INSTANCE. It is not really an issue, such information can be synced manually but would be idle to have a way to overwrite using env variables.

[trask]

@ajaythakran -

(1) should be fixed now
(2) should be fixed now also - agent's ikey, role name, and role instance are now applied

In case you have a chance to try it out again, I'm attaching the latest CI build: applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

Let us know what else you find!

[ajaythakran]

Hi @trask , Last reported issues are fixed but following issue came back:

• All custom request telemetry is appearing without Operation Name in application insights. Which causes wrong telemetry aggregation under Performance in Application Insights.

It would be nice to have operation method by default part of operation name for http request/dependencies (same as v2.x) as it plays important role in respect with REST services. I saw preview of processors to enable it but doesn't seem to work for me.

[trask]

All custom request telemetry is appearing without Operation Name in application insights

How are you setting the Operation Name on your custom request telemetry? I'm not seeing the same issue when testing it:

RequestTelemetry requestTelemetry = new RequestTelemetry(...);
requestTelemetry.getContext().getOperation().setName("my-operation-name");
telemetryClient.trackRequest(requestTelemetry);

I saw preview of processors to enable it but doesn't seem to work for me.

It sounds like you already tried the example here? https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-upgrade-from-2x#prefix-the-operation-name-with-the-http-method-get-post-etc

Can you provide an example of an auto-collected request telemetry that it's not being applied to? that might give us some clues why it's not working for you. E.g. when I try on spring-petclinic, I see this:

Thanks!

[ajaythakran]

Hi @trask,

All previously mentioned issues are resolved.

• Your observation was correct, I was not setting Operation Name. Earlier it used to copy from request name. I added Operation Name and is working.
• I was able to use preview of processors to add REST Method to Http Request Telemetry. I noticed my applicationinsights.json was having syntax issue.

New Issue:

• I noticed source is not captured from RequestTelemetry. It is useful in case we would like to capture the address of message source.

[trask]

Earlier it used to copy from request name

👍 This feature is coming back as part of #1467 (for both auto-collected telemetry and 2.x SDK collected telemetry)

EDIT: I'm not sure about this, because it won't play nicely with configuring the operation name via telemetry processors, need to think on this some more...

I noticed source is not captured from RequestTelemetry.

Do you mean the request telemetry you are viewing in the portal is missing that data, or when you call

ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry().getSource()

it is not working?

[ajaythakran]

Do you mean the request telemetry you are viewing in the portal is missing that data, or when you call

ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry().getSource()

it is not working?

Lets say for custom request telemetry, the source is set using below code:

RequestTelemetryContext requestCtx = new RequestTelemetryContext(System.currentTimeMillis());
RequestTelemetry requestTelemetry = requestCtx.getHttpRequestTelemetry();
requestTelemetry.setSource("endpoint:sb://samplenamespace.servicebus.windows.net | name:sampleTopic/subscriptions/sampleSubscription");
...
ThreadContext.setRequestTelemetryContext(requestCtx);

With agent 3.0.3, it is not available in application insights.

[trask]

hi @ajaythakran, this should work now in the latest SNAPSHOT: applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

[ajaythakran]

hi @ajaythakran, this should work now in the latest SNAPSHOT: applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

Hi @trask , I tried this version but still not able to find Source in Telemetry details at Azure Application Insights.

[trask]

oh no, I didn't read carefully, and solved this different problem 😂:

ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry().setSource("my source");

[trask]

hey @ajaythakran, here's the latest SNAPSHOT from CI that I believe addresses what you are looking for:

applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

[ajaythakran]

Hi @trask

applicationinsights-agent-3.0.3-BETA-SNAPSHOT.jar.zip

It worked!

When are you planning to release this version 3.0.3. I was thinking to submit this version to our performance team for review.
Thanks!

[trask]

hey @ajaythakran, we are planning to release 3.0.3-BETA end of this week or beginning of next week.

[trask]

3.0.3-BETA is released!