(AWS SDK for Java) Automatically disable segments in SDK background processes

[millems]

On the AWS SDK for Java 2.x, we sometimes make calls to AWS services as async background tasks, for example to cloudwatch to upload SDK metrics or to STS to refresh assumed-role credentials.

Customers that are using the automatic SDK instrumentation for x-ray are getting failures or warnings about a segment not being started in these background tasks because the SDK isn't starting a segment, and the customer doesn't control these service calls.

The AWS SDK for Java doesn't compile against the x-ray SDK, so we can't start a segment on behalf of the customer, but could we provide a "background task" flag to the x-ray execution interceptor to let it know that a segment isn't required because it isn't related to an ongoing trace?

I could even provide a pull request for this, as it should be relatively simple to implement on both sides myself.

Additional context from an affected user: aws/aws-sdk-java-v2#2919

[pingleig]

There is also v2 SDK integration in OpenTelemetry, not sure if the same problem would happen there https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/aws-sdk/aws-sdk-2.2/library/src/main/java/io/opentelemetry/instrumentation/awssdk/v2_2 @willarmiros

[Zhenye-Na]

Any updates for issue aws/aws-sdk-java-v2#3217 ?

Thanks!

[atshaw43]

Can you try removing the submodule aws-sdk-instrumentor and instrumenting only the clients you need?
https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-awssdkclients.html

You can also ignore missing context strategies to remove this from the logs, but this could hide issues in the future.
https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html

[mjgp2]

We've got multiple issues with this happening where we have having to put workarounds in place where there isn't a context in background jobs/threads, e.g. publishing cloudwatch metrics, using the S3 client, or even using quartz jobs.

It's really poor that we have this experience, and I am hoping that we can get some visibility on this somehow to figure out how we can get a working "out of the box" experience for xray.

Perhaps could we configure xray to create a segment if one is missing - although I am unsure where the segment name would come from, perhaps the thread name?

CC @atshaw43