Incorrect event coupling in Trace view?

[RsZoli]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

Calling a REST API enpoint from a .NET MAUI application when the internet connection switches from cellular to wifi, there is an exception thrown. After the internet connection is stablizied, and re-calling the API, the exception's trace is linked to the second API call attempt, they share trace ids!

Expected Result

Don't show the exception from the previous failed attempt on the actual trace that went okay.

Actual Result

As you can see here, the Exception is a mobile event, shows up in the issues page of the mobile project, happened before the 3 http request metric event and they are from 2 separate attempts to call the API!

Product Area

Performance

Link

https://rszoli.sentry.io/performance/trace/11ac128e425e4d10af003a46670d4105/?limit=100&pageEnd=2024-04-09T05%3A54%3A27.640&pageStart=2024-04-08T05%3A54%3A27.640

DSN

https://dcbc75e91c8801842d8d759d8f708a55@o4506944792035328.ingest.us.sentry.io/4507045300273152

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[bitsandfoxes]

Just so I get that right:

1. Mid-requests the internet connection switches
2. This triggers an exception in the request
3. The app makes the same request again
4. The trace ID got "reused" and multiple requests show up as part of the same trace

[RsZoli]

@bitsandfoxes Indeed! Also, since some time passed, it got even more mixed up:

As you can see, the lime rectangle are events from the mobile application, those events didn't reach the API. The yellow rectangle are events from the same day, from another mobil session, since there was no error on the mobile, they have all reached the API. The blue rectangles are mixed events, there was some errors on the mobile but those are not related to the API calls, but still, this is good that i can see them couple!

However, this 3 separate colored rectangles contains events that should not be coupled together in the same trace!

[jamescrosswell]

@RsZoli how are you creating the transactions that represent that trace? Do these start in the MAUI client application? And how/when are those transactions getting finished? Is it possible that you have a transaction that remains open for extended periods of time (e.g. days)?

[RsZoli]

@jamescrosswell In the MAUI application, i simply use this code: SentrySdk.CaptureException(ex);

This is that generates the exceptions that you can see in the lime box on my attached picture! These don't have counterparts on the API, but the traces in the yellow boxes are generated automatically by your Nuget packages, i'm using this code in the API:

        **builder.Services.AddOpenTelemetry().WithTracing(p =>
        {
            p.SetSampler(new AlwaysOnSampler());
            p.AddAspNetCoreInstrumentation();
            p.AddHttpClientInstrumentation();
            p.AddSentry();
        });

        SentrySdk.Init(p =>
        {
            p.Dsn = ***;
            p.TracesSampleRate = 1.0;
            p.UseOpenTelemetry();
        });**

As you can see, 2 days passed in between the transactions, the API of course run in 24/7, but the application on the mobile device always get restarted, hence the events indication in the different colored boxes always went through a newly created httpclient!

[jamescrosswell]

the traces in the yellow boxes are generated automatically by your Nuget packages, i'm using this code in the API

Hi @RsZoli, so the MAUI client isn't instrumented with Sentry?

My thinking was that if the MAUI client is instrumented with Sentry, then it's the client that creates transactions. It would then pass a trace id for the transaction in (via headers) to any HTTP requests that it was making to your HTTP server. If the MAUI client held those transactions open for days, then separate HTTP requests to your ASP.NET Core server would all be made using the same correlation ids and that would explain the behaviour you're seeing in your trace logs.

In fact, since you're using OpenTelemetry, the MAUI client doesn't even need to be instrumented with Sentry... if it's instrumented via OpenTelemetry then it would also pass in a trace id (via the OpenTelemetry headers), which Sentry would pick up and run with in your ASP.NET Core app.

[RsZoli]

@jamescrosswell Hey! Sorry, i forgot to mention, i do use this code in my MAUI project per your documentation: using var httpClient = new HttpClient(new SentryHttpMessageHandler()); I also have AutoSessionTracking and CaptureFailedRequests set to true, but those doesn't seem to do anything for me, from the MAUI application i see data only gathered by the SentrySdk.CaptureException() function.

The OpenTelemetry is only used in the API, not in the MAUI project!

[bitsandfoxes]

Hey @RsZoli, thanks for the additional details. We'll be revisiting this.

[bitsandfoxes]

So my hunch is that all of this is happening within the same scope, reusing the same trace propagation context and leading to all of those being part of the same trace ID.

[bitsandfoxes]

Hey @RsZoli,
it looks to us like the MAUI client is keeping the transaction open while doing multiple requests spanning days. Are you creating the transaction yourself in MAUI?

[RsZoli]

@bitsandfoxes Hey, i'm sorry for the late response!

No, the only thing, i use is capturing httpclient errors per your documentation:
using var httpClient = new HttpClient(new SentryHttpMessageHandler());

This my configuration:

builder.UseSentry(p =>
{
    p.AutoSessionTracking = true;
    p.CaptureFailedRequests = true;
    p.Dsn = GetConfiguration($"SentryDsns:{DeviceInfo.Current.Platform.ToString()}:{GetConfiguration("Environment")}");
    p.ExperimentalMetrics = new ExperimentalMetricsOptions
    {
        EnableCodeLocations = true
    };
    p.TracesSampleRate = 1.0;
    p.ProfilesSampleRate = 1.0;
});

Maybe it is worth to mention that from MAUI, no metrics are gathered, also i see nothing on the performance page either but this was always the case, however since your latest update to the trace UI, i no longer see the related API calls to the MAUI exception!

[bitsandfoxes]

From what I gathered, this seems to be a bug or a case that is not properly covered right now. The outgoing spans all refer back to one single trace ID that is held on the scope. That ID gets created during initialization and currently is not getting updated through the lifetime of the application.
As a workaround you could either push a new scope or wrap your outgoing requests in a transaction.

[jamescrosswell]

I think this behaviour was changed with:

• revert: Tracing for SentryHttpMessageHandler with no active transaction #3367

The SentryHttpMessageHandler will no longer create a transaction if one does not exist... so unless @RsZoli explicitly creates a transaction in the MAUI front end, none of these traces (spanning multiple days) would be sent.

I think we can close this issue then.