Kestrel stops listening/responding to requests after some time (only after updating to .NET 7.0)

[shaynevanasperen]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I have a Web Application which runs some background services and hosts some UI pages using Razor Pages. It was working perfectly fine on .NET 6.0, but after simply updating the target framework to net7.0 and redeploying to a folder on my machine, I started to encounter this problem.

Immediately after deploying the application, it works fine and continues to respond to requests from my browsers (Firefox, Chrome, Edge). But anywhere from 10 minutes to 6 hours later, the site no longer responds to HTTP requests (hitting refresh in my browser results in the connection timing out), and then it stays this way until I stop and restart it from the command line.

I have tried several things to attempt to narrow down the problem, but everything I have tried has failed. It's extremely difficult to do this because I have to wait so long between each try. As soon as I revert the target framework back to .NET 6.0, the problem goes away again and the site remains responsive permanently. So unfortunately I am unable to produce a minimalistic project which reproduces the issue.

I have a Prometheus /metrics endpoint exposed through the application and I have Grafana Agent configured on my machine to scrape the metrics every 15 seconds (from http://localhost/metrics). I have a branch in my request pipeline to exclude that endpoint from Authorization. The rest of the application is secured with MicrosoftIdentity authentication pointing to my Azure Active Directory tenant using the OpenIdConnect authentication scheme.

The really strange thing is that when the browser requests start to show connection timeouts, my Grafana Agent scraper is still able to scrape the /metrics endpoint every 15 seconds, which I can confirm from the logs. Here is a snippet from my Program.cs:

builder.WebHost.UseUrls($"https://{DomainName}", $"http://{DomainName}", "http://localhost");

if (!builder.Environment.IsDevelopment())
    builder.Services
        .AddLettuceEncrypt(LetsEncrypt.Options.Configure)
        .PersistCertificatesToAzureKeyVault(LetsEncrypt.AzureKeyVault.Configure);

builder.Services
    .AddRazorPages()
    .AddMicrosoftIdentityUI();

builder.Services
    .AddAuthorization(options => options.FallbackPolicy = options.DefaultPolicy)
    .AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(Authentication.MicrosoftIdentity.Configure);

I've tried disabling all my background sevices, and I even tried disabling authentication/authorization completely, and using self-signed certificates instead of LetsEncrypt, but none of these have had any effect.

Expected Behavior

The web application should continue to respond to HTTP requests indefinitely.

Steps To Reproduce

Host a .NET 7.0 Razor Pages web application on Windows 11. Wait for the site to stop responding to HTTP requests.

Exceptions (if any)

Nothing. Logs show nothing either.

.NET Version

7.0.100

Anything else?

No response

[davidfowl]

Host a .NET 7.0 Razor Pages web application on Windows 11. Wait for the site to stop responding to HTTP requests.

I'm not sure this is enough to reproduce the problem.

A couple more questions:

• Are you running on IIS?
• Can you capture a process dump when the application gets into a bad state?

[shaynevanasperen]

I'm not sure this is enough to reproduce the problem.

I will continue my attempts to narrow the problem down. However it is very frustrating because I have to wait so long each time before it get's into a bad state. I am currently running a test on another machine, as perhaps it has something to do with my environment. But even if that IS the case, why should it behave any differently than it does with .NET 6.0? (that's the only variable).

Are you running on IIS?

No. It is hosted with only with Kestrel and I launch the application from the command line using dotnet .\ApplicationName.dll

Can you capture a process dump when the application gets into a bad state?

I shall do this the next time that happens. I assume I should use this tool? https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump. Shall I upload the process dump somewhere after I have captured it?

[Tratcher]

Once you have the dump, open it in VS and look at the threads window for blocked threads, and the parallel tasks window for blocked tasks. That should give you an idea of where things are stalling.

[shaynevanasperen]

I'm not quite sure how to diagnose the issue using the dump, but when I view the Parallel Stacks, I can see that there are some blocked Async Logical Stacks:

Here is another view:

The source code for the Serilog.Sinks.Async.BackgroundWorkerSink looks like this:

The source code for Crypto.Websocket.Extensions.Core.OrderBooks.Sources.OrderBookSourceBase looks like this:

I noticed that both of them use:

Task.Factory.StartNew(...)

Could this be the culprit? Otherwise, could somebody point me in the right direction of what I should look for next?

[Tratcher]

StartNew is fine. _bufferPauseEvent.WaitOne() isn't great. What unblocks that event? There are async structures you can use to avoid blocking the thread, like SemaphoreSlim, Channels, ConcurrentQueue, etc.

[Tratcher]

Does BackgroundWorkerSink.Pump do something similar?

[shaynevanasperen]

_bufferPauseEvent.WaitOne()

I already tried excluding this code path entirely, and it didn't help

Does BackgroundWorkerSink.Pump do something similar?

If you follow the link to the source code, you can see what it does. But that's also irrelevant because I tried excluding this code as well and it didn't help either.

[shaynevanasperen]

In other news, ever since a few weeks ago my Visual Studio credentials keep expiring on me (like every few minutes) and then my app fails to launch until I refresh the credentials. If this happens overnight after my machine reboots to install Windows updates, then my app won't relaunch until I wake up and realize the problem. Prior to a few weeks ago, my credentials remained cached for months at a time.

And now, whenever I have to refresh the credentials, it hardly even works at all: I just get a loading spinner:

I think it all started happening after installing a recent update for Visual Studio. Could this have anything to do with it?

I have now tried running my app on a different machine, and the problem still occurs there too.

I am at my wits' end. I've probably spent over 40 hours trying to resolve this problem and I am no further than when I started. 😭

[BrennanConroy]

Can you share the parallel stacks (not tasks) window after disabling the serilog logging? Also, is it possible to share the .dmp file? (this can contain PII, so if the app contains customer info don't share it)

[shaynevanasperen]

Can you share the parallel stacks (not tasks) window after disabling the serilog logging?

This is the Parallel Stacks view after my most recent incident (which ran for about 7.5 hours before the problem occurred):

In this version, I have removed all of my own background services and the only services added to the container are done via the following lines of code:

builder.Logging.ClearProviders();
builder.Host.UseSerilog(configure);

builder.Services
    .AddLettuceEncrypt(LetsEncrypt.Options.Configure)
    .PersistCertificatesToAzureKeyVault(LetsEncrypt.AzureKeyVault.Configure);

services.AddSingleton<IPolicyEvaluator, AuthorizationPolicyEvaluator>();
services.AddPrometheusCounters();
services.AddPrometheusAspNetCoreMetrics();
services.AddPrometheusHttpClientMetrics();
services.AddPrometheusSqlClientMetrics();

builder.Services
    .AddRazorPages()
    .AddMicrosoftIdentityUI();

builder.Services
    .AddAuthorization(options => options.FallbackPolicy = options.DefaultPolicy)
    .AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(Authentication.MicrosoftIdentity.Configure);

My AuthorizationPolicyEvaluator looks like this:

class AuthorizationPolicyEvaluator : PolicyEvaluator
{
    public AuthorizationPolicyEvaluator(IAuthorizationService authorization) : base(authorization) { }

    public override async Task<PolicyAuthorizationResult> AuthorizeAsync(AuthorizationPolicy policy, AuthenticateResult authenticationResult, HttpContext context, object? resource)
    {
        if (context.Request.Host.HasValue && context.Request.Host.Value.StartsWith("localhost", StringComparison.OrdinalIgnoreCase) &&
            context.Request.Path.HasValue && context.Request.Path.Value == "/metrics")
            return PolicyAuthorizationResult.Success();

        return await base.AuthorizeAsync(policy, authenticationResult, context, resource);
    }
}

This version also includes Serilog logging, but I have now replaced the Serilog.Sinks.Async with Serilog.Sinks.Channels which doesn't contain the Task.Factory.StartNew(...) call I showed previously.

Also, is it possible to share the .dmp file?

This version contains code which loads App Configuration from Azure, which includes sensitive information (API keys) so unfortunately I can't share it this time. I will now remove that as well, and also remove Serilog logging and see if I can reproduce it that way. If I can reproduce it that way, how can I share the .dmp file (as it is about 1GB in size)? Perhaps if you give me an email address I can share the file with you via Google Drive?

[danmoseley]

The VS creds issue should be unrelated - please report it though the VS feedback button so they can help with that.

[shaynevanasperen]

I have now managed to create a minimalistic project which reproduces this issue: https://github.com/shaynevanasperen/WebApplication

I suspect that the problem has something to do with the LettuceEncrypt.Azure library not liking .NET 7.0. I wonder if it could actually be related to the VS credentials issue, because from my understanding, the LettuceEncrypt.Azure library will check my certificate for renewal once every 24 hours (maybe less) and when it decides that it is time to check again, it tries to load the certificate from Azure KeyVault, but when it does that, my credentials are expired and need to be refreshed manually? But then this should also be happening on .NET 6.0, so maybe not then. Maybe it is just some async code in LettuceEncrypt.Azure that causes a deadlock.

[shaynevanasperen]

Actually, I just re-read my initial comment where I had mentioned that I tried it with self-signed certificates instead of using LettuceEncrypt.Azure and that it showed the problem even then. I am now running a test with self-signed certificates in my minimalistic project. Will report back in several hours, if and when I see the problem again.

[shaynevanasperen]

I've just managed to capture it failing immediately upon startup. I'm not sure if this is the same issue but it smells like it could be related. This was with a version of the code where all the background services are disabled, but still using Serilog and LettuceEncrypt.

The log output is as follows:

PS C:\Alphabot\ProductionG20> dotnet .\MissionControl.dll
[17:03:10.332 INF] Loading Azure App Configuration ...
[17:03:12.131 INF] Loading Azure App Configuration from https://missioncontrol.azconfig.io/kv/?key=%2A&label=%00&api-version=1.0 ...
[17:03:14.001 INF] Finished loading Azure App Configuration.
[17:03:14.082 INF] WebApplication built.
[17:03:14.186 INF] Starting WebApplication ...
[17:03:14.204 DBG] Searching for certificate in KeyVault for alphabot.shaynevanasperen.net {"SourceContext": "LettuceEncrypt.Azure.Internal.AzureKeyVaultCertificateRepository"}
[17:03:21.049 INF] Found certificate for alphabot.shaynevanasperen.net from Azure Key Vault with thumbprint C287F457A433F3B92AEE7B1859F34EAE506A63D9 {"SourceContext": "LettuceEncrypt.Azure.Internal.AzureKeyVaultCertificateRepository"}
[17:03:21.054 DBG] Certificate for alphabot.shaynevanasperen.net already found. {"SourceContext": "LettuceEncrypt.Internal.AcmeStates.ServerStartupState"}
[17:03:21.055 DBG] Checking certificates' renewals for alphabot.shaynevanasperen.net {"SourceContext": "LettuceEncrypt.Internal.AcmeStates.CheckForRenewalState"}

Here is a screenshot of my Parallel Stacks when opening the .dmp file:

And this is the Tasks window:

And this is the Threads window:

[davidfowl]

Nothing there looks inherently bad. When you get into this state, do new requests get stuck? Do requests to specific URLs get stuck?
You mentioned this

The really strange thing is that when the browser requests start to show connection timeouts, my Grafana Agent scraper is still able to scrape the /metrics endpoint every 15 seconds, which I can confirm from the logs.

Can you pinpoint the requests that are failing? If it's only the https requests, then that might be a clue that its lettuce encrypt.

[shaynevanasperen]

When you get into this state, do new requests get stuck? Do requests to specific URLs get stuck?

No requests seem to be able to make it into my app from "outside", meaning by using the public DNS name. But requests from "inside" seem to work fine.

To be clear: "outside" means Browser requests to ANY application URLs starting with https://my.domain.net/. The same goes for HTTP, where the URL starts with http://my.domain.net/. If the HTTP requessts did make it to my application, then it would have just returned a redirect to HTTPS anyway, but it never even gets that far.

My Grafana Agent (Windows Service) scrapes my application using the URL http://localhost/metrics, and when the Browser requests start to time out as described, then somehow the scraper is still able to reach my application using the http://localhost/metrics URL. But what is really strange is that when I then try to hit that http://localhost/metrics from my Browser, it also times out (with the caveat that I did manage on one occasion to successfully hit the http://localhost/metrics URL from my Browser after the https://my.domain.net/ URLs had begun showing the timeout problem, but this only lasted a few minutes before they also timed out the same way).

It seems like Kestrel just stops listening on the port or something. But that can't be right because the Grafana Agent's requests still make it into my application. Could this be by virtue of the HTTP connection being kept alive because the Grafana Agent is hitting it so frequently (every 15 seconds), whereas the user-interface URLs are only being hit when I decide to press F5 in my browser, which can be after several hours of no requests from my browser?

Can you pinpoint the requests that are failing? If it's only the https requests, then that might be a clue that its lettuce encrypt.

No. ALL requests fail (except those coming from Grafana Agent). Even HTTP (not HTTPS) request fail, because I've opened the Network tab in my browser and I can confirm that the HTTP request did not make it into my application, because otherwise I would see the redirect to HTTPS.

The thing is that I have managed to reproduce this issue while using self-signed certificates (with my LettuceEncrypt code completely commented-out), so I know that it can't be due to LettuceEncrypt.

All clues currently seem to be pointing towards some sort of "KeepAlive" on the HTTP connection. Because the Grafana Agent is continually polling the http://localhost/metrics endpoint, it keeps the TCP connection alive. But my browser does not do this and so the connection "dies" after some time, afterwhich new requests try to establish a new TCP connection and then it fails.

To test this theory out, I will now try running the site as before but keep one of the UI pages open where I happen to have a JavaScript infinite loop running to fetch a partial view and update part of the page every 1 second. If my theory is correct, then this will prevent the timeout problem from occurring. I should know within the next 24 hours or so, and then I will report back with my findings.

[davidfowl]

Disable https completely and see if you can reproduce the issue

[shaynevanasperen]

Disable https completely and see if you can reproduce the issue

Unfortunately, this is not possible without ALSO disabling authentication completely. See screenshot below:

Nevertheless, I have now deployed a version of my application that has no capability for modifying any data via the UI, so that I can test it out with HTTPS and Authentication disabled completely.

Meanwhile, the test I described above (where I keep one of the web pages open which continually requests a partial view every 1 second) ran for a whole 48 hours without issue (the timeout problem did NOT occur). This seems to prove my theory correct, which is that the issue has something to do with the HTTP connection KeepAlive: If I keep the connection alive, the problem never occurs.

[shaynevanasperen]

Nevertheless, I have now deployed a version of my application that has no capability for modifying any data via the UI, so that I can test it out with HTTPS and Authentication disabled completely.

This test ran for about 2 hours before it started exhibiting the timeout issue. At least that's how much time passed before I checked it and noticed. It may have lasted far less than 2 hours. This further proves that my theory regarding the HTTP KeepAlive is probably correct.

I've now re-launched it in this cut-down mode with HTTPS and Authentication disabled completely, but now also with all my background services disabled. Let's see how long it lasts this time.

[shaynevanasperen]

I've now re-launched it in this cut-down mode with HTTPS and Authentication disabled completely, but now also with all my background services disabled. Let's see how long it lasts this time.

Yep, less than 2 hours passed before the timeout issue occurred again. This is with no HTTPS, no Authentication, and just the following code for building the WebApplication instance:

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.UseUrls($"http://{DomainName}:8020");

var options = new DefaultAzureCredentialOptions
{
    VisualStudioTenantId = tenantId,
    InteractiveBrowserTenantId = tenantId,
    ExcludeManagedIdentityCredential = true
};
var credentials = new DefaultAzureCredential(options);

builder.Configuration.AddAzureAppConfiguration(options => options
    .ConfigureClientOptions(x => x.AddPolicy(new LoggingPolicy(logger), HttpPipelinePosition.PerRetry))
    .Connect(new(endpoint), credentials)
    .ConfigureRefresh(x => x.Register("Sentinel", true))
    .ConfigureKeyVault(x => x.SetCredential(credentials)));

builder.Logging.ClearProviders();

builder.Host.UseSerilog(configure);

builder.Services.AddSingleton<IPolicyEvaluator, AuthorizationPolicyEvaluator>();
builder.Services.AddPrometheusCounters();
builder.Services.AddPrometheusAspNetCoreMetrics();
builder.Services.AddPrometheusHttpClientMetrics();
builder.Services.AddPrometheusSqlClientMetrics();

builder.Services
    .AddRazorPages()
    .AddMicrosoftIdentityUI();
	
var app = builder.Build();

This HTTP connection timeout problem does NOT occur under .NET 6.0 and it does NOT occur under .NET 7.0 on condition that I keep refreshing the web page frequently

Surely this is enough for you guys to pinpoint the problem now?

[davidfowl]

Surely this is enough for you guys to pinpoint the problem now?

Not really but appreciate the narrowing down of the issue thus far.

builder.WebHost.UseUrls($"http://{DomainName}:8020");

What is DomainName?

This repro looks pretty minimal. So the only external dependencies left are:

• Serilog
• Azure app config
• Prometheus.NET?

Can you tell me about the infrastructure around this application? When does an external request look like? I'd like to understand what it means when the request is stuck.

If requests aren't making it to the machine, where could it be getting stuck? Have you tried on verbose logging for Kestrel? This might show you if connections are being established but requests aren't being made over those connections.

Since you have Prometheus metrics, what do they look like when this situation happens?

From those dumps, I don't see anything that stands out with respect to something hanging or getting stuck in the request pipeline.

[shaynevanasperen]

Not really but appreciate the narrowing down of the issue thus far.

I had already linked to a minimalistic project which reproduces this issue. I have now further simplified this minimalistic project all the way down to the basic project template that you get from creating a new ASP.NET Core Web Application from the Visual Studio "new project" menu. So anyone should be able to reproduce this now by simply creating the barebones project from Visual Studio's File > New > Project... menu without adding any other code whatsoever!

What is DomainName?

Have a look here at my minimalistic sample application. You will see that it is test.shaynevanasperen.net.

This repro looks pretty minimal. So the only external dependencies left are:

• Serilog
• Azure app config
• Prometheus.NET?

I have now removed these dependencies as well, and I am still able to reproduce the problem.

Can you tell me about the infrastructure around this application? When does an external request look like? I'd like to understand what it means when the request is stuck.

Please have a look at the README file in my minimalistic sample application. It goes into great detail explaining the infrastructure and how the requests are being made etc.

If requests aren't making it to the machine, where could it be getting stuck? Have you tried on verbose logging for Kestrel? This might show you if connections are being established but requests aren't being made over those connections.

I have enabled verbose logging, but there is nothing showing in the logs.

Since you have Prometheus metrics, what do they look like when this situation happens?

There is nothing unusual to be seen.

Something else

As an aside which is unrelated to this issue (hopefully), a friend of mine recently alerted me to some alarming code in EF Core 7.0. Please have a look at the code here, and here. You can see that in both of those places, there is a completely unnecessary loop that is repeated 10000 times! And yet the pull request was merged by the maintainers of that project anyway! If the code in this repository is maintained with the same level of care and attention to detail, then frankly I am not surprised that problems like the one I am facing have surfaced.

As a result, this friend and I have now determined that .NET 7.0 is unfortunately not yet ready for production applications.

[davidfowl]

We'll take a deeper look at your application and see if we can reproduce given the instructions. For what I read on this repository though it's not clear if the reproduction of the issue means hitting f5 in the browser continuously (sorry if I missed that). After building the application, deploying it somewhere, and mapping it via a google domain, do I need to send requests for a day to see this issue happen?

As an aside which is unrelated to this issue (hopefully), a friend of mine recently alerted me to some alarming code in EF Core 7.0. Please have a look at the code here, and here. You can see that in both of those places, there is a completely unnecessary loop that is repeated 10000 times! And yet the pull request was merged by the maintainers of that project anyway! If the code in this repository is maintained with the same level of care and attention to detail, then frankly I am not surprised that problems like the one I am facing have surfaced.

Ouch dotnet/efcore#29642

As a result, this friend and I have now determined that .NET 7.0 is unfortunately not yet ready for production applications.

There are people running on .NET 7 and aren't having this problem. This isn't to say the problem doesn't exist, but there's something about it that we don't quite understand yet.

[shaynevanasperen]

For what I read on this repository though it's not clear if the reproduction of the issue means hitting f5 in the browser continuously (sorry if I missed that).

The way I have been testing it is to simply deploy it and then "check in" on it every few hours or so by hitting f5 in the browser. In the several hours of my absense (between each time that I "check in" on it) my Grafana Agent keeps hitting the /metrics endpoint every 15 seconds. So there may be hundreds of requests issued by Grafana Agent in the background by the time I come back to test it again with my browser.

What I have noticed however, is that if I set up a script to essentially hit f5 in my browser every 1 second, then when I come back to it after many hours, or even after a day or so, the manual requests from my browser continue to work (the problem never happens). It is ONLY when I go away for a long time and then come back after a long period of NOT making ANY browser requests, that the problem occurs.

After building the application, deploying it somewhere, and mapping it via a google domain, do I need to send requests for a day to see this issue happen?

No. You need to do the OPPOSITE of that. Leave the website IDLE with no requests for a day or more, and THEN try to make a request from your browser. This is with the caveat that requests are continuously being made from Grafana Agent in the background (not from the browser). I wonder if I should try to see if this problem still occurs when I disable my Grafana Agent from making those background requests?

[davidfowl]

OK so I don't plan to setup a grafana agent. The repro steps you suggest are to deploy, hit it a few times and then leave it for a while (couple hours) then come back and try to hit it again, both from an external URL and an the internal URL until one of those don't respond. Is that right?

[shaynevanasperen]

The repro steps you suggest are to deploy, hit it a few times and then leave it for a while (couple hours) then come back and try to hit it again, both from an external URL and an the internal URL until one of those don't respond. Is that right?

Yes. It might take 2 or 3 days though, not just a few hours.

[shaynevanasperen]

OK so I don't plan to setup a grafana agent.

You could write a shell script to hit /metrics every 15 seconds.

[adityamandaleeka]

@shaynevanasperen It would be great if you could narrow down a repro that's a bit easier to run. As you can imagine, it's challenging for us to reproduce issues that might take 2 or 3 days.

[adityamandaleeka]

Alternatively, if you could capture a dump and/or log info when it's in the bad state, that could help us diagnose the issue.

[shaynevanasperen]

It would be great if you could narrow down a repro that's a bit easier to run. As you can imagine, it's challenging for us to reproduce issues that might take 2 or 3 days.

I already have narrowed down the repro. It's as basic as possible already. Unfortunately the nature of this issue is that it can take that long to manifest.

Alternatively, if you could capture a dump and/or log info when it's in the bad state, that could help us diagnose the issue.

I have already done this before and there is nothing that stands out (see above).

[adityamandaleeka]

I see, thanks @shaynevanasperen. Unfortunately, we don't have enough information to productively debug this at this point.

If you end up getting more information about it (or if anyone else hits this issue), please let us know (here or in a new issue) and we can help investigate.

[adityamandaleeka]

Closing this issue for now.