Deadlock between GarbageCollectGeneration and HttpClient on Linux #12345
Comments
|
We have fixed similar bug for 2.1 : https://github.com/dotnet/coreclr/issues/9745 Could you please check that this is really happening on 2.2 runtime? The CurlHandler is not enable by default on 2.2 - are you explicitly enabling it for your app? |
|
Yes, this is happening on 2.2.1 runtime. The stack traces I copied were actually from running on 2.0.0 but they are identical on 2.2.1. I am not aware of any configuration regarding whether the CurlHandler has been explicitly enabled and it isn't something I have done intentionally. How is this done? I would be interested in disabling it as a possible workaround? |
|
The call stacks look strange. First, the "(MethodDesc 00007fd2c927c808 + 0x.........." for all managed functions is something I've never seen before. Then the middle frame in the following sequence of stack frames doesn't make sense: It seems as if you have incorrect debug info. Can you please share how you've got these stack traces? Are they from a live session or a core? If it was from core, have you loaded the core into lldb in the docker container or outside of it? |
|
I docker exec into the container and attach to the live but deadlocked process with lldb. I use the libsosplugin.so from the installed runtime. Apologies for dumping my session here but it is the best way to answer your questions. Please note that I have truncated the GC thread to remove references to my application code. |
|
@darrenod thank you for the details. Dumpstack is not the best command to get the managed stack trace. Can you please use |
|
Thanks for the tip, |
|
@darrenod this really looks like the issue that @jkotas has mentioned and that was fixed. I wonder - is it possible that you are executing the app using an old .NET core runtime that's copied in the application directory? |
|
The way we are using I am not sure how best to prove that it is a 2.2 version but I can get the commit hash if it helps |
|
@darrenod thank you for the commit hash. That commit is from Jul 19, 2017, so pretty old. Looks like 2.0.0 runtime. dotnet publish publishes stuff from runtime that you have specified in your .csproj files. So when your .csproj files contain netcoreapp2.0, then even if you build with 2.2 or 3.0 SDK, your app will be published with 2.0 runtime. |
|
@janvorli That is my problem. It is adding the unintended libcoreclr.so to the app directory. I had this in the .csproj file... I removed the I trust, as you say, that the issue I reported is most likely resolved in 2.2 and I can forget about it. Thanks very much for the help with this! |
|
@darrenod you're welcome! |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I am experiencing a reasonably frequent deadlock between a thread initiating garbage collection and the CurlHandler worker thread which is part of the HttpClient implementation on Linux.
This is running in docker on CentOS 7 with base image microsoft/dotnet:2.2.1-runtime. Also occurs with the 2.0.0-runtime image version.
When the deadlock occurs the whole application freezes and requires killing and restarting.
When attaching to the process there are about 20 threads in the application and the majority are inside
libcoreclr.so!Thread::RareDisablePreemptiveGC()which I assume (perhaps incorrectly?) means they are suspended for GC. However, I always see the same pattern in the following two stacks; the thread which initiated the GC and the CurlHandler thread. My comments inline.Having looked at the stacks and the coreclr code I have come to the conclusion that the CurlHandler thread has acquired a read lock here inside
libcoreclr.so!ExecutionManager::IsManagedCodeWithLock. Then GC is initiated in another thread which causes the CurlHandler thread to be suspended while it holds the read lock. The GC thread then proceeds to attempt to acquire the write lock on the same resource and we have a deadlock.This is causing quite a headache in our production application so would appreciate some guidance as to whether this is a genuine bug or if there is something I should be doing to avoid this scenario.
Thanks.
The text was updated successfully, but these errors were encountered: