System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

karelz · 2022-08-27T14:42:56Z

Frequency as of:

9/9 - last 30 days of System.Net.Security.Tests (with Core Dump) - all arm64-NativeAOT only
9/9 - last 30 days of System.Net.WebSockets.Client.Tests (with Core Dump) - all arm64-NativeAOT only
9/9 - last 30 days of System.Net.Http.Functional.Tests (with Core Dump) - mostly arm64-NativeAOT with 2 exceptions crossed out in the table
9/9 - last 30 days of System.Security.Cryptography.Tests (with Core Dump) - all arm64-NativeAOT only

Day	Test suite	Run	OS	Details
9/8	System.Security.Cryptography.Tests	PR 9648 - PR #75306 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
9/8	System.Security.Cryptography.Tests	PR 9492 - PR #75213 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
9/4	System.Net.WebSockets.Client.Tests	Rolling run 4489 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
9/2	System.Net.Http.Functional.Tests	Rolling run 3461 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
9/1	System.Net.Http.Functional.Tests	Rolling run 1981002 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/28	System.Net.Security.Tests	Rolling run 1972481 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
~~8/28~~	~~System.Net.Http.Functional.Tests~~	~~Rolling run 1972750~~	~~net7.0-windows-Release-x64-NativeAOT_Release-Windows.81.Amd64.Open~~	~~Core Dump - Unhandled Exception: System.NullReferenceException~~
8/28	System.Net.Http.Functional.Tests	Rolling run 1972413 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/27	System.Net.WebSockets.Client.Tests	Rolling run 1972127 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
8/26	System.Net.Security.Tests	Rolling run 1969936 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
~~8/25~~	~~System.Net.Http.Functional.Tests~~	~~Merged PR 1966082~~	~~net7.0-windows-Release-x64-NativeAOT_Release-Windows.81.Amd64.Open~~	~~Core Dump - [FAIL] System.Net.Http.Functional.Tests.SyncHttpHandlerTest_AutoRedirect.AllowAutoRedirect_True_ValidateNewMethodUsedOnRedirection~~
8/21	System.Net.Http.Functional.Tests	Rolling run 1957446 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/20	System.Net.Security.Tests	Rolling run 1956604 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/20	System.Net.WebSockets.Client.Tests	Rolling run 1955765 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL

Not quite blocking CI yet, but close ...

ghost · 2022-08-27T14:44:38Z

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Frequency as of 8/27 - last 30 days (with Core dump):

Day	Run	OS	Details
8/26	Rolling run 1969936 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump
8/20	Rolling run 1956604 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump

Not quite blocking CI yet, but close ...

Author:	karelz
Assignees:	-
Labels:	`arch-arm64`, `area-System.Net.Security`
Milestone:	-

jkotas · 2022-08-27T16:28:46Z

Stacktrace:

0x0000007f`84350000
libcrypto_so_1!X509_get_pubkey_parameters+0xf18
libcrypto_so_1!X509_verify_cert+0x78
System_Net_Security!System_Security_Cryptography_Interop_Crypto___CryptoNative_X509VerifyCert_g____PInvoke_275_0+0x3c [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 7264] 
System_Net_Security!System_Security_Cryptography_Interop_Crypto::CryptoNative_X509VerifyCert+0x38 [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 5839] 
System_Net_Security!System_Security_Cryptography_Interop_Crypto::X509VerifyCert+0x1c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.X509.cs @ 196] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::BuildWorkingChain+0x98 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 653] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::Finish+0x58 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 688] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChainCore+0x320 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 198] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChain+0xb4 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 49] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_X509Chain::Build+0x308 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Chain.cs @ 126] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStreamCertificateContext::Create+0x184 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamCertificateContext.cs @ 56] 
System_Net_Security!System_Net_Security_System_Net_Security_SslAuthenticationOptions::UpdateOptions+0x174 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.cs @ 123] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x24 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 426] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x8c [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 419] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_AsyncSslStreamSystemDefaultTest::AuthenticateServerAsync+0x48 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 231] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest___c__DisplayClass9_0::<ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails>b__0+0x70 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync_d__95::MoveNext+0x38 [/_/src/xunit.assert/Asserts/Record.cs @ 104] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync+0x30 [/_/src/xunit.assert/Asserts/Record.cs @ 75] 
System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync_d__67_1<System___Canon>::MoveNext+0x48 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 132] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x64 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync<System___Canon>+0x44 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 119] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails_d__9::MoveNext+0x1ac [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails+0x38 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 30]

jkotas · 2022-08-27T16:51:20Z

A function pointer in libcrypto data structures is bad. Use-after-free bug?

84350000 is the corrupted function pointer. 8490... are the good function pointers around it.

0000007f`7c0dbaf0  0000007f`8490f1b0 0000007f`84350000
0000007f`7c0dbb00  0000007f`8490b8e0 0000007f`8490e290

karelz · 2022-09-01T16:23:08Z

Triage: We should treat it as high pri for 7.0 until we have further details about root cause.

karelz · 2022-09-09T00:29:47Z

cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...

jkotas · 2022-09-09T05:59:24Z

I have taken a look at the two most recent core dumps. It is still same pattern - function pointer overwritten in libcrypto
data structure.

My guess is that this crash is another manifestation of the problem that is behind other mysterious crashes in networking tests: #74795 and #72830 . There seems to be a stray memory write in networking that corrupts random memory.

MichalStrehovsky · 2022-09-09T06:07:41Z

I was looking at the System.Security.Cryptography.Tests crash in #75306 that also looks similar (we end up in some location that is not valid code) - https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75306-merge-171bd00681b64e6184/System.Security.Cryptography.Tests/1/console.108589f8.log?helixlogtype=result.

That one is not a networking test, but libcrypto is the common theme. Maybe a crypto bug instead of networking bug?

cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...

I'm not aware of any active ARM64 issues with NativeAOT. The one we had was fixed two weeks ago.

MichalStrehovsky · 2022-09-09T06:12:07Z

The stack for the System.Security.Cryptography test failure is (notice the offset of the last frame is very far away from anything known):

00 0000007f`6affacb0 0000007f`84be2294     libcrypto_so_1!shadow_DES_check_key+0x73ee640
01 0000007f`6affacb0 0000007f`84be2f44     libcrypto_so_1!X509_get_pubkey_parameters+0x6d4
02 0000007f`6affad50 00000055`7dc1e48c     libcrypto_so_1!X509_verify_cert+0xb0
03 0000007f`6affad90 00000055`7dc16e48     System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto___CryptoNative_X509VerifyCert_g____PInvoke_275_0+0x3c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.Cipher.cs @ 235] 
04 0000007f`6affae20 00000055`7dc16ecc     System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto::CryptoNative_X509VerifyCert+0x38 [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 5824] 
05 0000007f`6affae50 00000055`7dc8fd48     System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto::X509VerifyCert+0x1c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.X509.cs @ 196] 
06 0000007f`6affae70 00000055`7dc8fe38     System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::BuildWorkingChain+0x98 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 653] 
07 0000007f`6affaea0 00000055`7dc70fd0     System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::Finish+0x58 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 688] 
08 0000007f`6affaee0 00000055`7dc70bf0     System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChainCore+0x320 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 198] 
09 0000007f`6affaf50 00000055`7dc80058     System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChain+0xb0 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 49] 
0a 0000007f`6affafd0 00000055`7dab5410     System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_X509Chain::Build+0x308 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Chain.cs @ 126] 
0b 0000007f`6affb090 00000055`7e4bf048     System_Security_Cryptography!System_Security_Cryptography_Tests_System_Security_Cryptography_X509Certificates_Tests_DynamicChainTests::BasicConstraints_ExceedMaximumPathLength+0x240 [/_/src/libraries/System.Security.Cryptography/tests/X509Certificates/DynamicChainTests.cs @ 291]

karelz · 2022-09-09T13:53:06Z

I added System.Security.Cryptography into the table in top post. There have been 2 PRs yesterday to hit it.
@bartonjs any ideas?

jkotas · 2022-09-09T15:35:05Z

System.Security.Cryptography

The interesting detail in this dump is that there is libmsquic_lttng_so_2_2 loaded, but libmsquic is not. It means that the process loaded libmsquic at some point in the past for the msquic IsSupported check, unloaded it, but libmsquic_lttng_so_2_2 stayed around.

@rzikm @wfurt @karelz Is it expected that libmsquic_lttng_so_2_2 stays loaded even after libmsquic is unloaded? Any chance that there is a dangling pointer in libmsquic_lttng_so_2_2 that gets written to by lttng and causes random crashes?

There was change in the msquic unloading done yesterday (#75163) that can explain why we have started seeing this crash in a new test (due to changes in timing of the corruption, etc.).

wfurt · 2022-09-09T16:28:10Z

cc: @nibanks. My understanding is that libmsquic_lttng is bridge from msquic to Linux tracing. AFAIK the main focus was to avoid active running threads more so than completely unload. The crashes seems older than #75163, right?

bartonjs · 2022-09-09T16:41:27Z

00 0000007f`6affacb0 0000007f`84be2294     libcrypto_so_1!shadow_DES_check_key+0x73ee640
01 0000007f`6affacb0 0000007f`84be2f44     libcrypto_so_1!X509_get_pubkey_parameters+0x6d4

I can think of no reason why X509_get_pubkey_parameters would call (shadow_)DES_check_key. https://github.com/openssl/openssl/blob/6e6aad333f26694ff39aba1e59b358e3f25a9a1d/crypto/x509/x509_vfy.c#L1948-L1981

DES would have nothing to do with a chain build. SO some sort of memory corruption is happening somewhere. (I don't even see anything there that would be calling an object-stored fnptr, so something has corrupted executable pages)

MichalStrehovsky · 2022-09-09T20:18:01Z

I can think of no reason why X509_get_pubkey_parameters would call (shadow_)DES_check_key.

Unless check_key is more than 0x73ee640 bytes long, we're not actually in check_key method body. It was just the symbol debugger picked up.

jkotas · 2022-09-09T22:39:23Z

Right, the above stacktrace is with export symbols only so it is not accurate. For reference, the stacktrace with resolved private symbols is:

0x0000007f`8c080000
libcrypto_so_1!verify_cb_cert+0xa8 [./build_shared/../crypto/x509/x509_vfy.c @ 162] 
libcrypto_so_1!build_chain+0xf68 [./build_shared/../crypto/x509/x509_vfy.c @ 3209] 
libcrypto_so_1!verify_chain+0xf84 [./build_shared/../crypto/x509/x509_vfy.c @ 218] 
libcrypto_so_1!X509_verify_cert+0x78 [./build_shared/../crypto/x509/x509_vfy.c @ 302] 
System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto___CryptoNative_X509VerifyCert_g____PInvoke_275_0+0x3c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EVP.Cipher.cs @ 235] 
System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto::CryptoNative_X509VerifyCert+0x38 [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 5824] 
System_Security_Cryptography!System_Security_Cryptography_Interop_Crypto::X509VerifyCert+0x1c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.X509.cs @ 196] 
System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::BuildWorkingChain+0x98 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 653] 
System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::Finish+0x58 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 688] 
System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChainCore+0x320 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 198] 
System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChain+0xb0 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 49] 
System_Security_Cryptography!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_X509Chain::Build+0x308 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Chain.cs @ 126] 
System_Security_Cryptography!System_Security_Cryptography_Tests_System_Security_Cryptography_X509Certificates_Tests_ChainTests::BuildChain_WithApplicationPolicy_Match+0x15c [/_/src/libraries/System.Security.Cryptography/tests/X509Certificates/ChainTests.cs @ 533]

The corrupted function pointer is verify_cb: https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/x509/x509_local.h#L116

The function pointers next to it look fine. It is just the low 4 bytes of verify_cb that are overwritten:

0000007f`54007570  0000007f`8d5a21b0 // x509_store_st::verify = libcrypto_so_1!internal_verify
0000007f`54007578  0000007f`8c080000 // x509_store_st::verify_cb - corrupted
0000007f`54007580  0000007f`8d59e8e0 // x509_store_st::get_issuer = libcrypto_so_1!X509_STORE_CTX_get1_issuer

bartonjs · 2022-09-09T22:57:02Z

We do hook that callback into managed:

runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs

Lines 644 to 677 in e55c908

    
           WorkingChain workingChain = new WorkingChain(); 
        
           WorkingChain? extraDispose = null; 
        
           Interop.Crypto.X509StoreCtxReset(_storeCtx); 
        
           Interop.Crypto.X509StoreVerifyCallback workingCallback = workingChain.VerifyCallback; 
        
           Interop.Crypto.X509StoreCtxSetVerifyCallback(_storeCtx, workingCallback); 
        
           bool verify = Interop.Crypto.X509VerifyCert(_storeCtx); 
        
           if (workingChain.AbortedForSignatureError) 
        
           { 
        
               Debug.Assert(!verify, "verify should have returned false for signature error"); 
        
               CloneChainForSignatureErrors(); 
        
               // Reset to a WorkingChain that won't fail. 
        
               extraDispose = workingChain; 
        
               workingChain = new WorkingChain(abortOnSignatureError: false); 
        
               workingCallback = workingChain.VerifyCallback; 
        
               Interop.Crypto.X509StoreCtxSetVerifyCallback(_storeCtx, workingCallback); 
        
               verify = Interop.Crypto.X509VerifyCert(_storeCtx); 
        
           } 
        
           // Keep the bound delegate alive until X509_verify_cert isn't going to call it any longer. 
        
           GC.KeepAlive(workingCallback); 
        
           // Because our callback tells OpenSSL that every problem is ignorable, it should tell us that the 
        
           // chain is just fine (unless it returned a negative code for an exception) 
        
           Debug.Assert(verify, "verify should have returned true"); 
        
           extraDispose?.Dispose(); 
        
           _workingChain = workingChain; 
        
           return workingChain;

The delegate is kept alive for the call via GC.KeepAlive. If NativeAOT/arm64 doesn't respect that, maybe we've jumped to where a delegate used to be pointing while we still needed it?

jkotas · 2022-09-09T23:37:33Z

Yes, it looks like a native aot problem then.

Fixes dotnet#74710

* Flush instruction cache after thunk pool allocation Fixes #74710 * More precise ifdef

Fixes #74710

karelz · 2022-09-12T15:34:02Z

Reopening to track the 7.0 backport ...

…5403) * Flush instruction cache after thunk pool allocation Fixes #74710 * More precise ifdef Co-authored-by: Jan Kotas <jkotas@microsoft.com>

jkotas · 2022-09-12T17:22:28Z

Backport has been merged.

karelz · 2022-09-12T20:53:52Z

Fixed in main (8.0) in PR #75393 and in 7.0 (RC2) in PR #75403.

karelz added arch-arm64 area-System.Net.Security labels Aug 27, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Aug 27, 2022

This comment was marked as duplicate.

Sign in to view

wfurt self-assigned this Aug 29, 2022

karelz added this to the 7.0.0 milestone Sep 1, 2022

karelz removed the untriaged New issue has not been triaged by the area owner label Sep 1, 2022

karelz changed the title ~~System.Net.Security.Tests crash on NativeAOT arm64~~ System.Net.* crash on NativeAOT arm64 Sep 9, 2022

karelz changed the title ~~System.Net.* crash on NativeAOT arm64~~ System.Net.* and System.Security.* crash on NativeAOT arm64 Sep 9, 2022

jkotas mentioned this issue Sep 9, 2022

[release/7.0] Unload MsQuic after checking for QUIC support to free resources (#75163) #75330

Closed

jkotas added area-NativeAOT-coreclr and removed area-System.Net.Security labels Sep 9, 2022

jkotas assigned jkotas and unassigned wfurt Sep 9, 2022

jkotas added a commit to jkotas/runtime that referenced this issue Sep 10, 2022

Flush instruction cache after thunk pool allocation

4581a71

Fixes dotnet#74710

jkotas mentioned this issue Sep 10, 2022

Flush instruction cache after thunk pool allocation #75393

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 10, 2022

jkotas closed this as completed in #75393 Sep 10, 2022

jkotas added a commit that referenced this issue Sep 10, 2022

Flush instruction cache after thunk pool allocation (#75393)

1f94e11

* Flush instruction cache after thunk pool allocation Fixes #74710 * More precise ifdef

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 10, 2022

github-actions bot pushed a commit that referenced this issue Sep 10, 2022

Flush instruction cache after thunk pool allocation

2f16613

Fixes #74710

rzikm mentioned this issue Sep 12, 2022

Don't unload MsQuic from the process #75441

Merged

karelz mentioned this issue Sep 12, 2022

[release/7.0] Flush instruction cache after thunk pool allocation #75403

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 12, 2022

karelz reopened this Sep 12, 2022

karelz added the tenet-reliability Reliability/stability related issue (stress, load problems, etc.) label Sep 12, 2022

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 12, 2022

jkotas closed this as completed Sep 12, 2022

ghost locked as resolved and limited conversation to collaborators Oct 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

karelz commented Aug 27, 2022 •

edited

Loading

ghost commented Aug 27, 2022

This comment was marked as duplicate.

jkotas commented Aug 27, 2022

jkotas commented Aug 27, 2022

karelz commented Sep 1, 2022

karelz commented Sep 9, 2022 •

edited

Loading

jkotas commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

karelz commented Sep 9, 2022

jkotas commented Sep 9, 2022 •

edited

Loading

wfurt commented Sep 9, 2022

bartonjs commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

jkotas commented Sep 9, 2022

bartonjs commented Sep 9, 2022

jkotas commented Sep 9, 2022

karelz commented Sep 12, 2022

jkotas commented Sep 12, 2022

karelz commented Sep 12, 2022

System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

Comments

karelz commented Aug 27, 2022 • edited Loading

ghost commented Aug 27, 2022

This comment was marked as duplicate.

jkotas commented Aug 27, 2022

jkotas commented Aug 27, 2022

karelz commented Sep 1, 2022

karelz commented Sep 9, 2022 • edited Loading

jkotas commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

karelz commented Sep 9, 2022

jkotas commented Sep 9, 2022 • edited Loading

wfurt commented Sep 9, 2022

bartonjs commented Sep 9, 2022

MichalStrehovsky commented Sep 9, 2022

jkotas commented Sep 9, 2022

bartonjs commented Sep 9, 2022

jkotas commented Sep 9, 2022

karelz commented Sep 12, 2022

jkotas commented Sep 12, 2022

karelz commented Sep 12, 2022

karelz commented Aug 27, 2022 •

edited

Loading

karelz commented Sep 9, 2022 •

edited

Loading

jkotas commented Sep 9, 2022 •

edited

Loading