Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure tracing\\eventpipe\\processenvironment\\processenvironment\\processenvironment.cmd #51477

Closed
VincentBu opened this issue Apr 19, 2021 · 8 comments · Fixed by #54769
Closed

Test failure tracing\\eventpipe\\processenvironment\\processenvironment\\processenvironment.cmd #51477

VincentBu opened this issue Apr 19, 2021 · 8 comments · Fixed by #54769
Labels
area-Tracing-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs GCStress os-windows
Milestone
6.0.0

Comments

@VincentBu
Copy link
Contributor

VincentBu commented Apr 19, 2021 •
edited
Loading

Run: runtime-coreclr gcstress0x3-gcstress0xc 20210418.1

Failed test:

CoreCLR windows arm64 Checked gcstress0x3 @ Windows.10.Arm64v8.Open
 -tracing\\eventpipe\\processenvironment\\processenvironment\\processenvironment.cmd

Error message:

Assert failure(PID 9676 [0x000025cc], Thread: 3416 [0x0d58]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x19C (0x00007ff897c508ec)
CORECLR! Object::Validate + 0xC0 (0x00007ff897c50728)
CORECLR! WKS::GCHeap::Promote + 0x9C (0x00007ff897e6e13c)
CORECLR! TransitionFrame::PromoteCallerStackHelper + 0x11C (0x00007ff897bfc454)
CORECLR! TransitionFrame::PromoteCallerStack + 0x1A0 (0x00007ff897bfc320)
CORECLR! GcStackCrawlCallBack + 0x34C (0x00007ff897d0830c)
CORECLR! Thread::MakeStackwalkerCallback + 0x70 (0x00007ff897c74fc0)
CORECLR! Thread::StackWalkFramesEx + 0x174 (0x00007ff897c762ac)
CORECLR! Thread::StackWalkFrames + 0x130 (0x00007ff897c760a0)
CORECLR! ScanStackRoots + 0x1D4 (0x00007ff897d070a4)
 File: D:\workspace\_work\1\s\src\coreclr\vm\object.cpp Line: 514
 Image: D:\h\w\9A030924\p\corerun.exe


Return code: 1
Raw output file: D:\h\w\9A030924\w\A64108D9\e\tracing\eventpipe\Reports\tracing.eventpipe\processenvironment\processenvironment\processenvironment.output.txt
Raw output:
BEGIN EXECUTION
 "D:\h\w\9A030924\p\corerun.exe" processenvironment.dll 
 0.0s: running sub-process: D:\h\w\9A030924\p\corerun.exe D:\h\w\9A030924\w\A64108D9\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.dll 0
 4.4s: subprocess started: True
 4.4s: subprocess PID: 10476
 15.5s: Test PID: 10476
 16.7s: Wrote: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=2; Reserved=0 }; }
 17.5s: Received: { Header={ Magic=System.Byte[]; Size=26; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x90 0x22 0x00 0x00 0x00 0x00 ]}
 17.6s: continuation size: 8848 bytes
 17.6s: future value: 0
 17.6s: Starting to copy continuation
 17.9s: Finished copying continuation
 17.9s: Total bytes in continuation: 8848
 17.9s: envCount: 90
 18.4s: finished parsing env
 18.5s: Saw test values in env
 18.5s: Sending 'exit' to subprocess stdin
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=D:\h\w\9A030924\p
> D:\h\w\9A030924\w\A64108D9\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.cmd
Expected: True
Actual: False

Stack trace
   at tracing_eventpipe._processenvironment_processenvironment_processenvironment_._processenvironment_processenvironment_processenvironment_cmd()

Historical failures of this test:

Queued OS Arch Pipeline
2021-05-23T14:36:28.409Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-23T05:17:13.546Z windows.10.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-05-22T14:44:02.494Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-22T14:42:50.584Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-17T07:50:49.573Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-05-17T04:54:31.141Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-05-16T15:15:59.913Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-16T15:15:37.567Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-16T07:43:56.936Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-05-15T22:39:57.779Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-05-15T14:44:38.035Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-10T05:07:57.183Z windows.10.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-05-09T22:51:18.278Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-05-09T14:50:18.076Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-05-08T14:43:40.295Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-03T04:16:45.475Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
More failures
Queued OS Arch Pipeline
2021-05-02T14:52:08.289Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-05-02T04:29:03.793Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-05-01T14:51:43.166Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-26T08:53:43.532Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-26T06:11:24.034Z windows.10.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-26T05:24:04.103Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-25T15:39:01.988Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-25T08:51:53.456Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-25T05:12:38.714Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-24T23:42:55.524Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-24T19:10:04.928Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-18T16:00:24.894Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-17T15:45:16.388Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-07T07:48:46.117Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-07T07:47:25.095Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-06T09:33:38.643Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 outerloop Checked-no_tiered_compilation
2021-04-06T07:47:50.416Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-06T07:46:00.726Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-05T07:45:37.05Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-05T07:43:16.253Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-05T07:43:03.279Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-04T07:48:08.096Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-04T07:44:29.755Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-04T07:40:43.903Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-03T07:45:05.194Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-03T07:41:56.711Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-02T07:52:20.614Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-02T07:50:05.638Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-02T07:47:04.071Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-01T07:58:14.852Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-01T07:56:44.714Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-04-01T07:56:34.509Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-31T07:43:15.865Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-31T07:38:08.136Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-31T07:37:48.248Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-30T09:56:44.004Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 outerloop Checked-no_tiered_compilation
2021-03-30T07:50:54.296Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-30T07:50:36.104Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-29T07:49:47.59Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-29T07:48:28.092Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-29T07:46:26.785Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-28T08:02:37.34Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-28T08:00:49.072Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-28T08:00:22.794Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-27T07:43:17.334Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-27T07:43:15.059Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-27T07:38:54.637Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-26T08:04:04.588Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-26T07:58:57.566Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-26T07:58:33.935Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-25T09:48:59.413Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 outerloop Checked-no_tiered_compilation
2021-03-25T08:19:22.436Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-25T08:15:39.8Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-25T08:10:14.552Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-24T07:52:46.43Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-24T07:51:02.228Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-24T07:49:05.385Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-23T07:40:22.005Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-23T07:39:51.634Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-23T07:36:39.776Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-22T07:48:56.33Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-22T07:46:34.352Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-21T07:46:36.51Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-21T07:45:36.208Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-21T07:41:41.713Z osx.1013.amd64.open x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-20T07:37:01.638Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
2021-03-20T07:35:37.622Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2 Checked-no_tiered_compilation
@dotnet-issue-labeler dotnet-issue-labeler bot added area-crossgen2-coreclr untriaged New issue has not been triaged by the area owner labels Apr 19, 2021
@tommcdon
Copy link
Member

@josalem

@tommcdon tommcdon added this to the 6.0.0 milestone Apr 19, 2021
@tommcdon tommcdon removed the untriaged New issue has not been triaged by the area owner label Apr 19, 2021
@josalem
Copy link
Contributor

josalem commented Apr 19, 2021

I'm not sure that the crossgen2 pipeline failures are actually this test failing.

/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh: line 280: -r:/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/IL-CG2/*.dll: No such file or directory
      rm: cannot remove '/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/common.dll.rsp': No such file or directory
      rm: cannot remove '/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll.rsp': No such file or directory
      
      Assert failure(PID 9250 [0x00002422], Thread: 9266 [0x2432]): Verify_FieldOffset 'ExtendedSocketException._endPoint' Field offset 128!=136(actual) || baseOffset 128!=136(actual)
          File: /__w/1/s/src/coreclr/vm/jitinterface.cpp Line: 14128
          Image: /home/helixbot/work/B4AF0978/p/corerun
      
      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh: line 401:  9250 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
      
      Return code:      1
      Raw output file:      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/Reports/tracing.eventpipe/reverseouter/reverseouter/reverseouter.output.txt
      Raw output:
      BEGIN EXECUTION
      in takeLock
      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/IL-CG2/common.dll
      Response file: /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/common.dll.rsp
      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/IL-CG2/common.dll
      -o:/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/common.dll
      -r:/home/helixbot/work/B4AF0978/p/System.*.dll
      -r:/home/helixbot/work/B4AF0978/p/Microsoft.*.dll
      -r:/home/helixbot/work/B4AF0978/p/mscorlib.dll
      --verify-type-and-field-layout
      --targetarch:x64
      -O
      Running CrossGen2:  dotnet /home/helixbot/work/B4AF0978/p/crossgen2/crossgen2.dll @/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/common.dll.rsp  
      Emitting R2R PE file: /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/common.dll
      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/IL-CG2/reverseouter.dll
      Response file: /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll.rsp
      /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/IL-CG2/reverseouter.dll
      -o:/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll
      -r:/home/helixbot/work/B4AF0978/p/System.*.dll
      -r:/home/helixbot/work/B4AF0978/p/Microsoft.*.dll
      -r:/home/helixbot/work/B4AF0978/p/mscorlib.dll
      --verify-type-and-field-layout
      --targetarch:x64
      -O
      Running CrossGen2:  dotnet /home/helixbot/work/B4AF0978/p/crossgen2/crossgen2.dll @/home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll.rsp  
      Emitting R2R PE file: /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll
      in ReleaseLock
      /home/helixbot/work/B4AF0978/p/corerun reverseouter.dll ''
        0.0s: ::== Running test: TEST_ReverseConnectionCanRecycleWhileTracing
        0.0s: Server name is '/tmp/DOTNET_TRACE_TESTS_0l03mfce.gkr'
        0.0s: running sub-process: /home/helixbot/work/B4AF0978/p/corerun /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll 0
        0.1s: subprocess started: True
        0.1s: subprocess PID: 9257
        0.3s: Starting EventPipeSession over standard connection
        0.3s: Waiting for connection
      Gathering state for process 9250 corerun
      Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.9250.dmp
      Written 97906688 bytes (23903 pages) to core file
      Dump successfully written
      Expected: 100
      Actual: 134
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/home/helixbot/work/B4AF0978/p
      > /home/helixbot/work/B4AF0978/w/9767089E/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh
      Expected: True
      Actual:   False
      Stack Trace:

It looks like corssgen2 fails on an assert, but then it still attempts to run the test which seemingly crashes partway through execution. I haven't looked at the dumps yet though. @mangod9 does this type of failure looks familiar to you?

As for the gcstress failure, I don't have any hunches. There's no dump to inspect so I'll have to see if this repros. It's attempting to validate the method table pointer for whatever object the GC is inspecting. Either that pointer was null or it failed the validation.

runtime/src/coreclr/vm/object.cpp

Lines 510 to 514 in 9f10999

MethodTable *pMT = GetGCSafeMethodTable();
lastTest = 1;
CHECK_AND_TEAR_DOWN(pMT && pMT->Validate());

The test itself seems to be in the shutdown phase, so this could be during the child process's shutdown.

@mangod9
Copy link
Member

mangod9 commented Apr 19, 2021

hmm.. interesting. Is this an intermittent failure? I would expect field layout asserts would be deterministic.

cc @dotnet/crossgen-contrib @trylek @davidwrighton

@trylek
Copy link
Member

trylek commented Apr 19, 2021 •
edited
Loading

For John's comment, the field layout validation error is a runtime crash, not a compile-time crash. One way or another this is super weird as I believe this is the exact bug I was fixing about 1-2 weeks back in

#50364

As you can see, I used the exact ExtendedSocketException layout to construct the regression test that seems to be passing now. The primary problem quoted by Vincent in the issue description is basically a heap corruption which naturally may occur upon inconsistent field layout calculation but I find it hard to understand how such a crash could occur without previously hitting the field offset verification check.

@VincentBu
Copy link
Contributor Author

Failed again in runtime-coreclr r2r-extra 20210502.1

Failed test:

R2R windows arm64 Checked gcstress0xf @ Windows.10.Arm64v8.Open
 -tracing\\eventpipe\\processenvironment\\processenvironment\\processenvironment.cmd

Error message:

Assert failure(PID 14940 [0x00003a5c], Thread: 13036 [0x32ec]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x19C (0x00007ffcefc1b5ac)
CORECLR! Object::Validate + 0xC0 (0x00007ffcefc1b3e8)
CORECLR! WKS::GCHeap::Promote + 0x9C (0x00007ffcefe4261c)
CORECLR! TransitionFrame::PromoteCallerStackHelper + 0x11C (0x00007ffcefbc6c44)
CORECLR! TransitionFrame::PromoteCallerStack + 0x1A0 (0x00007ffcefbc6b10)
CORECLR! GcStackCrawlCallBack + 0x34C (0x00007ffcefcdc50c)
CORECLR! Thread::MakeStackwalkerCallback + 0x70 (0x00007ffcefc40400)
CORECLR! Thread::StackWalkFramesEx + 0x174 (0x00007ffcefc416ec)
CORECLR! Thread::StackWalkFrames + 0x130 (0x00007ffcefc414e0)
CORECLR! ScanStackRoots + 0x1D4 (0x00007ffcefcdb2b4)
 File: D:\workspace\_work\1\s\src\coreclr\vm\object.cpp Line: 514
 Image: D:\h\w\AC52091D\p\corerun.exe


Return code: 1
Raw output file: D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\Reports\tracing.eventpipe\processenvironment\processenvironment\processenvironment.output.txt
Raw output:
BEGIN EXECUTION
 1 file(s) copied.
" "D:\h\w\AC52091D\p\crossgen.exe" /Platform_Assemblies_Paths D:\h\w\AC52091D\p;D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment /in D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.org /out D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment\\processenvironment.dll"
Microsoft (R) CoreCLR Native Image Generator - Version 6.0.0-ci
Copyright (c) Microsoft Corporation. All rights reserved.

Native image D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment\\processenvironment.dll generated successfully.
 "D:\h\w\AC52091D\p\corerun.exe" processenvironment.dll 
 0.0s: running sub-process: D:\h\w\AC52091D\p\corerun.exe D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.dll 0
 4.4s: subprocess started: True
 4.5s: subprocess PID: 11820
 11.5s: Test PID: 11820
 12.6s: Wrote: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=2; Reserved=0 }; }
 13.6s: Received: { Header={ Magic=System.Byte[]; Size=26; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x02 0x26 0x00 0x00 0x00 0x00 ]}
 13.8s: continuation size: 9730 bytes
 13.8s: future value: 0
 13.8s: Starting to copy continuation
 14.0s: Finished copying continuation
 14.0s: Total bytes in continuation: 9730
 14.1s: envCount: 94
 14.7s: finished parsing env
 14.8s: Saw test values in env
 14.9s: Sending 'exit' to subprocess stdin
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=D:\h\w\AC52091D\p
> D:\h\w\AC52091D\w\9F6C08E4\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.cmd
Expected: True
Actual: False

Stack trace
   at tracing_eventpipe._processenvironment_processenvironment_processenvironment_._processenvironment_processenvironment_processenvironment_cmd()

Historical failures of this test has updated in this comment.

@VincentBu
Copy link
Contributor Author

Failed again in runtime-coreclr gcstress0x3-gcstress0xc 20210523.1

Failed test:

CoreCLR windows x64 Checked gcstress0x3 @ Windows.10.Amd64.Open

- tracing\\eventpipe\\processenvironment\\processenvironment\\processenvironment.cmd

Error message:

Assert failure(PID 4976 [0x00001370], Thread: 692 [0x02b4]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x120 (0x00007ffb2d5ca1e0) CORECLR! Object::Validate + 0xB9 (0x00007ffb2d5ca089)
CORECLR! WKS::GCHeap::Promote + 0xA0 (0x00007ffb2da5a0c0) CORECLR! TransitionFrame::PromoteCallerStackHelper + 0xF5 (0x00007ffb2d50eef1)
CORECLR! TransitionFrame::PromoteCallerStack + 0x174 (0x00007ffb2d50ede4) CORECLR! GcStackCrawlCallBack + 0x379 (0x00007ffb2d74e959)
CORECLR! Thread::MakeStackwalkerCallback + 0x6B (0x00007ffb2d61c987) CORECLR! Thread::StackWalkFramesEx + 0x164 (0x00007ffb2d61e250)
CORECLR! Thread::StackWalkFrames + 0x184 (0x00007ffb2d61e04c) CORECLR! ScanStackRoots + 0x22E (0x00007ffb2d74d43e)
File: D:\workspace_work\1\s\src\coreclr\vm\object.cpp Line: 514
Image: C:\h\w\B3CA0974\p\corerun.exe

Return code:      1
Raw output file:      C:\h\w\B3CA0974\w\B75809CB\e\tracing\eventpipe\Reports\tracing.eventpipe\processenvironment\processenvironment\processenvironment.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\B3CA0974\p\corerun.exe"  processenvironment.dll
0.0s: running sub-process: C:\h\w\B3CA0974\p\corerun.exe C:\h\w\B3CA0974\w\B75809CB\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.dll 0
3.8s: subprocess started: True
3.9s: subprocess PID: 4040
16.4s: Test PID: 4040
17.6s: Wrote: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=2; Reserved=0 }; }
18.5s: Received: { Header={ Magic=System.Byte[]; Size=26; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x7A 0x20 0x00 0x00 0x00 0x00  ]}
18.7s: continuation size: 8314 bytes
18.7s: future value: 0
18.7s: Starting to copy continuation
18.9s: Finished copying continuation
18.9s: Total bytes in continuation: 8314
18.9s: envCount: 87
19.6s: finished parsing env
19.6s: Saw test values in env
19.6s: Sending 'exit' to subprocess stdin
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\B3CA0974\p
C:\h\w\B3CA0974\w\B75809CB\e\tracing\eventpipe\processenvironment\processenvironment\processenvironment.cmd
Expected: True
Actual:   False


Stack trace
   at tracing_eventpipe._processenvironment_processenvironment_processenvironment_._processenvironment_processenvironment_processenvironment_cmd()

Historical failures of this test has updated in this comment.

@BruceForstall
Copy link
Member

This continues to fail for Windows x64, Windows x86, Windows arm64 in either GCStress=3 or GCStress=C:

https://dev.azure.com/dnceng/public/_build/results?buildId=1198527&view=ms.vss-test-web.build-test-results-tab&runId=35925072&paneView=debug

BruceForstall added a commit to BruceForstall/runtime that referenced this issue Jun 22, 2021
@BruceForstall
Disable failing tests under GCStress
748c264 
dotnet#51477
dotnet#53359
@BruceForstall BruceForstall mentioned this issue Jun 22, 2021
Merged
@BruceForstall BruceForstall added the blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs label Jun 22, 2021
@runfoapp runfoapp bot mentioned this issue Jun 22, 2021
Closed
BruceForstall added a commit that referenced this issue Jun 22, 2021
@BruceForstall
Disable failing tests under GCStress (#54532)
62b8d02 
#51477
#53359
@davidwrighton
Copy link
Member

@josalem, I took a chance too look at this due to the comment about crossgen behavior above. Fortunately it appears to be an entirely different issue. I'm putting together a PR now.

@davidwrighton davidwrighton mentioned this issue Jun 25, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 25, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 26, 2021
@tommcdon tommcdon mentioned this issue Jun 28, 2021
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Jul 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Tracing-coreclr blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs GCStress os-windows
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Fix gc hole in IO thread pool davidwrighton/runtime
7 participants
@davidwrighton @BruceForstall @tommcdon @trylek @josalem @VincentBu @mangod9