Intermittent failure in CriticalFinalizer test #76041

AntonLapounov · 2022-09-22T21:35:56Z

Affected tests:

baseservices/finalization/CriticalFinalizer/CriticalFinalizer.sh
baseservices\\finalization\\CriticalFinalizer\\CriticalFinalizer.cmd

Runfo failures in last 30 days - .sh and last 30 days - .cmd:

9/23 - 188 failures
- First failure in Rolling run https://runfo.azurewebsites.net/view/build/?number=24556 on 9/20 ... more than 50x failures per day

CoreCLR: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75573-merge-b0042e942c0145f38e/baseservices.finalization/1/console.e135272f.log?helixlogtype=result

BEGIN EXECUTION
"C:\h\w\A68508F4\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  CriticalFinalizer.dll 
Finalized 0 Normal and 0 Critical objects.
The first Critical object was finalized after 0 Normal objects.
Expected: 100
Actual: 101
END EXECUTION - FAILED

Mono: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74586-merge-662a1339f83d41b786/baseservices.finalization/1/console.0af36b40.log?helixlogtype=result

BEGIN EXECUTION
/tmp/helix/working/A1850911/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false CriticalFinalizer.dll ''
Finalized 100 Normal and 99 Critical objects.
The first Critical object was finalized after 100 Normal objects.
Expected: 100
Actual: 101
END EXECUTION - FAILED

{ "ErrorMessage":"CriticalFinalizer" }

Report

Build	Definition	Step Name	Console log
28655	dotnet/runtime	LLVM AOT compile CoreCLR tests	Log

Build	Definition	Test
29097	dotnet/runtime	baseservices.finalization.WorkItemExecution
28963	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28964	dotnet/runtime	baseservices.finalization.WorkItemExecution
29051	dotnet/runtime	baseservices.finalization.WorkItemExecution
29016	dotnet/runtime	baseservices.finalization.WorkItemExecution
29012	dotnet/runtime	baseservices.finalization.WorkItemExecution
29010	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28977	dotnet/runtime	baseservices.finalization.WorkItemExecution
28942	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28861	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28621	dotnet/runtime	baseservices.finalization.WorkItemExecution
28899	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28892	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28655	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28841	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28819	dotnet/runtime	baseservices.finalization.WorkItemExecution
28749	dotnet/runtime	baseservices.finalization.WorkItemExecution
28766	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28692	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28682	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28649	dotnet/runtime	baseservices.finalization.WorkItemExecution
28606	dotnet/runtime	baseservices.finalization.WorkItemExecution
28615	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28597	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28592	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28591	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27258	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28513	dotnet/runtime	baseservices.finalization.WorkItemExecution
28494	dotnet/runtime	baseservices.finalization.WorkItemExecution
28476	dotnet/runtime	baseservices.finalization.WorkItemExecution
28414	dotnet/runtime	baseservices.finalization.WorkItemExecution
27149	dotnet/runtime	baseservices.finalization.WorkItemExecution
28383	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28313	dotnet/runtime	baseservices.finalization.WorkItemExecution
28293	dotnet/runtime	baseservices.finalization.WorkItemExecution
28278	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28251	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28306	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28149	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27779	dotnet/runtime	baseservices.finalization.WorkItemExecution
27987	dotnet/runtime	baseservices.finalization.WorkItemExecution
27102	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28056	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
28047	dotnet/runtime	baseservices.finalization.WorkItemExecution
28023	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27880	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27960	dotnet/runtime	baseservices.finalization.WorkItemExecution
27940	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27866	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
26219	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27820	dotnet/runtime	baseservices.finalization.WorkItemExecution
27804	dotnet/runtime	baseservices.finalization.WorkItemExecution
27855	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27831	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27789	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27742	dotnet/runtime	baseservices.finalization.WorkItemExecution
27754	dotnet/runtime	baseservices.finalization.WorkItemExecution
27703	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
26661	dotnet/runtime	baseservices.finalization.WorkItemExecution
27657	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27643	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27588	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
27630	dotnet/runtime	baseservices.finalization.WorkItemExecution
27557	dotnet/runtime	baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
33	65	65

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2022-09-22T21:36:01Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

HJLeee · 2022-09-22T23:58:36Z

@AntonLapounov Please take a look. #76046

karelz · 2022-09-23T13:31:30Z

@AntonLapounov it seems it may have been a regression caused by #75952.
This is blocking significantly CI (see top post with update and links) -- can you please roll back the change, or disable the affected test ASAP? (definitely before the weekend) Thanks!

buyaa-n · 2022-09-23T20:58:29Z

Failed in #75965
Log: https://helix.dot.net/api/2019-06-17/jobs/ab2fe355-3fb8-48fd-bf26-f1ab51bd2648/workitems/baseservices.finalization/console

baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd [FAIL]
      
      Return code:      1
      Raw output file:      C:\h\w\BD950A05\w\B05A0982\uploads\Reports\baseservices.finalization\CriticalFinalizer\CriticalFinalizer.output.txt
      Raw output:
      BEGIN EXECUTION
       "C:\h\w\BD950A05\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  CriticalFinalizer.dll 
      Finalized 0 Normal and 0 Critical objects.
      The first Critical object was finalized after 0 Normal objects.
      Expected: 100
      Actual: 101
      END EXECUTION - FAILED
      FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=C:\h\w\BD950A05\p
      > C:\h\w\BD950A05\w\B05A0982\e\baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
      Expected: True
      Actual:   False
      Stack Trace:
           at baseservices_finalization._CriticalFinalizer_CriticalFinalizer_._CriticalFinalizer_CriticalFinalizer_cmd()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      C:\h\w\BD950A05\w\B05A0982\uploads\Reports\baseservices.finalization\CriticalFinalizer\CriticalFinalizer.output.txt
        Raw output:
        BEGIN EXECUTION
         "C:\h\w\BD950A05\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  CriticalFinalizer.dll 
        Finalized 0 Normal and 0 Critical objects.
        The first Critical object was finalized after 0 Normal objects.
        Expected: 100
        Actual: 101
        END EXECUTION - FAILED
        FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=C:\h\w\BD950A05\p
        > C:\h\w\BD950A05\w\B05A0982\e\baseservices\finalization\CriticalFinalizer\CriticalFinalizer.cmd
  Finished:    baseservices.finalization.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   baseservices.finalization.XUnitWrapper  Total: 1, Errors: 0, Failed: 1, Skipped: 0, Time: 1.235s

Contributes to dotnet#76041

jkotas · 2022-09-24T17:51:38Z

Disabling the test in #76131. Please re-enable once fixed.

* Disable frequently failing CriticalFinalizer test Contributes to #76041 * Delete coreclr specific exclusion, the test is failing everywhere

AustinWise · 2022-09-25T07:32:51Z

If my system is doing something CPU intensive, like compiling CoreCLR, the test failure reproduces more reliably. It fails rough 1 in 100 runs.

Calling GC.Collect and GC.WaitForPendingFinalizers several times in a loop seems to fix the problem. I'm not sure though that is really a reliable way to fix the test.

AntonLapounov · 2022-09-25T07:46:56Z

There are two issues:

The lambda used in Parallel.For creates a closure object (DisplayClass) and places the arr reference there. That closure object may be kept alive by one of thread pool threads.
In some Mono runs the test reports only 99 Critical objects collected out of 100. I suspect that GC.WaitForPendingFinalizers() might not synchronize the current thread with the finalizer thread regarding visibility of memory stores performed by finalizers.

AntonLapounov · 2022-09-25T08:13:14Z

Looking at the Mono code, there may be a race condition in sgen_have_pending_finalizers for checking the pending_unqueued_finalizer flag:

runtime/src/mono/mono/sgen/sgen-gc.c

Lines 2906 to 2912 in c986a12

    
           gboolean 
        
           sgen_have_pending_finalizers (void) 
        
           { 
        
           	if (sgen_suspend_finalizers) 
        
           		return FALSE; 
        
           	return pending_unqueued_finalizer || !sgen_pointer_queue_is_empty (&fin_ready_queue) || !sgen_pointer_queue_is_empty (&critical_fin_queue); 
        
           }

mangod9 · 2023-07-29T01:19:59Z

I am closing this based on #76046. We can reopen if this continues to be an issue.

AustinWise · 2023-08-13T18:16:33Z

@mangod9 This test is still disabled, referencing this issue:

https://github.com/dotnet/runtime/blob/main/src/tests/issues.targets#L14-L16

So I'm don't think the test was ever fixed to be reliable.

jkotas · 2023-08-14T04:43:54Z

We should either delete the test or fix it to be reliable.

AntonLapounov added this to the 8.0.0 milestone Sep 22, 2022

AntonLapounov self-assigned this Sep 22, 2022

AntonLapounov mentioned this issue Sep 22, 2022

Fix critical finalization test #75952

Merged

AntonLapounov changed the title ~~Intermittent failures in CriticalFinalizer test~~ Intermittent failure in CriticalFinalizer test Sep 22, 2022

filipnavara added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Sep 22, 2022

runfoapp bot mentioned this issue Sep 22, 2022

Infrastructure - Status/Health #702

Closed

AndyAyersMS mentioned this issue Sep 23, 2022

JIT: fix issue with unreachable blocks in redundant branch opt #76040

Merged

MichalStrehovsky mentioned this issue Sep 23, 2022

Disable EventSource support in NativeAOT by default #76000

Merged

vargaz mentioned this issue Sep 23, 2022

[mono][aot] Prefer specific instances instead of gshared for methods … #76033

Merged

karelz added the area-VM-coreclr label Sep 23, 2022

This was referenced Sep 23, 2022

JIT: Use gtCloneExpr in fgMorphModToSubMulDiv for potentially complex trees #76061

Merged

SPMI: Add and utilize number of contexts with diffs #76011

Merged

kunalspathak mentioned this issue Sep 23, 2022

Implement DivRem intrinsic for X86 #66551

Merged

VSadov mentioned this issue Sep 23, 2022

Disabling finalizer tests pending investigation. #76089

Merged

buyaa-n mentioned this issue Sep 23, 2022

[main] Update dependencies from dotnet/roslyn-analyzers dotnet/hotreload-utils #75965

Merged

TIHan mentioned this issue Sep 23, 2022

[JIT] CodeGen verification testing #75102

Merged

4 tasks

En3Tho mentioned this issue Sep 24, 2022

Add more optimizations for (https://github.com/dotnet/runtime/issues/61412) #74806

Merged

huoyaoyuan mentioned this issue Sep 24, 2022

Implement IEEE754 totalOrder comparer #75517

Merged

jkotas added the Known Build Error Use this to report build issues in the .NET Helix tab label Sep 24, 2022

jkotas added a commit to jkotas/runtime that referenced this issue Sep 24, 2022

Disable frequently failing CriticalFinalizer test

d2ffee9

Contributes to dotnet#76041

jkotas mentioned this issue Sep 24, 2022

Disable frequently failing CriticalFinalizer test #76131

Merged

jkotas added the disabled-test The test is disabled in source code against the issue label Sep 24, 2022

AndyAyersMS mentioned this issue Sep 25, 2022

JIT: refactor jump threading a bit #76108

Merged

VSadov pushed a commit that referenced this issue Sep 25, 2022

Disable frequently failing CriticalFinalizer test (#76131)

c986a12

* Disable frequently failing CriticalFinalizer test Contributes to #76041 * Delete coreclr specific exclusion, the test is failing everywhere

jkotas removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Sep 25, 2022

jkotas removed the Known Build Error Use this to report build issues in the .NET Helix tab label Sep 25, 2022

AntonLapounov mentioned this issue Sep 25, 2022

Ensure gc collects memory for CriticalFinalizer test #76046

Closed

MichalStrehovsky mentioned this issue Sep 26, 2022

Test failure baseservices\\finalization\\CriticalFinalizer\\CriticalFinalizer.cmd #76161

Closed

jakobbotsch mentioned this issue Sep 26, 2022

Fold SIMD-typed partial access to locals; delete base types from vector constants #74580

Merged

mangod9 closed this as completed Jul 29, 2023

jkotas unassigned AntonLapounov Aug 14, 2023

jkotas modified the milestones: 8.0.0, 9.0.0 Aug 14, 2023

jkotas added the test-bug Problem in test source code (most likely) label Aug 14, 2023

jkotas reopened this Aug 14, 2023

mangod9 modified the milestones: 9.0.0, 10.0.0 Aug 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent failure in CriticalFinalizer test #76041

Intermittent failure in CriticalFinalizer test #76041

AntonLapounov commented Sep 22, 2022 •

edited by build-analysis bot

Loading

dotnet-issue-labeler bot commented Sep 22, 2022

HJLeee commented Sep 22, 2022

karelz commented Sep 23, 2022 •

edited

Loading

buyaa-n commented Sep 23, 2022

jkotas commented Sep 24, 2022

AustinWise commented Sep 25, 2022

AntonLapounov commented Sep 25, 2022

AntonLapounov commented Sep 25, 2022

mangod9 commented Jul 29, 2023

AustinWise commented Aug 13, 2023

jkotas commented Aug 14, 2023

Intermittent failure in CriticalFinalizer test #76041

Intermittent failure in CriticalFinalizer test #76041

Comments

AntonLapounov commented Sep 22, 2022 • edited by build-analysis bot Loading

Report

Summary

dotnet-issue-labeler bot commented Sep 22, 2022

HJLeee commented Sep 22, 2022

karelz commented Sep 23, 2022 • edited Loading

buyaa-n commented Sep 23, 2022

jkotas commented Sep 24, 2022

AustinWise commented Sep 25, 2022

AntonLapounov commented Sep 25, 2022

AntonLapounov commented Sep 25, 2022

mangod9 commented Jul 29, 2023

AustinWise commented Aug 13, 2023

jkotas commented Aug 14, 2023

AntonLapounov commented Sep 22, 2022 •

edited by build-analysis bot

Loading

karelz commented Sep 23, 2022 •

edited

Loading