[apple] Mobile failures in System.Diagnostics.DiagnosticSource runtime metrics

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=747439
Build error leg or test failing: System.Diagnostics.DiagnosticSource.Tests
Affected CI: [ios][tvos]simulator_x64_release_allsubsets_mono, maccatalyst_[x64/arm64]_release_allsubsests_mono (runtime-extra-platforms)
Range of commits: 0806470...7cdd2e9

Stack trace example from MacCatalyst x64:

2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10] 1) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements<Int64>(metricName: "dotnet.jit.compiled_il.size", valueAssertion: Func`2 { Method = System.ValueTuple`2[System.Boolean,System.String] <.cctor>b__21_1(Int64), Target = <>c { } }, beforeRecord: null)   Test case: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Exception messages: Expected value to be greater than zero. Actual value was: 0.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateSingleMeasurement[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
   at System.Object.InvokeStub_RuntimeMetricsTests.ValidateMeasurements(Object , Span`1 )
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.1050582
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10] 2) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements<Int64>(metricName: "dotnet.jit.compiled_methods", valueAssertion: Func`2 { Method = System.ValueTuple`2[System.Boolean,System.String] <.cctor>b__21_1(Int64), Target = <>c { } }, beforeRecord: null)   Test case: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Exception messages: Expected value to be greater than zero. Actual value was: 0.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateSingleMeasurement[Int64](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
   at System.Object.InvokeStub_RuntimeMetricsTests.ValidateMeasurements(Object , Span`1 )
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.1024496
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10] 3) 	[FAIL] System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements   Test name: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements<Double>(metricName: "dotnet.jit.compilation.time", valueAssertion: Func`2 { Method = System.ValueTuple`2[System.Boolean,System.String] <.cctor>b__21_3(Double), Target = <>c { } }, beforeRecord: null)   Test case: System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateMeasurements
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Assembly:  [System.Diagnostics.DiagnosticSource.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]
2024-07-19 06:05:58.025 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    Exception messages: Expected value to be greater than zero. Actual value was: 0.   Exception stack traces:    at System.Diagnostics.Metrics.Tests.RuntimeMetricsTests.ValidateSingleMeasurement[Double](String metricName, Func`2 valueAssertion, Func`1 beforeRecord)
   at System.Object.InvokeStub_RuntimeMetricsTests.ValidateMeasurements(Object , Span`1 )
2024-07-19 06:05:58.026 Df System.Diagnostics.DiagnosticSource.Tests[5417:db10]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   Execution time: 0.1059831

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Expected value to be greater than zero. Actual value was: 0.",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=747439
Error message validated: [Expected value to be greater than zero. Actual value was: 0.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/21/2024 4:35:30 PM UTC

Report

Build	Definition	Test	Pull Request
758343	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
755747	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
751577	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
750924	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
748618	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
747439	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	6

[dotnet-policy-service]

Tagging subscribers to 'os-ios': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'os-tvos': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to 'os-maccatalyst': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[matouskozak]

Possibly related to the new tests introduced in #104680. Do you know if these tests should be disabled or same changes are needed for them to work on Apple mobile targets? @stevejgordon @tarekgh ?

[stevejgordon]

@matouskozak I think we can safely skip this platform in the runtime metrics tests.

[matouskozak]

@matouskozak I think we can safely skip this platform in the runtime metrics tests.

Thank you, would you like to open a PR to disable them or should I take a look?

Do you know if this is an issue that should be fixed on the Mono runtime side or is it by design of the test? It looks like only a few tests are failing so I suppose the RuntimeMetricsTests suit isn't CoreCLR specific.

[stevejgordon]

@matouskozak I've opened a PR and took a different approach after reviewing the failures more closely.