Test failure JIT\\Regression\\JitBlue\\GitHub_18362\\GitHub_18362\\GitHub_18362.cmd

[JulieLeeMSFT]

Failed in run: runtime-coreclr outerloop 20230122.3

Failed tests:

coreclr windows arm Checked @ Windows.11.Arm64.Open
  - JIT\\Regression\\JitBlue\\GitHub_18362\\GitHub_18362\\GitHub_18362.cmd
  - JIT\\Regression\\JitBlue\\GitHub_12037\\GitHub_12037\\GitHub_12037.cmd
  - JIT\\Regression\\JitBlue\\GitHub_36905\\GitHub_36905\\GitHub_36905.cmd

Error message:

Return code:      1
Raw output file:      C:\h\w\AF0D0915\w\A4BB090F\uploads\Reports\JIT.Regression\JitBlue\GitHub_18362\GitHub_18362\GitHub_18362.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\AF0D0915\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  GitHub_18362.dll
Failure at line 94. Expected real: 0. Actual real: 2
Expected: 100
Actual: -1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\AF0D0915\p
C:\h\w\AF0D0915\w\A4BB090F\e\JIT\Regression\JitBlue\GitHub_18362\GitHub_18362\GitHub_18362.cmd
Expected: True
Actual:   False

Stack trace

   at JIT_Regression._JitBlue_GitHub_18362_GitHub_18362_GitHub_18362_._JitBlue_GitHub_18362_GitHub_18362_GitHub_18362_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failed in run: runtime-coreclr outerloop 20230122.3

Failed tests:

coreclr windows arm Checked @ Windows.11.Arm64.Open
  - JIT\\Regression\\JitBlue\\GitHub_18362\\GitHub_18362\\GitHub_18362.cmd
  - JIT\\Regression\\JitBlue\\GitHub_12037\\GitHub_12037\\GitHub_12037.cmd
  - JIT\\Regression\\JitBlue\\GitHub_36905\\GitHub_36905\\GitHub_36905.cmd

Error message:

Return code:      1
Raw output file:      C:\h\w\AF0D0915\w\A4BB090F\uploads\Reports\JIT.Regression\JitBlue\GitHub_18362\GitHub_18362\GitHub_18362.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\AF0D0915\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  GitHub_18362.dll
Failure at line 94. Expected real: 0. Actual real: 2
Expected: 100
Actual: -1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\AF0D0915\p
C:\h\w\AF0D0915\w\A4BB090F\e\JIT\Regression\JitBlue\GitHub_18362\GitHub_18362\GitHub_18362.cmd
Expected: True
Actual:   False

Stack trace

   at JIT_Regression._JitBlue_GitHub_18362_GitHub_18362_GitHub_18362_._JitBlue_GitHub_18362_GitHub_18362_GitHub_18362_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Author:	JulieLeeMSFT
Assignees:	jakobbotsch
Labels:	area-CodeGen-coreclr, blocking-outerloop
Milestone:	-

[JulieLeeMSFT]

Hi @jakobbotsch, it is blocking outerloop. PTAL.
cc @markples, just a question if we are not getting enough error messages because of test consolidation.

[markples]

@JulieLeeMSFT JIT\Regression is not merged in main, so this particular helix job will not be impacted that way.

Did you see something in JIT\Regression that made it seem like something is missing? Or do you think something might be happening with JIT\Methodical? Or is this just a general concern?

[JulieLeeMSFT]

Just a general question. @kunalspathak, I am wondering why these tests do not generate more detailed error messages related to why it failed. Is there anything we can do better to expose error messages?

[markples]

I think everything is there for this one from an infrastructure perspective. The log includes:

Failure at line 94. Expected real: 0. Actual real: 2
Expected: 100
Actual: -1

From the test - the failure is detected by the test code itself. Here it prints the message and throws an exception.

    private static void VerifyRealImaginaryProperties(Complex complex, double real, double imaginary, [CallerLineNumber] int lineNumber = 0)
    {
        if (!real.Equals(complex.Real) && !IsDiffTolerable(complex.Real, real))
        {
            Console.WriteLine("Failure at line {0}. Expected real: {1}. Actual real: {2}", lineNumber, real, complex.Real);
            throw new Exception();
        }

Here is the caller where the line number comes from

    public static void Conjugate(double real, double imaginary)
    {
        int returnVal = 100;

        var complex = new Complex(real, imaginary);
        Complex result = Complex.Conjugate(complex);

[94]    VerifyRealImaginaryProperties(result, real, -imaginary);
        VerifyMagnitudePhaseProperties(result, complex.Magnitude, -complex.Phase);
    }

And then in the test Main the exception is converted to an exit code:

    public static int Main()
    {
        try
        {
            Conjugate(2.0, 3.0);
        }
        catch (Exception e)
        {
            return -1;
        }
        return 100;

From the infrastructure's perspective, those three lines are all that really happen.

Rewriting individual tests is an expensive undertaking, though theoretically more tracking / earlier assertions could be added.

One somewhat more systematic thing we could do is convert to using xunit more than the "return 100" pattern. In this case, Main would become a [Fact] void function - then the harness could catch the Exception and perhaps automatically print a stack trace. Also, it eliminates some test boilerplate.

(This could go a bit further and even remove Main and replace with a [Theory] [StaticData] combination on Conjugate itself - then we'd automatically get that "Conjugate 2.0 3.0" failed, but it wouldn't really add more information about the failure.)

[jakobbotsch]

I cannot reproduce any of these on my win-arm32 machine. One interesting data point here is that these three tests are all testing floating point arithmetic and only seem to have failed in this particular test run.

Given that the failures are win-arm32 (a non-shipping configuration) and isolated to this particular test run I'm going to unmark it as blocking.

[jakobbotsch]

Haven't seen this since then, so will close it.