[JIT] CodeGen verification testing

docs/workflow/testing/coreclr/disasm-checks.md

@@ -0,0 +1,85 @@
+# Disassembly output verification checks
+There are tests that the runtime executes that will be able to verify X64/ARM64 assembly output from the JIT.
+The tools used to accomplish this are LLVM FileCheck, SuperFileCheck, and the JIT's ability to output disassembly using `DOTNET_JitDisasm`. LLVM FileCheck is built in https://www.github.com/dotnet/llvm-project and provides several packages for the various platforms. See more about LLVM FileCheck and its syntax here: https://llvm.org/docs/CommandGuide/FileCheck.html. SuperFileCheck is a custom tool located in https://www.github.com/dotnet/runtime. It wraps LLVM FileCheck and provides a simplified workflow for writing these tests in a C# file by leveraging Roslyn's syntax tree APIs.
+# What is FileCheck?
+From https://www.llvm.org/docs/CommandGuide/FileCheck.html:
+
+> **FileCheck** reads two files (one from standard input, and one specified on the command line) and uses one
+to verify the other. This behavior is particularly useful for the testsuite, which wants to verify that the
+output of some tool (e.g. **llc**) contains the expected information (for example, a movsd from esp or
+whatever is interesting). This is similar to using **grep**, but it is optimized for matching multiple
+different inputs in one file in a specific order.
+# Converting an existing test to use disassembly checking
+We will use the existing test `JIT\Regression\JitBlue\Runtime_33972` as an example. The test's intent is to verify that on ARM64, the method `AdvSimd.CompareEqual` behaves correctly when a zero vector is passed as the second argument. Below are snippets of its use:
+```csharp
+    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
+    {
+        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
+    }
+...
+...
+        if (!ValidateResult_Vector64<byte>(AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte>.Zero), Byte.MaxValue))
+            result = -1;
+```
+Currently, the test only verifies that the behavior is correct. It does not verify that the optimal ARM64 instruction was actually used. So now we will add this verification.
+First we need to modify the project file `Runtime_33972.csproj`:
+```xml
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+  </PropertyGroup>
+  <PropertyGroup>
+    <DebugType>None</DebugType>
+    <Optimize>True</Optimize>
+    <AllowUnsafeBlocks>True</AllowUnsafeBlocks>
+  </PropertyGroup>
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs" />
+  </ItemGroup>
+</Project>
+```
+Looking at the `ItemGroup`:
+```xml
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs" />
+  </ItemGroup>
+```
+We want to add `<HasDisasmCheck>true</HasDisasmCheck>` as a child of the `Compile` tag:
+```xml
+  <ItemGroup>
+    <Compile Include="$(MSBuildProjectName).cs">
+      <HasDisasmCheck>true</HasDisasmCheck>
+    </Compile>
+  </ItemGroup>
+```
+Doing this lets the test builder and runner know that this test has assembly that needs to be verified. Finally, we need to write the assembly check and put the `[MethodImpl(MethodImplOptions.NoInlining)]` attribute on the method `AdvSimd_CompareEqual_Vector64_Byte_Zero`:
+```csharp
+    [MethodImpl(MethodImplOptions.NoInlining)]
+    static Vector64<byte> AdvSimd_CompareEqual_Vector64_Byte_Zero(Vector64<byte> left)
+    {
+        // ARM64-FULL-LINE: cmeq v0.8b, v0.8b, #0
+        return AdvSimd.CompareEqual(left, Vector64<byte>.Zero);
+    }
+```
+And that is it. A few notes about the above example:
+- `ARM64-FULL-LINE` checks to see if there is an exact line that matches the disassembly output of the method `AdvSimd_CompareEqual_Vector64_Byte_Zero` - leading and trailing spaces are ignored.
+- Method bodies that have FileCheck syntax, e.g. `ARM64-FULL-LINE:`/`X64:`/etc, must have the attribute `[MethodImpl(MethodImplOptions.NoInlining)]`. If it does not, then an error is reported.
+- FileCheck syntax outside of a method body will also report an error.
+# Additional functionality
+LLVM has a different setup where each test file is passed to `lit`, and `RUN:` lines inside the test specify
+configuration details such as architectures to run, FileCheck prefixes to use, etc.  In our case, the build
+files handle a lot of this with build conditionals and `.cmd`/`.sh` file generation.  Additionally, LLVM tests
+rely on the order of the compiler output corresponding to the order of the input functions in the test file.
+When running under the JIT, the compilation order is dependent on execution, not the source order.
+
+Functionality that has been added or moved to MSBuild:
+- Conditionals controlling test execution
+- Automatic specificiation of `CHECK` and `<architecture>` as check prefixes
+
+Functionality that has been added or moved to SuperFileCheck:
+- Each function is run under a separate invocation of FileCheck. SuperFileCheck adds additional `CHECK` lines
+  that search for the beginning and end of the output for each function. This ensures that output from
+  different functions don't contaminate each other. The separate invocations remove any dependency on the
+  order of the functions.
+- `<check-prefix>-FULL-LINE:` - same as using FileCheck's `<check-prefix>:`, but checks that the line matches exactly; leading and trailing whitespace is ignored.
+- `<check-prefix>-FULL-LINE-NEXT:` - same as using FileCheck's `<check-prefix>-NEXT:`, but checks that the line matches exactly; leading and trailing whitespace is ignored.

eng/Subsets.props

@@ -313,6 +313,8 @@
 
     <ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and '$(TargetArchitecture)' != '$(BuildArchitecture)'" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />
     <ProjectToBuild Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)'" Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_crossarch.csproj" Category="clr" />
+
+    <ProjectToBuild Condition="'$(TargetOS)' == 'windows' or ('$(TargetOS)' == 'Linux' and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')) or '$(TargetOS)' == 'OSX'" Include="$(CoreClrProjectRoot)tools\SuperFileCheck\SuperFileCheck.csproj" Category="clr" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subset.Contains('+clr.toolstests+'))">

eng/Version.Details.xml

@@ -48,6 +48,38 @@
       <Uri>https://github.com/dotnet/llvm-project</Uri>
       <Sha>8688ea2538ef1287c6619d35a26b5750d355cc18</Sha>
     </Dependency>
+    <Dependency Name="runtime.linux-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.linux-musl-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>754d13817d834b716d339183e21aac7d2489c496</Sha>
+    </Dependency>
+    <Dependency Name="runtime.win-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.win-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.osx.11.0-arm64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
+    <Dependency Name="runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.JIT.Tools" Version="1.0.0-alpha.1.22431.4">
+      <Uri>https://github.com/dotnet/llvm-project</Uri>
+      <Sha>c1304304028d603e34f6803b0740c34441f80d2e</Sha>
+    </Dependency>
     <Dependency Name="System.CommandLine" Version="2.0.0-beta4.22355.1">
       <Uri>https://github.com/dotnet/command-line-api</Uri>
       <Sha>5618b2d243ccdeb5c7e50a298b33b13036b4351b</Sha>

eng/Versions.props

@@ -204,5 +204,14 @@
     <SwixPackageVersion>1.1.87-gba258badda</SwixPackageVersion>
     <WixPackageVersion>1.0.0-v3.14.0.5722</WixPackageVersion>
     <MonoWorkloadManifestVersion>6.0.0-preview.5.21275.7</MonoWorkloadManifestVersion>
+    <!-- JIT Tools -->
+    <runtimelinuxarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxmuslarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxmuslarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimelinuxmuslx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimelinuxmuslx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimewinarm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimewinarm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimewinx64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimewinx64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimeosx110arm64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimeosx110arm64MicrosoftNETCoreRuntimeJITToolsVersion>
+    <runtimeosx1012x64MicrosoftNETCoreRuntimeJITToolsVersion>1.0.0-alpha.1.22431.4</runtimeosx1012x64MicrosoftNETCoreRuntimeJITToolsVersion>
   </PropertyGroup>
 </Project>

src/coreclr/scripts/superpmi_collect_setup.py

@@ -167,6 +167,9 @@
     "vcruntime140_1.dll",
     "R2RDump.exe",
     "R2RTest.exe",
+    "FileCheck.exe",
+    "SuperFileCheck.exe",
+    "llvm-mca.exe",
     "superpmi.exe",
     "superpmi-shim-collector.dll",
     "superpmi-shim-counter.dll",