JITServer throughput regression

[mpirvu]

I noticed a 4.8% throughput regression when running the AcmeAir application with JITServer. The JVM (Java17) was pinned to just 1 CPU.
To triage it I used nightly builds and found the following points of change:

2023/07/11 --> 3969 pages/sec
2023/07/12 --> 3815 pages/sec ==> 3.9% regression
2023/07/14 --> 3824 pages/sec
2023/07/18 --> 3365 pages/sec ==> 12% regression
2023/07/19 --> 3527 pages/sec ==> a recovery
2023/08/04 --> 3511 pages/sec
2023/08/07 --> 3779 pages/sec ==> A recovery, but still 4.8% slower than 07/11

Java version info:

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-4c75373dfe0, JRE 17 Linux amd64-64-Bit Compressed References 20230711_531 (JIT enabled, AOT enabled)
OpenJ9   - 4c75373dfe0
OMR      - bcc94e42952
JCL      - 78eae25a717 based on jdk-17.0.8+6)

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-13ea57d6495, JRE 17 Linux amd64-64-Bit Compressed References 20230712_532 (JIT enabled, AOT enabled)
OpenJ9   - 13ea57d6495
OMR      - bcc94e42952
JCL      - 78eae25a717 based on jdk-17.0.8+6)

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-d3a0aafa044, JRE 17 Linux amd64-64-Bit Compressed References 20230714_534 (JIT enabled, AOT enabled)
OpenJ9   - d3a0aafa044
OMR      - b1af0b4c8fb
JCL      - 0ccb615c987 based on jdk-17.0.8+6)

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-e3003d9490f, JRE 17 Linux amd64-64-Bit Compressed References 20230718_535 (JIT enabled, AOT enabled)
OpenJ9   - e3003d9490f
OMR      - b1af0b4c8fb
JCL      - 0ccb615c987 based on jdk-17.0.8+6)

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-3dddeecdd80, JRE 17 Linux amd64-64-Bit Compressed References 20230804_548 (JIT enabled, AOT enabled)
OpenJ9   - 3dddeecdd80
OMR      - 43a2b1d3416
JCL      - 31bcd9f4205 based on jdk-17.0.8+7)

OpenJDK Runtime Environment (build 17.0.8-internal+0-adhoc.jenkins.BuildJDK17x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-b03193f9bd4, JRE 17 Linux amd64-64-Bit Compressed References 20230807_549 (JIT enabled, AOT enabled)
OpenJ9   - b03193f9bd4
OMR      - 43a2b1d3416
JCL      - 31bcd9f4205 based on jdk-17.0.8+7)

[mpirvu]

These are the OpenJ9 PRs for the first regression 07/11-->07/12

• Abort compilations for exceptions during invokedynamic resolution Abort compilations for exceptions during invokedynamic resolution #17625
• Modify the implementation of isUnnamedClass() Modify the implementation of isUnnamedClass() #17766
• Add Preload and ImplicitCreation attributes Add Preload and ImplicitCreation attributes #17655

A likely culprit is #17625 because it involves some JITServer changes.

These are the OpenJ9 PRs for the second regression 07/14 --> 07/18

• Update the default mode for adding to the continuation list Update the default mode for adding to the continuation list #17802
• Adjust call target weight estimate for callees with const args Adjust call target weight estimate for callees with const args #17539
• Add flag to indicate method was FSD compiled Add flag to indicate method was FSD compiled #17798
• Initialize JITServer SSL context post CRIU restore Initialize JITServer SSL context post CRIU restore #17797
• Enable value types escape analysis Enable value types escape analysis #17594

A likely culprit is #17539 because inliner changes can have big impact on performance.
It's important to note that #17539 did have a negative performance impact on non-JITServer tests because the change was disabled by default in #17900. This did improve performance (see 08/04-->08/7 gain) but not to the full extent.

[mpirvu]

Investigation showed that the reason for the first regression is a missing break statement for a switch case. This problem was introduced by #17625.
The second regression was introduced by #17539 and was reverted by #17900
In my local performance measurements, the fix from #17942 completely restores the remaining throughput regression, so this issue can be closed.