Enable reproducible comparing tests on jdk21+ macos

[sophia-guo]

No description provided.

[sophia-guo]

I have tried both static mac and orka. Both no jq installed.

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10327/console
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10328

Error: jq is not installed. Please install jq before running this script

Might be easy to install jq on static mac and orka?

[sophia-guo]

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/

17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified

Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?

[steelhead31]

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/

17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified

Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?

Yup, that machines OS I think is too old for Xcode 15.2, and I think it has to be the same version of Xcode for reproducibility to work, @andrew-m-leonard can confirm this I think :)

[andrew-m-leonard]

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/

17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified

Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?

Yup, that machines OS I think is too old for Xcode 15.2, and I think it has to be the same version of Xcode for reproducibility to work, @andrew-m-leonard can confirm this I think :)

Yes, correct Xcode 15.2 requires MacOS 13.5 minimum : https://developer.apple.com/support/xcode/

[sophia-guo]

So we need extra label for macos to ensure the reproducing tests run on correct environment. Ether xcode15.2 or sw.os.osx.13_5. Two Mac static agents wont work for reproducing tests. https://ci.adoptium.net/label/ci.role.test&&hw.arch.x86&&%28sw.os.osx%7C%7Csw.os.mac%29/

Current orka ones are label as xcode15.0.1&&build&&mac&&x64 so can't work for reproducing either. Should we open an issue in infra to see if it's possible to update orka's os version?

[sophia-guo]

Interesting orka ones with label xcode15.0.1&&build&&mac&&x64 can do the reproducing build. https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/ https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/artifact/workspace/target/OpenJDK21U-sbom_x64_mac_hotspot_2024-05-31-22-18.json

[andrew-m-leonard]

Interesting orka ones with label xcode15.0.1&&build&&mac&&x64 can do the reproducing build. https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/ https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/artifact/workspace/target/OpenJDK21U-sbom_x64_mac_hotspot_2024-05-31-22-18.json

@sophia-guo so the Orka nodes are using Xcode 15.2, the 15.0.1 label has not been changed just because of the hassle involved! should have been called Xcode15 !

[sophia-guo]

orka updated with jq rerun adoptium/infrastructure#3593

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10470/console

[sophia-guo]

16:32:23  Make JDK Any Platform Argument List = 
16:32:23  --clean-git-repo --jdk-boot-dir /Users/admin/comp-jdk-build/jdk-20.0.2+9/Contents/Home --target-file-name OpenJDK21U-jdk_x64_mac_hotspot_21.0.4_5-ea.tar.gz --disable-shallow-git-clone -bjdk-21.0.4+5_adopt --create-jre-image --create-sbom --build-reproducible-date "2024-06-25 11:04:23"--user-openjdk-build-root-directory /Users/admin/comp-jdk-build/temurin-build/workspace/build/openjdkbuild/ --freetype-dir bundled --use-jep319-certs --create-debug-image  --configure-args "--disable-warnings-as-errors --openjdk-target=x86_64-apple-darwin --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk --with-extra-cxxflags='-mmacosx-version-min=10.9' --enable-dtrace --with-version-opt=ea"" --build-variant temurin jdk21u 
16:32:23  
16:32:23  Parameters Parsed Successfully
16:32:23  ---------------------------------------------
16:32:23  Building JDK...
16:32:23  sh: line 1: unexpected EOF while looking for matching `"'
16:32:23  sh: line 2: syntax error: unexpected end of file
16:32:23  -----------------------------------
16:32:23  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED

parsing issue as SBOM --configure-args value format updated with double quotes. Parsing in mac script also need to update.

[sophia-guo]

New grinder with latest and label https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ to make sure running on orka

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10769/ - waiting

[sophia-guo]

Running with orka, issue message complains the Xcode can't be found.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10771/console.

Test is running on orka node , which should have Xcode 15.2 available #3844 (comment)
https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/

11:27:51  Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
11:27:51  Checking For Xcode Compilers In : /Applications/
11:27:51  Error - An Xcode Installaton Could Not Be Found In The Default Path : /Applications / Xcode 
11:27:51  -----------------------------------
11:27:51  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED

[sophia-guo]

#3844 (comment)

@steelhead31 If it's related with how the Xcode installation is checked? #3844 (comment)

[sophia-guo]

rerun on build orka node https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ got different error from running on test orka node https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10775/console

12:01:08  ---------------------------------------------
12:01:08  Building JDK...
12:01:08  sh: line 1: unexpected EOF while looking for matching `"'
12:01:08  sh: line 2: syntax error: unexpected end of file
12:01:08  -----------------------------------
12:01:08  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED
12:01:08  -----------------------------------

[steelhead31]

@sophia-guo I've put some debugging in, and it does appear the script is checking correctly, but not finding an Xcode installation on that Orka node..

00:02:46.365  ---------------------------------------------
00:02:46.365  System Architecture: 64-bit (Intel)
00:02:46.365  SBOM & SYSTEM Architectures Match - All OK
00:02:46.365  ---------------------------------------------
00:02:46.365  Checking For Supported Compiler Versions...
00:02:46.365
00:02:46.365  Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
00:02:46.366  Checking For Xcode Compilers In : /Applications/
00:02:46.366  Debug Xcode Find
00:02:46.366  total 0
00:02:46.366  drwxr-xr-x  3 root  wheel  96 Oct 20  2023 Utilities
00:02:46.366  lrwxr-xr-x@ 1 root  wheel  54 Oct 20  2023 Safari.app -> ../System/Cryptexes/App/System/Applications/Safari.app
00:02:46.366  drwxr-xr-x  3 root  wheel  96 Jul 10 03:52 Packages.app
00:02:46.366  End Debug
00:02:46.366  Error - An Xcode Installaton Could Not Be Found In The Default Path : /Applications / Xcode

[steelhead31]

@sophia-guo I've run a grinder here https://ci.adoptium.net/view/Test_grinder/job/Grinder/10782/console with these labels.. ( borrowed from a build job )

xcode15.0.1&&build&&mac&&x64&&!build-orka-macos14-arm64-fwc8f
 

So this issue is down to the orka test nodes not having Xcode installed... I'll take a look at the other issue now

[steelhead31]

@sophia-guo I've fixed the issue with the repro script, and it now attempts to build, but its complaining about a missing SPEC.mk file..

Error: Cannot locate spec.gmk, given by SPEC=osx_x86-64.

@andrew-m-leonard may also be able to help in my absence.

See the grinder https://ci.adoptium.net/view/Test_grinder/job/Grinder/10791/console

[sophia-guo]

I have updated the parsing of makejdk_any_platform_args to align with linux. It seems no issue with the arguments. but I got

14:01:15  BUILD_CONFIG[WORKSPACE_DIR]="/Users/admin/comp-jdk-build/temurin-build/workspace"
14:01:15  /Users/admin/comp-jdk-build/temurin-build/sbin/build.sh: line 1968: cd: /Users/admin/comp-jdk-build/temurin-build/workspace/./build//src: No such file or directory
14:01:15  fatal: '/Users/admin/comp-jdk-build/temurin-build/workspace/./build//src' does not appear to be a git repository
14:01:15  fatal: Could not read from remote repository.
14:01:15 

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10836/consoleFull

@andrew-m-leonard any idea about this? I haven't tried to reset SPEC, if reset what should be the correct value SPEC=osx_aarch64 for cross compiling?

[andrew-m-leonard]

I have updated the parsing of makejdk_any_platform_args to align with linux. It seems no issue with the arguments. but I got

14:01:15  BUILD_CONFIG[WORKSPACE_DIR]="/Users/admin/comp-jdk-build/temurin-build/workspace"
14:01:15  /Users/admin/comp-jdk-build/temurin-build/sbin/build.sh: line 1968: cd: /Users/admin/comp-jdk-build/temurin-build/workspace/./build//src: No such file or directory
14:01:15  fatal: '/Users/admin/comp-jdk-build/temurin-build/workspace/./build//src' does not appear to be a git repository
14:01:15  fatal: Could not read from remote repository.
14:01:15 

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10836/consoleFull

@andrew-m-leonard any idea about this? I haven't tried to reset SPEC, if reset what should be the correct value SPEC=osx_aarch64 for cross compiling?

For an OpenJDK make, SPEC is not set, so in reproducible.mk trying unsetting it ?

[sophia-guo]

@andrew-m-leonard what should we do with option --assemble-exploded-image on mac? Should we just ignore it?

[sophia-guo]

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10922/console @andrew-m-leonard . This is the results by reproduced on arm64 by comparing with repro_compare.sh. Anything wrong?

Regarding to run this x64 tests on arm64, I haven't got a good solution for this. Right now we can enable as part of dev.system and set a special rule like dev.system on mac only run on arm64, as there is no other tests in this group right now. But going forward this is not correct.

[andrew-m-leonard]

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10922/console @andrew-m-leonard . This is the results by reproduced on arm64 by comparing with repro_compare.sh. Anything wrong?

Regarding to run this x64 tests on arm64, I haven't got a good solution for this. Right now we can enable as part of dev.system and set a special rule like dev.system on mac only run on arm64, as there is no other tests in this group right now. But going forward this is not correct.

@sophia-guo It looks like it's not picking up a good JDK for running the Java compare classes:

18:31:20  Cleaning any Temurin build-scripts release file BUILD_INFO from src_jdk/Contents/Home
18:31:21  Error: Fatal error: Unsupported class file version: 67

[sophia-guo]

@sophia-guo It looks like it's not picking up a good JDK for running the Java compare classes:

@andrew-m-leonard I don't quite understand this. Source jdk is copied from build https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23/job/jdk23-mac-x64-temurin/16/artifact/workspace/target/OpenJDK23U-jdk_x64_mac_hotspot_23_37-ea.tar.gz. We can successfully rebuild the compared one. Both have this issue Error: Fatal error: Unsupported class file version: 67

But the compare results is exactly same as when we use the independent jenkins job (thought it's jdk21) https://ci.adoptium.net/view/ReproducibleBuild/job/jdk21u-mac-x64-temurin_reproduce_compare/74/consoleFull

So I'm thinking this might not be issues from test view?

[andrew-m-leonard]

@sophia-guo It looks like it's not picking up a good JDK for running the Java compare classes:

@andrew-m-leonard I don't quite understand this. Source jdk is copied from build https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23/job/jdk23-mac-x64-temurin/16/artifact/workspace/target/OpenJDK23U-jdk_x64_mac_hotspot_23_37-ea.tar.gz. We can successfully rebuild the compared one. Both have this issue Error: Fatal error: Unsupported class file version: 67

But the compare results is exactly same as when we use the independent jenkins job (thought it's jdk21) https://ci.adoptium.net/view/ReproducibleBuild/job/jdk21u-mac-x64-temurin_reproduce_compare/74/consoleFull

So I'm thinking this might not be issues from test view?

The problem is occuring invoking "javap" to process the System*.class files, here:

temurin-build/tooling/reproducible/repro_common.sh

Line 104 in 13b306f

javap -v -sysinfo -l -p -c -s -constants "$ff" > "$f.javap.tmp"

The System*.class will be compiled with the given JDK internal class version, so for jdk-23 that is "67". I would guess the "javap" being picked up off PATH by the test is not jdk-23, so it will not know how to read a jdk-23(67) class file.
The reason it works for jdk21u is probably because the test is picking up a jdk-21 jdk in the PATH?

To resolve, I think we need to put a copy of the source or target JDK in the PATH, to resolve to a version of "javap" that knows how to process class files of the same internal version.

[sophia-guo]

For cross compiling issue discussion is here adoptium/TKG#618

[sophia-guo]

Enable aarch64_mac https://ci.adoptium.net/view/Test_grinder/job/Grinder/11050/console #3972

Interesting , test will archive the reproduced jdk and compare result. For other platform it's around 260M, but Mac_aarch64 is around 4.57G. Will download to see what's the difference and cause such a large aritfact. Caused by wrong workdir, updated.

[sophia-guo]

Update all javap with testJDK.

Also there are 150 dylib differences, which seems removeMacOSNonComparableData() ( remove the macos dylib non-compararable data) was not called at all. Updated to call removeMacOSNonComparableData() and removeWindowsNonComparableData() before the final patchManifests step.

@andrew-m-leonard removeMacOSNonComparableData() get the error, what's the issue here?

00:18:25  Removing MacOS dylib non-comparable UUID from src_jdk/Contents/Home
00:18:45  Error occurred during initialization of VM
00:18:45  Failed setting boot class path.
00:18:45    FAILED ==> java "temurin.tools.BinRepl" --inFile "src_jdk/Contents/Home/../../Contents/Home/bin/jwebserver" --outFile "src_jdk/Contents/Home/../../Contents/Home/bin/jwebserver" --hex "8C:04:09:F5:4F:35:3A:EB:B0:C1:42:BF:D8:09:D4:D5-AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA:AA" --firstOnly

https://ci.adoptium.net/view/Test_grinder/job/Grinder/11059/console