Reproduciable comparing tests on linux failed with Invalid build.sh option: --with-version-opt=ea

[sophia-guo]

https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_x86-64_linux/13/console

15:59:25  Local jdk dir
15:59:25  Starting ./makejdk-any-platform.sh to configure, build (Adoptium)OpenJDK binary
15:59:25  Parsing opt: --clean-git-repo
15:59:25  Possible opt arg: --jdk-boot-dir
15:59:25  Parsing opt: --jdk-boot-dir
15:59:25  Possible opt arg: /usr/lib/jvm/jdk-20.0.2+9
15:59:25  Parsing opt: --configure-args
15:59:25  Possible opt arg: --disable-warnings-as-errors --enable-dtrace
15:59:25  Parsing opt: --with-version-opt=ea
15:59:25  Possible opt arg: --target-file-name
15:59:25  Invalid build.sh option: --with-version-opt=ea

[sophia-guo]

This is parsing issue. I've updated parsing the build parameters, which will correct set --configure-args with quotes. With ea or release build I got following message

14:01:15  Running /home/jenkins/temurin-build/workspace/./build//src/configure with arguments 'bash /home/jenkins/temurin-build/workspace/./build//src/configure --verbose  --with-vendor-name="Eclipse Adoptium" --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-pre=beta --with-version-build=3 --with-vendor-version-string=Temurin-21.0.4+3-202405151902 --with-boot-jdk=/usr/lib/jvm/jdk-20.0.2+9 --with-devkit=/home/jenkins/temurin-build/workspace/./build//devkit --with-debug-level=release --with-native-debug-symbols=external --with-alsa=/home/jenkins/temurin-build/workspace/./build//installedalsa --with-source-date=1715799725 --with-hotspot-build-time='2024-05-15 19:02:05' --disable-ccache --with-build-user=admin --with-jvm-variants=server --with-cacerts-src=/home/jenkins/temurin-build/sbin/../security/certs --disable-warnings-as-errors  --enable-dtrace  --with-version-opt=ea --with-freetype=bundled --with-zlib=bundled'
14:01:15  RPM version 4.11.3
14:01:16  package strace is not installed
14:01:16  Strace is not available on system

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10088/console

@andrew-m-leonard is this build environment setup issue?

[andrew-m-leonard]

This is parsing issue. I've updated parsing the build parameters, which will correct set --configure-args with quotes. With ea or release build I got following message

14:01:15  Running /home/jenkins/temurin-build/workspace/./build//src/configure with arguments 'bash /home/jenkins/temurin-build/workspace/./build//src/configure --verbose  --with-vendor-name="Eclipse Adoptium" --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-pre=beta --with-version-build=3 --with-vendor-version-string=Temurin-21.0.4+3-202405151902 --with-boot-jdk=/usr/lib/jvm/jdk-20.0.2+9 --with-devkit=/home/jenkins/temurin-build/workspace/./build//devkit --with-debug-level=release --with-native-debug-symbols=external --with-alsa=/home/jenkins/temurin-build/workspace/./build//installedalsa --with-source-date=1715799725 --with-hotspot-build-time='2024-05-15 19:02:05' --disable-ccache --with-build-user=admin --with-jvm-variants=server --with-cacerts-src=/home/jenkins/temurin-build/sbin/../security/certs --disable-warnings-as-errors  --enable-dtrace  --with-version-opt=ea --with-freetype=bundled --with-zlib=bundled'
14:01:15  RPM version 4.11.3
14:01:16  package strace is not installed
14:01:16  Strace is not available on system

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10088/console

@andrew-m-leonard is this build environment setup issue?

@sophia-guo it looks like the arguments are not quoted correctly, where is this build being called from?
Maybe should be using single quotes rather than double?

[sophia-guo]

It's from https://ci.adoptium.net/job/build-scripts/job/openjdk21-pipeline/283/artifact/target/linux/x64/temurin/OpenJDK21U-jdk_x64_linux_hotspot_21.0.4_3-ea.tar.gz https://ci.adoptium.net/job/build-scripts/job/openjdk21-pipeline/283/artifact/target/linux/x64/temurin/OpenJDK21U-sbom_x64_linux_hotspot_21.0.4_3-ea.json, which is not able anymore by jenkins build, should be available from binary repo https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.4%2B3-ea-beta/OpenJDK21U-jdk_x64_linux_hotspot_21.0.4_3-ea.tar.gz https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.4%2B3-ea-beta/OpenJDK21U-sbom_x64_linux_hotspot_21.0.4_3-ea.json. Buy it's a common issue with release one https://ci.adoptium.net/job/build-scripts/job/jobs/job/release/job/jobs/job/jdk21u/job/jdk21u-release-linux-x64-temurin/6/consoleFull.
Double quote should not be a problem
@andrew-m-leonard which arguments?
13:58:48 BUILD_CONFIG[USER_SUPPLIED_CONFIGURE_ARGS]="--disable-warnings-as-errors --enable-dtrace --with-version-opt=ea"? Double quote should not be a problem, the only difference I can see is there are two spaces between each options '--disable-warnings-as-errors --enable-dtrace --with-version-opt=ea', will see if that's a problem.

[sophia-guo]

Original ea build Args https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-x64-temurin/200/consoleFull

15:02:06  Running /home/jenkins/workspace/build-scripts/jobs/jdk21u/jdk21u-linux-x64-temurin/workspace/./build//src/configure with arguments 'bash /home/jenkins/workspace/build-scripts/jobs/jdk21u/jdk21u-linux-x64-temurin/workspace/./build//src/configure --verbose  --with-vendor-name="Eclipse Adoptium" --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-pre=beta --with-version-build=3 --with-vendor-version-string=Temurin-21.0.4+3-202405151902 --with-boot-jdk=/usr/lib/jvm/jdk-20 --with-devkit=/usr/local/devkit/gcc-11.3.0-Centos7.9.2009-b02 --with-debug-level=release --with-native-debug-symbols=external  --with-alsa=/home/jenkins/workspace/build-scripts/jobs/jdk21u/jdk21u-linux-x64-temurin/workspace/./build//installedalsa --with-source-date=1715799725 --with-hotspot-build-time='2024-05-15 19:02:05' --disable-ccache --with-build-user=admin --with-jvm-variants=server --with-cacerts-src=/home/jenkins/workspace/build-scripts/jobs/jdk21u/jdk21u-linux-x64-temurin/sbin/../security/certs  --disable-warnings-as-errors --enable-dtrace --with-version-opt=ea --with-freetype=bundled --with-zlib=bundled'

Reproduced Args https://ci.adoptium.net/view/Test_grinder/job/Grinder/10088/console

14:01:15  Running /home/jenkins/temurin-build/workspace/./build//src/configure with arguments 'bash /home/jenkins/temurin-build/workspace/./build//src/configure --verbose  --with-vendor-name="Eclipse Adoptium" --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-pre=beta --with-version-build=3 --with-vendor-version-string=Temurin-21.0.4+3-202405151902 --with-boot-jdk=/usr/lib/jvm/jdk-20.0.2+9 --with-devkit=/home/jenkins/temurin-build/workspace/./build//devkit --with-debug-level=release --with-native-debug-symbols=external --with-alsa=/home/jenkins/temurin-build/workspace/./build//installedalsa --with-source-date=1715799725 --with-hotspot-build-time='2024-05-15 19:02:05' --disable-ccache --with-build-user=admin --with-jvm-variants=server --with-cacerts-src=/home/jenkins/temurin-build/sbin/../security/certs --disable-warnings-as-errors  --enable-dtrace  --with-version-opt=ea --with-freetype=bundled --with-zlib=bundled'

Release build https://ci.adoptium.net/job/build-scripts/job/jobs/job/release/job/jobs/job/jdk21u/job/jdk21u-release-linux-x64-temurin/6/consoleFull

02:45:55  Running /home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/./build//src/configure with arguments 'bash /home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/./build//src/configure --verbose  --with-vendor-name="Eclipse Adoptium" --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-opt=LTS --without-version-pre --with-version-build=9 --with-vendor-version-string=Temurin-21.0.3+9 --with-boot-jdk=/usr/lib/jvm/jdk-20 --with-devkit=/home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/./build//devkit --with-debug-level=release --with-native-debug-symbols=external  --with-source-date=version --disable-ccache --with-build-user=admin --with-extra-cflags='-fdebug-prefix-map=/home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/build/openjdkbuild/=' --with-extra-cxxflags='-fdebug-prefix-map=/home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/build/openjdkbuild/=' --with-jvm-variants=server --with-cacerts-src=/home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/sbin/../security/certs  --disable-warnings-as-errors --enable-dtrace --with-freetype=bundled --with-zlib=bundled'

I didn't see the difference args between original and reproduced builds except:
ea build

--with-devkit=/usr/local/devkit/gcc-11.3.0-Centos7.9.2009-b02
15:02:06  RPM version 4.11.3
15:02:06  strace-4.24-7.el7_9.x86_64
15:02:06  Strace and rpm is available on system

reproduced build:

--with-devkit=/home/jenkins/temurin-build/workspace/./build//devkit
14:01:15  RPM version 4.11.3
14:01:16  package strace is not installed
14:01:16  Strace is not available on system

release build

--with-devkit=/home/jenkins/workspace/build-scripts/jobs/release/jobs/jdk21u/jdk21u-release-linux-x64-temurin/workspace/./build//devkit
02:45:55  RPM version 4.11.3
02:45:55  strace-4.24-7.el7_9.x86_64
02:45:55  Strace and rpm is available on system

Seems package strace is not installed ? @andrew-m-leonard

[sophia-guo]

installed strace and got https://ci.adoptium.net/view/Test_grinder/job/Grinder/10108/console

12:31:15  strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
12:31:15  strace: PTRACE_SETOPTIONS: Operation not permitted
12:31:15  strace: detach: waitpid(5330): No child processes
12:31:15  OpenJDK make failed, archiving make failed logs
12:31:15  Archiving and compressing with pigz

Seems related with docker version and be fixed by '--cap-add=SYS_PTRACE' ? https://jvns.ca/blog/2020/04/29/why-strace-doesnt-work-in-docker/

[sophia-guo]

Is this also related with 14:01:16 Strace is not available on system? @andrew-m-leonard

[andrew-m-leonard]

Is this also related with 14:01:16 Strace is not available on system? @andrew-m-leonard

@sophia-guo not related to that issue
the problem is the x86_64 linux "test" nodes don't have "strace" enabled like we do on our build nodes.... this means you can't run an x86_64 (only arch currently running strace).

What I suggest is to disable running strace on the reproduce build, as you don't need to. Remove this build arg "--enable-sbom-strace"

[sophia-guo]

adoptium/temurin-build#3834