Error generating native-image with --allow-incomplete-classpath

[ilopmar]

This is related to #812. I've tried with latest GraalVM master and also with tag release/graal-vm/1.0 (that currently is rc11-snapshot) and when building the native image I get the following exception:

[fresh-graal:9388]    classlist:   8,397.07 ms
[fresh-graal:9388]        (cap):   1,861.29 ms
[fresh-graal:9388]        setup:   7,494.86 ms
[fresh-graal:9388]     analysis:  77,128.87 ms
fatal error: java.lang.NoClassDefFoundError: rx/Observable
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at jdk.vm.ci.hotspot.HotSpotJDKReflection.getDeclaredMethod(HotSpotJDKReflection.java:560)
	at jdk.vm.ci.hotspot.HotSpotJDKReflection.getMethod(HotSpotJDKReflection.java:601)
	at jdk.vm.ci.hotspot.HotSpotJDKReflection.getMethodAnnotation(HotSpotJDKReflection.java:225)
	at jdk.vm.ci.hotspot.HotSpotResolvedJavaMethodImpl.getAnnotation(HotSpotResolvedJavaMethodImpl.java:526)
	at com.oracle.svm.hosted.cenum.CEnumCallWrapperSubstitutionProcessor.lookup(CEnumCallWrapperSubstitutionProcessor.java:51)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:380)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:360)
	at com.oracle.graal.pointsto.meta.AnalysisType.resolveMethod(AnalysisType.java:811)
	at com.oracle.graal.pointsto.meta.AnalysisType.resolveMethod(AnalysisType.java:73)
	at jdk.vm.ci.meta.ResolvedJavaType.resolveConcreteMethod(ResolvedJavaType.java:238)
	at com.oracle.graal.pointsto.meta.AnalysisType.resolveConcreteMethod(AnalysisType.java:823)
	at com.oracle.graal.pointsto.DefaultAnalysisPolicy$DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultAnalysisPolicy.java:174)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:347)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:389)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:508)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:174)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.ClassNotFoundException: rx.Observable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 29 more
Error: Image building with exit status 1

[cstancu]

I am unable to replicate this while using the example code in #812 , Micronaut 1.0.3, latest GraalVM master, i.e., GraalVM Version 1.0.0-rc12-dev, and labsjdk1.8.0_192-jvmci-0.53 as my base JDK.

[ilopmar]

Actually you're right. I have repeated all steps one more time and it also works for me using the same versions as you. The only difference was that I was using labsjdk1.8.0_192-jvmci-0.50 instead of 0.53.
I can confirm that with 0.50 it fails and using 0.53 it works, so it was my bad 🙏

Just to make sure I'm doing everything right for the future, this is what I do to compile everything:

• Clone https://github.com/oracle/graal and https://github.com/graalvm/mx
• Add mx to the path
• In graal/compiler directory: mx clean, mx build
• In graal/substratevm directory: mx clean, mx build
• In graal/vm directory: mx clean, mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm build
• Then I have the distribution in graal/vm/latest_graalvm_home directory, so in the Micronaut terminal I set JAVA_HOME and PATH to use that version and check that everything is correct:

micronaut-graal-experiments/fresh-graal $ java -version
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
GraalVM 1.0.0-rc12-dev (build 25.192-b12-jvmci-0.53, mixed mode)

Are these the right steps to build everything?

Another question. In the Micronaut sample application we have compileOnly "com.oracle.substratevm:svm" that takes the version from Micronaut's BOM. At this moment is 1.0.0-rc8. We can check that:

$ ./gradlew dependencyInsight --configuration compileOnly --dependency svm

> Task :dependencyInsight
com.oracle.substratevm:svm:1.0.0-rc8 (selected by rule)
   variant "default" [
      org.gradle.status = release (not requested)
   ]

com.oracle.substratevm:svm -> 1.0.0-rc8
\--- compileOnly

com.oracle.substratevm:svm-hosted-native-darwin-amd64:1.0.0-rc8
   variant "runtime" [
      org.gradle.status = release (not requested)
   ]

com.oracle.substratevm:svm-hosted-native-darwin-amd64:1.0.0-rc8
\--- com.oracle.substratevm:svm:1.0.0-rc8
     \--- compileOnly

com.oracle.substratevm:svm-hosted-native-linux-amd64:1.0.0-rc8
   variant "runtime" [
      org.gradle.status = release (not requested)
   ]

com.oracle.substratevm:svm-hosted-native-linux-amd64:1.0.0-rc8
\--- com.oracle.substratevm:svm:1.0.0-rc8
     \--- compileOnly

What I've done is publish locally the latest jars and use them:

• In graal/substratevm/mxbuild/dists/jdk1.8/ I install in my mavenLocal svm.jar, objectfile.jar and pointsto.jar.
• In graal/substratevm/mxbuild/linux-amd64/dists I install in my mavenLocal svm-hosted-native-linux-amd64.tar.gz

And then I update the dependency in build.gradle as com.oracle.substratevm:svm:1.0.0-rc12.BUILD-SNAPSHOT:

$ ./gradlew dependencyInsight --configuration compileOnly --dependency svm

> Task :dependencyInsight
com.oracle.substratevm:svm:1.0.0-rc12.BUILD-SNAPSHOT (selected by rule)
   variant "default" [
      org.gradle.status = integration (not requested)
   ]

com.oracle.substratevm:svm:1.0.0-rc12.BUILD-SNAPSHOT
\--- compileOnly

com.oracle.substratevm:svm-hosted-native-linux-amd64:1.0.0-rc12.BUILD-SNAPSHOT (selected by rule)
   variant "default" [
      org.gradle.status = integration (not requested)
   ]

com.oracle.substratevm:svm-hosted-native-linux-amd64:1.0.0-rc12.BUILD-SNAPSHOT
\--- compileOnly

And build the fatjar again and the native-image. I was wondering if this is really necessary.

Thank you very much for your time and sorry for the noise.

[cstancu]

We should increase the minimum JVMCI required version from 50 to 53. @dougxc recently made some changes on the JVMCI side to help with our efforts to fix the incomplete classpath issues.

The steps you use to build are correct, however you don't need to run mx clean and mx build in graal/compiler and graal/substratevm. It is enough to run it in graal/vm since the first two are transitive dependecies and mx will automatically clean/build them. Yes, mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm build is what you need to build a minimal GraalVM if you are only interested in native-image.

Your JAVA_HOME doesn't necessary need to point to graal/vm/latest_graalvm_home if you only use native-image, you just need to have GraalVM on the PATH. I usually set GRAALVM_HOME to point to graal/vm/latest_graalvm_home and add that to the PATH. My JAVA_HOME points to the labsjdk, e.g., labsjdk1.8.0_192-jvmci-0.53. Pointing JAVA_HOME to GraalVM in the Micronaut terminal should be ok though, but make sure you don't attempt to build GraalVM with GraalVM, that is still tricky.

Regarding the com.oracle.substratevm:svm dependency there haven't been any API changes since 1.0.0-rc8, at least not in the parts of the API the Micronaut example uses, i.e., the substitutions annotations, that's why it should also work with 1.0.0-rc8 - and it does, I tried it. Since it is a compileOnly dependency, as long as the interface signatures match, the build should succed. (However, in general, I think that build.gradle should specify a concrete version of the library, ideally the latest, and avoid the default.) More importantly the version of svm.jar that native-image uses is not the one from your local mavenLocal. It is actually the one in your GRAALVM_HOME, i.e., where native-image is. You can check that by running native-image --no-server --verbose HelloWorld (the HelloWorld class doesn't need to exist) and then inspecting the -cp argument which is generated automatically by the native-image tool. You should see something like:

Executing [
${GRAALVM_HOME}/bin/java \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-XX:-UseJVMCICompiler \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-d64 \
-noverify \
-XX:-UseJVMCIClassLoader \
-Xss10m \
-Xms1g \
-Xmx14g \
-Duser.country=US \
-Duser.language=en \
-Dgraalvm.version=1.0.0-rc12-dev \
-Dorg.graalvm.version=1.0.0-rc12-dev \
-Dcom.oracle.graalvm.isaot=true \
-Djvmci.class.path.append=${GRAALVM_HOME}/jre/lib/jvmci/graal.jar \
-Xbootclasspath/a:${GRAALVM_HOME}/jre/lib/boot/graal-sdk.jar \
-cp \
${GRAALVM_HOME}/jre/lib/svm/builder/objectfile.jar:${GRAALVM_HOME}/jre/lib/svm/builder/svm.jar:${GRAALVM_HOME}/jre/lib/svm/builder/pointsto.jar:${GRAALVM_HOME}/jre/lib/jvmci/jvmci-hotspot.jar:${GRAALVM_HOME}/jre/lib/jvmci/graal-management.jar:${GRAALVM_HOME}/jre/lib/jvmci/graal.jar:${GRAALVM_HOME}/jre/lib/jvmci/jvmci-api.jar \
com.oracle.svm.hosted.NativeImageGeneratorRunner \
-watchpid \
17328 \
-H:Path=${CWD} \
-H:CLibraryPath=${GRAALVM_HOME}/jre/lib/svm/clibraries/linux-amd64 \
-H:Class=HelloWorld \
-H:Name=helloworld \
-imagecp \
${GRAALVM_HOME}/jre/lib/boot/graal-sdk.jar:${GRAALVM_HOME}/jre/lib/svm/builder/objectfile.jar:${GRAALVM_HOME}/jre/lib/svm/builder/svm.jar:${GRAALVM_HOME}/jre/lib/svm/builder/pointsto.jar:${GRAALVM_HOME}/jre/lib/jvmci/jvmci-hotspot.jar:${GRAALVM_HOME}/jre/lib/jvmci/graal-management.jar:${GRAALVM_HOME}/jre/lib/jvmci/graal.jar:${GRAALVM_HOME}/jre/lib/jvmci/jvmci-api.jar:${GRAALVM_HOME}/jre/lib/svm/library-support.jar:${CWD}
]

The native-image tool is just a simple application (which by the way is written in Java and is packaged into an executable using itself) that runs com.oracle.svm.hosted.NativeImageGeneratorRunner on ${GRAALVM_HOME}/bin/java. If you built the fatjar with the right version of the API you shouldn't need to rebuild it, even if implementation details might change.

I hope I answered all your questions. We will try to make all this more explicit in our documentation. I will close this issue now.

[ilopmar]

@cstancu Thank you very much for your detailed explanation. Now I understand a little bit more how this works!