Micronaut support for SubstrateVM and native image

[graemerocher]

I have been investigating support for SubstrateVM and the nativeimage tool for Micronaut http://micronaut.io/

Micronaut produces largely reflection free DI and AOP, however does allow soft loading of classes.

Currently only one error occurs when preparing the image, which is probably a matter of supplying a correct reflection.json:

RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of io.micronaut.caffeine.cache.UnsafeRefArrayAccess. Add a RecomputeFieldValue.ArrayIndexScale manual substitution for io.micronaut.caffeine.cache.UnsafeRefArrayAccess. 

More fundamentally however, we rely on ServiceLoader.load(..) and classLoader.getResources() in a few places and these seem to be unimplemented? Calls to getClass().getClassLoader() return null

Is there an alternative way to load resources other than through the classloader? Can we provide an alternative to ServiceLoader.load(..)?

We could certainly abstract the strategies for loading resources and service descriptors if there was an alternative way to get at these and a way to detect you are running on SubstrateVM (is there a way?)

[cstancu]

@graemerocher loading resources is currently available through the Class.getResourceAsStream(String) and Class.getResource(String) API. The resources need to be specified at image build time using the option -H:IncludeResources=<resource-path-regexp> (we will add this option to the documentation too).

We are also currently working on a solution to provide a custom class loader at runtime that can be used to load any class that has been observed during image building. This class loader will be available through Thread.getContextClassLoader(), Class.getClassLoader() and ClassLoader.getSystemClassLoader(), and will also enable you to load resources directly.

[cstancu]

Regarding the automatic substitution failed warning, that is not about reflection, it is about the sun.misc.Unsafe API. You can read more about SubstrateVM support of sun.misc.Unsafe and about how those warnings can be avoided in this blog post, the Unsafe Memory Access section.

[graemerocher]

@cstancu Thanks for the reply. Is there a way to use wildcards to include resources, such as -H:IncludeResources=META-INF/services/* ?

Specifying every resource by name is not really practical as many are generated at compile time.

[cstancu]

Yes, wildcards should work, the option takes a regexp as an argument. It also works for resources in both directories and jars on the classpath. We don't have documentation for this yet, but the implementation is straightforward.

[graemerocher]

@cstancu I have progressed past the resources issue. Our next problem is with regards to ServiceLoader. The following throws an exception:

     ServiceLoader<BeanDefinitionReference> services = ServiceLoader.load(BeanDefinitionReference.class);
        System.out.println("services.iterator().hasNext() = " + services.iterator().hasNext());

        for (BeanDefinitionReference beanDefinitionReference : services) {
            System.out.println("beanDefinitionReference = " + beanDefinitionReference);
        }

The above results in:

Caused by: java.util.ServiceConfigurationError: io.micronaut.inject.BeanDefinitionReference: Provider example.$HelloControllerDefinitionClass not found
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at java.util.ServiceConfigurationError.<init>(ServiceConfigurationError.java:72)
	at java.util.ServiceLoader.fail(ServiceLoader.java:239)
	at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:69)
	at ex

I assume this related to limitations with regards to dynamic class loading, although in this case the classes are on the classpath when the image is created. Is this a limitation that can be overcome?

[cstancu]

Yes, this is related to the dynamic class loading limitation. Currently Thread.currentThread().getContextClassLoader() returns null. As I mentioned above we are currently working on a solution for this.

[graemerocher]

Gotcha. Thanks

[graemerocher]

@cstancu do you have a timeline for when the feature you describe will be available?

[cstancu]

We will try to make it available in the next release, which as of now is planned for June 29th.

[graemerocher]

Thanks for the update

[cstancu]

As of ef56566 we have support for dynamic class loading thanks to @vjovanov. That is ClassLoader.getSystemClassLoader() and Thread.getContextClassLoader() will return a class loader that can be used to load classes seen during native image building. Please note that this is not "true" dynamic class loading. The classes need to be visible during image building and need to be configured for reflective access. For services you need to configure all the classes implementing the service since those are loaded with Class.forName(). It is also worth mentioning that Class.getClassLoader() currently returns null which, as per the JDK comment on the original method, is valid and signifies that the class was loaded by the bootstrap class loader. We may change this behavior in the future and return a non-null class loader. This doesn't affect the ServiceLoader implementation as it falls back to ClassLoader.getSystemClassLoader() when you provide a null ClassLoader. Please try it and let us know how it works.

[cstancu]

By the way, now that you have access to a ClassLoader you can also use ClassLoader.getResource(String) (and all the other related methods) directly. You need a similar configuration as for Class.getResource(String) as they are backed by the same implementation.

[cstancu]

I also created a simple demo project for loading services with ServiceLoader at
https://github.com/cstancu/native-image-service-loader-demo.

[graemerocher]

@cstancu Great thanks! Will try it out shortly

[graemerocher]

Are there instructions somewhere to build from source on Mac OS X?

[cstancu]

@graemerocher AFAIK building from sources on Mac OS X shouldn't be any different than building on Linux. Please see the quick start guide for quick building for development purposes. You can also build a distribution of GraalVM by following the info in the VM suite. If you just care about native-image you can:

$ git clone https://github.com/oracle/graal.git
$ cd graal/vm
$ mx --dy /substratevm build

This will take a while as it builds several components. After that you will find a VM distribution in graal/vm/latest_graalvm.

[graemerocher]

@cstancu Currently when I try this with the latest source it fails with:

Archiving GRAAL_RUNTIME... [dependency org.graalvm.compiler.runtime updated]
/Users/graemerocher/dev/oss/graal/truffle/src/com.oracle.truffle.api.vm/src/com/oracle/truffle/api/vm/PolyglotImpl.java:92: error: PolyglotImpl is not abstract and does not override abstract method buildEngine(OutputStream,OutputStream,InputStream,Map<String,String>,long,TimeUnit,boolean,long,boolean,boolean) in AbstractPolyglotImpl
public final class PolyglotImpl extends AbstractPolyglotImpl {
             ^
/Users/graemerocher/dev/oss/graal/truffle/src/com.oracle.truffle.api.vm/src/com/oracle/truffle/api/vm/PolyglotImpl.java:157: error: method does not override or implement a method from a supertype
    @Override
    ^
/Users/graemerocher/dev/oss/graal/truffle/src/com.oracle.truffle.api.vm/src/com/oracle/truffle/api/vm/PolyglotEngineImpl.java:79: error: PolyglotEngineImpl is not abstract and does not override abstract method createContext(OutputStream,OutputStream,InputStream,boolean,boolean,boolean,boolean,boolean,Predicate<String>,Map<String,String>,Map<String,String[]>,String[],FileSystem) in AbstractEngineImpl
class PolyglotEngineImpl extends org.graalvm.polyglot.impl.AbstractPolyglotImpl.AbstractEngineImpl implements com.oracle.truffle.api.vm.PolyglotImpl.VMObject {
^
/Users/graemerocher/dev/oss/graal/truffle/src/com.oracle.truffle.api.vm/src/com/oracle/truffle/api/vm/PolyglotEngineImpl.java:946: error: method does not override or implement a method from a supertype
    @Override
    ^
Compiling com.oracle.svm.core with javac-daemon(JDK 1.8)... [dependency GRAAL_RUNTIME updated]
Compiling com.oracle.graal.pointsto with javac-daemon(JDK 1.8)... [dependency GRAAL_RUNTIME updated]
4 errors

Compiling com.oracle.truffle.api.vm with javac-daemon(JDK 1.8) failed
error while killing subprocess 2877 "/Users/graemerocher/.sdkman/candidates/java/graalvm-ee-1.0.0-rc2/bin/java -d64 -cp /Users/graemerocher/dev/oss/mx/mxbuild/java/com.oracle.mxtool.compilerserver/bin:/Users/graemerocher/.sdkman/candidates/java/graalvm-ee-1.0.0-rc2/lib/tools.jar com.oracle.mxtool.compilerserver.JavacDaemon -j 16": [Errno 3] No such process
1 build tasks failed

I am using graalvm-ee-1.0.0-rc2 to build though. Is that correct?

[cstancu]

You need to use a labsjdk. You can get one from http://www.oracle.com/technetwork/oracle-labs/program-languages/downloads/index.html the JVMCI JDK Downloads section. We currently don't support self bootstrapping GraalVM

[graemerocher]

That was it. Got it compiling. Thanks for the help

[graemerocher]

I have gotten a little further with getting Micronaut running for the project https://github.com/micronaut-projects/micronaut-examples/tree/master/hello-world-kotlin

After doing ./gradlew assemble and cd build/libs and running:

$ native-image --class-path hello-world-kotlin-0.1-all.jar -H:ReflectionConfigurationFiles=../../reflect.json -H:EnableURLProtocols=http -H:IncludeResources=META-INF/services/*.* -H:Name=hw -H:Class=example.Application -H:+ReportUnsupportedElementsAtRuntime

I am hitting this error when running the application:

java.util.ServiceConfigurationError: io.micronaut.inject.BeanConfiguration: Provider io.micronaut.http.server.cors.$BeanConfiguration not found
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at java.util.ServiceConfigurationError.<init>(ServiceConfigurationError.java:72)
	at java.util.ServiceLoader.fail(ServiceLoader.java:239)
	at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at io.micronaut.context.DefaultBeanContext.readAllBeanConfigurations(DefaultBeanContext.java:1585)

The class io.micronaut.http.server.cors.$BeanConfiguration is contained within the hello-world-kotlin-0.1-all.jar being used to build the native image so I am not 100% sure why the error is happening.

I assume it is because I need to configure the class for reflective access as per https://github.com/oracle/graal/blob/master/substratevm/REFLECTION.md

We don't actually need to do any reflection on the class however, just instantiate it. What would a appropriate reflection.json look like in this case?

[graemerocher]

ok so doing:

[ 
  {
    "name" : "io.micronaut.http.server.cors.$BeanConfiguration",
  "allDeclaredConstructors" : true
  },
  {
    "name": "io.netty.channel.socket.nio.NioServerSocketChannel",
    "methods": [
      { "name": "<init>", "parameterTypes": [] }
    ]
  }
]

Seems to get me further. I will have to figure out a way to make all META-INF/services be declared in reflection.json

It seems like it would be nice if there was an option in Graal to do this automatically? Since the classes are already declared in META-INF/services files it seems a odd constraint to have to duplicate that content in a reflection JSON file?

[cstancu]

Yes, currently all classes used to allocate objects via Class.newInstance(), which is what ServiceLoader does for all service providers, need to be explicitly declared via the reflection configuration as I demonstrated in the demo project. (Not sure if Class.newInstance() is actually considered to be part of the Java reflection API but it is often used with other reflective calls and we treat it as such to simplify configuration.) In the long run we will automate discovery of classes/methods/fields that need reflective access, for example detecting classes loaded via Class.forName(name) as long as the name is a constant. This will include classes declared in META-INF/services as well. This is not a strong limitation, just needs more work.

[graemerocher]

Understood, I think I can probably write a build to that generates the reflection.json specifically for Micronaut but yeah it seems like from a user experience point of view that it would be better not to have to need a build tool.

[graemerocher]

Ok so I wrote a Gradle build task that generates the reflect.json https://github.com/graemerocher/micronaut-graal-experiments/blob/master/hello-world-java/build.gradle#L66

The steps I am taking with to build the image are here:

https://github.com/graemerocher/micronaut-graal-experiments/tree/master/hello-world-java

It is now failing with

Caused by: java.lang.ArrayStoreException: sun.reflect.annotation.TypeNotPresentExceptionProxy
	at sun.reflect.annotation.AnnotationParser.parseClassArray(AnnotationParser.java:724)
	at sun.reflect.annotation.AnnotationParser.parseArray(AnnotationParser.java:531)
	at sun.reflect.annotation.AnnotationParser.parseMemberValue(AnnotationParser.java:355)
	at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:286)
	at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
	at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
	at java.lang.Class.createAnnotationData(Class.java:3521)
	at java.lang.Class.annotationData(Class.java:3510)
	at java.lang.Class.getAnnotation(Class.java:3415)
	at jdk.vm.ci.hotspot.HotSpotResolvedObjectTypeImpl.getAnnotation(HotSpotResolvedObjectTypeImpl.java:79

Is there a way we can get a better error to find out the class causing that exception?

[cstancu]

When you build from source for development (i.e., running mx build in graal/substratevm) you can find svm.jar under graal/substratevm/mxbuild/dists/svm.jar. If you build a GraalVM distribution as I described above (i.e., running mx --dy /substratevm build in graal/vm) you can find svm.jar under graal/vm/latest_graalvm/graalvm-cmp-gu-gvm-ins-libpoly-poly-polynative-pro-rgx-svm-tfl-1.0.0-rc3-dev/jre/lib/svm/builder/svm.jar.

Edit: You can deploy it locally running mvn install:install-file -Dfile=./mxbuild/dists/svm.jar -DgroupId=com.oracle.svm -DartifactId=svm -Dversion=GraalVM-1.0.0-rc3-dev -Dpackaging=jar from graal/substratevm.

[cstancu]

After deploying svm.jar from 1.0.0-rc3-dev locally, pulling your latest changes and updating the dependency in build.gradle I cannot see the error you describe above. However I hit the dreaded error:

Error: Detected a started Thread in the image heap. This is not supported. The object was reached from a static initializer. All static class initialization is done during native image construction, thus a static initializer cannot contain code that captures state dependent on the build machine. Write your own initialization methods and call them explicitly from your main entry point.
Trace: 	object io.netty.util.internal.ObjectCleaner$AutomaticCleanerReference
	object java.util.concurrent.ConcurrentHashMap$Node
	object java.util.concurrent.ConcurrentHashMap$Node[]
	object java.util.concurrent.ConcurrentHashMap
	object io.netty.util.internal.ConcurrentSet
	method io.netty.util.internal.ObjectCleaner.register(Object, Runnable)
Call path from entry point to io.netty.util.internal.ObjectCleaner.register(Object, Runnable): 
	at io.netty.util.internal.ObjectCleaner.register(ObjectCleaner.java:95)
	at io.netty.util.concurrent.FastThreadLocal.registerCleaner(FastThreadLocal.java:158)
	at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:144)
	at io.netty.channel.ChannelOutboundBuffer.clearNioBuffers(ChannelOutboundBuffer.java:356)
	at io.netty.channel.ChannelOutboundBuffer.close(ChannelOutboundBuffer.java:686)
	at io.netty.channel.ChannelOutboundBuffer$3.run(ChannelOutboundBuffer.java:653)
	at com.oracle.svm.core.jdk.RuntimeSupport.executeHooks(RuntimeSupport.java:137)
	at com.oracle.svm.core.jdk.RuntimeSupport.executeStartupHooks(RuntimeSupport.java:91)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:172)
	at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eJavaMainWrapper_002erun_0028int_002corg_002egraalvm_002enativeimage_002ec_002etype_002eCCharPointerPointer_0029(generated:0)

It looks like the micronaut code makes some part of netty reachable that we haven't tried before. We don't have a general solution for this type of errors yet and the workaround in the error message, to " Write your own initialization methods and call them explicitly from your main entry point", cannot be easily applied to third party libraries. We are investigating the possibility to push some static initializers to runtime to avoid issues like this, but that feature is only at the drawing board stage.

[graemerocher]

Is there a way to identify what part of Netty is starting threads in the static initialiser? We can then potentially report an issue to the Netty project to make it configurable somehow

[sdeleuze]

Hey @graemerocher, it has already been fixed in Netty snapshots, see https://twitter.com/normanmaurer/status/1014461371168247808.

[graemerocher]

@sdeleuze Thanks for the pointer, will try out the snapshots

[graemerocher]

I have made some more progress on getting Micronaut operational on Graal nativeimage, it is getting much further in running the application. Currently I am hitting a problem with this line of code:

return beansOfType.toArray((Object[]) Array.newInstance(argumentType.getComponentType(), beansOfType.size()));

Which leads to:

Class io.micronaut.web.router.RouteBuilder[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.RuntimeReflection

It seems there is no way in nativeimage to create an array programatically? Does that mean every time dynamic array creating occurs we have to register the class for reflection? 😱

[graemerocher]

FWIW I tried to add io.micronaut.web.router.RouteBuilder[] to reflect.json and it reports class not found. Adding io.micronaut.web.router.RouteBuilder doesn't work either.

[cstancu]

You should use the [Lio.micronaut.web.router.RouteBuilder; format. However there is no reason to not support the io.micronaut.web.router.RouteBuilder[] format too. We will do that in the future.

[cstancu]

The changes in 3c85875 might also interest you. It improves native-image reflection support by automatically detecting reflective calls and intrinsifying them to the corresponding target elements statically. Please read the updated REFLECTION.md for details.

[graemerocher]

Ok, I will take a look, in the meantime what would really help is a fix for the sun.reflect.annotation.TypeNotPresentExceptionProxy issue discussed previously. Can --report-unsupported-elements-at-runtime also be made to work with annotations that reference types not on the class path?

[graemerocher]

@cstancu I tried to make some changes to get passed the array issue. However now the native imagine I built is exiting with:

Segmentation fault: 11

How do I debug this issue? The application is at https://github.com/graemerocher/micronaut-graal-experiments

[graemerocher]

Any thoughts on how to debug the Segmentation fault issue?

[cstancu]

I tried building your app but I hit the sun.reflect.annotation.TypeNotPresentExceptionProxy exceptions. Can you push your changes that avoid that?

[cstancu]

If you use GraalVM EE then you can pass the -g flag which will generate debugging information and then you can use GDB to debug.

[graemerocher]

@cstancu I pushed the latest changes, btw something broke in rc5 because I get

Build on Server(pid: 12115, port: 59905)
   classlist:   1,749.07 ms
       setup:      51.03 ms
error: Already registered: sun.misc.PerfCounter.add(long)
Error: Processing image build request failed

When using rc5.

With rc4 I get the segmentation fault still.

[cstancu]

The reason for the Already registered: ... error is that you have svm.jar classes twice and we try to register the same substitution class twice. First it is included in your hello-world-java-0.1-all.jar and then it is added to the native-image classpath automatically. (You can run with --verbose flag to see the -imagecp value.) You should change your GraalVM dependency in build.gradle from compile to compileOnly: compileOnly 'com.oracle.substratevm:svm:GraalVM-1.0.0-rc5'.

svm.jar was not by default on the native-image -imagecp before rc5 and it might change back in the future. In general it is better to add SVM API dependencies to the native-image explicitly rather than adding them to your jar.
The previous paragraph was misleading, here is an edit: The native-image tool adds its own API dependencies automatically, via the -imagecp, option. If your app depends on any of the SVM APIs you need to make sure that those dependencies are compileOnly and not added to your app jar.

[graemerocher]

@cstancu I have made significant progress with Micronaut on Graal nativeimage.

First the good news is that Micronaut now starts up and takes 40ms to do so consuming 30mb of total memory. The application at https://github.com/graemerocher/micronaut-graal-experiments/tree/master/hello-world-java demonstrates this.

To make it work I wrote a class loading reporter that generates the reflect.json file when the applications tests are executed https://github.com/micronaut-projects/micronaut-core/blob/master/graal/src/main/java/io/micronaut/graal/reflect/GraalClassLoadingReporter.java#L49

So effectively you run ./gradlew test assemble and then ./build-native-image.sh to build a native image.

The bad news is that things are still not working 100% and request processing fails with the following exception:

10:27:31.400 [nioEventLoopGroup-2-3] ERROR i.m.h.s.netty.RoutingInBoundHandler - Unexpected error occurred: Code that was considered unreachable by closed-world analysis was reached
com.oracle.svm.core.jdk.UnsupportedFeatureError: Code that was considered unreachable by closed-world analysis was reached
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at com.oracle.svm.core.jdk.UnsupportedFeatureError.<init>(UnsupportedFeatureError.java:31)
	at com.oracle.svm.core.jdk.Target_com_oracle_svm_core_util_VMError.unsupportedFeature(VMErrorSubstitutions.java:109)
	at com.oracle.svm.core.snippets.SnippetRuntime.unreachedCode(SnippetRuntime.java:199)
	at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:354)
	

Which refers to this line of code https://github.com/micronaut-projects/micronaut-core/blob/30826975035f0dcfbc4fd3fb615b3cb25adc374d/http-server-netty/src/main/java/io/micronaut/http/server/netty/RoutingInBoundHandler.java#L354

I don't really understand why that line is causing the issue since it is simply calling the get method on a collection. Any tips?

You can reproduce the error by building the native image, running ./hw and then hitting http://localhost:8080/hello.

[graemerocher]

One other thing is that I still get intermittent segmentation faults which I don't know how to resolve. They seem to be related to logback. If you change the log level in logback.xml to info https://github.com/graemerocher/micronaut-graal-experiments/blob/master/hello-world-java/src/main/resources/logback.xml#L11

Then I consistently get a Segmentation fault: 11 error every time I run the app whilst switching back to debug logging I only get the error occasionally. It seems the late initialisation of logback with lesser log levels causes the issue? Either way there appears to be issues with using nativeimage and logback. It may be that switching to another logging framework resolves the issue.

[graemerocher]

Adding a recompute for Caffeine seems to have alleviated the Segmentation fault: 11 issue graemerocher/micronaut-graal-experiments@b5e58a0

Now I am just stuck on the UnsupportedFeatureError referenced in comment
#470 (comment)

If there are any thoughts on how I can alter the code to get past the error, or whether I should file a separate issue report please let me know.

[cstancu]

@graemerocher good to hear that you got past the segmentation fault. I will take a look at the Code that was considered unreachable by closed-world analysis was reached error in the next days.

[graemerocher]

@cstancu Thanks for looking into it. I tried changing the code from UriRouteMatch<Object> establishedRoute = uriRoutes.get(0); to UriRouteMatch<Object> establishedRoute = uriRoutes.iterator().next(); but the same error occurs so maybe it is related to the object contained within the collection, but I am not sure.

[graemerocher]

Micronaut now works on Graal native image with RC6. See https://github.com/graemerocher/micronaut-graal-experiments/tree/master/hello-world-java for a working example

Thanks for all the great feedback

[cstancu]

That's great news. It looks like --delay-class-initialization-to-runtime is what you needed.