Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception handling not working correctly in native-image master with --allow-incomplete-classpath #812

Closed
graemerocher opened this issue Nov 16, 2018 · 25 comments
Assignees

Comments

@graemerocher
Copy link
Member

graemerocher commented Nov 16, 2018

Steps to reproduce using GraalVM master:

  1. git clone git@github.com:graemerocher/micronaut-graal-experiments.git
  2. cd fresh-graal
  3. build-native-image.sh

The image builds but when starting the app the exception is thrown:

Caused by: java.lang.NoClassDefFoundError: reactor.core.publisher.Mono
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at java.lang.LinkageError.<init>(LinkageError.java:55)
	at java.lang.NoClassDefFoundError.<init>(NoClassDefFoundError.java:59)
	at com.oracle.svm.core.Exceptions.throwNoClassDefFoundError(Exceptions.java:43)
	at io.micronaut.reactive.reactor.converters.$PublisherToMonoConverterDefinition.$micronaut_load_class_value_0(Unknown Source)
	at io.micronaut.reactive.reactor.converters.$PublisherToMonoConverterDefinition.<init>(Unknown Source)

The exception originates from generated byte code, that looks like

AnnotationClassValue $micronaut_load_class_value_0() {
    try {
         return new AnnotationClassValue(reactor.core.publisher.Mono.class);
    } catch(Throwable e) {
         return new AnnotationClassValue("reactor.core.publisher.Mono");
    }
}

The exception should hence never be thrown and be caught and handled. This worked fine in rc8.

@graemerocher graemerocher changed the title Exception handling not working correctly in master with --allow-incomplete-classpath Exception handling not working correctly in native-image master with --allow-incomplete-classpath Nov 16, 2018
@cstancu
Copy link
Member

cstancu commented Nov 21, 2018

Thanks for reporting and for the easy to use reproducer. The issue is now fixed in master. However, now I run into another runtime error:

20:34:44.657 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: Error instantiating bean of type [io.micronaut.core.convert.TypeConverter]: Multiple possible bean candidates found: [io.micronaut.http.server.netty.converters.ByteBufConverters, io.micronaut.http.server.netty.converters.ByteBufConverters]
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.micronaut.core.convert.TypeConverter]: Multiple possible bean candidates found: [io.micronaut.http.server.netty.converters.ByteBufConverters, io.micronaut.http.server.netty.converters.ByteBufConverters]
	at java.lang.Throwable.<init>(Throwable.java:287)
	at java.lang.Exception.<init>(Exception.java:84)
	at java.lang.RuntimeException.<init>(RuntimeException.java:80)
	at io.micronaut.context.exceptions.BeanContextException.<init>(BeanContextException.java:32)
	at io.micronaut.context.exceptions.BeanInstantiationException.<init>(BeanInstantiationException.java:67)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1345)
	at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:2237)
	at io.micronaut.context.DefaultBeanContext.getBeansOfTypeInternal(DefaultBeanContext.java:2165)
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:752)
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:522)
	at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:313)
	at io.micronaut.context.DefaultApplicationContext.initializeTypeConverters(DefaultApplicationContext.java:330)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:184)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:2033)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:156)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:138)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:67)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:271)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:257)
	at fresh.graal.Application.main(Application.java:8)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:164)

Everything seems to run ok when I run $ java -jar build/libs/fresh-graal-0.1-all.jar so is likely a native-image issue.

@graemerocher
Copy link
Member Author

@cstancu Thanks, I will look into it. Maybe it because the bean for ByteBufConverters is loaded twice because it appears in both reflect.json and then is also loaded by the new service loader feature?

@graemerocher
Copy link
Member Author

@cstancu I updated the example to use a minimal reflect.json graemerocher/micronaut-graal-experiments@1f590fc

So it is definitely not the reflect.json that is the issue. I am wondering if there is a bug in the ServiceLoader implementation and multiple instances of the same type are being returned?

@cstancu
Copy link
Member

cstancu commented Nov 22, 2018

You are right, it looks like we are registering the META-INF/services/io.micronaut.inject.BeanDefinitionReference resource twice. Once when we parse the -H:IncludeResources="logback.xml|application.yml|META-INF/services/*.*" option and again via the ServiceLoaderFeature when it is processing the io.micronaut.inject.BeanDefinitionReference service.
( You need -H:Log=registerResource and -H:+TraceServiceLoaderFeature if you want to look at the trace for those options.) Then, at runtime io.micronaut.core.io.service.SoftServiceLoader loads the same class twice. This is clearly an issue on our side and we should fix it. (Alternativelly, io.micronaut.context.DefaultBeanContext.beanDefinitionsClasses could be implemented as a set instead of a queue. I am just sugesting this as an workaround, I am not familiar with the internals of micronaut and I don't know what the impact would be.)

For now, just removing |META-INF/services/*.* from -H:IncludeResources= and relying on the ServiceLoaderFeature should work. Doing so makes the executable run past that point, however it crashes later with:

20:25:03.374 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: reactor.core.publisher.Mono
java.lang.NoClassDefFoundError: reactor.core.publisher.Mono
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at java.lang.LinkageError.<init>(LinkageError.java:55)
	at java.lang.NoClassDefFoundError.<init>(NoClassDefFoundError.java:59)
	at com.oracle.svm.core.Exceptions.throwNoClassDefFoundError(Exceptions.java:43)
	at io.micronaut.reactive.reactor.converters.$PublisherToMonoConverterDefinition.getTypeArgumentsMap(Unknown Source)
	at io.micronaut.context.AbstractBeanDefinition.getTypeArguments(AbstractBeanDefinition.java:210)
	at io.micronaut.inject.BeanDefinition.getTypeArguments(BeanDefinition.java:193)
	at io.micronaut.context.DefaultApplicationContext.initializeTypeConverters(DefaultApplicationContext.java:333)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:184)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:2033)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:156)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:138)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:67)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:271)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:257)
	at fresh.graal.Application.main(Application.java:8)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:164)

reactor.core.publisher.Mono is clearly missing, running javap -v -cp build/libs/fresh-graal-0.1-all.jar reactor.core.publisher.Mono yields Error: class not found: reactor.core.publisher.Mono, however this is probably another corner case that --allow-incomplete-classpath needs to support. I'll investigate further.

@graemerocher
Copy link
Member Author

Interesting, so looks like we can probably get rid of META-INF/services/*.* however the original bug still stands, so would be good to get that fixed.

Regarding the Mono exception, there is definitely still something going wrong, it should not get to the point where that exception occurs, since the bean has a @Requires which means it would only be loaded if the class is not present

See https://github.com/micronaut-projects/micronaut-core/blob/master/runtime/src/main/java/io/micronaut/reactive/reactor/converters/PublisherToMonoConverter.java#L35

I think somehow Graal is reporting Mono as present where it is not, which is why it doesn't work in the native image.

@graemerocher
Copy link
Member Author

This is the code that is failing, it should be reporting the classes as missing on the class path (which they are):

https://github.com/micronaut-projects/micronaut-core/blob/master/inject/src/main/java/io/micronaut/context/RequiresCondition.java#L404

Instead the log doesn't output any "Class xxx is not present" messages, as Graal reports them as being present.

It seems there is still something wrong with the $micronaut_load_class.. methods I mentioned earlier. They must be returning classes for this code:

AnnotationClassValue $micronaut_load_class_value_0() {
    try {
         return new AnnotationClassValue(reactor.core.publisher.Mono.class);
    } catch(Throwable e) {
         return new AnnotationClassValue("reactor.core.publisher.Mono");
    }
}

What should happen is the string constructor should be used, but instead it seems the class constructor is used even if the class is not there?

@graemerocher
Copy link
Member Author

graemerocher commented Dec 17, 2018

@cstancu @thomaswue I have looked into this further to try and identify the problem and it seems at image compilation time Graal is representing classes that are not present as being present. I added some printlns to this constructor (which should never be invoked as the class is not present):

https://github.com/micronaut-projects/micronaut-core/blob/master/core/src/main/java/io/micronaut/core/annotation/AnnotationClassValue.java#L56

diff --git a/core/src/main/java/io/micronaut/core/annotation/AnnotationClassValue.java b/core/src/main/java/io/micronaut/core/annotation/AnnotationClassValue.java
index 739d51549..2db7d7081 100644
--- a/core/src/main/java/io/micronaut/core/annotation/AnnotationClassValue.java
+++ b/core/src/main/java/io/micronaut/core/annotation/AnnotationClassValue.java
@@ -42,6 +42,7 @@ public final class AnnotationClassValue<T> implements CharSequence, Named {
      */
     @UsedByGeneratedCode
     public AnnotationClassValue(String name) {
+        System.out.println("name = " + name);
         this.name = name.intern();
         this.theClass = null;
     }
@@ -53,6 +54,7 @@ public final class AnnotationClassValue<T> implements CharSequence, Named {
      */
     @UsedByGeneratedCode
     public AnnotationClassValue(Class<T> theClass) {
+        System.out.println("theClass = " + theClass);
         this.name = theClass.getName().intern();
         this.theClass = theClass;
     }

At image compilation time I see output like from the println like:

theClass = interface reactor.core.publisher.Mono

However the output should be:

name = interface reactor.core.publisher.Mono

The constructor that takes a class reference should never be invoked because the class is not there, but somehow the incorrect constructor is being invoked for code like this:

AnnotationClassValue $micronaut_load_class_value_0() {
    try {
         // should not be invoked
         return new AnnotationClassValue(reactor.core.publisher.Mono.class);
    } catch(Throwable e) {
        // exception should cause this constructor to be invoked
         return new AnnotationClassValue("reactor.core.publisher.Mono");
    }
}

This indicates that GraalVM native image tool thinks the class is there, but in fact it is not there. When I then run the application this is resulting in false positives at runtime and breaking the application.

Could you explain why or provide a solution to why GraalVM native image is returning class references when these class references are not on the class path?

@vjovanov
Copy link
Member

When --allow-incomplete-classpath option is set we now generate classes for every class that is not on the classpath. In this case, this happens when the analysis reaches reactor.core.publisher.Mono.class and tries to load this class from HotSpot, we generate the ghost class that appears to the rest of the world similar to the original class.

Now, I am concerned with why and how this example fails at runtime? The snippet you provided should behave exactly the same on HotSpot as on SVM in case reactor.core.publisher.Mono is not present.

Is there a way to reproduce the runtime failure?

@graemerocher
Copy link
Member Author

@vjovanov The example in the issue report has been updated with steps to reproduce https://github.com/graemerocher/micronaut-graal-experiments/tree/master/fresh-graal

Simply clone it run ./build-native-image.sh then run ./fresh-graal and you will get an exception like:

20:25:03.374 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: reactor.core.publisher.Mono
java.lang.NoClassDefFoundError: reactor.core.publisher.Mono
	at java.lang.Throwable.<init>(Throwable.java:265)
	at java.lang.Error.<init>(Error.java:70)
	at java.lang.LinkageError.<init>(LinkageError.java:55)
	at java.lang.NoClassDefFoundError.<init>(NoClassDefFoundError.java:59)
	at com.oracle.svm.core.Exceptions.throwNoClassDefFoundError(Exceptions.java:43)
	at io.micronaut.reactive.reactor.converters.$PublisherToMonoConverterDefinition.getTypeArgumentsMap(Unknown Source)
	at io.micronaut.context.AbstractBeanDefinition.getTypeArguments(AbstractBeanDefinition.java:210)
	

This should never happen because there are checks in the real code for the existence of the class

@graemerocher
Copy link
Member Author

@vjovanov What appears to be happening is that code run in static initialisers that reference classes not on the class path is not executed again at runtime, this results in the app thinking the classes are there at runtime. Why are these ghost classes loadable from the class loader? Surely that would break all static initialisers that do class loading?

@vjovanov
Copy link
Member

True, this is the bug that we are hitting here; thanks for figuring it out. I am working on a fix now and I can see that wrong decisions are made during static initialization based on ghost classes. Unfortunately, I can't reproduce the issue on GraalVM running on OS X. When I follow the instructions I get:

io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.micronaut.core.convert.TypeConverter]: Multiple possible bean candidates found: [io.micronaut.http.server.netty.converters.ByteBufConverters, io.micronaut.http.server.netty.converters.ByteBufConverters]
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1345)
        at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:2237)
        at io.micronaut.context.DefaultBeanContext.getBeansOfTypeInternal(DefaultBeanContext.java:2165)
<...>
Caused by: io.micronaut.context.exceptions.NonUniqueBeanException: Multiple possible bean candidates found: [io.micronaut.http.server.netty.converters.ByteBufConverters, io.micronaut.http.server.netty.converters.ByteBufConverters]
        at io.micronaut.context.DefaultBeanContext.findConcreteCandidate(DefaultBeanContext.java:1401)
        at io.micronaut.context.DefaultApplicationContext.findConcreteCandidate(DefaultApplicationContext.java:313)
        at io.micronaut.context.DefaultBeanContext.lastChanceResolve(DefaultBeanContext.java:1893)
        at io.micronaut.context.DefaultBeanContext.findConcreteCandidateNoCache(DefaultBeanContext.java:1834)

@graemerocher
Copy link
Member Author

@vjovanov I have pushed a fix for that, sorry it was a duplication of the classes in META-INF/services

See graemerocher/micronaut-graal-experiments@b3a739a

@cstancu
Copy link
Member

cstancu commented Dec 21, 2018

@graemerocher thank you for your patience! I made some progress on this, the changes will be merged into master soon. Using mock-up classes, i.e., the so called ghost classes, to model the missing classes turned out to be problematic because they could leak into the executable, so I completely removed that idea and reimplemented our support for code that references missing types. Now the example program executes past the previous step. However, it chockes later with:

01:04:50.662 [main] DEBUG i.m.context.DefaultBeanContext - Finding candidate beans for type: interface io.micronaut.runtime.server.EmbeddedServer
01:04:50.662 [main] DEBUG i.m.context.DefaultBeanContext - Resolved bean candidates [Definition: io.micronaut.http.server.netty.NettyHttpServer] for type: interface io.micronaut.runtime.server.EmbeddedServer
01:04:50.662 [main] DEBUG i.m.context.DefaultBeanContext - Finalized bean definitions candidates: [Definition: io.micronaut.http.server.netty.NettyHttpServer]
01:04:50.662 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: BeanDefinition.requiresMethodProcessing() returned true but method has no @Executable definition. This should never happen. Please report an issue.
java.lang.IllegalStateException: BeanDefinition.requiresMethodProcessing() returned true but method has no @Executable definition. This should never happen. Please report an issue.
	at io.micronaut.context.DefaultBeanContext.lambda$null$23(DefaultBeanContext.java:1087)
	at io.micronaut.context.DefaultBeanContext$$Lambda$5e07e938b11862cb33cf7ff5206bd3f6053ad9a0.get(Unknown Source)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at io.micronaut.context.DefaultBeanContext.lambda$initializeContext$24(DefaultBeanContext.java:1086)
	at io.micronaut.context.DefaultBeanContext$$Lambda$29f6c1cea9a02eb2d58dca201e674a20329d9771.apply(Unknown Source)
	at java.util.stream.Collectors.lambda$groupingBy$45(Collectors.java:907)
	at java.util.stream.Collectors$$Lambda$340637087fa4286e0452b77c431b9ea00388bbed.accept(Unknown Source)
	at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1083)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:190)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:2033)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:156)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:138)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:67)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:271)
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:257)
	at fresh.graal.Application.main(Application.java:8)

After some debugging I figured out that the method for which this fails is io.micronaut.health.HeartbeatTask.pulsate, but my understanding of micronaut internals is limited so I am not quite sure what the error message means. Do you have any insights in what could go wrong here?

It could have something to do with annotations that reference missing types, but I am not sure. I am currently trying to understand how micronaut processes annotations, especially in the context of https://bugs.openjdk.java.net/browse/JDK-7183985 which could be triggered for example by io.micronaut.context.annotation.Requires.classes() when it references some missing classes. I came across io.micronaut.core.annotation.AnnotationMetadata and as far as I understand micronaut uses its own annotation processing so you probably don't run into JDK-7183985. In SubstrateVM we pre-process all anotations AOT using the java.lang.reflect.AnnotatedElement API and encode the results in the image. In case of JDK-7183985 I am currently catching and encoding the ArrayStoreException at image build time and re-throwing it at run-time, to match the JVM behavior, until this bug which has fixed in JDK11 is back-ported to JDK8. This works as expected in unit tests but maybe it interfers with some micronaut mechanism that I am not aware of.

@graemerocher
Copy link
Member Author

@cstancu I am going to build from master and try reproduce to get back to you.

Micronaut doesn't use the annotation API from Java directly, it computes annotation metadata at compilation time so that it can deal with merging meta annotations, referencing classes not present etc.

The most likely cause of this problem is is the code is arriving here https://github.com/micronaut-projects/micronaut-core/blob/bde43d75b1848ab14c170cf79606e3e1471e7eba/core/src/main/java/io/micronaut/core/annotation/AnnotationMetadata.java#L325

Which triggers a Class.forName call as a last resort. It shouldn't arrive there, I will check if it is for some reason.

@graemerocher
Copy link
Member Author

@cstancu I found a place where we were still doing Class.forName(..) so the issue may have simply been an incomplete reflect.json. I have pushed an update to Micronaut that removes this case micronaut-projects/micronaut-core@45881ef

And updated the example at https://github.com/graemerocher/micronaut-graal-experiments/tree/master/fresh-graal (you can do git pull to get the changes)

However I have not been able to test if the change is working with the master version of GraalVM, I guess as you said you have not yet pushed the changes.

@graemerocher
Copy link
Member Author

Related issue #851

@thomaswue
Copy link
Member

Thanks @graemerocher.

@cstancu
Copy link
Member

cstancu commented Dec 22, 2018

@graemerocher my changes are now merged (see 39b9b49) and with your latest update I was able to successfully build a working image. Accessing http://localhost:8080/ returns a Page Not Found message but that's expected since the example doesn't define any routes. So I copied the controller from https://github.com/micronaut-projects/micronaut-examples/blob/master/hello-world-java/src/main/java/example/HelloController.java and it worked as expected. I will add this to our integration tests to make sure we don't regress on it again.

@cstancu cstancu closed this as completed Dec 22, 2018
@graemerocher
Copy link
Member Author

Fantastic news!! Thanks!

@graemerocher
Copy link
Member Author

Did this change make it into rc10? I am seeing the same behaviour as in this comment
#812 (comment)

Which indicates the ghost class issue is still present?

@graemerocher
Copy link
Member Author

Ah no, I see RC10 was released earlier, so this will be in RC11. Do you have a release date yet?

@ansalond
Copy link
Member

ansalond commented Jan 8, 2019

Hi @graemerocher, we do not have an official release date, but ideally RC11 will be available at some point next week.

@graemerocher
Copy link
Member Author

Good news!

@graemerocher
Copy link
Member Author

btw we have now released 1.0.3 of Micronaut so the integration test @cstancu added can be updated to use a non-snapshot version

@cstancu
Copy link
Member

cstancu commented Jan 8, 2019

@graemerocher thanks for the heads up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants