Support dynamic class loading and serialization

[ziyilin]

This patch provides supports for dynamic class loading and serialization/deserialization features for substratevm.
Overview
The basic idea for supporting dynamic class loading is dumping classes generated at run time in a beforehand run with native-image-agent to a specified directory which is added to the classpath to build native-image. Serialization/deserialization is basically dynamic class loading + reflection.
This is implemented in 3 steps:

1. Intercept java.lang.ClassLoader.postDefineClass and sun.reflect.ClassDefiner.defineClass by native-image-agent to dump the dynamically generated class to file system.
2. Make sure the generated classes' names are fixed through Hotspot version runtime and native-image version runtime.
3. Substitute relevant classes in SubstrateVM to load the previously dumped classes.

Features:

• Add new agent option -agentlib:native-image-agent=dynmaic-class-dump-dir= to specify where to dump the dynamically generated classs, the default value is "dynClass" if not specified.
• Both dynamically generated class and its stack trace are dumped.
• Dynamically generated class's name could be decided at runtime (e.g. runtime sequence number as postfix) or null when defineClass is invoked. The former is supported by using same rule to generate fixed names for both Agent runtime and native-image runtime. The latter is supported by retrieving class name from dumped class bytecode at native-image runtime.
• Support JDK serialization/deserialization which replies on dynamic class loading and reflection.

Limitations:

1. Don't support serializing proxied class. The proxied class is generated and cached in native-image at build time. The generated class name is profixed with a sequence number which could be different from Agent run and native-image build time run.
2. It is possible the jar on classpath has a different signature file from dynamically generated class and fail the check in java.lang.ClassLoader.checkCerts(String name, CodeSource cs) at native-image build time. Current solution is to delete jar's signature file before building.
3. Warning messages such as "WARNING: Method java.lang.Object.() not found." will be reported at native-image build time. Because such method has been accessed via JNI calls at serialization time to calculate serializeVersionUID and the Agent has intercepted and recorded them.

Tests

We have prepared the serial test cases in SPECjvm2008 to test this patch. Simply unzip specjvm2008-for-serial.zip, change $GRAALVM_HOME to your GraalVM home and run

./SVMBenchmarks/serial.sh -cr

The script builds and runs spec.benchmarks.serial.Main in SPECjvm2008.
"TestProxy" was excluded from the serial tests because of the 1st limitation mentioned above.

Dependency

This patch depends on PR #2322

[kristofdho]

Can this be updated? It also only supports java8 so can java11 be included? I tried to get it working locally but I don't know enough of both java & graal internals to manage setting it up myself.

[ziyilin]

Can this be updated? It also only supports java8 so can java11 be included? I tried to get it working locally but I don't know enough of both java & graal internals to manage setting it up myself.

It seems many classes have changed since I committed last time. I will update the commit. The development and testing of this PR are all based on JDK8. We didn't check if it works with JDK11.

[kristofdho]

If it helps, this is a patch of what I ended up with. However it crashes the agent at runtime, i just don't know why. https://we.tl/t-rxorTEeQs2

[ziyilin]

@kristofdho I have updated the PR to the lastest commit(fd11c68) on master branch.

[vjovanov]

I greatly support this PR and I believe it is quite important to get this right--thank you very much! This PR introduces two distinct features: (1) support for dynamic class loading and (2) serialization.

In order to get this right, I propose that we merge dynamic class loading and serialization
separately. The class loading feature can be developed as an individual feature and should not hold restrictions that currently specialize it for Java serialization. Once we merge and test this PR, we can use this feature for development of serialization.

[ziyilin]

I committed this PR as one patch because it was started to solve the serialization requirement at the very beginning, and the serial tests in SPECjvm2008 were choosen to test the correctness. It makes sense to seperate this patch into two, but may need to prepare extra tests for dynamic class loading. Should I start two new PRs for each feature or modify this one to dynamic class loading and start a new for serialization?

[kristofdho]

Biggest issue i have with this, is after adding the .class files to my jar, I get the following exception during image building:
Caused by: java.lang.IllegalAccessError: class jdk.internal.reflect.GeneratedSerializationConstructorAccessor$javax_crypto_spec_SecretKeySpec loaded by com.oracle.svm.hosted.NativeImageClassLoader @754ba872 cannot access jdk/internal/reflect superclass jdk.internal.reflect.SerializationConstructorAccessorImpl

This probably has something to do with the module system in java11 and I can't figure out how to work around that.

Edit: I seem to be able to get around it by adding -J"--patch-module=java.base=jarfile.jar" to the native-image command, jarfile containing the generated class files. But this is hardly a clean solution.

[ziyilin]

Biggest issue i have with this, is after adding the .class files to my jar, I get the following exception during image building:
Caused by: java.lang.IllegalAccessError: class jdk.internal.reflect.GeneratedSerializationConstructorAccessor$javax_crypto_spec_SecretKeySpec loaded by com.oracle.svm.hosted.NativeImageClassLoader @754ba872 cannot access jdk/internal/reflect superclass jdk.internal.reflect.SerializationConstructorAccessorImpl
This probably has something to do with the module system in java11 and I can't figure out how to work around that.
Edit: I seem to be able to get around it by adding -J"--patch-module=java.base=jarfile.jar" to the native-image command, jarfile containing the generated class files. But this is hardly a clean solution.

It does not support JDK11 yet. Not only module, but also runtime behaviors are different. Can you try on JDK 8 if possible?

[kristofdho]

My project is in java 11 so sadly no, cannot try it on 8. What I'm using locally is a modified version to include java 11 based on this. Basically add the package switching between sun.reflect and jdk.internal.reflect and add some naive basic support for nestmates as the classloader seems to depend on this. It does generate the required classes & reflection config and with the --patch-module argument it picks them up but it doesn't work.
So the agent seems to work fine on 11, but something is lacking at runtime.

It would be great if java 11 could be included in this, or if you know what has to change I can try to do it myself.

[ziyilin]

You need to check the JDK methods required for serialization in JDK11 and provide proper substitutions.
For example, com.oracle.svm.core/src/com/oracle/svm/core/jdk/Target_jdk_internal_reflect_AccessorGenerator.java deletes original jdk.internal.reflect.MethodAccessorGenerator(jdk11) and sun.reflect.MethodAccessorGenerator(jdk8) which will dynamically generate class for serialization. I have provided com.oracle.svm.core/src/com/oracle/svm/core/jdk/serialize/Target_sun_reflect_MethodAccessorGenerator.java to substitute sun.reflect.MethodAccessorGenerator to look for the class prepared in advance instead of dynamically generating. You need do the same thing for the jdk11 and maybe other relavant modifications.

[vjovanov]

I committed this PR as one patch because it was started to solve the serialization requirement at the very beginning, and the serial tests in SPECjvm2008 were choosen to test the correctness. It makes sense to seperate this patch into two, but may need to prepare extra tests for dynamic class loading. Should I start two new PRs for each feature or modify this one to dynamic class loading and start a new for serialization?

We should open start a separate PR for this and add a few basic tests. I have a few minor comments, but I can put them on that PR.

[ziyilin]

I committed this PR as one patch because it was started to solve the serialization requirement at the very beginning, and the serial tests in SPECjvm2008 were choosen to test the correctness. It makes sense to seperate this patch into two, but may need to prepare extra tests for dynamic class loading. Should I start two new PRs for each feature or modify this one to dynamic class loading and start a new for serialization?

We should open start a separate PR for this and add a few basic tests. I have a few minor comments, but I can put them on that PR.

I'll start this work next week.

[kristofdho]

You need to check the JDK methods required for serialization in JDK11 and provide proper substitutions.
For example, com.oracle.svm.core/src/com/oracle/svm/core/jdk/Target_jdk_internal_reflect_AccessorGenerator.java deletes original jdk.internal.reflect.MethodAccessorGenerator(jdk11) and sun.reflect.MethodAccessorGenerator(jdk8) which will dynamically generate class for serialization. I have provided com.oracle.svm.core/src/com/oracle/svm/core/jdk/serialize/Target_sun_reflect_MethodAccessorGenerator.java to substitute sun.reflect.MethodAccessorGenerator to look for the class prepared in advance instead of dynamically generating. You need do the same thing for the jdk11 and maybe other relavant modifications.

That is what I ment with "the package switching between sun.reflect and jdk.internal.reflect".
Everything you added for java 8, I generalised to work for both. This issue is whatever extra is required in 11 I suppose.

[ziyilin]

I committed this PR as one patch because it was started to solve the serialization requirement at the very beginning, and the serial tests in SPECjvm2008 were choosen to test the correctness. It makes sense to seperate this patch into two, but may need to prepare extra tests for dynamic class loading. Should I start two new PRs for each feature or modify this one to dynamic class loading and start a new for serialization?

We should open start a separate PR for this and add a few basic tests. I have a few minor comments, but I can put them on that PR.

Started a new PR for dynamic class loading only: #2442