Fatal NoClassDefErrror com/fasterxml/jackson/core/util/JacksonFeature

[liza-mae]

Elasticsearch version (bin/elasticsearch --version): 7.14.0-SNAPSHOT

Plugins installed: []

JVM version (java -version): Unknown - Cloud

OS version (uname -a if on a Unix-like system): Unknown - Cloud

Description of the problem including expected versus actual behavior:
Unable to start Elasticsearch, getting a Java NoClassDefFound error

Steps to reproduce:
Start 7.14.0-SNAPSHOT on ESS (This broke today, was working yesterday)

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Create ESS Deployment 7.14.0-SNAPSHOT

Provide logs (if relevant):

[instance-0000000001] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/util/JacksonFeature
	at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:655) ~[?:?]
	at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:558) ~[?:?]
	at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:37) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:45) ~[?:?]

On-prem installation appears to have been successful.

[mark-vieira]

The issue here seems to be that the jackson-core jar is missing from the plugin bundle:

$ unzip -l plugins/repository-s3/build/distributions/repository-s3-7.14.0-SNAPSHOT.zip 
Archive:  plugins/repository-s3/build/distributions/repository-s3-7.14.0-SNAPSHOT.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     1711  2021-06-09 15:21   plugin-descriptor.properties
    69089  2021-06-09 15:21   repository-s3-7.14.0-SNAPSHOT.jar
  1048013  2020-10-12 22:56   aws-java-sdk-s3-1.11.749.jar
  1004543  2020-10-12 22:55   aws-java-sdk-core-1.11.749.jar
    27585  2020-10-12 22:56   jmespath-java-1.11.749.jar
   774640  2020-10-12 22:55   httpclient-4.5.10.jar
   328347  2020-10-12 22:55   httpcore-4.4.12.jar
    62050  2020-10-12 22:55   commons-logging-1.1.3.jar
    63731  2020-10-12 22:55   log4j-1.2-api-2.11.1.jar
   335042  2020-10-12 22:55   commons-codec-1.11.jar
  1404874  2020-10-12 22:55   jackson-databind-2.10.4.jar
    68083  2020-10-12 22:55   jackson-annotations-2.10.4.jar
   105134  2020-10-12 22:55   jaxb-api-2.2.2.jar
   135090  2021-06-09 15:21   NOTICE.txt
     1528  2021-04-29 08:05   plugin-security.policy
        0  2021-06-09 15:21   config/
      817  2020-10-19 08:53   config/log4j2.properties
    34447  2021-02-02 18:07   LICENSE.txt
---------                     -------
  5464724                     18 files

Looking at the build logic here you can see were are removing compileOnly dependencies from the bundled plugin. In this build scan you can see that :server brings in jackson-core so we are transitively removing it.

Couple of questions then:

1. Has anything changed in this plugin build logic or has this JAR always been missing from the plugin bundle?
2. Shouldn't this plugin have visibility to jackson-core in the libs directory brought in by the server jar?

@rjernst perhaps you can provide some insight on question (2). Has anything changes regarding plugin classloaders here? Why would this only happen in ESS deployments?

[mark-vieira]

Ok, so I just checked 7.13 and there's no jackson-core jar in the plugin there either. So that means for some reason the S3 plugin can't see the JAR in the distribution libs dir. Could be a) some classloader goofiness or b) the jar is missing.

[rjernst]

@liza-mae Can we get the complete log file? The ClassDefNotFound usually indicates a failure to load the class earlier in the log, often with a static init exception.

@mark-vieira jackson-core is a dependency of server, it is on the application classpath, which is why plugins do not bundled it (it would cause jarhell).

[mark-vieira]

@mark-vieira jackson-core is a dependency of server, it is on the application classpath, which is why plugins do not bundled it (it would cause jarhell).

Understood, which confirms my understanding. Thanks.

[rjernst]

Also, I see this is a 7.14 snapshot. Was that snapshot built prior to a few days ago? We just reverted back to jackson 2.10 from 2.12. I wonder if there was some issue with 2.12 there.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

We just reverted back to jackson 2.10 from 2.12. I wonder if there was some issue with 2.12 there.

Yeah, could just be a coincidence but seemed awfully suspicious. This build had to have been post 2.12 upgrade since the JacksonFeature was introduced in 2.12 and doesn't exist in 2.10. Perhaps the next build which includes the revert to 2.10 will just sort this out. Would be interesting to know what exactly went wonky here though.

[rjernst]

Ok, I have a theory. The plugins are pulled separately from the rest of Elasticsearch. Since we are talking about snapshots, this could have been a case where ES itself got a snapshot with the reversion to 2.10, but the plugins were still built against 2.12. They then try to find JacksonFeature, but can't.

[mark-vieira]

I was just going to say the same thing. That would definitely explain what's happening here. We might want to confirm offline with Cloud folks on how this stuff is orchestrated. If we aren't using the same commit for ES and plugins we are going to get all sorts of funny behavior in these test environments.

[liza-mae]

@rjernst @mark-vieira thanks, I don't any other error messages in the log, but I have attached it.

The ESS 7.14.0-SNAPSHOT that was working for me yesterday was built on June 6 and if related, should have contained PR: Upgrade SDK and Jackson and the build today June 9 should have contained the PR: Revert Upgrade SDK and Jackson

Log:
a610c36c56eb41c3b644c9036e2364af-2021.06.09.log

I agree we should talk to the Cloud team about how the staging environment is orchestrated, I cross-posted the issue in cloud slack.

[mark-vieira]

Since we're tracking this separately in the linked infra issue and we've identified this isn't actually a bug in Elasticsearch I'm going to close this issue.