[BUG] Scala 2.13 build with JDK 11 or 17 fails OpcodeSuite tests

[NVnavkumar]

From the Scala 2.13 PR #8592 (comment). There are some unresolved unit test failures in OpcodeSuite when building against JDK 11 or 17 (not with JDK 8).

 JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -Pjdk11 test -pl udf-compiler -Dbuildver=330
- java lang string builder test - append *** FAILED ***
  OpcodeSuite.this.udfIsCompiled[T](ds1) was false (OpcodeSuite.scala:52)
- string test - + concat *** FAILED ***
  OpcodeSuite.this.udfIsCompiled[T](ds1) was false (OpcodeSuite.scala:52)
...
- throw a SparkException object *** FAILED ***
  OpcodeSuite.this.udfIsCompiled[T](ds1) was false (OpcodeSuite.scala:52)
- Conditional array buffer processing *** FAILED ***
  OpcodeSuite.this.udfIsCompiled[T](ds1) was false (OpcodeSuite.scala:52)

Tests: succeeded 161, failed 4, canceled 0, ignored 0, pending 0

The underlying issue seems to be a problem with invokedynamic which starts showing up in JDK 11 and 17 builds. Should fix these so JDK 11 and JDK 17 builds can pass with Scala 2.13.

The code does run fine though with Java 11 and 17 runtime environments if built with JDK 8.

[abellina]

@seanprime7 for visibility. We may need some help here, and we'll also need to fork this part of the compiler so it can work with JDK8 and JDK11+, because we have lots of spark versions built on JDK8 we support.

[abellina]

We are looking at the bytecode produced with @seanprime7 for the four udfs. The differences I see are actually between scala 2.12 and 2.13, not between jdk versions within a scala version. (e.g. scala 2.12+jdk 8 is the same as scala 2.12+jdk17, but scala 2.13+jdk 8 is not the same as scala 2.12+jdk8).

[abellina]

@NVnavkumar I am not an expert at the 2.13 builds. Does this mean that for 2.13 the UDF compiler is being built but the tests are getting skipped? E.g. what is the state of the UDF compiler today (other than these tests are failing...)

[NVnavkumar]

The artifacts are built and tested with Scala 2.13 and JDK 8. We don't built with JDK 11 or 17, and that's what causes the failed tests (ie the Scala 2.13+JDK11/17 built artifact fails the unit tests in OpcodeSuite, but Scala2.13+JDK8 built artifact passes thoses tests)

The nightly artifacts I believe are all built with JDK 8 (correct me if I'm wrong @pxLi )

[abellina]

@NVnavkumar @pxLi would it make sense to (for now) skip the scalatests so we unblock 2.13 + JDK11/17?

I'd hate to hold that CI because of these few tests that are failing.

[pxLi]

The nightly artifacts I believe are all built with JDK 8 (correct me if I'm wrong @pxLi )

Yes, the nightly one is built against jdk8 and run the with jdk17 runtime.

would it make sense to (for now) skip the scalatests so we unblock 2.13 + JDK11/17?

currently we should only have CI in github actions to test against scala213 build against JDK11&17 (no test runs)
and also they should be optional ones, feel free to ignore those if confirm known issue

[abellina]

The status is that we'll try to get these tests fixed in the next sprint but that this is not a blocker for 23.12 release.