[BUG] nightly integration test failed OOM kill in JDK11 ENV

[pxLi]

Describe the bug
internal pipeline rapids_integration-dev-github #758 #762

the executor failed test cases due to lost executor.
After investigation, we found that the memory usage of testing w/ jdk11 is abnormal, and it exceeded the resource limitation of our test ENV.

all other combinations took around 45~50G memory at most,
but jdk11 one consumed more than 55G and caused the OOM kill, this seems always reproducible

Steps/Code to reproduce bug
Please provide a list of steps or a code sample to reproduce the issue.
Avoid posting private or sensitive data.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

• Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
• Spark configuration settings related to the issue

Additional context
Add any other context about the problem here.

[andygrove]

The default garbage collector is different for different JDK versions:

• JDK 8 default is Parallel.
• JDK 11 default is G1
• JDK 12+ default is Shenandoah

@pxLi Could you try setting spark.executor.extraJavaOptions="-XX:+UseParallelGC" and see if that helps?

[pxLi]

The default garbage collector is different for different JDK versions:

• JDK 8 default is Parallel.
• JDK 11 default is G1
• JDK 12+ default is Shenandoah

@pxLi Could you try setting spark.executor.extraJavaOptions="-XX:+UseParallelGC" and see if that helps?

sure, I will take a try in the CI

Just tried w/

 export PYSP_TEST_spark_driver_extraJavaOptions="-Duser.timezone=UTC -XX:+UseParallelGC"
 export PYSP_TEST_spark_executor_extraJavaOptions="-Duser.timezone=UTC -XX:+UseParallelGC"

but still failed OOM kill (FYI last night's regular run passed w/ mem footprint peak close to 55GB, but did not reach that)
I am going to limit the test parallelism (currently 6) to 4 in JDK11 nightly IT to unblock downstream tests for now

[pxLi]

no more oom reporting. closing this for now