[BUG] test_non_empty_ctas fails on yarn

[tgravescs]

Describe the bug
06:31:10 integration_tests/src/main/python/parquet_write_test.py::test_non_empty_ctas[True][ALLOW_NON_GPU(DataWritingCommandExec,HiveTableScanExec)] FAILED [ 99%]
06:31:10 integration_tests/src/main/python/parquet_write_test.py::test_non_empty_ctas[False][ALLOW_NON_GPU(DataWritingCommandExec,HiveTableScanExec)] FAILED [100%]
06:31:10

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o106.sql.
06:31:10  E                   : java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType
06:31:10  E                   	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
06:31:10  E                   	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
06:31:10  E                   	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
06:31:10  E                   	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
06:31:10  E                   	at org.apache.****.sql.hive.client.Shim_v3_0.clazzLoadFileType$lzycompute(HiveShim.scala:1272)
06:31:10  E                   	at org.apache.****.sql.hive.client.Shim_v3_0.clazzLoadFileType(HiveShim.scala:1271)
06:31:10  E                   	at org.apache.****.sql.hive.client.Shim_v3_0.loadTable(HiveShim.scala:1355)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.$anonfun$loadTable$1(HiveClientImpl.scala:885)
06:31:10  E                   	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:276)
06:31:10  E                   	at org.apache.****.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:880)
06:31:10  E                   	at org.apache.****.sql.hive.HiveExternalCatalog.$anonfun$loadTable$1(HiveExternalCatalog.scala:884)
06:31:10  E                   	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
06:31:10  E                   	at org.apache.****.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:103)
06:31:10  E                   	at org.apache.****.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:878)
06:31:10  E                   	at org.apache.****.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:167)
06:31:10  E        

[tgravescs]

this failed again after kicking a second time.

[jlowe]

This looks like the same stacktrace as reported at apache/spark#26619 (comment). The stacktrace shows that a Hive 3.x shim is getting loaded, but it looks like a Hive 2.x jar is being used underneath it which goes about as well as one would expect.

[tgravescs]

seeing some tests on ucx egx yarn after this:

[2021-09-17T08:09:26.387Z] integration_tests/src/main/python/parquet_write_test.py::test_non_empty_ctas[True][ALLOW_NON_GPU(DataWritingCommandExec,HiveTableScanExec)] [31mFAILED[0m[31m [ 99%][0m
[2021-09-17T08:09:26.387Z] integration_tests/src/main/python/parquet_write_test.py::test_non_empty_ctas[False][ALLOW_NON_GPU(DataWritingCommandExec,HiveTableScanExec)] [31mFAILED[0m[31m [100%][0m

[2021-09-17T08:09:26.388Z] �[1m�[31mE : java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType�[0m

[tgravescs]

actually the nightly didn't push last night so reclose this and see if it fails again

[jlowe]

This is still happening.

[jlowe]

Note that the failure that is occurring happens when the test is performing a CPU-only table write. There's no GPU operations being performed, so it seems like the Spark YARN cluster is misconfigured somehow to cause this type of error.