Additional unit tests for GeneratedInternalRowToCudfRowIterator

[jbrennan333]

Adding some additional tests to GeneratedInternalRowToCudfRowIteratorRetrySuite.
The existing GPU OOM retry tests were actually testing CPU ooms, after the CPU Host Memory retry changes went in.
Now that we have improved the forced OOM interfaces, we can fix these.
I changed the existing tests to target the correct GPU operations, and added some new tests for the Host Memory allocations.

[jbrennan333]

build

[revans2]

Just FYI, some of the tests that were added with this PR are failing in the premerge tests. It appears to be for -Dbuildver=333

[2023-12-20T22:05:23.226Z] �[31m- a split and retry when allocating dataBuffer is handled *** FAILED ***�[0m
[2023-12-20T22:05:23.227Z] �[31m  com.nvidia.spark.rapids.jni.CpuSplitAndRetryOOM: CPU OutOfMemory: could not split inputs and retry�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$AutoCloseableAttemptSpliterator.split(RmmRapidsRetryIterator.scala:460)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryIterator.next(RmmRapidsRetryIterator.scala:588)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryAutoCloseableIterator.next(RmmRapidsRetryIterator.scala:517)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.drainSingleWithVerification(RmmRapidsRetryIterator.scala:291)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.withRetryNoSplit(RmmRapidsRetryIterator.scala:132)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1(GpuColumnarToRowExec.scala:262)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1$adapted(GpuColumnarToRowExec.scala:260)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at scala.Option.foreach(Option.scala:437)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:260)�[0m
[2023-12-20T22:05:23.227Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:304)�[0m
[2023-12-20T22:05:23.227Z] �[31m  ...�[0m
[2023-12-20T22:05:24.594Z] �[31m- a split and retry when allocating offsetsBuffer is handled *** FAILED ***�[0m
[2023-12-20T22:05:24.595Z] �[31m  com.nvidia.spark.rapids.jni.CpuSplitAndRetryOOM: CPU OutOfMemory: could not split inputs and retry�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$AutoCloseableAttemptSpliterator.split(RmmRapidsRetryIterator.scala:460)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryIterator.next(RmmRapidsRetryIterator.scala:588)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryAutoCloseableIterator.next(RmmRapidsRetryIterator.scala:517)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.drainSingleWithVerification(RmmRapidsRetryIterator.scala:291)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.withRetryNoSplit(RmmRapidsRetryIterator.scala:132)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1(GpuColumnarToRowExec.scala:262)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$loadNextBatch$1$adapted(GpuColumnarToRowExec.scala:260)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at scala.Option.foreach(Option.scala:437)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:260)�[0m
[2023-12-20T22:05:24.595Z] �[31m  at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:304)�[0m
[2023-12-20T22:05:24.595Z] �[31m  ...�[0m

[jbrennan333]

build

[jbrennan333]

I am having trouble reproducing the premerge failure locally. I can step through the unit tests in the debugger, and the CpuSplitAndRetryOOM appears to be throwing on the correct allocation, but clearly in the premerge test it is hitting in an allocation in ColumnarToRowIterator.loadNextBatch. It doesn't seem to be a count issue, because it's hitting for both of these unit tests.

[jbrennan333]

I figured out what was happening here. In premerge, another test (HostAllocSuite) was calling HostAlloc.initialize(), which was setting the default allocator in cudf. When I was running this test in isolation, that wasn't happening, so the alloc that was hitting the exceptions in premerge was not using going through our host memory code.
I added a call to HostAlloc.initialize() in RmmSparkRetrySuiteBase, and verified that the other tests that derive from it still pass.

[jbrennan333]

build

[jbrennan333]

@revans2 this should be good now.

[jbrennan333]

build