[QST] GPU Memory is completely consumed in AWS-EMR

[akmalmasud96]

I am trying to run Spark-Rapids with AWS-EMR. I am facing a problem that the GPU memory is completely consumed when the processing is initiated. And there is no space left in the GPU memory to perform processing.
Attached is an image, showing this phenomenon.

The AWS-EMR version is 7.1.0.
I want to ask that how can I solve this problem ?

[akmalmasud96]

The issue was due to configuring the following parameter
"spark.plugins":"com.nvidia.spark.SQLPlugin"
Removing it, resolved the issue.

[abellina]

@akmalmasud96 just making sure I understand, do you want to run the the spark-rapids plugin? If so the spark.plugins flag is required. Thanks!

[jlowe]

The RAPIDS Accelerator consumes almost all of the GPU memory by default. It does not expect to share the GPU with another process. You can configure the RAPIDS Accelerator to consume much less GPU memory, although that often has detrimental effects on performance due to extra spilling to fit within the smaller amount of GPU memory. See the spark.rapids.memory.gpu.maxAllocFraction or spark.rapids.memory.gpu.reserve configs to limit the amount of memory used, either by setting a maximum fraction or the amount of memory to leave reserved, respectively.

[akmalmasud96]

@abellina , @jlowe ,Thanks for describing. I had figured it out. I want to ask that can we use Raft in spark-rapids for vector operations ?

[jlowe]

Apologies for the delay, I accidentally missed this. Working with RAPIDS RAFT should be possible, but details will depend on how you're planning on using it from Spark. I assume this is via a UDF, so are you planning on a Python UDF calling the RAPIDS RAFT Python APIs or something lower-level like the C++ RAFT APIs via a Java UDF that allows the data to stay in the JVM process? Python UDF will require sharing the GPU across processes, which is a bit tricky as mentioned above.