[BUG] GC/OOM on parquet_test.py::test_small_file_memory
#9135
Assignees
Labels
bug
Something isn't working
Comments
mythrocks
added
bug
|
I should point out that this is a pre-existing test (to stress-test coalesced reads in Parquet). Literally 3 years ago: 7ac919b. |
|
I'm wondering if this message has anything to do with the problem: |
|
I've done some more digging. I think this might have to do with resetting the thread pool size or something. Here is an easy repro: // Write corpus.
spark.conf.set("spark.rapids.sql.enabled", false)
(0 to 2048).toDF.repartition(2000).write.mode("overwrite").parquet("file:///tmp/myth_parq_ints")
// Read for boom.
spark.conf.set("spark.rapids.sql.enabled", true)
spark.read.parquet("file:///tmp/myth_parq_ints").show
I'm not yet convinced that this behaviour is specific to CDH. I wonder if this could be an artifact of having a large number of threads, maybe? |
|
More experiments:
|
|
What is the executor core count set to on the cluster? Is the test running in local mode or in cluster mode? |
|
I think this is related to #9051. I suspect we were using the incorrect driver core count instead of the (I suspect larger) executor core count when setting up the multithreaded reader pool. Easiest fix is to give the executors more heap space on the cloudera cluster setup if that's possible. Long-term fix is budgeting host memory usage. This initiative is underway for off-heap memory, but this is a heap OOM. Would be interesting to get a heap dump on OOM to see what's taking up all the heap space. |
I've run it in local mode, for both CDH and Apache Spark. I have set I've captured a heap-dump on OOM. Analyzing it now. |
|
Heap dump indicates that there are 128 I see that the This is with There might be a couple of ways around the OOM:
I'm inclined to try the latter. I'll update here with results. Edit: There might be a longer term solution here, to size the |
|
We may want to revisit the 8MB heap buffer that is being used. This was copied from some existing Parquet reading code and could make sense to produce large, chunky reads, especially for cloud applications, but asking for 8MB when there are hundreds of threads is...problematic. We may want to automatically scale that buffer size based on the number of threads or simply use a smaller value for the temporary buffer. |
|
Ah, it turns out that one can't simply set One option is to set |
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this issue
Sep 1, 2023
Fixes NVIDIA#9135. (By workaround.) This change sets `spark.executor.cores` to `10`, if it is unset. This allows integration tests to work around the failure seen in `parquet_test.py:test_small_file_memory`, where the `COALESCING` Parquet reader's thread pool accidentally uses 128 threads with 8MB memory each, thus consuming the entire heap. Note that this is a bit of a workaround. A more robust solution would be to scale the Parquet reader's buffers based on the amount of available memory, and the number of threads. Signed-off-by: MithunR <mythrocks@gmail.com>
|
There is a tentative workaround to get this test to run on the large CDH nodes here: The more robust fix (to size the Parquet reader's buffers "appropriately") may be tackled later on. |
This was referenced Sep 19, 2023
pxLi
changed the title
parquet_test.py::test_small_file_memory on CDHparquet_test.py::test_small_file_memory
|
We also saw this case failed OOM constantly while testing in arm environment |
|
It stands to reason that it fails. :/ High core count, with low heap allocation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
parquet_test.py::test_small_file_memorytest runs out of memory on CDH (at least on the equivalent of Spark 3.3):The failing test takes about a minute to run on my workstation, but considerably longer on CDH, before it fails.
As indicated above, this test was run manually on CDH. As a control group, I was able to run the
window_function_test.pyon the same setup, without failures.Additionally,
parquet_test.py::test_small_file_memoryruns properly on Apache Spark 3.2.x:It appears that something is indeed up.
The text was updated successfully, but these errors were encountered: