Fix Databricks 13.3 udf_test failures #9493
Assignees
Labels
task
Work required that improves the product but is not user facing
Comments
sameerz
added
the
task
|
This is the exception that I see when I run either of the tests above. |
jlowe
changed the title
|
I was not able to reproduce the exception listed in the description, but I did see hangs. It appears Databricks 13.3 has pulled in SPARK-44705 which changes the Python worker handling to use a single thread rather than separate reader and writer threads. That leads to hangs in UDF tests because our support assumes there are reader and writer threads. |
|
I think I am quite close to the root cause, and the fix is likely to be a big one and take two or three days. Because I have to deal with all the existing shims and prepare for the coming 4.0 which is using the new |
|
I have fixed the hangs, but there are still some failing tests. Will continue debugging. Current change is on https://github.com/firestarman/spark-rapids/tree/udf-db133 |
firestarman
added a commit
that referenced
this issue
Nov 30, 2023
fix #9493 fix #9844 The python runner uses two separate threads to write and read data with Python processes, however on DB13.3, it becomes single-threaded, which means reading and writing run on the same thread. Now the first reading is always ahead of the first writing. But the original BatchQueue will wait on the first reading until the first writing is done. Then it will wait forever. Change made: - Update the BatchQueue to support asking for a batch instead of waiting unitl one is inserted into the queue. This can eliminate the order requirement of reading and writing. - Introduce a new class named BatchProducer to work with the new BatchQueue to support rows number peek on demand for the reading. - Apply this new BatchQueue to relevant plans. - Update the Python runners to support writing one batch one time for the singled-threaded model. - Found an issue about PythonUDAF and RunningWindoFunctionExec, it may be a bug specific to DB 13.3, and add a test (test_window_aggregate_udf_on_cpu) for it. - Other small refactors --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com>
razajafri
pushed a commit
to razajafri/spark-rapids
that referenced
this issue
Jan 25, 2024
fix NVIDIA#9493 fix NVIDIA#9844 The python runner uses two separate threads to write and read data with Python processes, however on DB13.3, it becomes single-threaded, which means reading and writing run on the same thread. Now the first reading is always ahead of the first writing. But the original BatchQueue will wait on the first reading until the first writing is done. Then it will wait forever. Change made: - Update the BatchQueue to support asking for a batch instead of waiting unitl one is inserted into the queue. This can eliminate the order requirement of reading and writing. - Introduce a new class named BatchProducer to work with the new BatchQueue to support rows number peek on demand for the reading. - Apply this new BatchQueue to relevant plans. - Update the Python runners to support writing one batch one time for the singled-threaded model. - Found an issue about PythonUDAF and RunningWindoFunctionExec, it may be a bug specific to DB 13.3, and add a test (test_window_aggregate_udf_on_cpu) for it. - Other small refactors --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com>
razajafri
added a commit
that referenced
this issue
Jan 26, 2024
* Download Maven from apache.org archives (#10225) Fixes #10224 Replace broken install using apt by downloading Maven from apache.org. Signed-off-by: Gera Shegalov <gera@apache.org> * Fix a hang for Pandas UDFs on DB 13.3[databricks] (#9833) fix #9493 fix #9844 The python runner uses two separate threads to write and read data with Python processes, however on DB13.3, it becomes single-threaded, which means reading and writing run on the same thread. Now the first reading is always ahead of the first writing. But the original BatchQueue will wait on the first reading until the first writing is done. Then it will wait forever. Change made: - Update the BatchQueue to support asking for a batch instead of waiting unitl one is inserted into the queue. This can eliminate the order requirement of reading and writing. - Introduce a new class named BatchProducer to work with the new BatchQueue to support rows number peek on demand for the reading. - Apply this new BatchQueue to relevant plans. - Update the Python runners to support writing one batch one time for the singled-threaded model. - Found an issue about PythonUDAF and RunningWindoFunctionExec, it may be a bug specific to DB 13.3, and add a test (test_window_aggregate_udf_on_cpu) for it. - Other small refactors --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com> * Fix a potential data corruption for Pandas UDF (#9942) This PR moves the BatchQueue into the DataProducer to share the same lock as the output iterator returned by asIterator, and make the batch movement from the input iterator to the batch queue be an atomic operation to eliminate the race when appending the batches to the queue. * Do some refactor for the Python UDF code to try to reduce duplicate code. (#9902) Signed-off-by: Firestarman <firestarmanllc@gmail.com> * Fixed 330db Shims to Adopt the PythonRunner Changes [databricks] (#10232) This PR removes the old 330db shims in favor of the new Shims, similar to the one in 341db. **Tests:** Ran udf_test.py on Databricks 11.3 and they all passed. fixes #10228 --------- Signed-off-by: raza jafri <rjafri@nvidia.com> --------- Signed-off-by: Gera Shegalov <gera@apache.org> Signed-off-by: Firestarman <firestarmanllc@gmail.com> Signed-off-by: raza jafri <rjafri@nvidia.com> Co-authored-by: Gera Shegalov <gera@apache.org> Co-authored-by: Liangcai Li <firestarmanllc@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tasks
The text was updated successfully, but these errors were encountered: