Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix a hanging issue when processing empty data. #841

Merged
merged 2 commits into from
Sep 25, 2020
Merged

Fix a hanging issue when processing empty data. #841

merged 2 commits into from
Sep 25, 2020

Conversation

firestarman
Copy link
Collaborator

@firestarman firestarman commented Sep 24, 2020 •
edited
Loading

The output iterator will wait on the batch queue when calling hasNext, and suppose to be waked up when the Python runner inserts something into the batch queue. But the insertion will never happen if the input data is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator after it finishes the data writing by calling the new added API finish().

Also added the test for it. The 'small_data' is small enough to let some tasks get no data when running.

Signed-off-by: Firestarman firestarmanllc@gmail.com

@firestarman firestarman linked an issue Sep 24, 2020 that may be closed by this pull request
[BUG] udf_cudf_test::test_with_column fails with IPC error #750
Closed
@sameerz sameerz added the bug Something isn't working label Sep 24, 2020
@firestarman
Copy link
Collaborator Author

build

revans2
revans2 previously approved these changes Sep 24, 2020
View reviewed changes
Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work finding this.

@firestarman
Copy link
Collaborator Author

build

@firestarman
Fix a hanging issue when processing empty data.
c199aaa 
The output iterator will wait on the batch queue when calling `hasNext`,
and suppose to be waked up when the Python runner inserts something into
the batch queue. But the insertion will never happen if the input data
is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator
after it finishes the data writing by calling the new added API `finish()`.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Add tests for processing empty data.
fa7d545 
The 'small_data' is small enough to let some tasks get
no data when running.

Now only test this for the Scalar type who just implements
the columnar pipeline.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman changed the title [WIP] Fix a hanging issue when processing empty data. Fix a hanging issue when processing empty data. Sep 25, 2020
@firestarman
Copy link
Collaborator Author

build

@firestarman
Copy link
Collaborator Author

@revans2 Added the test for it. Could you take another look?

@firestarman
Copy link
Collaborator Author

build

1 similar comment
@revans2
Copy link
Collaborator

revans2 commented Sep 25, 2020

build

@firestarman firestarman merged commit 779b9fa into NVIDIA:branch-0.3 Sep 25, 2020
@firestarman firestarman deleted the fix-hang-issue branch September 25, 2020 23:39
NvTimLiu pushed a commit to NvTimLiu/spark-rapids that referenced this pull request Oct 16, 2020
@firestarman @NvTimLiu
Fix a hanging issue when processing empty data. (NVIDIA#841)
7d195c3 
* Fix a hanging issue when processing empty data.

The output iterator will wait on the batch queue when calling `hasNext`,
and suppose to be waked up when the Python runner inserts something into
the batch queue. But the insertion will never happen if the input data
is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator
after it finishes the data writing by calling the new added API `finish()`.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Add tests for processing empty data.

The 'small_data' is small enough to let some tasks get
no data when running.

Now only test this for the Scalar type who just implements
the columnar pipeline.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
sperlingxx pushed a commit to sperlingxx/spark-rapids that referenced this pull request Nov 20, 2020
@firestarman @sperlingxx
Fix a hanging issue when processing empty data. (NVIDIA#841)
9557718 
* Fix a hanging issue when processing empty data.

The output iterator will wait on the batch queue when calling `hasNext`,
and suppose to be waked up when the Python runner inserts something into
the batch queue. But the insertion will never happen if the input data
is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator
after it finishes the data writing by calling the new added API `finish()`.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Add tests for processing empty data.

The 'small_data' is small enough to let some tasks get
no data when running.

Now only test this for the Scalar type who just implements
the columnar pipeline.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@firestarman
Fix a hanging issue when processing empty data. (NVIDIA#841)
7fda2ca 
* Fix a hanging issue when processing empty data.

The output iterator will wait on the batch queue when calling `hasNext`,
and suppose to be waked up when the Python runner inserts something into
the batch queue. But the insertion will never happen if the input data
is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator
after it finishes the data writing by calling the new added API `finish()`.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Add tests for processing empty data.

The 'small_data' is small enough to let some tasks get
no data when running.

Now only test this for the Scalar type who just implements
the columnar pipeline.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@firestarman
Fix a hanging issue when processing empty data. (NVIDIA#841)
851af53 
* Fix a hanging issue when processing empty data.

The output iterator will wait on the batch queue when calling `hasNext`,
and suppose to be waked up when the Python runner inserts something into
the batch queue. But the insertion will never happen if the input data
is empty. So it hangs forever.

The solution is to let the Python runner always wake up the output iterator
after it finishes the data writing by calling the new added API `finish()`.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Add tests for processing empty data.

The 'small_data' is small enough to let some tasks get
no data when running.

Now only test this for the Scalar type who just implements
the columnar pipeline.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
@nvauto
Update submodule cudf to 450d684bb377797d69432df0250fde71818752da (NV…
3b3a9ab 
…IDIA#841)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] udf_cudf_test::test_with_column fails with IPC error
3 participants
@firestarman @revans2 @sameerz