Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock due to race condition in download thread when using Bigquery Storage API #2032

Open
kien-truong opened this issue Sep 26, 2024 · 0 comments · May be fixed by #2034
Open

Deadlock due to race condition in download thread when using Bigquery Storage API #2032

kien-truong opened this issue Sep 26, 2024 · 0 comments · May be fixed by #2034
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery API.

Comments

@kien-truong
Copy link

There is a race condition in the download thread and the main thread when using Bigquery Storage API to fetch data.

for page in rowstream.pages:
if download_state.done:
return
item = page_to_item(page)
worker_queue.put(item)

finally:
# No need for a lock because reading/replacing a variable is
# defined to be an atomic operation in the Python language
# definition (enforced by the global interpreter lock).
download_state.done = True
# Shutdown all background threads, now that they should know to
# exit early.
pool.shutdown(wait=True)

When the download thread is blocked on worker_queue.put(item), if the main thread exit, causing the pool to shut down, the download thread will be stuck. This behavior prevents the program from exiting.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants