-
-
Notifications
You must be signed in to change notification settings - Fork 335
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make it easier to farm out work #685
Comments
How big is The reason I ask: right now you can write: limiter = trio.CapacityLimiter(max_workers)
async with trio.open_nursery() as nursery:
for item in worklist:
nursery.start_soon(partial(trio.run_in_worker_thread, do_work, item, limiter=limiter)) and... I suspect this will actually work pretty well? The downside compared to your hypothetical code is that it allocates a bunch of Trio tasks in memory up front, which then sit around waiting to acquire the capacity limiter before actually putting work onto a thread. A sleeping Trio task isn't free, but it's fairly cheap – just some Python data structures that take up a bit of memory, on the order of 5-10 kilobytes per task. That's wasting some RAM compared to your hypothetical approach, and if your In the longer run, this is an example of a general problem that lots of people run into, regardless of threads: how do I run a bunch of tasks concurrently, but not too much. After #586 gets finished and we have a solid API for sending work into/out of this kind of task pool, the next thing I want to do is implement a kind of generic "concurrent |
Typically worklist is less than a million, so there wouldn't be a memory problem. I think I would still go with the longer code though, to me it feels rather verbose but not as hacky. |
Closing because I don't think there are any action items. |
I have a bunch of work items that need processing. This involves calling expensive functions that release the GIL, so I want to use worker threads. Ideally, I would like to write code like this:
This should go through
worklist
, making sure that allmax_workers
threads are busy. At the end of the while statement, all workers should have completed.Sadly, there seems to be no
something_something
that I could use for with with block, nor is there arun_in_worker_thread
that is not synchronous. So I think what I would need to write instead is something like:This is rather verbose. I was wondering if Trio might be able to offer something more concise, along the lines of the first (currently hypothetical) code?
The text was updated successfully, but these errors were encountered: