Prevent experiment hanging when worker is killed by OS [Resolves #501] #506
Conversation
Codecov Report
@@ Coverage Diff @@
## master #506 +/- ##
==========================================
- Coverage 82.63% 82.51% -0.13%
==========================================
Files 81 81
Lines 4629 4660 +31
==========================================
+ Hits 3825 3845 +20
- Misses 804 815 +11
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Looks reasonable to me.
But I'm curious, is the OOM-handling specific to Pebble, rather than to all the concurrency stuff that was added in Python 3? There's so much high-level concurrency stuff in Python 3, I was just a little surprised that another library would be necessary.
It looks like the big change is that we use a future, and can handle exceptions in child processes. But, abstractly, that's not unique to Pebble.
|
I did look into vanilla concurrent.futures. It does claim to provide the process dying functionality. But it does not allow a |
| num_failures += 1 | ||
| results.append(result) | ||
| logging.exception('Child failure') |
There was a problem hiding this comment.
I really prefer the clarity of try/except/else but NBD:
try:
result = next(iterator)
except StopIteration:
break
except Exception:
logging.exception(...)
num_failures += 1
else:
results.append(result)
num_successes += 1
Converts the MulticoreExperiment to use Pebble instead of multiprocessing. This allows us to wrap the ProcessExpired exception in the case of the operating system (or something else) killing the subprocess. This ensures the process pool can keep going and provide useful information in the logs.
thcrock
force-pushed
the
concurrent_futures
branch
from
49b5d60
to
c15d51c
Compare
Converts the MulticoreExperiment to use Pebble instead of
multiprocessing.
This allows us to wrap the ProcessExpired exception in the case of
the operating system (or something else) killing the subprocess. This
ensures the process pool can keep going and provide useful information
in the logs.