Jupyter notebook for Parameter Sweep tutorial occasional cell timeout failures on Linux/Python 3.8

[lbianchi-lbl]

[adam-a-a]

Just noting that I am seeing a similar issue in checks for PR #1244 :

[lbianchi-lbl]

Based on my testing in #1314, the failures occur on Linux with Python 3.8 up for (up to) about half of the runs:

[bknueven]

It looks like the demo uses the "ConcurrentFutures" backend:

watertap/tutorials/parameter_sweep_demo.ipynb

Line 224 in a3eb077

" \"parallel_back_end\": \"ConcurrentFutures\", # Multiprocessing, MPI, Ray available\n",

.

[bknueven]

Given that Python 3.9 seems to be working, here are the release notes for Python 3.9 about concurrent.futures: https://docs.python.org/3/whatsnew/3.9.html#concurrent-futures

[bknueven]

Based on our experience with #1211 it's almost certainly the case that the deadlock is occurring in the gather method:
https://github.com/watertap-org/watertap/blob/main/watertap/tools/parallel/concurrent_futures_parallel_manager.py#L104-L115

[bknueven]

As a hack, the multiprocessing parallel back end seems to work fine: https://github.com/watertap-org/watertap/actions/runs/8080874093/job/22078190530

[bknueven]

Given the available evidence I am leaning towards this being a bug in Python 3.8. Python 3.8 hits EOL in October 2024.

How much resources should we devote to fixing this issue, when a work-around (use MultiProcessing) is readily available? It also raises the question of the utility of having both a multiprocessing parallel back end and a concurrent.futures parallel back end. If both are included with Python, why maintain both?

[lbianchi-lbl]

Thanks for running the checks @bknueven. Personally, I'd be more than happy to just consider this a Python 3.8 limitation, and not do anything until the time when we drop support for Python 3.8 beyond excluding the notebook from being tested in Python 3.8 CI jobs.

[bknueven]

Okay, I will implement the work around for Python 3.8 so this doesn't hold up further jobs.

[lbianchi-lbl]

Just to be clear, as far as I'm concerned the "workaround" can be

if: matrix.python-version == '3.8' && matrix.os == 'linux'`
run: rm tutorials/parameter_sweep_demo.ipynb

in the GHA job where notebooks are tested, meaning I don't think we need to come up with a "proper" fix.

[bknueven]

Just to be clear, as far as I'm concerned the "workaround" can be

if: matrix.python-version == '3.8' && matrix.os == 'linux'`
run: rm tutorials/parameter_sweep_demo.ipynb

in the GHA job where notebooks are tested, meaning I don't think we need to come up with a "proper" fix.

I already had a work-around slightly more complex than this implemented (#1316) -- which should help any users of Python 3.8 not hit this same issue.