Remove upper limit for tornado dependency by replacing it with asyncio

[simonpintarelli]

The current jupyter-notebook (version 6.0.0) is incompatible with AiiDA 1.0.0x.

pip install -e ".[notebook]" yields the following errors:

ERROR: notebook 6.0.0 has requirement pyzmq>=17, but you'll have pyzmq 16.0.4 which is incompatible.
ERROR: notebook 6.0.0 has requirement tornado>=5.0, but you'll have tornado 4.5.3 which is incompatible.
ERROR: jupyter-console 6.0.0 has requirement prompt-toolkit<2.1.0,>=2.0.0, but you'll have prompt-toolkit 1.0.16 which is incompatible.

plumpy is incompatible with tornado 5 (the stack_context module has been removed).

[sphuber]

It is mostly our dependency on the circus library, which is our daemonizer, that causes the problem, since it breaks for tornado>=5.0.0 and pyzmq>=17. They released 0.15.0 to explicitly set an upper limit but development has been stagnant and it is not sure when they will make a release that is compatible. In the meantime, we do not have the manpower to switch to another library unfortunately

[zooks97]

The following version stack is the latest that I've been able to get working with aiida 1.0.x

However, this does not work with python > 3.7.x

jupyter core     : 4.6.1
jupyter-notebook : 5.4.1
qtconsole        : 4.5.5
ipython          : 5.8.0
ipykernel        : 5.1.3
jupyter client   : 5.3.4
jupyter lab      : 1.1.4
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 4.4.0
traitlets        : 4.3.3

[dev-zero]

for the record: supervisord does not natively support raising number of workers at runtime without re-reading a configuration file. Someone created an extension to the XML-RPC interface, but it is unmaintained. So, the options would be: either generate files and do configuration re-loads or revive the linked supervisord extension.

[sphuber]

I have started addressing the problems in circus to solve this dependency block. While fixing the problem I noticed their CI builds were broken which I first had to fix. There is an open PR for this. On top of that I managed to lift the restriction of pyzmq that circus had and updated their code. Also this has an open PR. If these get merged, we can maybe get a release done, which will fix this problem. Note that it will then still require tornado<5.0 which is bound to become a problem in the future. Updating circus to newer versions of tornado is more complicated though, so it will have to be addressed in the future, but this initial fix will give us some breathing room.

[ltalirz]

Just to mention that the pyzmq issue has been addressed, and @unkcpz is working on upgrading both plumpy and circus to work with asyncio (or a later tornado version).

Just ran into a similar issue again today (bokeh 2.0 requires a later tornado version) - very much looking forward to this!