[Linux/1.5.2] A new constraint is needed for multidict in python requirements.txt

[jean-christophe-manciot]

multidict is a dependency of aiohttp & pbr. When updating python 3 packages, we encounter an issue with GNS3: aiohttp needs multidict<2, but pbr does not have such constraint, which means that if it is updated after aiohttp, we end with multidict==2.0.1, which breaks aiohttp requirement regarding multidict.

pip3 list | cut -d " " -f 1 | grep -v -f update-python-packages-exclude-list.txt | xargs -n1 pip3 install -U
...
Requirement already up-to-date: aiohttp in /usr/local/lib/python3.5/dist-packages
Requirement already up-to-date: multidict<2,>=1.2.2 in /usr/local/lib/python3.5/dist-packages (from aiohttp)
...
Requirement already up-to-date: pbr>=0.11 in /usr/local/lib/python3.5/dist-packages (from mock)
Collecting multidict
  Using cached multidict-2.0.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: multidict
  Found existing installation: multidict 1.3.0a1
    Uninstalling multidict-1.3.0a1:
      Successfully uninstalled multidict-1.3.0a1
Successfully installed multidict-2.0.1
...

gns3
...
Starting local server process with "/usr/local/bin/gns3server" --host=127.0.0.1 --port=8000 --local --controller --log="/root/.config/GNS3/gns3_server.log" --pid="/root/.config/GNS3/gns3_server.pid"
2016-08-31 16:36:05 INFO servers.py:550 Local server process has started (PID=27737)
Local server process has started (PID=27737)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 660, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (multidict 2.0.1 (/usr/local/lib/python3.5/dist-packages), Requirement.parse('multidict<2,>=1.2.2'), {'aiohttp'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/gns3server", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2976, in <module>
    @_call_aside
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2962, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2989, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 662, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 675, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 854, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'multidict<2,>=1.2.2' distribution was not found and is required by aiohttp

However, if we now downgrade multidict to 1.3.0a1, gns3-server is happy:

pip3 install multidict==1.3.0a1
Collecting multidict==1.3.0a1
  Using cached multidict-1.3.0a1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: multidict
  Found existing installation: multidict 2.0.1
    Uninstalling multidict-2.0.1:
      Successfully uninstalled multidict-2.0.1
Successfully installed multidict-1.3.0a1

gns3
...
Starting local server process with "/usr/local/bin/gns3server" --host=127.0.0.1 --port=8000 --local --controller --log="/root/.config/GNS3/gns3_server.log" --pid="/root/.config/GNS3/gns3_server.pid"
2016-08-31 16:39:10 INFO servers.py:550 Local server process has started (PID=29020)
Local server process has started (PID=29020)
2016-08-31 16:39:11 INFO __init__.py:181 VPCS module reset
VPCS module reset
2016-08-31 16:39:11 INFO __init__.py:322 Dynamips module reset
Dynamips module reset
2016-08-31 16:39:11 INFO __init__.py:262 IOU module reset
IOU module reset
2016-08-31 16:39:11 INFO __init__.py:256 QEMU module reset
QEMU module reset
2016-08-31 16:39:11 INFO __init__.py:302 VirtualBox module reset
VirtualBox module reset
2016-08-31 16:39:11 INFO __init__.py:369 VMware module reset
VMware module reset
2016-08-31 16:39:11 INFO __init__.py:193 Docker module reset
Docker module reset
2016-08-31 16:39:11 INFO __init__.py:70 Built-in module reset
Built-in module reset
2016-08-31 16:39:11 INFO topology.py:408 Topology reset
Topology reset
2016-08-31 16:39:13 INFO servers.py:575 Stopping local server (PID=29020)
Stopping local server (PID=29020)

[julien-duponchelle]

Hi,

It's not pbr which have trigger the upgrade to the version 2.0. The way you upgrade is wrong. pip3 list will give you the list of all package installed on the system including multidict. This mean you tell to pip to upgrade multidict to the last version whatever other package requirements.

When upgrading via pip you need to ask pip to upgrade the top level package like gns3-server. Unlike apt pip doesn't manage a global requirements database. That why developers use tools like virtualenv to isolate projects.

You can see that pip3 install -U gns3-server will process again the requirements for GNS3 server and will revert to the correct multidict version.

aiohttp will have support for multidict 2 soon this will avoid some problem:
aio-libs/aiohttp#1015