Unclean package installation when virtualenv option is enabled

[mueslo]

Details

• Read the Docs project URL: https://readthedocs.org/projects/fplore/
• Build URLs:
  • https://readthedocs.org/projects/fplore/builds/8212795/
  • https://readthedocs.org/projects/fplore/builds/8213217/

Expected Result

Documentation builds fine as it did previously and as it does locally.

Actual Result

Build errors.

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/bin/sphinx-build", line 7, in <module>
    from sphinx.cmd.build import main
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/sphinx/cmd/build.py", line 25, in <module>
    from sphinx.application import Sphinx
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/sphinx/application.py", line 34, in <module>
    from sphinx.environment import BuildEnvironment
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 38, in <module>
    from sphinx.util.websupport import is_commentable
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/sphinx/util/websupport.py", line 11, in <module>
    from sphinxcontrib.websupport.utils import is_commentable  # NOQA
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/sphinxcontrib/websupport/__init__.py", line 13, in <module>
    __import__('pkg_resources').declare_namespace(__name__)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3017, in <module>
    @_call_aside
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2577, in activate
    declare_namespace(pkg)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2151, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2091, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort'

and sometimes

Using base prefix '/usr'
New python executable in /home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/bin/python3.5
Not overwriting existing python script /home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/bin/python (you must use /home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/bin/python3.5)
Installing setuptools, pip, wheel...
  Complete output from command /home/docs/checkouts...latest/bin/python3.5 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/utils/__init__.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 3018, in <module>
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 3004, in _call_aside
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 3046, in _initialize_master_working_set
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 2578, in activate
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 2152, in declare_namespace
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 2092, in _handle_ns
  File "/usr/local/lib/python3.5/dist-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py", line 2121, in _rebuild_mod_path
AttributeError: '_NamespacePath' object has no attribute 'sort'
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 2328, in <module>
    main()
  File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/docs/checkouts...latest/bin/python3.5 - setuptools pip wheel failed with error code 1

[stsewd]

Can you try wiping the environment before build? Versions -> Edit -> Wipe

[mueslo]

That potentially fixed it. Edit: this only revealed the underlying cause.

I was looking for an option like that under the admin part of the control panel, but since it wasn't there, I assumed it clears it every run.

Now I'm getting new issues which do not occur locally, but I'm not sure yet why they occur (https://readthedocs.org/projects/fplore/builds/8213711/)

Edit: I think I've figured out the origin of the new build failure. I use sphinx-gallery, which is in my docs/requirements.txt. The python /home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/bin/pip install --exists-action=w --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/fplore/.cache/pip -r docs/requirements.txt build step then installs those. sphinx-gallery has matplotlib in its requirements.txt, but without a version, so the latest 3.0.2 is installed. I explicitly specified ~=2.2 in my setup.py for Python 2 backwards compatibility. During both steps I believe packages are installed to /home/docs/checkouts/readthedocs.org/user_builds/fplore/envs/latest/lib/python3.5/site-packages/, so I assume an unclean downgrade of the matplotlib package is occurring. See also this issue.

How would I fix this elegantly? Adding matplotlib with an explicit version string to docs/requirements.txt wouldn't qualify, since this would add redundant information that's already present in my setup.py.

Edit 2: And then rerunning the build without clearing the cache gives the initial errors from this issue again (https://readthedocs.org/projects/fplore/builds/8215731/). So this does not fix it.

Edit 3: I also cannot reproduce this locally. My attempt was to first run pip install -r docs/requirements.txt, pip install seaborn (due to this) and then python setup.py --force. The docs build fine on my machine after running this setup sequence. This makes it rather hard to debug.

[mueslo]

I've created a minimum failing example based on matplotlib. To get it to fail, I had enable the following non-standard options in RTD:

• Python Interpreter: CPython 2.x -> CPython 3.x
  (otherwise matplotlib 2.2.3 is installed initially, since matplotlib 3.x is not Python 2 compatible)
• Virtualenv: Disabled -> Enabled
  (which is required for the fplore project, since otherwise dependencies aren't properly installed from the setup.py)

My dirty workaround also succeeds (explicitly specifying matplotlib~=2.2 in docs/requirements.txt). So it is definitely related to the way RTD first installs matplotlib 3.x and then 2.x.

I presume the following is happening:

the from matplotlib.cbook from within the virtualenv site package matplotlib 2.2.3 (where the matplotlib.cbook._backports module exists) does not attempt to load from itself, but rather from the previously installed matplotlib 3.0.2 (where it doesn't).

  File "/home/docs/checkouts/readthedocs.org/user_builds/rtd-min-fail/envs/latest/lib/python3.5/site-packages/matplotlib-2.2.3-py3.5-linux-x86_64.egg/mpl_toolkits/mplot3d/axes3d.py", line 32, in <module>
    from matplotlib.cbook import _backports
ImportError: cannot import name '_backports'

Edit: This erroneous import behavious is caused by enabling the virtualenv option on readthedocs. So in essence, RTD does not properly isolate virtualenv packages. When the option is disabled, project dependencies successfully overwrite previously installed dependencies. When the option is enabled, they are installed side by side.

Edit 2: matplotlib-3.0.2.dist-info, matplotlib-2.2.3-py3.5-linux-x86_64.egg, matplotlib, matplotlib-3.0.2-py3.5-nspkg.pth are all located side by side in the same site-packages directory.

[mueslo]

@stsewd I believe this would qualify as a bug, not a support request.

I've cleanly circumvented this bug for now by moving away from readthedocs online configuration to the config file, using pip instead of setup.py install, and forgoing docs/requirements.txt for extras_require in my setup.py. Then the dependency resolution is done as a whole.

.readthedocs.yml:

python:
   version: 3
   pip_install: true
   extra_requirements:
    - docs

[stsewd]

This erroneous import behavious is caused by enabling the virtualenv option on readthedocs. So in essence, RTD does not properly isolate virtualenv packages.

Do you mean the install option? I'm guessing some residuals from the previous installation (if I understand correctly you change the versions, right?). If so, you should only need to wipe the environment to resolve that, we can't do that automatically because we don't know what packages are going to be installed, if we cache the virtualenv installation to save resources.

[mueslo]

Yes, the option " Install your project inside a virtualenv using setup.py install".

No, this has nothing to do with residuals from a previous installation/build of rtd. It installs two different versions at the same time when you have a cleanly wiped environment (due to different installation methods). I gave you a minimum failing example here: https://github.com/mueslo/rtd_min_fail/tree/6f3c33990bf3e1b7e4c0e66b477fd344561e7761

I've already explained it above, but for your convenience, here it is again. docs/requirements.txt leads to matplotlib 3.0.2 being installed. setup.py leads to matplotlib 2.2.3 being installed alongside it. Import matplotlib imports matplotlib version 2.2.3. When matplotlib tries to import other parts of matplotlib, it apparently imports the 3.0.2 version.

[stsewd]

I believe this is a sphinx-galery issue, when the package is installed in rtd they install an additional package https://github.com/sphinx-gallery/sphinx-gallery/blob/29d317b1674a08fc36a4e2ad35e183cdce8f1e39/setup.py#L20-L22, that package has a different version of matplolib https://github.com/mwaskom/seaborn/blob/f0349b1a7f1e022cb1da8b94f2e019f04eeed397/setup.py#L35, I'll try to replicate this locally setting a env variable, but not sure what we can do here.

[mueslo]

Sure, that's the cause, but it shouldn't be an issue if implemented correctly, as they do not fix a version requirement for matplotlib. My approach to fixing this would be to first get all the requirements from setup.py, requirements, etc., evaluate them as a whole, and then try to install them in one coherent way. But I'm unsure how feasible that is.

[stsewd]

I don't think that's possible or something we would implement. I have a question, this problem would go away if we change the installation order? Like first calling setup.py and then installing from requeriments.txt? If so, we have that coming in our v2 config file :) #4740

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further. Thanks!