Debian 12 install virtualenv #85

ignis-draco · 2023-06-27T06:50:20Z

Hey,

With a fresh Debian 12, the following error occurs during the first installation:

python3 -m pip.__main__ install --upgrade virtualenv
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

The problem is due to the workaround for Ubuntu 20.04 .
This should not be used with Debian.

The text was updated successfully, but these errors were encountered:

Tronde · 2023-06-27T19:47:33Z

I confirm the issue @ignis-draco reported. It happened on two fresh Debian 12 installations here.

In case you need more information to triage/reproduce this, please describe what you need to know, and I try to gather that information.

Edit 2023-06-28T08:30+02

I would like to add some information that may help to track down the issue. First my playbook call with the config used:

$ ansible-playbook --ask-vault-pass -K setup_paperless_ngx.yml -vvv
ansible-playbook [core 2.14.6]
  config file = /path/to/ansible/ansible.cfg
  configured module search path = ['/home/tronde/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /usr/share/ansible/collections:/path/to/ansible/my_collections:/home/tronde/.ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.11.3 (main, May 24 2023, 00:00:00) [GCC 13.1.1 20230511 (Red Hat 13.1.1-2)] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True
Using /path/to/ansible/ansible.cfg as config file

The error happens in TASK [ansible-role-paperless-ngx : Install python virtualenv]:

TASK [ansible-role-paperless-ngx : Install python virtualenv] ******************
task path: /path/to/ansible/roles/ansible-role-paperless-ngx/tasks/base_dependencies/repo_packages.yml:76
<host@example.com> ESTABLISH SSH CONNECTION FOR USER: root
<host@example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' host@example.com '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<host@example.com> (0, b'/root\n', b'')
<host@example.com> ESTABLISH SSH CONNECTION FOR USER: root
<host@example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' host@example.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225 `" && echo ansible-tmp-1687933250.9170494-4652-19925889384225="` echo /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225 `" ) && sleep 0'"'"''
<host@example.com> (0, b'ansible-tmp-1687933250.9170494-4652-19925889384225=/root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225\n', b'')
Using module file /usr/lib/python3.11/site-packages/ansible/modules/pip.py
<host@example.com> PUT /home/tronde/.ansible/tmp/ansible-local-4043iridjc3x/tmp21x6mlfl TO /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/AnsiballZ_pip.py
<host@example.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' '[host@example.com]'
<host@example.com> (0, b'sftp> put /home/tronde/.ansible/tmp/ansible-local-4043iridjc3x/tmp21x6mlfl /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/AnsiballZ_pip.py\n', b'')
<host@example.com> ESTABLISH SSH CONNECTION FOR USER: root
<host@example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' host@example.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/ /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/AnsiballZ_pip.py && sleep 0'"'"''
<host@example.com> (0, b'', b'')
<host@example.com> ESTABLISH SSH CONNECTION FOR USER: root
<host@example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' -tt host@example.com '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/AnsiballZ_pip.py && sleep 0'"'"''
<host@example.com> (1, b'\r\n{"cmd": ["/usr/bin/python3", "-m", "pip.__main__", "install", "--upgrade", "virtualenv"], "failed": true, "msg": "\\n:stderr: error: externally-managed-environment\\n\\n\\u00d7 This environment is externally managed\\n\\u2570\\u2500> To install Python packages system-wide, try apt install\\n    python3-xyz, where xyz is the package you are trying to\\n    install.\\n    \\n    If you wish to install a non-Debian-packaged Python package,\\n    create a virtual environment using python3 -m venv path/to/venv.\\n    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\\n    sure you have python3-full installed.\\n    \\n    If you wish to install a non-Debian packaged Python application,\\n    it may be easiest to use pipx install xyz, which will manage a\\n    virtual environment for you. Make sure you have pipx installed.\\n    \\n    See /usr/share/doc/python3.11/README.venv for more information.\\n\\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\\nhint: See PEP 668 for the detailed specification.\\n", "invocation": {"module_args": {"name": ["virtualenv"], "extra_args": "--upgrade", "state": "present", "virtualenv_site_packages": false, "virtualenv_command": "virtualenv", "editable": false, "version": null, "requirements": null, "virtualenv": null, "virtualenv_python": null, "chdir": null, "executable": null, "umask": null}}}\r\n', b'Shared connection to host@example.com closed.\r\n')
<host@example.com> Failed to connect to the host via ssh: Shared connection to host@example.com closed.
<host@example.com> ESTABLISH SSH CONNECTION FOR USER: root
<host@example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/tronde/.ssh/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/tronde/.ansible/cp/fed8c60353"' host@example.com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1687933250.9170494-4652-19925889384225/ > /dev/null 2>&1 && sleep 0'"'"''
<host@example.com> (0, b'', b'')
fatal: [host@example.com]: FAILED! => {
    "changed": false,
    "cmd": [
        "/usr/bin/python3",
        "-m",
        "pip.__main__",
        "install",
        "--upgrade",
        "virtualenv"
    ],
    "invocation": {
        "module_args": {
            "chdir": null,
            "editable": false,
            "executable": null,
            "extra_args": "--upgrade",
            "name": [
                "virtualenv"
            ],
            "requirements": null,
            "state": "present",
            "umask": null,
            "version": null,
            "virtualenv": null,
            "virtualenv_command": "virtualenv",
            "virtualenv_python": null,
            "virtualenv_site_packages": false
        }
    },
    "msg": "\n:stderr: error: externally-managed-environment\n\n× This environment is externally managed\n╰─> To install Python packages system-wide, try apt install\n    python3-xyz, where xyz is the package you are trying to\n    install.\n    \n    If you wish to install a non-Debian-packaged Python package,\n    create a virtual environment using python3 -m venv path/to/venv.\n    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n    sure you have python3-full installed.\n    \n    If you wish to install a non-Debian packaged Python application,\n    it may be easiest to use pipx install xyz, which will manage a\n    virtual environment for you. Make sure you have pipx installed.\n    \n    See /usr/share/doc/python3.11/README.venv for more information.\n\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\nhint: See PEP 668 for the detailed specification.\n"
}

Information regarding target node:

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ python3 --version
Python 3.11.2

Tronde · 2023-06-28T08:10:43Z

I can confirm that the same playbook runs without errors on a fresh Debian 11.7 install. Here are some parameters of the target host:

# cat /etc/os-release && cat /etc/debian_version && python3 --version
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
11.7
Python 3.9.2

Tronde · 2023-06-28T08:22:05Z

During an apt-get upgrade to upgrade Debian 11.7 to Debian 12 I found this news in apt-listchanges regarding python-pip:

python-pip (23.0+dfsg-2) unstable; urgency=medium

  This version of pip introduces PEP 668 support. Debian's python3.11
  interpreter will soon (>= 3.11.2-3) declare the installation to be
  EXTERNALLY-MANAGED, instructing pip to disallow package installation outside
  virtualenvs.

  See: https://peps.python.org/pep-0668/

  Practically, this means that you can't use pip to install packages outside a
  virtualenv, on Debian's Python interpreter by default, any more.

  You can override this behaviour by passing --break-system-packages to pip
  install, but be aware that if you are running pip as root, doing so can
  break your system.

  See /usr/share/doc/python3.11/README.venv for more details.

 -- Stefano Rivera <stefanor@debian.org>  Sun, 05 Feb 2023 13:01:43 -0400

I'm not sure how that affects this role. Generally speaking I think using --break-system-packages is not a good idea as it bears the risk of damaging the whole system. But it might be OK if the task [ansible-role-paperless-ngx : Install python virtualenv] runs as {{ paperless_ngx_system_user }} without root permissions.

Tronde · 2023-06-28T08:41:46Z

FYI: Starting from a working paperless-ngx installation on Debian 11.7 I ran an upgrade to Debian 12 as described in [0]. After the upgrade succeeded paperless-ngx services cannot start anymore due to missing modules:

# systemctl status paperless-*.service --no-pager
× paperless-scheduler.service - Paperless Celery Beat
     Loaded: loaded (/etc/systemd/system/paperless-scheduler.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 10:34:18 CEST; 3min 0s ago
   Duration: 63ms
    Process: 501 ExecStart=/opt/paperless-ngx/.venv/bin/celery --app paperless beat --loglevel INFO (code=exited, status=1/FAILURE)
   Main PID: 501 (code=exited, status=1/FAILURE)
        CPU: 22ms

Jun 28 10:34:18 paperless-t2 systemd[1]: Started paperless-scheduler.servic…eat.
Jun 28 10:34:18 paperless-t2 celery[501]: Traceback (most recent call last):
Jun 28 10:34:18 paperless-t2 celery[501]:   File "/opt/paperless-ngx/.venv/…ule>
Jun 28 10:34:18 paperless-t2 celery[501]:     from celery.__main__ import main
Jun 28 10:34:18 paperless-t2 celery[501]: ModuleNotFoundError: No module na…ery'
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-scheduler.service: Main …LURE
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-scheduler.service: Faile…de'.

× paperless-webserver.service - Paperless webserver
     Loaded: loaded (/etc/systemd/system/paperless-webserver.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 10:34:18 CEST; 3min 0s ago
   Duration: 79ms
    Process: 541 ExecStart=/opt/paperless-ngx/.venv/bin/gunicorn -c /etc/paperless.d/gunicorn.conf.py paperless.asgi:application (code=exited, status=1/FAILURE)
   Main PID: 541 (code=exited, status=1/FAILURE)
        CPU: 21ms

Jun 28 10:34:18 paperless-t2 systemd[1]: Started paperless-webserver.servic…ver.
Jun 28 10:34:18 paperless-t2 gunicorn[541]: Traceback (most recent call last):
Jun 28 10:34:18 paperless-t2 gunicorn[541]:   File "/opt/paperless-ngx/.venv…le>
Jun 28 10:34:18 paperless-t2 gunicorn[541]:     from gunicorn.app.wsgiapp im…run
Jun 28 10:34:18 paperless-t2 gunicorn[541]: ModuleNotFoundError: No module n…rn'
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-webserver.service: Main …LURE
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-webserver.service: Faile…de'.

× paperless-task-queue.service - Paperless Celery Workers
     Loaded: loaded (/etc/systemd/system/paperless-task-queue.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 10:34:18 CEST; 3min 0s ago
   Duration: 65ms
    Process: 502 ExecStart=/opt/paperless-ngx/.venv/bin/celery --app paperless worker --loglevel INFO (code=exited, status=1/FAILURE)
   Main PID: 502 (code=exited, status=1/FAILURE)
        CPU: 21ms

Jun 28 10:34:18 paperless-t2 systemd[1]: Started paperless-task-queue.servi…ers.
Jun 28 10:34:18 paperless-t2 celery[502]: Traceback (most recent call last):
Jun 28 10:34:18 paperless-t2 celery[502]:   File "/opt/paperless-ngx/.venv/…ule>
Jun 28 10:34:18 paperless-t2 celery[502]:     from celery.__main__ import main
Jun 28 10:34:18 paperless-t2 celery[502]: ModuleNotFoundError: No module na…ery'
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-task-queue.service: Main…LURE
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-task-queue.service: Fail…de'.

× paperless-consumer.service - Paperless consumer
     Loaded: loaded (/etc/systemd/system/paperless-consumer.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 10:34:18 CEST; 3min 0s ago
   Duration: 56ms
    Process: 499 ExecStart=/opt/paperless-ngx/.venv/bin/python3 manage.py document_consumer (code=exited, status=1/FAILURE)
   Main PID: 499 (code=exited, status=1/FAILURE)
        CPU: 15ms

Jun 28 10:34:18 paperless-t2 systemd[1]: Started paperless-consumer.service…mer.
Jun 28 10:34:18 paperless-t2 python3[499]: Traceback (most recent call last):
Jun 28 10:34:18 paperless-t2 python3[499]:   File "/opt/paperless-ngx/src/m…ule>
Jun 28 10:34:18 paperless-t2 python3[499]:     from django.core.management …line
Jun 28 10:34:18 paperless-t2 python3[499]: ModuleNotFoundError: No module n…ngo'
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-consumer.service: Main p…LURE
Jun 28 10:34:18 paperless-t2 systemd[1]: paperless-consumer.service: Failed…de'.
Hint: Some lines were ellipsized, use -l to show in full.

The release upgrade broke the paperless-ngx installation. This, of course, is not an issue with this ansible role. I just wanna share it here to complete the picture. I guess we have to stick with Debian 11 for a while, don't we?

[0] https://wiki.debian.org/DebianUpgrade

stevenengland · 2023-07-04T19:00:35Z

Thank you for all the information, I will have a look as fast as I can. I must admit that because of vacation this actually means about in a week at best.

stevenengland · 2023-08-04T13:15:25Z

Hello @Tronde @ignis-draco , first of all sorry for the delay. Then: Would you like to check, wether this is working for you? https://github.com/paperless-ngx/ansible/tree/feat-support_debian12

Here I added functionality that I wanted to implement a long time before but for which now it seems the time has arrived :) The point is, that I wanted to assure, that the virtual environment always uses the same python version as the docker container would. In that case 3.9. This is why in this branch I introduced steps that fetch and compile the corresponding python binaries and then use that for creation of the virtual environment.

Besides of that the virtualenv installation is now explicitely done in the scope of the user (extra_args: --upgrade --user).

Tronde · 2023-08-08T19:29:29Z

Hello @stevenengland ,
I successfully tested https://github.com/paperless-ngx/ansible/tree/feat-support_debian12 and confirm that it's working.

Cheers,
Tronde

stevenengland · 2023-08-09T05:26:30Z

Thank you very much for testing

ignis-draco · 2023-08-10T12:40:17Z

Sorry for my late reply.

For a new installation under Debian 12, the changes work.
However, when upgrading from 11 to 12, the version in the .venv is not changed and thus the check
"{{ paperless_ngx_dir_python }}/bin/python{{ paperless_ngx_python_version_short }} --version" does not work.

I have now managed to upgrade my Debian11 to Debian 12. My steps were as follows:

Debian update
remove all apt package with python3.9
remove .venv
create .venv with 3.11
run playbook
remove .venv
create .venv (/opt/python/3.9/bin/python3.9 -m venv .venv)
source .venv/bin/activate
pip install -r requirements.txt
run playbook

it may well be that you could omit a few steps but that's how it worked for me.

stevenengland · 2023-08-20T17:51:35Z

Oh, I am sorry that I forget that scenario. I will consider this together with this issue: #82
Because this is kind of related. The role is not really idempotent. The intention was to save steps. But that leads to side effects like this one.

ignis-draco · 2023-08-21T11:16:42Z

I think that for this big change it would be okay to write a small howto and do it by hand.

Tronde · 2023-08-22T08:50:22Z

To do something by hand is against all automation concepts. It would improve this role a lot when this case would be handled by the role as well.

ignis-draco · 2023-08-24T08:26:22Z

@Tronde You are right, of course, but this situation only occurs when you perform a distribution upgarde, which I would always do manually.

stevenengland mentioned this issue Aug 4, 2023

feat: add support for debian 12 #89

Merged

stevenengland closed this as completed Aug 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debian 12 install virtualenv #85

Debian 12 install virtualenv #85

ignis-draco commented Jun 27, 2023

Tronde commented Jun 27, 2023 •

edited

Loading

Tronde commented Jun 28, 2023

Tronde commented Jun 28, 2023

Tronde commented Jun 28, 2023

stevenengland commented Jul 4, 2023

stevenengland commented Aug 4, 2023

Tronde commented Aug 8, 2023

stevenengland commented Aug 9, 2023

ignis-draco commented Aug 10, 2023

stevenengland commented Aug 20, 2023

ignis-draco commented Aug 21, 2023

Tronde commented Aug 22, 2023

ignis-draco commented Aug 24, 2023

Debian 12 install virtualenv #85

Debian 12 install virtualenv #85

Comments

ignis-draco commented Jun 27, 2023

Tronde commented Jun 27, 2023 • edited Loading

Edit 2023-06-28T08:30+02

Tronde commented Jun 28, 2023

Tronde commented Jun 28, 2023

Tronde commented Jun 28, 2023

stevenengland commented Jul 4, 2023

stevenengland commented Aug 4, 2023

Tronde commented Aug 8, 2023

stevenengland commented Aug 9, 2023

ignis-draco commented Aug 10, 2023

stevenengland commented Aug 20, 2023

ignis-draco commented Aug 21, 2023

Tronde commented Aug 22, 2023

ignis-draco commented Aug 24, 2023

Tronde commented Jun 27, 2023 •

edited

Loading