Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python:3.9 - Failed run apt update from the last version of the image #837

Closed
aviadpriel opened this issue Jun 15, 2023 · 29 comments
Closed

Comments

@aviadpriel
Copy link

as titled

today morning we found that our builds failed
we are using python:3.9

when we run apt update the build failed

it looks that it fails because of the upgrade to bookworm

we get this errors

#5 [build 2/4] RUN apt-get update &&     apt-get install -y --no-install-recommends     poppler-utils
#5 sha256:fdb97fe2ebc3fcf79626c6628756f871cc0d6b0fc02ba21a2949fe33d31a2373
#5 0.240 Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
#5 0.247 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
#5 0.248 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#5 0.277 Err:1 http://deb.debian.org/debian bookworm InRelease
#5 0.277   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.303 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#5 0.303   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.329 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#5 0.329   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.330 Reading package lists...
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.337 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#5 0.337 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#5 0.337 E: Sub-process returned an error cod
@aviadpriel aviadpriel changed the title Failed run apt update from the last version of the build from image python:3.9 python:3.9 - Failed run apt update from the last version of the image Jun 15, 2023
@Cookiehook
Copy link

We're seeing the same issues on python:3.10. Worked around by using python:3.10-bullseye for now.

@aviadpriel
Copy link
Author

yes, same for python:3.9-bullseye

but we have couple of security fixes on Debian 12 that we want to get

@a1723
Copy link

a1723 commented Jun 15, 2023

The main question is: Will it be fixed in python:xx images or now we must always used images with "bullseye" postfix ?

@yosifkit
Copy link
Member

I am unable to reproduce. I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

This is similar to the update to Ubuntu focal: docker-library/mongo#606 (comment)

$ docker run -it python:3.9-bookworm bash
Unable to find image 'python:3.9-bookworm' locally
3.9-bookworm: Pulling from library/python
bba7bb10d5ba: Already exists
ec2b820b8e87: Already exists
284f2345db05: Already exists
fea23129f080: Already exists
7c62c924b8a6: Already exists
b2210932934e: Pull complete
ee9c01829d92: Pull complete
d6285f41f1b6: Pull complete
Digest: sha256:98f018a1afd67f2e17a4abd5bfe09b998734ba7c1ee54780e7ed216f8b8095c3
Status: Downloaded newer image for python:3.9-bookworm
root@0b679f272394:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 1s (7771 kB/s)
Reading package lists... Done
root@0b679f272394:/#

@InterferencePattern
Copy link

InterferencePattern commented Jun 15, 2023

I see the same for python:3.8-slim and others after the update from buster -> bookworm.
I think the image needs an update of the GPG public keys for the bookworm apt repos.

EDIT: Until this is fixed, you can use the -buster versions of images, such as python:3.8-slim-buster

@jmmclean
Copy link

hitting us as well, using python:3.7

@keever
Copy link

keever commented Jun 15, 2023

This broke our ci docker build jobs that used python:3.9-slim We used docker v24.0.2 to do the build.

@muasif80
Copy link

muasif80 commented Jun 16, 2023

It seems like the pointers for the latest image tags are moved to bookworm. The python:3.9-slim-bullseye might work instead of python:3.9-slim or maybe python:3.9-slim-buster will work too. The official 3.9 image github update shows that the version is updated from buster to bookworm.

@jacobeturpin
Copy link

Can confirm that our builds were also impacted using python:3.8-slim. Modifying to python:3.8-slim-bullseye resolved the issue, so we're temporarily using that until the issue is resolved.

@mitsuo0114
Copy link

mitsuo0114 commented Jun 19, 2023

Same here in out CI tool. I could solved with downgrading.

I'm assuming following change in Debian may have some affect to CI. (I haven't checked deeply)

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#python3-pep-668


5.2.2. Python Interpreters marked externally-managed
The Debian provided python3 interpreter packages (python3.11 and pypy3) are now marked as being externally-managed, following PEP-668. The version of python3-pip provided in Debian follows this, and ?will refuse to manually install packages on Debian's python interpreters, unless the --break-system-packages option is specified.

If you need to install a Python application (or version) that isn't packaged in Debian, we recommend that you install it with pipx (in the pipx Debian package). pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.


@aebm
Copy link

aebm commented Jun 19, 2023

Hi, that is a "problem" with the change of the debian base image used. It seems related to seccomp and it is mentioned here debuerreotype/docker-debian-artifacts#197 (comment)

Example (PS: Don't run you containers with more privileges that necessary =) )

docker run --name=dummy --rm -ti python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4697 kB/s)                         
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

vs

docker run --name=dummy --rm -ti --privileged python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4844 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (3313 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

@aebm
Copy link

aebm commented Jun 19, 2023

I can confirm that it "works" when unconfined. Don't try this at home

docker run --name=dummy --rm -ti --security-opt seccomp=unconfined python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4821 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (2989 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

@tianon
Copy link
Member

tianon commented Jun 20, 2023

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

@yosifkit
Copy link
Member

(basically the same comment as redis/docker-library-redis#365 (comment))

Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked. (Thanks for confirming @aebm)

libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

  • verify that it is libseccomp by running the Bookworm-based image with --security-opt seccomp=unconfined
  • one fix:
    • update libseccomp and docker on the host running the containers
  • one workaround:
    • switch to the *bullseye images (in the python images, these will continue to be maintained/updated until the respective Python end of life or the next Debian release, Debian Trixie)

@chazapp
Copy link

chazapp commented Jun 21, 2023

We've encountered this issue during docker builds on CircleCI runners using an old machine image (ubuntu-2004:202101-01). Updating the machine config to ubuntu-2004:current solved the issue for us.

@aporwal18
Copy link

Facing this same issue, any proper fix to this?

@yosifkit
Copy link
Member

Facing this same issue, any proper fix to this?

#837 (comment):

  • update libseccomp and docker on the host running the containers

@mke21
Copy link

mke21 commented Sep 21, 2023

I've got this problem running in gitlab-ci with the worker running on a private Kubernetes cluster. I'm not sure how to fix it apart from using the older bullseye image which obviously is a stopgap solution. This is the setup for the build:

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - name: docker:stable-dind

image: docker:stable

@sergei-lobanov
Copy link

Any plans for a normal fix?

ds283 added a commit to ds283/mps-project that referenced this issue Oct 4, 2023
- we are being bitten by this issue on the Sussex VM deployment: docker-library/python#837 (comment) (other deployments work fine, as far as I can tell)

- the issue seems to be that old versions of Docker use an old version of libseccomp, which means that some syscalls are blocked when executing linux containers on a linux host. This bites us on Sussex VM

- to fix, we need to upgrade the version of Docker installed on the ITS virtual machine, but only ITS can do this. So we will need assistance from them. Until then, we need to revert to bullseye.
@tianon
Copy link
Member

tianon commented Dec 7, 2023

Any plans for a normal fix?

#837 (comment)

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

@fgoura

This comment was marked as resolved.

@tianon

This comment was marked as resolved.

@fgoura

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests