Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade v5.1.1 #91

Merged
merged 2 commits into from
Jun 17, 2024
Merged

Upgrade v5.1.1 #91

merged 2 commits into from
Jun 17, 2024

Conversation

mgoltzsche
Copy link
Owner

@mgoltzsche mgoltzsche commented May 20, 2024 •
edited
Loading

Upgrade to Podman 5.1.1 and drop CNI networking and slirp4netns in favour of netavark (statically linked) and pasta.

This is a follow-up of #87 and #90.

@mgoltzsche mgoltzsche marked this pull request as draft May 20, 2024 00:58
@mgoltzsche
Copy link
Owner Author

mgoltzsche commented May 20, 2024 •
edited
Loading

PROBLEM: Even after upgrading to podman 5.0.3, many e2e tests are still failing, particularly those trying to run podman as root which appears to use pasta and fails as follows:

Error: setting up Pasta: pasta failed with exit code 1:
Don't run as root. Changing to nobody...

This is how to reproduce the error directly (using the changes within this PR/branch):

docker run --rm --privileged mgoltzsche/podman:5.0.3-local-dev podman run alpine:3.20 echo hello

I wonder whether it should be using pasta at all when running as root?

Rootless networking seems to work, except for port-forwarding.
Though, for some reason running podman play kube as non-root user also fails.

@mgoltzsche mgoltzsche force-pushed the upgrade-v5.0 branch 2 times, most recently from 54507de to 4315ec1 Compare May 20, 2024 23:26
kb2ma added a commit to kb2ma/podman-static that referenced this pull request Jun 3, 2024
@kb2ma
Use Podman v5, including netavark
edfb423 
Code copied from mgoltzsche/podman-static PR mgoltzsche#91, with some alterations.
@mgoltzsche mgoltzsche mentioned this pull request Jun 4, 2024
Closed
@mgoltzsche mgoltzsche force-pushed the upgrade-v5.0 branch 2 times, most recently from 2059bd6 to ae522af Compare June 4, 2024 23:13
@mgoltzsche mgoltzsche changed the title Upgrade v5.0.3 Upgrade v5.1.1 Jun 4, 2024
@mgoltzsche
Copy link
Owner Author

mgoltzsche commented Jun 4, 2024 •
edited
Loading

Upgrading podman to version 5.1.1 made the rootful network connectivity test pass (where pasta failed previously when run as root).
Remaining problems:

  • The port mapping tests fail on my machine but not within the CI build (maybe it is due to my Linux kernel (6.5.0-35 vs 6.5.0-1021-azure on GHA), docker version (26.1.3 vs 24.0.9 on GHA) or something else differs in my network stack - nftables?): the podman command creates the container successfully but does not forward the port. This happens in both cases when podman is run as non-root user and also when it is run as root.
  • The podman play kube command fails to start the Pod when it is trying to mount /tmp/storage-run-1000/containers/networks/rootless-netns/run into the Pod's containers at /run because the directory /tmp/storage-run-1000/containers/networks/rootless-netns does not exist at that point. Though, using the quay.io/podman/stable:v5.0.3 image I cannot reproduce the problem, indicating the problem is due to the setup within this repository. See discussion within the upstream podman repo.

@mgoltzsche mgoltzsche force-pushed the upgrade-v5.0 branch 6 times, most recently from 08a9833 to 0312cb1 Compare June 5, 2024 23:40
@mgoltzsche mgoltzsche mentioned this pull request Jun 5, 2024
Closed
mgoltzsche
mgoltzsche commented Jun 5, 2024
View reviewed changes
test/rootless.bats
Comment on lines +63 to +75
# The rootless-netns directory is created explicitly here as a workaround to make the test pass.
# See https://github.com/containers/podman/discussions/22903#discussioncomment-9675638
# and https://github.com/containers/common/pull/2042
$DOCKER run --rm --privileged -u podman:podman \
-v "$PODMAN_ROOT_DATA_DIR:/podman/.local/share/containers/storage" \
--mount="type=bind,src=`pwd`/test/pod.yaml,dst=/pod.yaml" \
--pull=never "${PODMAN_IMAGE}" \
podman play kube /pod.yaml
sh -c '
set -ex;
mkdir -pm700 /tmp/storage-run-1000/containers/networks/rootless-netns/run;
podman play kube /pod.yaml
'
Copy link
Owner Author

@mgoltzsche mgoltzsche Jun 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the workaround made it work.
I am inclined to release it as is but I'll let it soak in another day or two in case some other idea comes to mind or the upstream fix gets released quickly.

@mattcen @mgoltzsche
feat: upgrade to podman 5.0.0 (#87)
a7fe098 
* Upgrade to podman 5
* Include netavark and pasta binaries (new podman network stack and CNI replacement)
* Makefile: Don't access image dir directly for copy out
@mgoltzsche mgoltzsche force-pushed the upgrade-v5.0 branch 3 times, most recently from de743e0 to 9b452b9 Compare June 11, 2024 22:48
@mgoltzsche mgoltzsche marked this pull request as ready for review June 11, 2024 22:50
@mgoltzsche mgoltzsche force-pushed the upgrade-v5.0 branch 2 times, most recently from e410832 to 288b008 Compare June 14, 2024 00:42
@mgoltzsche
Copy link
Owner Author

Now that I rebuilt the PR without changes really, the port-forwarding tests are failing also within the CI build as on my machine (docker 26.1.4).
Looks like a docker upgrade caused the problem since the last working build from 2 days ago logged the following environment info:

Linux fv-az1567-837 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Docker version 24.0.9, build 2936816

but the failing build from today logs:

Linux fv-az1543-982 6.5.0-1021-azure #22~22.04.1-Ubuntu SMP Tue Apr 30 16:08:18 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Docker version 26.1.3, build b72abbb

Also, the rootful port-forwarding test fails on my machine when I run the tests against the v4.9.5 build - rootless port-forwarding still works on my machine with podman 4.9.5, using the old network stack, though.

@mgoltzsche @trentapple
fix: upgrade podman 5.1.1, drop CNI support
75f5141 
* podman 5.1.1
* drop CNI networking and slirp4netns in favour of netavark + pasta
* build statically linked netavark, pasta and aardvark
* conmon 2.1.11
* crun 1.15
* crun is also shipped with the normal (non-minimal) image now.
* build using golang 1.22, use alpine 3.19 base container

Co-authored-by: Trent Apple <trentapple@users.noreply.github.com>
@mgoltzsche
Copy link
Owner Author

mgoltzsche commented Jun 16, 2024 •
edited
Loading

Turns out that the newer docker version adds an /etc/hosts to the container that makes localhost resolve to IPv6 but the port was available on IPv4 only. To make the test pass, I've let it use 127.0.0.1 now instead of localhost.

@mgoltzsche mgoltzsche merged commit 56108ae into master Jun 17, 2024
1 check passed
@mgoltzsche mgoltzsche mentioned this pull request Jun 17, 2024
Closed
@mgoltzsche mgoltzsche deleted the upgrade-v5.0 branch June 17, 2024 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mgoltzsche @mattcen