Update payloads to using the ubuntu base image #180

fidencio · 2023-03-14T12:02:27Z

enclave-cc: Fix preInstall image tag / hostPath / mountPath

We've updated the preInstall image to use Ubuntu 20.04 instead of
CentOS, but we didn't fully reflect the changes for Enclave CC.

payloads: Update to an image using ubuntu as base

Both Enclave CC and Kata Containers have switched to using a base image
for their payloads, and we should make sure it works on our side.

As part of the change, we need to update the hostPath / mountPath to
reflect what's needed in order to be able to call systemctl restart ... using the new Ubuntu base image.

fidencio · 2023-03-14T12:02:36Z

/test

fidencio · 2023-03-14T14:41:47Z

/test

wainersm · 2023-03-14T17:41:57Z

The SEV job failed because it hit pull limit to docker.io:

13:28:22 TASK [Start a docker registry] *************************************************
13:28:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error pulling docker.io/library/registry - code: None message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: [https://www.docker.com/increase-rate-limit"}](https://www.docker.com/increase-rate-limit%22%7D)

Have you seen that problem often, @ryansavino , @UnmeshDeodhar ?

wainersm

LGTM. IIUC it is going to fix the operator that got problem by changes on kata-deploy (http://jenkins.katacontainers.io/blue/organizations/jenkins/confidential-containers-ci/detail/confidential-containers-ci/16/pipeline)

fidencio · 2023-03-14T19:32:41Z

LGTM. IIUC it is going to fix the operator that got problem by changes on kata-deploy (http://jenkins.katacontainers.io/blue/organizations/jenkins/confidential-containers-ci/detail/confidential-containers-ci/16/pipeline)

Yep, thanks @wainersm! I still need to figure out why the enclave-cc test is not passing though.

ryansavino · 2023-03-14T20:36:57Z

The SEV job failed because it hit pull limit to docker.io:

13:28:22 TASK [Start a docker registry] *************************************************
13:28:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error pulling docker.io/library/registry - code: None message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: [https://www.docker.com/increase-rate-limit"}](https://www.docker.com/increase-rate-limit%22%7D)

Have you seen that problem often, @ryansavino , @UnmeshDeodhar ?

I've retriggered the sev job. Looks like the docker hub rate limit affects the operator ci. Maybe we should consider pulling from GHCR or quay?

wainersm · 2023-03-15T14:03:14Z

The SEV job failed because it hit pull limit to docker.io:
13:28:22 TASK [Start a docker registry] *************************************************
13:28:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error pulling docker.io/library/registry - code: None message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: [https://www.docker.com/increase-rate-limit"}](https://www.docker.com/increase-rate-limit%22%7D)
Have you seen that problem often, @ryansavino , @UnmeshDeodhar ?
I've retriggered the sev job. Looks like the docker hub rate limit affects the operator ci. Maybe we should consider pulling from GHCR or quay?

I've opened an issue to track that issue: tests/e2e: hitting docker.io/library/registry pull rate limit and @UnmeshDeodhar will be working on a fix.

mythi · 2023-03-20T05:49:52Z

I debugged the enclave-cc failure and my conclusion is it will not work: kind nodes don't have dbus running so all systemctl commands are failing. Not sure why it worked with CentOS 7. It's using systemd-219 which is from 2015 so it could be systemctl was just very different then?

I managed to test a workaround but it has implications: when running without dbus, the pods deployed by the operator could run with hostPID: true to get access to PID 1. After that, SYSTEMD_IGNORE_CHROOT=1 systemctl works.

One observation while debugging this was that we restart systemd twice. I believe pre/post-install daemon could run unprivileged if it just copied containerd...

fidencio · 2023-03-20T15:21:28Z

I debugged the enclave-cc failure and my conclusion is it will not work: kind nodes don't have dbus running so all systemctl commands are failing. Not sure why it worked with CentOS 7. It's using systemd-219 which is from 2015 so it could be systemctl was just very different then?

I managed to test a workaround but it has implications: when running without dbus, the pods deployed by the operator could run with hostPID: true to get access to PID 1. After that, SYSTEMD_IGNORE_CHROOT=1 systemctl works.

One observation while debugging this was that we restart systemd twice. I believe pre/post-install daemon could run unprivileged if it just copied containerd...

Hmm. This ia bad, sincerely bad.

mythi · 2023-03-20T16:57:51Z

@fidencio I could try to ask kind to add dbus and explain our use-case (install custom runtimehandlers)

fidencio · 2023-03-25T12:38:18Z

/test

fidencio · 2023-03-27T08:54:37Z

/test

stevenhorsman · 2023-03-27T11:39:48Z

The TDX failures are do to a pre-existing error: kata-containers/kata-containers#6529

stevenhorsman

A couple of questions about versions

stevenhorsman · 2023-03-27T11:43:45Z

config/samples/ccruntime/default/kustomization.yaml

-  newName: quay.io/confidential-containers/runtime-payload
-  newTag: kata-containers-129e43d1ea5cca528b7b97234b7561219208a244-x86_64
+  newName: quay.io/confidential-containers/runtime-payload-ci
+  newTag: kata-containers-c9d6b46ea0923672a89e85a01e0d380249e3993f-x86_64


The runtime-payload tags are 13 and 14 days old now. Can we bump this to the latest and re-run the tests, or do you want to do that as a separate PR?

Hmmm. I'd prefer having this one merged and unblock possible work, but I'm fine on waiting a little bit on that and actually have the TDX fix merged on Kata Containers first.

That works fine for me - as long as we have it on the list to update this soon :)

config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml

.github/workflows/e2e.yaml

fidencio · 2023-03-27T19:13:26Z

/test

We've updated the preInstall image to use Ubuntu 20.04 instead of CentOS, but we didn't fully reflect the changes for Enclave CC. Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>

fidencio · 2023-03-28T08:42:02Z

/test

Both Enclave CC and Kata Containers have switched to using a base image for their payloads, and we should make sure it works on our side. As part of the change, we need to update the hostPath / mountPath to reflect what's needed in order to be able to call `systemctl restart ...` using the new Ubuntu base image. Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>

fidencio · 2023-03-28T09:36:19Z

/test

stevenhorsman

Thanks for bumping the runtime payload versions. Assuming the tests are passing, LGTM!

- The upstream runtime-payload is based on ubuntu now, so we need to update systemd and dbus paths to be compatible - See confidential-containers/operator#180 Fixes: confidential-containers#869 Signed-off-by: stevenhorsman <steven@uk.ibm.com>

- The upstream runtime-payload is based on ubuntu now, so we need to update systemd and dbus paths to be compatible - See confidential-containers/operator#180 Fixes: #869 Signed-off-by: stevenhorsman <steven@uk.ibm.com>

- The upstream runtime-payload is based on ubuntu now, so we need to update systemd and dbus paths to be compatible - See confidential-containers/operator#180 Fixes: confidential-containers#869 Signed-off-by: stevenhorsman <steven@uk.ibm.com>

arronwy mentioned this pull request Mar 14, 2023

tests/e2e: test changes to the pre-install-payload image #179

Merged

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from 12208e8 to b3a6dbd Compare March 14, 2023 14:41

wainersm approved these changes Mar 14, 2023

View reviewed changes

mythi mentioned this pull request Mar 20, 2023

cc-operator-daemon-install fails with upstream bundle #183

Closed

wainersm mentioned this pull request Mar 20, 2023

config: Update payloads to the latest CI image #169

Merged

fitzthum mentioned this pull request Mar 21, 2023

Add SEV-ES Test kata-containers/tests#5543

Merged

stevenhorsman mentioned this pull request Mar 22, 2023

Merge peer-pods and CoCo operator/payloads together confidential-containers/cloud-api-adaptor#561

Closed

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from eb9178a to 5d9b391 Compare March 25, 2023 12:23

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from 79e4252 to dc8d03d Compare March 27, 2023 08:53

stevenhorsman reviewed Mar 27, 2023

View reviewed changes

mythi reviewed Mar 27, 2023

View reviewed changes

.github/workflows/e2e.yaml Show resolved Hide resolved

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from dc8d03d to 3d418f4 Compare March 27, 2023 19:13

enclave-cc: Fix preInstall image tag / hostPath / mountPath

1fa442f

We've updated the preInstall image to use Ubuntu 20.04 instead of CentOS, but we didn't fully reflect the changes for Enclave CC. Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from 3d418f4 to a61ba8f Compare March 28, 2023 08:41

fidencio and others added 2 commits March 28, 2023 11:23

e2e: build custom kind node image

c0d0080

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>

fidencio force-pushed the topic/update-kata-payload-image-to-using-ubuntu-as-base branch from a61ba8f to c0d0080 Compare March 28, 2023 09:23

stevenhorsman approved these changes Mar 28, 2023

View reviewed changes

fidencio merged commit 973a9f9 into confidential-containers:main Mar 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update payloads to using the ubuntu base image #180

Update payloads to using the ubuntu base image #180

fidencio commented Mar 14, 2023

fidencio commented Mar 14, 2023

fidencio commented Mar 14, 2023

wainersm commented Mar 14, 2023

wainersm left a comment

fidencio commented Mar 14, 2023

ryansavino commented Mar 14, 2023

wainersm commented Mar 15, 2023

mythi commented Mar 20, 2023

fidencio commented Mar 20, 2023

mythi commented Mar 20, 2023

fidencio commented Mar 25, 2023

fidencio commented Mar 27, 2023

stevenhorsman commented Mar 27, 2023

stevenhorsman left a comment

stevenhorsman Mar 27, 2023

fidencio Mar 27, 2023

stevenhorsman Mar 27, 2023

fidencio commented Mar 27, 2023

fidencio commented Mar 28, 2023

fidencio commented Mar 28, 2023

stevenhorsman left a comment

Update payloads to using the ubuntu base image #180

Update payloads to using the ubuntu base image #180

Conversation

fidencio commented Mar 14, 2023

fidencio commented Mar 14, 2023

fidencio commented Mar 14, 2023

wainersm commented Mar 14, 2023

wainersm left a comment

Choose a reason for hiding this comment

fidencio commented Mar 14, 2023

ryansavino commented Mar 14, 2023

wainersm commented Mar 15, 2023

mythi commented Mar 20, 2023

fidencio commented Mar 20, 2023

mythi commented Mar 20, 2023

fidencio commented Mar 25, 2023

fidencio commented Mar 27, 2023

stevenhorsman commented Mar 27, 2023

stevenhorsman left a comment

Choose a reason for hiding this comment

stevenhorsman Mar 27, 2023

Choose a reason for hiding this comment

fidencio Mar 27, 2023

Choose a reason for hiding this comment

stevenhorsman Mar 27, 2023

Choose a reason for hiding this comment

fidencio commented Mar 27, 2023

fidencio commented Mar 28, 2023

fidencio commented Mar 28, 2023

stevenhorsman left a comment

Choose a reason for hiding this comment