You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I attempted to run e2e tests on a Fedora s390x Pod VM image built using mkosi, and found the case TestCreatePeerPodWithLargeImage fails on the Fedora image but succeeds on an Ubuntu image built using Packer.
The root cause is there is no enough space to store the image data.
The large test image exceeds 2GB in size and is pulled into the directory "/run/kata-containers/image/layers". However, the mount point "/run" is only allocated 1.6GB of space via tmpfs:
So it run out of space when pulling an image larger than 1.6GB.
There is no such problem for Ubuntu image built using Packer, because the mount service "run-kata\x2dcontainers.mount" will mount "/run/kata-containers" to hard disk "/kata-containers".
However, this mount service was removed for mkosi. I can find the related PR and commit: #1606 c6ac908
Since "/" should be read only, should we allocate more space to "/run" for runtime data?
Based on my investigation, Fedora typically allocates 20% of physical RAM to "/run" via tmpfs by default. We can increase this allocation to 50% or more in /etc/fstab:
tmpfs /run tmpfs rw,nosuid,nodev,size=50% 0 0
Moreover, we can allocate an even larger size than physical RAM by enabling swap. However, we must prepare the swap file on the hard disk in advance.
I hope we can discuss the following options:
increase size percentage to 50% or 75%
enable swap and allocate a larger size.
keep the default size for "/run" but reduce the size of the large test image.
The text was updated successfully, but these errors were encountered:
@genjuro214mkosi generated images are more closer to what we eventually want, ie
verity based ro rootfs
All writeable stuff in memory (since memory is encrypted)
Mount encrypted storage if needed.
Now for large images we would want the VM memory to be sufficient to hold the image content and correspondingly the fs size should be adjusted as needed.
I think increasing the size of /run to 75% of mem is good option and run the tests to see if there are any isses.
Enabling swap will result in the risk of leaking sensitive data without additional protection in place and better avoided
Reducing the size of the large test image doesn't solve the real problem :-)
All writeable stuff in memory (since memory is encrypted)
I'm not necessarily against this. But I remember that we had a discussion about the security implication before adding the /run/kata-containers -> /kata-containers and Azure and SE were using encrypted disks, so effectively gave us encrypted storage for free, which is why we were comfortable with this change. Has this position changed?
All writeable stuff in memory (since memory is encrypted)
I'm not necessarily against this. But I remember that we had a discussion about the security implication before adding the /run/kata-containers -> /kata-containers and Azure and SE were using encrypted disks, so effectively gave us encrypted storage for free, which is why we were comfortable with this change. Has this position changed?
The position remains the same. If the disk is encrypted then we put the sensitive data there.
Since encrypted root disk with end user key needs quite a few manual steps (afaik), it's not safe to mount /run/kata-containers into a disk storage by default for mkosi based builds imho.
I attempted to run e2e tests on a Fedora s390x Pod VM image built using mkosi, and found the case
TestCreatePeerPodWithLargeImage
fails on the Fedora image but succeeds on an Ubuntu image built using Packer.The root cause is there is no enough space to store the image data.
The large test image exceeds
2GB
in size and is pulled into the directory "/run/kata-containers/image/layers". However, the mount point "/run" is only allocated1.6GB
of space via tmpfs:So it run out of space when pulling an image larger than
1.6GB
.There is no such problem for Ubuntu image built using Packer, because the mount service "run-kata\x2dcontainers.mount" will mount "/run/kata-containers" to hard disk "/kata-containers".
However, this mount service was removed for mkosi. I can find the related PR and commit:
#1606
c6ac908
Since "/" should be read only, should we allocate more space to "/run" for runtime data?
Based on my investigation, Fedora typically allocates 20% of physical RAM to "/run" via tmpfs by default. We can increase this allocation to
50%
or more in/etc/fstab
:Moreover, we can allocate an even larger size than physical RAM by enabling swap. However, we must prepare the swap file on the hard disk in advance.
I hope we can discuss the following options:
50%
or75%
The text was updated successfully, but these errors were encountered: