-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use CAPI image builder for base images #30
Comments
Notes: #35 |
FINE i will do this now |
Here are some notes and thoughts about switching our Microvm OS images. See the end for my wrap-up. TLDRControl plane node Microvms with the CAPI images become ready 19.36 seconds (39.84%) faster than those based on our own. Worker node Microvms with the CAPI images become ready 4.02 seconds (30.90%) slower than our own. CAPI based images win on speed for control plane node readiness, but lose on worker node readiness. Control plane node Microvms with the CAPI images start with 2.4GB (82.76%) more disk already claimed at boot. Worker node Microvms with the CAPI images start with 3.2GB (177.78%) more disk already claimed at boot. CAPI based images lose across the board for disk hogging. Image detailsOG OS imageOur OG Microvm OS is a container image. The This image is 1.27GB. CAPI OS imageThis proposed microvm OS image uses the kubernetes image-builder, more specifically the This resulting image is ready to use with all the binaries installed and ready to go, any sys config set, and pre-pulled kubeadm images. Reading the image-builder ansible (fun times), and inspecting the image shows that the builder does not install anything else, but because it is based of a machine image, which is based off a full-on ubuntu with a kernel and everything, the image is huge. 4.24GB to be exact. Some of that can be attributed to the pre-pulled images. Some can be attributed to the kernel modules, which we can remove. With a bit of What does this mean?This size impacts in 3 ways:
On the other hand... having the Test environment and constraintsTo illustrate these notes, I got some data. I created a single
I created several clusters (not at the same time) with just 1 control plane and 1 worker node each. They used the same kernel image: I tested the difference in boot times and init disk usage for Microvm k8s nodes created with two OS images:
The I used the "Ready" message which we supply in user-data as a somewhat lazy but easily found way to determine the overall "readiness" time of a Microvm+Node. FindingsSpeedControl plane node Microvms created with our original base images were on average in a "ready" state within 48.60 seconds. Control plane node Microvms created with the CAPI-based images were on average in a "ready" state within 29.24 seconds. Control plane node Microvms with the CAPI images become ready 19.36 seconds (39.84%) faster than those based on our own. Worker node Microvms with our original base images become ready 4.02 seconds (30.90%) faster than the CAPI ones. The Microvms created with CAPI-based images boot slower in general, but start k8s faster. Disk claim at bootNote that the default Microvm disk size (as controlled by Note also that Control plane node Microvms created with the CAPI-based images report 5.3GB of used disk space at boot. This leaves just 4.1GB available for applications.
Worker node Microvms created with the CAPI-based images report 5GB of used disk space at boot. This leaves just 4.4GB available for applications.
Control plane node Microvms created with our original base images report 2.9GB of used disk space at boot. This leaves 6.5GB available for applications.
Worker node Microvms created with our original base images report 1.8GB of used disk space at boot. This leaves 7.6GB available for applications.
Control plane node Microvms with the CAPI images start with 2.4GB (82.76%) more disk already claimed at boot. Worker node Microvms with the CAPI images start with 3.2GB (177.78%) more disk already claimed at boot. What do we want from an OS image?We went into this wanting to "stay aligned with the ecosystem", which is fair, but it is more important to consider the requirements for our usecase. So what does a good OS image for Microvms on a high-performant resource-constrained environment look like:
We need to decide on numbers for the first 2, and agree to keep on top of any excess. Like 2GB size and <30s control plane start, <15s worker node start, or something. For the last one, keeping in line with the ecosystem is all well and good, but if someone comes to us saying "why is the OS taking nearly half my disk here?" do we really want to say "afraid it has to be like this because that is how all the other CAPI images look."? If we stick with the CAPI image plan here, we kind of end up with a "you can only have 2 of the 3 options at any one time" situation. Options moving forward
|
@Callisto13 - what fantastic work, bravo. This type of analysis and real quantifiable numbers is invaluable 🙇 It's strange that the control plane node ready up quicker with the CAPI but not the worker nodes. I would've expected them both to be the same (either faster or slower). I'd love to know why this was the case. When we did the original image builder work (and flintlock/capmvm) we deliberately didn't do early optimization, so even if we stick with our own image building process, we could think about optimizing it. I agree with the 3 requirement of size, speed and customisable. There is another aspect to using the CAPI images other than "stay aligned with the ecosystem" as a principal which is the CAPI images are designed for use with CAPBPK (kubeadm bootstrap) and so if changes are made to the provider that require something from the base image then we need to make sure we reflect that in our own images. The size requirement is also interesting. Having less available space to the guest isn't such a problem (unless you are on really resource constrained machines) as we can just use bigger snapshot sizes which will give more space to the apps in the microvms.....but agreed that the initial image size on the host takes up more space (and takes more time to download). My thoughts on the options:
I would like to add another option:
For me, I favour options 2, 5 and 7 (no in that order) |
Ha yes forgot to add option 7. I think that is my preference, but I am going to step away from this to now and complete the "mounting the kernel mod and bins separately" part of this image work. Then I'll circle back and think about this piece some more. |
Is liquidmetal-dev/flintlock#610 ready to be moved out of WIP @richardcase ? |
@Callisto13 - i need to do another pass through it. Will try and get that done today 🤞 |
🙏 I can get pretty far without it on the CAPMVM side with some guesswork and placeholders, but would be good to have by end of next week. I can pick up if you don't have time. |
Basically replace the method we have now and use the more standard one.
@richardcase to add any spike stuff, here or on a branch, even if untidy and unfinished.
The text was updated successfully, but these errors were encountered: