RFC: Simpler management of VM instances and PeerPod objects

[yoheiueda]

cloud-api-adaptor (CAA) and peerpod-ctrl have some similar functionality regarding VM instance management, and we can simplify CAA by moving some of its functionality to peerpod-ctrl.

Current design

When CAA creates a cloud VM instance using a cloud API as a peer pod VM, it also creates a PeerPod CRD object with a finalizer. A PeerPod object also has a owner reference to the corresponding Pod object. PeerPod objects were introduced to avoid dangling VM instances.

CAA runs as a DaemonSet, so a CAA pod runs on each worker node. CAA manages a mapping between sandbox ID and VM instance ID, and forwards kata agent API calls from kata shim to a corresponding peer pod VM using it.

When a user deletes the Pod object, CAA receives a StopVM request from the kata shim. Then, CAA deletes a cloud VM instance using a cloud API, and it also deletes the finalizer of the PeerPod object. The PeerPod object is eventually deleted since it has owner reference to the deleted Pod object.

This mechanism of VM deletion does not work when CAA crashes or is restarted. peerpod-ctrl handles such dangling VM instances. When it detects deleted PeerPod object, it checks whether the PeerPod object has a finalizer or not. If it does not have a finalizer, it probably means that CAA did not delete the VM instance, so peerpod-ctrl deletes the VM instance on behalf of CAA.

We also need a special care when CAA restarts. The mapping information between sandbox ID and VM instance ID is stored in the memory of the CAA process, so it will be lost when CAA restarts. Kubelet fails to monitor status of the containers running in the pod when CAA restarts, and then it tries to recreate a sandbox and containers in it. In this case, the Pod object is kept and not recreated. When CAA creates a new peer pod VM, the old peer pod VM is still alive since the old PeerPod object is also still alive. peerpod-ctrl monitors PeerPod objects, and delete the stale PeerPod objects as well as old peer pod VM instances. In this current design, peer pod VMs needs to be recreated when CAA restarts.

The following table summarizes which component creates/deletes/monitors which entity. Note that both CAA and peerpod-ctrl manage VM instances.

	Created by	Deleted By	Monitored by
Pod object	User	User
PeerPod object	CAA	Kubernetes or peerpod-ctrl	peerpod-ctrl
VM instance	peerpod-ctrl	CAA or peerpod-ctrl

Proposed new design

We can simplify the design as follows. In this new design, only peerpod-ctrl manages VM instances. CAA only watches PeerPod objects, and does not directly manage VM instances. The mapping information is stored in each PeerPod object, instead of CAA's process memory.

	Created by	Deleted By	Monitored by
Pod object	User	User	peerpod-ctrl
PeerPod object	peerpod-ctrl	Kubernetes	peerpod-ctrl and CAA
VM instance	peerpod-ctrl	peerpod-ctrl

A PeerPod object has the following information.

• Owner reference to a Pod object
• VM instance ID
• VM instance IP address
• other VM information
• Sandbox ID (provided from kata shim)

The peerpod-ctrl controller manages objects and VMs as follows.

• Watches Pod objects with a specific runtimeClass value
• Watches PeerPod objects
• Creates a new peer pod VM instance as well as a PeerPod object when it detects a new Pod object
• Deletes the peer pod VM instance when it detects deletion of a PeerPod object

CAA mainly manages request forwarding from kata shim to peer pod VMs as follows.

• Updates a PeerPod object with a sandbox ID when it receives a StartVM request from kata shim
• Monitors PeerPod objects, and retrieves the mapping information between sandbox ID and VM instance ID

Advantages of the new design

• No need to recreate peer pod VMs when CAA crashes or is restarted
• No need to handle dangling peer pod VMs
• No need to import the cloud provider packages in the CAA code

Related discussions

#328 (comment)
#375 (comment)
#1517 (comment)
#1224