Skip to content

Latest commit

 

History

History
303 lines (225 loc) · 13.5 KB

20190517-image-decryption.md

File metadata and controls

303 lines (225 loc) · 13.5 KB
Raw
title authors owning-sig participating-sigs reviewers approvers creation-date status
Adding the Support for Encrypted Images
@harche
sig-node
sig-architecture
smarterclayton
tallclair
yujuhong
smarterclayton
tallclair
2019-05-16
provisional

Adding the Support for Encrypted Images

Table of Contents

Summary

The underlying specification for the containers, the OCI spec, is soon going to support encrypted images. Kubernetes should be able to support decryption of these encrypted images with the addition of a new type of Secret, which we would like to call ImageDecryptSecret.

Along with OCI spec, there is an ongoing effort to enable the support for encrypted images in containerd.

OCI Spec Issue - opencontainers/image-spec#747

OCI Spec PR - opencontainers/image-spec#775

POC - https://github.com/harche/kubernetes/commit/8a0ef9898a55110d7e41f8e1a846c66cfc2fa691

POC Demo - https://youtu.be/S3FK4y5McOk

Motivation

The kubernetes worker nodes are where container images are pulled by a runtime such as containerd. If the images pulled are encrypted then containerd will have to decrypt them before running the pod. In order to be able to decrypt the images, the worker node needs to have access to corresponding the private keys.

Kubernetes Secrets are used to securely deliver sensitive data to pods in corresponding worker nodes. We need to have a secret that can be utilized before the pod is provisioned. Regular Kubernetes secret gets attached to the pod as tmpfs mount after the pod is started. However, there exists another type of kubernetes secret called ImagePullSecrets. ImagePullSecrets are used to pull the images from the private container image registry, hence they contain login credentials. ImagePullSecrets get utilized before the pod is started, this is exactly the same kind of requirement for being able to decrypt the encrypted container image. We need to have a secret that can be used before the pod is provisioned to decrypt the image. Hence, we are submitting this KEP to kubernetes community to propose a new type of kubernetes secret that is modeled after ImagePullSecret, called ImageDecryptSecret. While the ImagePullSecret holds the login credentials for the private registry, the ImageDecryptSecret will hold the private keys to decrypt encrypted images.

Goals

  • Introduce a new type of secret, ImageDecryptSecret to represent the necessary key(s) required to decrypt the contents of the image.
  • Define how ImageDecryptSecret can be used by configuration yaml(s) of the Pod (or Deployments)
  • Define how ImageDecryptSecret can be integrated into the service accounts
  • Define the Image Authorization process to prevent unauthorized access to the cached encrypted images.

Non-Goals

  • Kubernetes should be able to decrypt the images. However, in a typical workflow kubernetes has no role to play to encrypt the images. This is similar to how kubernetes plays a role in downloading and using the images instead of building and uploading container images to the registry.

User Stories

  • As a cluster user, I want to create a secret that carries the private keys required for my encrypted images
  • As a cluster user, I want to run the encrypted container images using the private keys carried by ImageDecryptSecret
  • As a cluster operator, I want to add the ImageDecryptSecret to the service account
  • As a cluster user, I want to run the encrypted images using the private keys from the service account
  • As an application developer, I want to encrypt my container images and be able to run them securely using kubernetes
  • As an application developer, I want to protect the content of my container images such that only me (as an application developer) and the execution runtime can read them. Any other third party, such as container registry, should NOT be able to read the content of my container image.

Proposal

The initial design includes:

  • ImageDecryptSecret API resource definition
  • ImageDecryptSecret pod field for specifying the ImageDecryptSecret the pod should be run with
  • Kubelet implementation for fetching & interpreting the ImageDecryptSecret
  • CRI API & implementation for passing along the ImageDecryptSecret

API

ImageDecryptSecret is a new cluster-scoped Secret

The private key is selected by the pod by specifying the ImageDecryptSecret in the PodSpec. Once the pod is scheduled, the ImageDecryptSecret cannot be changed.

(This is a simplified declaration, syntactic details will be covered in the API PR review)

// Container represents a single container that is expected to be run on the host.
type Container struct {
  <snip>
  // ImageDecryptSecrets is an optional list of references to secrets in the same namespace to use for decrypting any of the images used by this PodSpec.
  // If specified, these secrets will be passed to individual puller implementations for them to use.
  // +optional
  ImageDecryptSecrets []LocalObjectReference
  </snip>
}

Reference Implementation of the Container struct

const (
  <snip>
  // SecretTypeDecryptKeys defines the type for the decrypt secrets
  SecretTypeDecryptKey SecretType = "kubernetes.io/decryptionkey"

  // ImageDecryptionKeys represent the key required to access secret data
  ImageDecryptionKey = ".imagedecryptionkey"
  </snip>
)

Reference Implementation for the constant that hold the actual private key required for decrypting the image

In order to support ImageDecryptSecret using service account, we need to define a struct that holds the ImageDecryptSecret and the corresponding image(s) that can be decrypted.

// ImageDecryptServiceSecret holds the name of the decrypt secret and the corresponding list of images that can be decrypted using that secret.
type ImageDecryptServiceSecret struct {
	// Required.
	Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`
	// Required.
	Images []string `json:"images,omitempty" protobuf:"bytes,8,rep,name=images"`
}

See the reference implementation for ImageDecryptServiceSecret

type ServiceAccount struct {
  <snip>
  // ImageDecryptSecrets is a list of references to secrets in the same namespace to use for decrypting any encrypted images
  // in pods that reference this ServiceAccount. ImageDecryptSecrets are distinct from Secrets because Secrets
  // can be mounted in the pod, but ImageDecryptSecrets are only accessed by the kubelet.
  // +optional
  ImageDecryptSecrets []ImageDecryptServiceSecret `json:"imageDecryptSecrets,omitempty" protobuf:"bytes,5,rep,name=imageDecryptSecrets"`
  </snip>

See the reference implementation for modification to the ServiceAccount struct

An unspecified nil or empty "" ImageDecryptSecret is equivalent to the backwards-compatible default behavior as if the ImageDecryptSecret feature is disabled.

Examples

Suppose we operate a cluster that lets users create a secret of type ImageDecryptSecret.

For the kubectl command line, we propose that the user create a secret of type image-decrypt and give it a name. One or multiple private keys may then be stored under this secret.

kubectl create secret image-decrypt <secret name> --decrypt-secret=<private key>[:<password>] [--decrypt-secret=<private key>[:<password>]]

private key - base64 representation of key

password (optional) - password in cleartext if the given private key is protected with the password

For example,

kubectl create secret image-decrypt keysecret --decrypt-secret="base64OfPrivateKey":"password"

See it in action in the demo

A variant for passing the ImageEncrypSecret is the following:

kubectl create secret image-decrypt <secret name> --from-file=<path to file>:[:<password>]

path to file - The path of the private key in binary format

password (optional) - password in cleartext if the given private key is protected with the password For example,

kubectl create secret image-decrypt keysecret --from-file="/path/to/private_key":"password"

Then when a user creates a workload, they can choose the desired ImageDecryptSecret to use

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: localhost:5000/nginx:enc
    imageDecryptSecrets:
    - name: <secret name>
    imagePullPolicy: Always
    ports:
    - containerPort: 80

ImageDecryptSecrets can be added to the service account by,

kubectl patch serviceaccount <account name> -p '{"imageDecryptSecrets":[{"name":<secret name>, "images":[<image name>}]}'

or while creating a secret account,

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2015-08-07T22:02:39Z
  name: default
  namespace: default
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: 052fb0f4-3d50-11e5-b066-42010af0d7b6
secrets:
- name: default-token-uudge
imagePullSecrets:
- name: myregistrykey
imageDecryptSecrets:
- name: <secret name>
- image: [<image name>]

For example,

kubectl patch serviceaccount default -p '{"imageDecryptSecrets":[{"name":"keysecret", "images":["localhost:5000/nginx:enc"]}]}'

Image Handler

The privake keys are extracted from ImageDecryptSecret and passed to the CRI as by bundling them in DecryptParams as a part of PullImageRequest:

// DecryptParams is used to send decryption parameters to the image service of the CRI
message DecryptParams {
    repeated string privateKeyPasswds = 1;
}
message PullImageRequest {
    // Spec of the image.
    ImageSpec image = 1;
    // Authentication configuration for pulling the image.
    AuthConfig auth = 2;
    // Config of the PodSandbox, which is used to pull image in PodSandbox context.
    PodSandboxConfig sandbox_config = 3;
    // DecryptParams for the images service of the CRI
    DecryptParams dcparams = 4;
}

See the reference implementation

Just like in the case of ImagePullSecrets, CRI forwards the private keys to containerd (or CRI-O) where the actual decryption of the image will take place.

Consumption of the ImageDecryptSecrets

In the diagram https://imgur.com/a/eVkUF9w, when the kubelet wants to create a new pod which has encrypted images it has to first retrieve the decryption keys referenced decryption keys (They can be referenced directly in the pod or deployment yaml or in the pod’s service account).

Kubelet sends the request to pull the image along with the decryption keys to CRI which then forwards it to containerd. Containerd looks up for the image in the snapshotter used by CRI, if the image doesn’t exist then it downloads it and decrypts it using the decryption keys that were passed by CRI.

In case containerd finds the image in the snapshotter cache which was already pulled and decrypted by earlier request, it performs a check we call ‘Image Authorization’. This check only attempts to unwrap the keys in the image manifest of the image. In simple terms it means everytime you want to use an encrypted image, you will have to prove that you have the necessary keys to decrypt it even if the actual image is present in the decrypted form in the snapshotter. This prevents an attack where a user, without having decryption keys, might get access to encrypted image content if their pod gets scheduled on a worker node where that particular encrypted image was already pulled and decrypted by earlier request.

There is no change required in CRI RuntimeService or Kubernetes

Graduation Criteria

Alpha:

  • [] Everything described in the current proposal:
    • [] Introduce the ImageDecryptSecret API resource
    • [] Add a ImageDecryptSecret field to the PodSpec
    • [] Add a ImageDecryptSecret field to the CRI PullImageRequest
    • [] Plumb through the ImageHandler in the Kubelet
  • [] ImageDecryptSecret support in CRI-Containerd.
    • [] An error is reported when the private key is invalid or is unknown or unsupported
  • [] Testing
    • [] Kubernetes E2E tests (only validating single image handler cases)

Beta:

  • [] ImageDecryptSecret support in CRI-O deprecated.
  • [] Comprehensive test coverage

Implementation History