Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add webhook certificate handling for k8s #114

Merged
merged 1 commit into from
May 7, 2021
+95 −4
Merged

Add webhook certificate handling for k8s #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ IMAGE_TAG?=nfvpe/$(APP_NAME):latest
MAIN_PKG=cmd/manager/main.go
export NAMESPACE?=openshift-sriov-network-operator
export WATCH_NAMESPACE?=openshift-sriov-network-operator
export ENABLE_ADMISSION_CONTROLLER?=true
export GOFLAGS+=-mod=vendor
export GO111MODULE=on
PKGS=$(shell go list ./... | grep -v -E '/vendor/|/test|/examples')
Expand Down Expand Up @@ -189,11 +188,12 @@ bundle: manifests
bundle-build:
docker build -f bundle.Dockerfile -t $(BUNDLE_IMG) .

deploy-setup: export ENABLE_ADMISSION_CONTROLLER?=true
deploy-setup: skopeo install
hack/deploy-setup.sh $(NAMESPACE)

deploy-setup-k8s: export NAMESPACE=sriov-network-operator
deploy-setup-k8s: export ENABLE_ADMISSION_CONTROLLER=false
deploy-setup-k8s: export ENABLE_ADMISSION_CONTROLLER?=false
deploy-setup-k8s: export CNI_BIN_PATH=/opt/cni/bin
deploy-setup-k8s: export OPERATOR_EXEC=kubectl
deploy-setup-k8s: export CLUSTER_TYPE=kubernetes
Expand Down
2 changes: 2 additions & 0 deletions bindata/manifests/operator-webhook/001-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ metadata:
name: operator-webhook-service
namespace: {{.Namespace}}
annotations:
{{- if eq .ClusterType "openshift" }}
service.alpha.openshift.io/serving-cert-secret-name: operator-webhook-service
{{- end }}
spec:
ports:
- port: 443
Expand Down
14 changes: 14 additions & 0 deletions bindata/manifests/operator-webhook/003-webhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@ metadata:
name: {{.SRIOVMutatingWebhookName}}
namespace: {{.Namespace}}
annotations:
{{- if eq .ClusterType "openshift" }}
service.beta.openshift.io/inject-cabundle: "true"
{{- else if and (not .CaBundle) (eq .ClusterType "kubernetes") }}
cert-manager.io/inject-ca-from: {{.Namespace}}/operator-webhook-service
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the docs on cert manager and I am wondering, why not use this?

cert-manager.io/inject-ca-from-secret

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That requires creating the secrets manually with a specific annotation which would be an actual extra step on all the procedure. It's is primarily used when bootstrapping cert-manager itself. After that, cert-manager creates the secrets from certificates for you so all you can directly annotate the certificate and save that extra step.

{{- end }}
webhooks:
- name: operator-webhook.sriovnetwork.openshift.io
failurePolicy: Fail
Expand All @@ -14,6 +18,9 @@ webhooks:
name: operator-webhook-service
namespace: {{.Namespace}}
path: "/mutating-custom-resource"
{{- if and (.CaBundle) (eq .ClusterType "kubernetes") }}
caBundle: "{{.CaBundle}}"
{{- end}}
rules:
- operations: [ "CREATE", "UPDATE" ]
apiGroups: ["sriovnetwork.openshift.io"]
Expand All @@ -27,7 +34,11 @@ metadata:
name: {{.SRIOVMutatingWebhookName}}
namespace: {{.Namespace}}
annotations:
{{- if eq .ClusterType "openshift" }}
service.beta.openshift.io/inject-cabundle: "true"
{{- else if and (not .CaBundle) (eq .ClusterType "kubernetes") }}
cert-manager.io/inject-ca-from: {{.Namespace}}/operator-webhook-service
{{- end }}
webhooks:
- name: operator-webhook.sriovnetwork.openshift.io
failurePolicy: Fail
Expand All @@ -36,6 +47,9 @@ webhooks:
name: operator-webhook-service
namespace: {{.Namespace}}
path: "/validating-custom-resource"
{{- if and (.CaBundle) (eq .ClusterType "kubernetes") }}
caBundle: "{{.CaBundle}}"
{{- end }}
rules:
- operations: [ "CREATE", "UPDATE", "DELETE" ]
apiGroups: ["sriovnetwork.openshift.io"]
Expand Down
2 changes: 2 additions & 0 deletions bindata/manifests/webhook/001-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ metadata:
name: network-resources-injector-service
namespace: {{.Namespace}}
annotations:
{{- if eq .ClusterType "openshift" }}
service.alpha.openshift.io/serving-cert-secret-name: network-resources-injector-secret
{{- end }}
spec:
ports:
- port: 443
Expand Down
7 changes: 7 additions & 0 deletions bindata/manifests/webhook/003-webhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,21 @@ metadata:
name: {{.SRIOVMutatingWebhookName}}
namespace: {{.Namespace}}
annotations:
{{- if eq .ClusterType "openshift" }}
service.beta.openshift.io/inject-cabundle: "true"
{{- else if and (not .CaBundle) (eq .ClusterType "kubernetes") }}
cert-manager.io/inject-ca-from: {{.Namespace}}/network-resources-injector-secret
{{- end }}
webhooks:
- name: network-resources-injector-config.k8s.io
clientConfig:
service:
name: network-resources-injector-service
namespace: {{.Namespace}}
path: "/mutate"
{{- if and (.CaBundle) (eq .ClusterType "kubernetes") }}
caBundle: "{{.CaBundle}}"
{{- end }}
rules:
- operations: [ "CREATE" ]
apiGroups: ["apps", ""]
Expand Down
2 changes: 2 additions & 0 deletions controllers/sriovoperatorconfig_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,8 @@ func (r *SriovOperatorConfigReconciler) syncWebhookObjs(dc *sriovnetworkv1.Sriov
data.Data["NetworkResourcesInjectorImage"] = os.Getenv("NETWORK_RESOURCES_INJECTOR_IMAGE")
data.Data["SriovNetworkWebhookImage"] = os.Getenv("SRIOV_NETWORK_WEBHOOK_IMAGE")
data.Data["ReleaseVersion"] = os.Getenv("RELEASEVERSION")
data.Data["ClusterType"] = utils.ClusterType
data.Data["CaBundle"] = os.Getenv("WEBHOOK_CA_BUNDLE")
objs, err := render.RenderDir(path, &data)
if err != nil {
logger.Error(err, "Fail to render webhook manifests")
Expand Down
2 changes: 2 additions & 0 deletions deploy/operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,5 @@ spec:
value: $CNI_BIN_PATH
- name: CLUSTER_TYPE
value: $CLUSTER_TYPE
- name: WEBHOOK_CA_BUNDLE
value: "$WEBHOOK_CA_BUNDLE"
65 changes: 63 additions & 2 deletions doc/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,72 @@ make deploy-setup

If you are running a Kubernetes cluster:
```bash
export OPERATOR_EXEC=kubectl
export ENABLE_ADMISSION_CONTROLLER=false
make deploy-setup-k8s
```

Webhooks are disabled when deploying on a Kubernetes cluster as per the instructions above. To enable webhooks on Kubernetes cluster, there are two options:

1. Create certificates for each of the two webhooks using a single CA whose cert you provide through an environment variable.

For example, given `cacert.pem`, `key.pem` and `cert.pem`:
```bash
kubectl create ns sriov-network-operator
kubectl -n sriov-network-operator create secret tls operator-webhook-service --cert=cert.pem --key=key.pem
jcaamano marked this conversation as resolved.
Show resolved Hide resolved
kubectl -n sriov-network-operator create secret tls network-resources-injector-secret --cert=cert.pem --key=key.pem
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be able to inject the secret, do we need this annotation in the secret?:

cert-manager.io/allow-direct-injection: "true"

injection of secrets Ref

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this option is assuming the users are creating the certificates manually and cert-manager is not being used.

export ENABLE_ADMISSION_CONTROLLER=true
export WEBHOOK_CA_BUNDLE=$(base64 -w 0 < cacert.pem)
make deploy-setup-k8s
```

2. Using https://cert-manager.io/, deploy it as:
```bash
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.0/cert-manager.yaml
```

Define the appropriate Issuer and Certificates, as an example:
```bash
kubectl create ns sriov-network-operator
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: sriov-network-operator-selfsigned-issuer
namespace: sriov-network-operator
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: operator-webhook-service
namespace: sriov-network-operator
spec:
secretName: operator-webhook-service
dnsNames:
- operator-webhook-service.sriov-network-operator.svc
issuerRef:
name: sriov-network-operator-selfsigned-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: network-resources-injector-service
namespace: sriov-network-operator
spec:
secretName: network-resources-injector-secret
dnsNames:
- network-resources-injector-service.sriov-network-operator.svc
issuerRef:
name: sriov-network-operator-selfsigned-issuer
EOF
```

And then deploy the operator:
```bash
export ENABLE_ADMISSION_CONTROLLER=true
make deploy-setup-k8s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  Warning  FailedMount  40s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[tls network-resources-injector-sa-token-zwls5]: timed out waiting for the condition
  Warning  FailedMount  35s (x9 over 2m43s)  kubelet            MountVolume.SetUp failed for volume "tls" : secret "network-resources-injector-secret" not found

Got the following error when tried this method on a fresh k8 cluster. I made sure WEBHOOK_CA_BUNDLEenv var wasnt exported.

Copy link
Contributor Author

@jcaamano jcaamano Apr 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange, this is working for me.

The secret network-resources-injector-secret should be created in namespace sriov-network-operator from the Certificate network-resources-injector-service if cert-manager is functioning correctly. Can you describe the Certificate and check what's its status? It is also strange that it happens for one of the webhooks and not the other. I wiull try it again on my side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried again and working for me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's working for me too on a fresh k8s install

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worked on a fresh install. Thanks!

```

By default, the operator will be deployed in namespace 'sriov-network-operator' for Kubernetes cluster, you can check if the deployment is finished successfully.

```bash
Expand Down
1 change: 1 addition & 0 deletions hack/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,4 @@ export RESOURCE_PREFIX=${RESOURCE_PREFIX:-openshift.io}
export ENABLE_ADMISSION_CONTROLLER=${ENABLE_ADMISSION_CONTROLLER:-"true"}
export CLUSTER_TYPE=${CLUSTER_TYPE:-openshift}
export NAMESPACE=${NAMESPACE:-"openshift-sriov-network-operator"}
export WEBHOOK_CA_BUNDLE=${WEBHOOK_CA_BUNDLE:-""}