operator-sdk run bundle with security-context-config restricted fails to spawn registry pod due to runAsNonRoot

[stephan48]

Bug Report

What did you do?

Ran the following to deploy my ansible(but it does not matter) based operator into my k8s cluster:
$ operator-sdk run bundle --security-context-config restricted kuberegistry.blub.tld/test/ansible-operator-dev/test1-bundle:v0.0.1

What did you expect to see?

What did you see instead? Under which circumstances?

The deployment fails directly with the following

INFO[0009] Creating a File-Based Catalog of the bundle "kuberegistry.blub.tld/test/ansible-operator-dev/test1-bundle:v0.0.1"
INFO[0010] Generated a valid File-Based Catalog
FATA[0010] Failed to run bundle: create catalog: error creating registry pod: error creating pod: pods "y-blub-tld-test-ansible-operator-dev-test1-bundle-v0-0-1" is forbidden: violates PodSecurity "restricted:latest": runAsNonRoot != true (pod or container "registry-grpc" must set securityContext.runAsNonRoot=true)

Environment

Operator type:
N/A

Kubernetes cluster type:

Talos Linux 1.3.x with K8s 1.26.3 and PodSecurity set to restricted.

$ operator-sdk version

operator-sdk version: "v1.28.0", commit: "484013d1865c35df2bc5dfea0ab6ea6b434adefa", kubernetes version: "1.26.0", go version: "go1.19.6", GOOS: "linux", GOARCH: "amd64"

$ kubectl version

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

As per the comment starting in line 244 the reasoning for not applying the runAsNonRoot container security context is that previous OpenShift and K8s versions did not support all needed options. We could probably by now assume that we don't need to support 1.19.x or possible even the older OpenShift versions anymore.

Possible we could add detection modi for OpenShift/Legacy K8s or enhance the switch to allow the user to choose the right security level.

Additional context

Together with @everettraven on Slack(https://kubernetes.slack.com/archives/C0181L6JYQ2/p1683845500758729) we identified the following points:

• https://github.com/operator-framework/operator-sdk/blob/v1.28.0/internal/olm/operator/registry/index/registry_pod.go#L144
• https://github.com/operator-framework/operator-sdk/blob/v1.28.0/internal/olm/operator/registry/index/registry_pod.go#L244

Temporary Workaround via Kyverno Policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: operator-sdk-patch-registry-grpc
spec:
  rules:
    - name: operator-sdk-patch-registry-grpc
      match:
        any:
        - resources:
            kinds:
            - Pod
            namespaces:
            - "*"
            names:
            - "*bundle*"
      mutate:
        patchStrategicMerge:
          spec:
            containers:
              # match images which end with :latest
              - (name): "registry-grpc"
                # set the imagePullPolicy to "IfNotPresent"
                securityContext:
                  runAsNonRoot: true

[varshaprasad96]

We would need help in implementing this feature. @everettraven would be able to guide in implementing this feature. Please free to assign yourself. Thank you

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ehvs]

@varshaprasad96
This issue still current for me, is there any other possible workarounds? Or was it already fixed and maybe Im missing something.

Using:

operator-sdk version: v1.32
OpenShift, Kubernetes version: v1.26
Kustomize Version: v4.5.7

❯ operator-sdk run bundle docker.io/myrepo/memcached-operator-bundle:v0.0.1 
INFO[0020] Creating a File-Based Catalog of the bundle "docker.io/songbird159/memcached-operator-bundle:v0.0.1" 
INFO[0022] Generated a valid File-Based Catalog  
FATA[0023] Failed to run bundle: create catalog: error creating registry pod: error creating pod: pods "docker-io-songbird159-memcached-operator-bundle-v0-0-1" is forbidden: 
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "registry-grpc-init", "registry-grpc" must set securityContext.allowPrivilegeEscalation=false),
 unrestricted capabilities (containers "registry-grpc-init", "registry-grpc" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "registry-grpc-init", 
"registry-grpc" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "registry-grpc-init", "registry-grpc" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"

[astoycos]

I'm also still running into this :/

[astoycos@nfvsdn-03 bpfman]$ operator-sdk run bundle quay.io/bpfman/bpfman-operator-bundle:latest --namespace openshift-bpfman
INFO[0012] Creating a File-Based Catalog of the bundle "quay.io/bpfman/bpfman-operator-bundle:latest" 
INFO[0013] Generated a valid File-Based Catalog         
FATA[0013] Failed to run bundle: create catalog: error creating registry pod: error creating pod: pods "quay-io-bpfman-bpfman-operator-bundle-latest" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "registry-grpc-init", "registry-grpc" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "registry-grpc-init", "registry-grpc" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "registry-grpc-init", "registry-grpc" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "registry-grpc-init", "registry-grpc" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") 

[kaovilai]

/reopen

[openshift-ci]

@kaovilai: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[acornett21]

@kaovilai There is a flag for this, have you tried to run with this flag?

--security-context-config SecurityContext   specifies the security context to use for the catalog pod. allowed: 'restricted', 'legacy'. (default legacy)

[everettraven]

@acornett21 This issue is about a bug in the functionality of that flag. It is missing setting the runAsNonRoot field in the SCC

[kaovilai]

@weshayutin said it didn't work on v1.33.0

[kaovilai]

Checking the master branch code, nothing is setting the RunAsNonRoot: true./

[kaovilai]

consider my team unblocked with the flag working on latest GA 1.34.2

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.