test(scorecard): add Scorecard test for Cryostat install

[ebaron]

This PR adds a new custom scorecard test that creates a default Cryostat CR and ensures that the operator brings up an Available Cryostat deployment. It then checks that the application URL field has been set in the CR's status.

In order to have a client for these tests that works with CRs, I've added a new CryostatClientset that extends kubernetes.Clientset with this extra functionality. This client can be used to work with both built-in objects and CRs.

I have test images prepared to make testing a bit easier:
$ make test-scorecard CUSTOM_SCORECARD_IMG=quay.io/ebaron/cryostat-operator-scorecard:2.3.0-20230317182833 BUNDLE_IMG=quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01

Example output for the new test:

--------------------------------------------------------------------------------
Image:      quay.io/ebaron/cryostat-operator-scorecard:2.3.0-20230317182833
Entrypoint: [cryostat-scorecard-tests cryostat-cr]
Labels:
	"test":"cryostat-cr"
	"suite":"cryostat"
Results:
	Name: cryostat-cr
	State: pass

	Log:
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is available
		Application is ready at https://cryostat-cr-test-cryostat-operator-scorecard.apps.example.com

Fixes: #456

[andrewazores]

Maybe I just need to keep waiting but it seems like it's stuck and no more output is coming:

$ make test-scorecard CUSTOM_SCORECARD_IMG=quay.io/ebaron/cryostat-operator-scorecard:2.3.0-20230317182833 BUNDLE_IMG=quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01 
oc create namespace cryostat-operator-scorecard
namespace/cryostat-operator-scorecard created
oc -n cryostat-operator-scorecard create -f internal/images/custom-scorecard-tests/rbac/
role.rbac.authorization.k8s.io/cryostat-scorecard created
rolebinding.rbac.authorization.k8s.io/cryostat-scorecard created
serviceaccount/cryostat-scorecard created
operator-sdk run bundle -n cryostat-operator-scorecard quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01
INFO[0004] Creating a File-Based Catalog of the bundle "quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01" 
INFO[0004] Generated a valid File-Based Catalog         
INFO[0009] Created registry pod: uay-io-ebaron-cryostat-operator-bundle-cryostat-cr-scorecard-01 
INFO[0009] Created CatalogSource: cryostat-operator-catalog 
INFO[0009] OperatorGroup "operator-sdk-og" created      
INFO[0009] Created Subscription: cryostat-operator-v2-3-0-dev-sub 
INFO[0012] Approved InstallPlan install-nxj8d for the Subscription: cryostat-operator-v2-3-0-dev-sub 
INFO[0012] Waiting for ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" to reach 'Succeeded' phase 
INFO[0012]   Waiting for ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" to appear 
INFO[0024]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Pending 
INFO[0026]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Installing 
INFO[0037]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Succeeded 
INFO[0037] OLM has successfully installed "cryostat-operator.v2.3.0-dev" 
operator-sdk scorecard -n cryostat-operator-scorecard -s cryostat-scorecard -w 5m quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01
W0317 18:27:23.006889  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.013512  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.013577  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.015352  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.025462  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040103  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040328  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040378  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

[ebaron]

Maybe I just need to keep waiting but it seems like it's stuck and no more output is coming:

$ make test-scorecard CUSTOM_SCORECARD_IMG=quay.io/ebaron/cryostat-operator-scorecard:2.3.0-20230317182833 BUNDLE_IMG=quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01 
oc create namespace cryostat-operator-scorecard
namespace/cryostat-operator-scorecard created
oc -n cryostat-operator-scorecard create -f internal/images/custom-scorecard-tests/rbac/
role.rbac.authorization.k8s.io/cryostat-scorecard created
rolebinding.rbac.authorization.k8s.io/cryostat-scorecard created
serviceaccount/cryostat-scorecard created
operator-sdk run bundle -n cryostat-operator-scorecard quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01
INFO[0004] Creating a File-Based Catalog of the bundle "quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01" 
INFO[0004] Generated a valid File-Based Catalog         
INFO[0009] Created registry pod: uay-io-ebaron-cryostat-operator-bundle-cryostat-cr-scorecard-01 
INFO[0009] Created CatalogSource: cryostat-operator-catalog 
INFO[0009] OperatorGroup "operator-sdk-og" created      
INFO[0009] Created Subscription: cryostat-operator-v2-3-0-dev-sub 
INFO[0012] Approved InstallPlan install-nxj8d for the Subscription: cryostat-operator-v2-3-0-dev-sub 
INFO[0012] Waiting for ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" to reach 'Succeeded' phase 
INFO[0012]   Waiting for ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" to appear 
INFO[0024]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Pending 
INFO[0026]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Installing 
INFO[0037]   Found ClusterServiceVersion "cryostat-operator-scorecard/cryostat-operator.v2.3.0-dev" phase: Succeeded 
INFO[0037] OLM has successfully installed "cryostat-operator.v2.3.0-dev" 
operator-sdk scorecard -n cryostat-operator-scorecard -s cryostat-scorecard -w 5m quay.io/ebaron/cryostat-operator-bundle:cryostat-cr-scorecard-01
W0317 18:27:23.006889  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.013512  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.013577  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.015352  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.025462  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040103  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040328  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0317 18:27:23.040378  540203 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "scorecard-untar", "scorecard-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "scorecard-untar", "scorecard-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "scorecard-untar", "scorecard-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

I think this is probably caused by not having cert-manager installed in the cluster. It should be an easy fix by making the test-scorecard target depend on check_cert_manager. I'm going to investigate a bit how we can make this work with our downstream scorecard testing first though.

[andrewazores]

Ah, that could be it. I don't remember if I had installed cert-manager on that cluster yet. I'll try again now.

[andrewazores]

Looks like that was indeed it, I created a new crc cluster, made sure to make cert_manager, and now I have more output reflecting a successful run of tests.

[ebaron]

I just pushed one line commit to add the check_cert_manager as a prerequisite for test-scorecard. I think I've found a way to make things work downstream, so this should be fine for upstream.