enclave-cc e2e updates

[mythi]

No description provided.

[ldoktor]

Hello Mikko, this does fixes the problem. Since on that would you consider adding some debug info when such issue happens (like kubectl describe pod/enclave-cc-pod-sim) to the workflow?

Also this is the first time I tried running this locally, is there a suggested way of doing that? I used a kcli vm with kind installed, then I ran tests/e2e/operator.sh to create the registry (and populate it), then I created the kind cluster, modified it's /etc/hosts to point registry to main route and then I was able to run the test successfully. Not really straight forward so I guess there ought to be a better way.

[mythi]

Hello Mikko, this does fixes the problem. Since on that would you consider adding some debug info when such issue happens (like kubectl describe pod/enclave-cc-pod-sim) to the workflow?

Good point, I can look into adding something there.

Also this is the first time I tried running this locally, is there a suggested way of doing that? I used a kcli vm with kind installed, then I ran tests/e2e/operator.sh to create the registry (and populate it), then I created the kind cluster, modified it's /etc/hosts to point registry to main route and then I was able to run the test successfully. Not really straight forward so I guess there ought to be a better way.

I don't have any good script. A kcli vm with kind installed is a good starting point since it's roughly what a Github runner is. All the steps after that are in

operator/.github/workflows/enclave-cc-e2e.yaml

Lines 35 to 70 in 016a7d4

	- name: Build and push the pre-install to local registry
	run: |
	pushd install/pre-install-payload
	make reqs-image registry="${REGISTRY}" extra_docker_manifest_flags="--insecure"
	popd
	env:
	REGISTRY: localhost:5000/reqs-payload
	- name: Setup kind cluster
	run: |
	kind create cluster --image "kindest/node:v1.27.3" -n coco-sgx --config tests/e2e/enclave-cc-kind-config.yaml --wait 120s
	kubectl label node coco-sgx-worker node.kubernetes.io/worker=
	docker network connect kind registry
	- name: Deploy operator from the local registry
	run: |
	cd config/default
	kustomize edit set image quay.io/confidential-containers/operator=localhost:5000/cc-operator:latest
	kubectl apply -k .
	sleep 1
	kubectl wait --for=jsonpath='{.status.conditions[0].status}'=True deployments/cc-operator-controller-manager -n confidential-containers-system
	img=$(kubectl get deployments/cc-operator-controller-manager -n confidential-containers-system -o jsonpath='{.spec.template.spec.containers[?(@.name == "manager")].image}')
	[ "${img}" = "localhost:5000/cc-operator:latest" ]
	- name: Install enclave-cc sim
	run: |
	cd config/samples/enclave-cc/sim
	kustomize edit set image quay.io/confidential-containers/reqs-payload=localhost:5000/reqs-payload:latest
	kubectl apply -k .
	sleep 1
	kubectl wait --for=jsonpath='{.status.runtimeClass}'=enclave-cc ccruntime/ccruntime-enclave-cc-sgx-mode-sim --timeout=90s
	- name: Deploy sample workload
	run: |
	kubectl apply -f tests/e2e/enclave-cc-pod-sim.yaml
	kubectl wait --for=condition=Ready pod/enclave-cc-pod-sim

and it should be possible to just copy-and-paste there. The tests I'm fixing here are not using tests/e2e/operator.sh

[ldoktor]

Hello Mikko, this does fixes the problem. Since on that would you consider adding some debug info when such issue happens (like kubectl describe pod/enclave-cc-pod-sim) to the workflow?

Good point, I can look into adding something there.

Thanks, could be a separate PR to not to block this fix, though

Also this is the first time I tried running this locally, is there a suggested way of doing that? I used a kcli vm with kind installed, then I ran tests/e2e/operator.sh to create the registry (and populate it), then I created the kind cluster, modified it's /etc/hosts to point registry to main route and then I was able to run the test successfully. Not really straight forward so I guess there ought to be a better way.

I don't have any good script. A kcli vm with kind installed is a good starting point since it's roughly what a Github runner is. All the steps after that are in

operator/.github/workflows/enclave-cc-e2e.yaml

Lines 35 to 70 in 016a7d4

	- name: Build and push the pre-install to local registry
	run: |
	pushd install/pre-install-payload
	make reqs-image registry="${REGISTRY}" extra_docker_manifest_flags="--insecure"
	popd
	env:
	REGISTRY: localhost:5000/reqs-payload
	- name: Setup kind cluster
	run: |
	kind create cluster --image "kindest/node:v1.27.3" -n coco-sgx --config tests/e2e/enclave-cc-kind-config.yaml --wait 120s
	kubectl label node coco-sgx-worker node.kubernetes.io/worker=
	docker network connect kind registry
	- name: Deploy operator from the local registry
	run: |
	cd config/default
	kustomize edit set image quay.io/confidential-containers/operator=localhost:5000/cc-operator:latest
	kubectl apply -k .
	sleep 1
	kubectl wait --for=jsonpath='{.status.conditions[0].status}'=True deployments/cc-operator-controller-manager -n confidential-containers-system
	img=$(kubectl get deployments/cc-operator-controller-manager -n confidential-containers-system -o jsonpath='{.spec.template.spec.containers[?(@.name == "manager")].image}')
	[ "${img}" = "localhost:5000/cc-operator:latest" ]
	- name: Install enclave-cc sim
	run: |
	cd config/samples/enclave-cc/sim
	kustomize edit set image quay.io/confidential-containers/reqs-payload=localhost:5000/reqs-payload:latest
	kubectl apply -k .
	sleep 1
	kubectl wait --for=jsonpath='{.status.runtimeClass}'=enclave-cc ccruntime/ccruntime-enclave-cc-sgx-mode-sim --timeout=90s
	- name: Deploy sample workload
	run: |
	kubectl apply -f tests/e2e/enclave-cc-pod-sim.yaml
	kubectl wait --for=condition=Ready pod/enclave-cc-pod-sim

and it should be possible to just copy-and-paste there.

Sure, that's what I followed, propagating the docker registry was the only troublemaker... Still would be nice to have a simple way to reproduce things locally...

The tests I'm fixing here are not using tests/e2e/operator.sh

I know, but it can serve well to setup the registry :-).