test(scorecard): log workload conditions/events upon failure

[ebaron]

Welcome to Cryostat! 👋

Before contributing, make sure you have:

• Read the contributing guidelines
• Linked a relevant issue which this PR resolves
• Linked any other relevant issues, PR's, or documentation, if any
• Resolved all conflicts, if any
• Rebased your branch PR on top of the latest upstream main branch
• Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
• Signed all commits: git commit -S -m "YOUR_COMMIT_MESSAGE"

---

Fixes: #559

Description of the change:

If an expected deployment doesn't become available within the time limit (10m), log the conditions and warning events of the deployment, its replica sets and pods.

I also increased the timeout for the scorecard tests since we use timeouts for blocking operations inside the tests themselves and I've found that if the Scorecard timeout occurs then we lose logs.

Motivation for the change:

If Scorecard tests are run on a cluster where we don't have access to logs, it would be helpful to include as much output as possible for why the tests failed.

How to manually test:

I built a test bundle and scorecard image from this PR. I also set the CORE_IMG to quay.io/ebaron/cryostat:does-not-exist in order to cause an error during the test.
make test-scorecard BUNDLE_IMG=quay.io/ebaron/cryostat-operator-bundle:replica-failure-03

Here's a sample of the output upon deployment failure:

--------------------------------------------------------------------------------
Image:      quay.io/ebaron/cryostat-operator-scorecard:replica-failure-03
Entrypoint: [cryostat-scorecard-tests cryostat-cr]
Labels:
	"suite":"cryostat"
	"test":"cryostat-cr"
Results:
	Name: cryostat-cr
	State: fail

	Errors:
		Cryostat main deployment did not become available: timed out waiting for the condition
	Log:
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet found
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		[...]		
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test is not yet available
		deployment cryostat-cr-test conditions:
			Available == False, MinimumReplicasUnavailable: Deployment does not have minimum availability.
			Progressing == True, ReplicaSetUpdated: ReplicaSet "cryostat-cr-test-69bc79d7df" is progressing.
		deployment cryostat-cr-test warning events:
		replica set cryostat-cr-test-69bc79d7df conditions:
		replica set cryostat-cr-test-69bc79d7df warning events:
		pod cryostat-cr-test-69bc79d7df-b4vjz phase: Pending
		pod cryostat-cr-test-69bc79d7df-b4vjz conditions:
			Initialized == True, : 
			Ready == False, ContainersNotReady: containers with unready status: [cryostat-cr-test]
			ContainersReady == False, ContainersNotReady: containers with unready status: [cryostat-cr-test]
			PodScheduled == True, : 
		pod cryostat-cr-test-69bc79d7df-b4vjz warning events:
			Failed: Failed to pull image "quay.io/ebaron/cryostat:does-not-exist": rpc error: code = Unknown desc = reading manifest does-not-exist in quay.io/ebaron/cryostat: manifest unknown: manifest unknown
			Failed: Error: ErrImagePull
			Failed: Error: ImagePullBackOff