test(itest): improve container startup health detection

[andrewazores]

Welcome to Cryostat! 👋

Before contributing, make sure you have:

• Read the contributing guidelines
• Linked a relevant issue which this PR resolves
• Linked any other relevant issues, PR's, or documentation, if any
• Resolved all conflicts, if any
• Rebased your branch PR on top of the latest upstream main branch
• Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
• Signed the last commit: git commit --amend --signoff

---

Fixes: #1533

Description of the change:

This adds Podman health probes to the containers started for integration testing, and changes the wait-for-X executions to query podman's view of container health as a startup check.

Motivation for the change:

Rather than running a timeout command that forks subprocess shells executing curl to try to check the container availability externally, this ends up using a process run within the container to check for the service availability, which is reported to podman. From outside the containers we ask podman what the status is. This is less prone to getting into a bad state with curl trying to open a connection to the services in the containers when they are not yet ready to accept connections and causing curl to time out and therefore fail the startup check and the integration test run. This should make itest startup much more reliable.

How to manually test:

1. bash repeated-integration-tests.bash N SomeTestsIT, where N is some large number and SomeTestsIT are a subset of tests run simply to ensure basic startup succeeded. bash repeated-integration-tests.bash 10 HealthIT for example.

[github-actions]

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1534-a64070ea2b968c391b3699964866cb9c02d71381 sh smoketest.sh

[andrewazores]

Seems like after one of the recent commits this actually breaks the CI runner entirely, it always seems to time out... unfortunately, not much in the way of logging.

[andrewazores]

I wonder if maybe the CI runner's grep doesn't accept -E or something.

[tthvo]

Ohh yeh just realized I just started the 7th attempt....What is version of grep on CI?

[tthvo]

Actually, I am experiencing the same issue locally too.

Been stuck at [INFO] --- exec-maven-plugin:3.1.0:exec (wait-for-cryostat) @ cryostat --- for a while.

[tthvo]

Maybe the regular expression here is invalid because the input is the while row including the container name, uptime and brackets?

$ podman ps --format '{{.Names}} {{.Status}}' | grep jfr-datasource | grep -E -i 'health'
jfr-datasource-itest Up About a minute (healthy)

$ podman ps --format '{{.Names}} {{.Status}}' | grep jfr-datasource | grep -E -i '^health'
$ echo $?
1

[github-actions]

Test image available:

$ CRYOSTAT_IMAGE=ghcr.io/cryostatio/cryostat:pr-1534-2343b4bd37c3ac739b4fe04788294d9486843dd2 sh smoketest.sh

[tthvo]

Looks good to me!