Feature: Adjust GitHub prune workflow so that it doesn't delete multi-arch release images

[chang-annie]

Description

The Github prune action which clears out our older Docker images is too aggressive and as a result, the release images are being deleted.

This ticket is to adjust the workflow so that the multi-arch release images are no longer being deleted as part of the prune process.

DOD

Acceptance criteria required to realize the requested feature

• Uncomment schedule for testing purposes
• Update prune workflow so that we're identifying SHAs of any multi-platform release images and exclude them from the clean up process
• Re-schedule the prune action and re-enable it
• Stretch goal: clean up older tagged images - only want to keep latest snapshot images and all release images
• Stretch goal: avoid pruning patch versions - for example, if current development version is 1.9.0-SNAPSHOT and we're also working on a patch version 1.8.1-SNAPSHOT at the same time, then we would want to keep the images for both snapshot versions

Test Strategy/Script

OTS:

• manually kick off the prune workflow (keep dry-run: true)
• Use the workflow logs for the prune_ghcr (image-name) steps to confirm the following
• The Prune ghcr > Runtime configuration section should include exclude-tags: 1.7.0,1.8.0. For example, using aissemble-vault:

  project owner               boozallen
  repository                  aissemble
  package                     aissemble-vault
  exclude-tags                1.7.0,1.8.0
  delete-untagged             false
  keep-n-tagged               2
  dry-run                     true
  validate                    false
  log-level                   INFO

• Review the Prune ghcr > Loaded Package Data section - note the two most recent NON-release versions found. For most images, this should be 1.9.0-SNAPSHOT and 1.8.0-rc8.
• NOTE: aissemble-airflow, aissemble-kafka, and aissemble-mlflow images are deprecated and will not have a 1.9.0-SNAPSHOT tag
• Review the Prune ghcr > Finding tagged images to delete, keeping 2 versions section and confirm that the list does not include any release versions (e.g., 1.8.0,1.7.0) or the two most recent non-release versions (e.g., 1.9.0-SNAPSHOT, 1.8.0-rc8)
• Review the Prune ghcr > Deleting packages section and confirm that the list does not include any release versions or the two most recent non-release versions

Final Test:

• Confirm Prune workflow is re-enabled
• Kick off the prune workflow with the dev branch and confirm successful
• Use the workflow logs for the prune_ghcr (image-name) steps to confirm the following
• The Prune ghcr > Runtime configuration section should include exclude-tags: 1.7.0,1.8.0. For example, using aissemble-vault:

  project owner               boozallen
  repository                  aissemble
  package                     aissemble-vault
  exclude-tags                1.7.0,1.8.0
  delete-untagged             false
  keep-n-tagged               2
  validate                    false
  log-level                   INFO

• Review the Prune ghcr > Loaded Package Data section - note the two most recent NON-release versions found. For most images, this should be 1.9.0-SNAPSHOT and 1.8.0-rc8.
• NOTE: aissemble-airflow, aissemble-kafka, and aissemble-mlflow images are deprecated and will not have a 1.9.0-SNAPSHOT tag
• Review the Prune ghcr > Finding tagged images to delete, keeping 2 versions section and confirm that the list does not include any release versions (e.g., 1.8.0,1.7.0) or the two most recent non-release versions (e.g., 1.9.0-SNAPSHOT, 1.8.0-rc8)
• Review the Prune ghcr > Deleting packages section and confirm that the list does not include any release versions or the two most recent non-release versions
• Successfully pull a 1.8.0, 1.7.0, and 1.9.0-SNAPSHOT docker image

docker pull ghcr.io/boozallen/[image-name]:1.8.0
docker pull ghcr.io/boozallen/[image-name]:1.7.0
docker pull ghcr.io/boozallen/[image-name]:1.9.0-SNAPSHOT

References/Additional Context

Container retention policy doc

[chang-annie]

DOD completed with @ewilkins-csi

[chang-annie]

Some interesting findings:

When trying to pull on the manifest SHA for the 1.8.0 release images, we cannot simply use something like:

docker buildx imagetools inspect ghcr.io/boozallen/aissemble-spark:1.8.0 --format "{{json .Manifest}}" | jq -r .digest

This will output:

ERROR: failed to copy: httpReadSeeker: failed open: content at https://ghcr.io/v2/boozallen/aissemble-spark/manifests/sha256:c0ea773c38265bf3a80a133211aba2468dec87b8ce5b341d82611e2c81252147 not found: not found

Where the aforementioned SHA is connected to the build attestations. These are created because we have buildx's provenance set to min aka not false (default setting). This creates two extra manifests in the manifest list with os: unknown and architecture: unknown. You can further inspect those manifests to retrieve basic provenance info.

[chang-annie]

OTS completed with @carter-cundiff and @jaebchoi

[ewilkins-csi]

OTS passed ✅

[J-Clingerman]

All tests passed. ✅

[chang-annie]

Reopening so we can test the fix to address docker images with no release versions (which is adding an extra comma at the front of our exclude-tags parameter - e.g., exclude-tags: ,1.7.0,1.8.0)

[cwoods-cpointe]

Final test: Passed. Did not delete any release version or the 2 latest dev versions.