Plan to deprecate packer based podvm image builds and switch to mkosi

[bpradipt]

In preparation for deprecating packer based podvm image builds and switching to mkosi based image builds, I'm proposing a restructuring of the current directory layout.

• Consolidate packer based build deps (eg, hcl files, configs, docker files etc) under podvm-packer sub-dir
• Consolidate mkosi based build deps (eg, configs, dockerfiles etc) under podvm-mkosi sub-dir
• Update the github workflows to reflect the new directory structure for packer based builds in CI
• Add/update github workflows to reflect the new directory structure for mkosi based builds in CI

Once this is done, we should decide on a timeline to switch to podvm-mkosi based builds in CI.

@mkulke @stevenhorsman @snir911 @katexochen @surajssd @wainersm and others are you ok with this approach?
Is there any other alternative approach you would like to recommend ?

[stevenhorsman]

We discussed this in the community call and we were happy with the plan, with a couple of specifics/additions

• Due to issues with build ibm cloud images with mkosi, the switch to podvm-mkosi might need to only be for the other cloud providers initially, so we'd keep packer around as deprecated for the IBM Cloud providers. DaLi has created a PR to pin the version, so that should avoid license issues in the short-term
• During the time period that we have CI for both packer based and mkosi based podvms, we might need to re-prioritise the kata-agent caching in order to reduce the overhead of running both builds