Reference images with only an image or only a tag to be cri-o/OCI compatible

[maxandersen]

Expected behavior

skaffold should have option to only use tags (or digest) when referring to generated images.
That is so it is compatible with the oci spec and will work in systems using cri-o (such as openshift/origin).

Today it references images like this:
docker.io/maxandersen/skaffold-example2:v0.41.0-175-g448e4014-dirty@sha256:f3e4ce9a4d629b78aef104253d6233132853e1c355dbdd04b2a9d309f14f20d5

That result in errors like this:

rpc error: code = Unknown desc = Invalid image name "docker.io/maxandersen/skaffold-example2:v0.41.0-175-g448e4014-dirty@sha256:f3e4ce9a4d629b78aef104253d6233132853e1c355dbdd04b2a9d309f14f20d5", unknown transport "docker.io/maxandersen/skaffold-example2"
Back-off pulling image "docker.io/maxandersen/skaffold-example2:v0.41.0-175-g448e4014-dirty@sha256:f3e4ce9a4d629b78aef104253d6233132853e1c355dbdd04b2a9d309f14f20d5"

This have previously been discussed in #2089
which was closed with the comment to reopen or create new issue if requests to cri-o based projects did not succeed in getting support for tag/digest. Both containers/buildah#1407 and cri-o/cri-o#2351 have been closed referencing the distribution spec:

The name and reference parameter identify the image and are REQUIRED. The reference MAY include a tag or digest.

I understand why skaffold uses both as that in other systems are used to state, pull from the tag and if the sha does not match cause a failure as a double insurance.

But for now that "useful loophole" is not honored by cri-o and thus I suggest skaffold at least provide an option to only add the tag so skaffold can be used on these systems.

Information

• Skaffold version: 1.0
• Operating system: Any

Steps to reproduce the behavior

1. git clone https://github.com/GoogleContainerTools/skaffold
2. cd skaffold/examples/getting-started
3. target an cri-o based system, i.e. openshift 4.2
4. skaffold dev
5. output will have the above error in it.

[maxandersen]

Note, I was hoping https://skaffold.dev/docs/pipeline-stages/taggers/ could be used to control this but it seems like skaffold always append @<shadigest> to whatever tag is generated.

[maxandersen]

I tried do a few changes to the codebase to strip the digest but turns out that since both gitcommit and sha256 taggers generates fully stable tags no matter the amount of changes you've made skaffold's logic relies on the digest to be present.

Then I tried to rewire the the manifest render which kinda worked but I'm not sure that would be the right level to fix it. Any guidance on where this would be fixed best would be appreciated and I'll happily try make a proper patch for this.

[dgageot]

@maxandersen We currently assume that the digest can safely be appended to the tag. This is unfortunately done in multiple places (look for + "@" + digest in the codebase).

The solution would be to extract that code to a function that appends the digest only to the basename.

If it's possible to activate that behaviour only when the underlying engine is cri-o, that would be a plus.

[dgageot]

@maxandersen Can you share the output of kubectl get nodes -oyaml. I'd like to see if we can auto-detect the runtime.

[maxandersen]

@dgageot it’s a rather big cluster i'm testing on so a rather large file :) but I assume this section is what you would be looking for:

nodeInfo:
      architecture: amd64
      bootID: 792d5736-a82e-4ef3-a9cf-aa0799558cac
      containerRuntimeVersion: cri-o://1.14.11-0.23.dev.rhaos4.2.gitc41de67.el8
      kernelVersion: 4.18.0-80.11.2.el8_0.x86_64
      kubeProxyVersion: v1.14.6+7e13ab9a7
      kubeletVersion: v1.14.6+7e13ab9a7
      machineID: 88a2ff55e9bd4363aa812e84c4b404dc
      operatingSystem: linux
      osImage: Red Hat Enterprise Linux CoreOS 42.80.20191022.0 (Ootpa)
      systemUUID: ec28902b-023d-9aa9-0026-0e76a938580a

particular containerRuntimeVersion: cri-o://1.14.11-0.23.dev.rhaos4.2.gitc41de67.el8 ?

[maxandersen]

Autodetection is fine but I would still think it should be something to explicitly enable/declare to do rendering without contacting the cluster?

[maxandersen]

Gave this a go but just replacing all places with + "@" + was not sufficient - the initial name:tag@sha seem to be created somewhere else. My current progress is in https://github.com/maxandersen/skaffold/tree/cleantagdigest if curious or can point to where else the tag@sha happens.

[mgoltzsche]

@maxandersen I tested your branch and it works with CRI-O when your docker.MakeFqn function just returns repository + "@" + digest instead of tag + "@" + digest. (Maybe the build cache was confusing you when you tested it?) Unfortunately this way image tags are not visible within k8s manifests. However using only tag seems currently not feasible with the gitCommit tagPolicy since the tag remains the same in dirty state and as such relies on the digest in order to actually do a rollout. Therefore I think the tagging strategy would need to be changed as well to provide both human-readable and unique tags - at least in a dirty git repo state the image digest could be part of the tag.

I agree with you that the target/CRI-O cannot always be auto-detected. A flag could be added to skaffold.yaml to enable this feature.
For simple target auto-detection within in-cluster builds like a tekton pipeline you could quickly check whether /proc/self/cgroup contains a crio-* cgroup if you want to avoid checking all cluster nodes - for skaffold dev you may still need to enable the feature explicitly or actually check the nodes.
UPDATE: To detect a local podman/buildah tool chain you could check whether docker -v output contains podman.

[maxandersen]

Yes, I basically reached same conclusion that this behavior was kinda needing to have some influence on the tag policy too - i tried swizzle the values in the generated manifests but it became way more messy than I felt it should be.

And yes, sha1 is what must be there.

[chanseokoh]

Recent Tekton releases start to use both a tag and a digest, and therefore they don't work for some platforms like the new OpenShift 4.x which uses cri-o. To support them, Tekton decided to provide notags yamls, which I think makes a lot of sense. It's likely that you just can't use Skaffold with OpenShift in this state.

[dgageot]

Looks like Cri-o will support those :tag@digest names: cri-o/cri-o#2351

[maxandersen]

correct - will hopefully soon be available in upcoming origin/openshift releases.

[dgageot]

I'm going to close this issue because there's not much we have to do on the skaffold side now. Thanks for reporting and investigating!