add semver tagging semantics

[vito]

fixes #151
closes #197 (supersedes it)

still a WIP - leaving lots of context in each commit message, will amend this with more commentary later

note: this is intended to be as backwards-compatible as possible, but the default behavior will change specifically when tag: is not specified. rather than tracking only the latest tag, the resource will track all semver tags present, only returning latest if there are no semver tags with the same digest

TODO:

• /check: support detecting "variants" (i.e. alpine, 3-alpine, 3.2-alpine, 3.2.4-alpine)
• /check: distinguish between multi-word variants (i.e. apache shouldn't match 1.2.3-foo-apache)
• /check: distinguish between prereleases and variants (i.e. apache SHOULD match 1.2.3-rc.2-apache)
• /check: skip prereleases by default, add flag
• /check: support checking "from" a version
• /check: refactor; there's a lot of branching with tag vs. non-tag and mirror vs. no mirror
• /out: support pushing versions and variant versions
• (bumping aliases #230): /out: implement optional semver tag bumping (i.e. bump 3.2, 3, and latest when shipping 3.2.4 assuming 3.2 is latest)
  • (bumping aliases #230) plus variant support (bump 3.2.0-alpine, and optionally 3.2-alpine if latest 3.2.x, 3-alpine if latest 3.x, alpine if latest overall)
• update README
• An initial check request which fetches digests of all the tags is a bit slow - it looks like go-containerregistry does a fresh auth flow for each call. Can we optimize this, either by avoiding requests when given a cursor version or by opening an issue on go-containerregistry? (opened issue: Is there a way to reuse token/transport to prevent repeated auth requests? google/go-containerregistry#740)
• /in: respect a version that only has digest: - this is may be from a user manually pinning, per Feature request: support digest in 'source' #45 (comment)
  • hmm, actually maybe we should just always check and then find the specified version?

[vito]

@xtreme-sameer-vohra Quick update: this is good to go! Just finished the last TODO item and fixed a broken test.

[vito]

@xtreme-sameer-vohra I merged master into this instead of rebasing. There are a bunch of commits here so rebasing would have been a nightmarish hellscape vs. merging and resolving all the conflicts at once. Hope that's OK.

[vito]

@xtreme-sameer-vohra This is ready to go again. We're thinking of jumping to v7.0 instead of v6.8 so I'd like to include this too since we're doing a major version bump and this involves a behaviour change, even though it's mostly backwards-compatible.

I wish I could make it easier to review but with how long this has been open it'd be a bit of a nightmare to rebase. All of my recent pushes are just to bring it in line with changes that have been made on master - mainly the registry mirror handling and the HEAD/GET fallback, and previously the whole commands/ restructuring. So hopefully there's not a whole lot to review from last time. 🙏

[aoldershaw]

Just skimmed through so far, but seems pretty nice!

Have you considered adding a semver constraint configuration to the source? Seems like it'd be super easy with Masterminds/semver

[vito]

Have you considered adding a semver constraint configuration to the source? Seems like it'd be super easy with Masterminds/semver

Yeah I think that makes a lot of sense. I'd rather leave it for a separate PR though, just 'cause this one's pretty big already.

[aoldershaw]

check is certainly doing a lot more work, given that it has to make a HEAD request for each (semver) tag. I wanted to see how much slower it is:

$ jq -n '{source:{repository:"nginx"}}' | \
    time docker run --rm -i concourse/registry-image-resource:pr-alpine-214 \
      /opt/resource/check
...
       25.66 real         0.05 user         0.02 sys

compared to latest:

$ jq -n '{source:{repository:"nginx"}}' | \
    time docker run --rm -i concourse/registry-image-resource \
      /opt/resource/check
...
        1.53 real         0.05 user         0.02 sys

It's quite a bit slower. I expect this won't be a huge problem for named resources, since they only need to do this slow check once (and whenever a digest disappears) - but will it be for anonymous task.image_resources?

edit: obviously, "problem" is subjective - things will still work, but if it tacks on an extra 20s to many task steps, that could be a bit annoying

edit take 2: I refreshed my memory on how the check step works, and see that it uses the latest version based on the resource config scope if a from version isn't explicitly set - so this won't actually be slow for subsequent task runs, will it? (the whole resource config scope thing still kinda eludes me)

[vito]

@aoldershaw Yup - that's what led to the whole image-fetching refactor. With how things used to work, it would definitely be too slow for task image resources - but now that check runs from the latest version, it should only be slow the first time the image is ever used.

Scopes are a little confusing, but in this case you can ignore them since scopes only apply to pipeline-level resources - otherwise it'll be the "null" (global) scope.

[aoldershaw]

LGTM!