Clean up language around tag listing and repository naming

[dmcgowan]

Tag listing should mention the endpoints and external methods relying on content addressability.
Image repositories should not be referred to as indexes, but rather as image repositories.

[stevvooe]

LGTM

[jonjohnsonjr]

Nit: s/compatible/compatibility/

[mikebrow]

see comments

[mikebrow]

What do you think about "Managing the grouping of image repository names" ?

[mikebrow]

definition is a narrow term.. I think best to just say spec.. or remove. Or did you mean to indicate that a non-definition discussion of some type may come out of the spec regarding said external methods?
/s/definition/this spec/

[dmcgowan]

I can remove it, what I was trying to say is that external methods will not be defined within this specification. Rather the interface external methods would need is already in place.

[mikebrow]

LGTM

[wking]

The distribution mandate needs to be broader than CAS. I don't think this sentence is broad enough to justify including the mutable, name-addressable tag-listing and tag -> manifest endpoints, which you list as in-scope above.

[dmcgowan]

This sentence is describing the protocol which is based on content-addressable objects. Naming just maps to these objects, the objects themselves are not mutable. It is clearly listed up above that these mapping endpoints are in scope and will be supported. The protocol however can be used without this tagging.

[wking]

This sentence is describing the protocol which is based on content-addressable objects.

The protocol uses content-addressable objects with a mutable, name-addressable layer on top (like Git's refs). Of the endpoints listed here, only /v2/<name>/blobs/<digest> is strictly content addressable. The stuff under /v2/<name>/blobs/uploads/ is how you feed data into CAS, so that's fine too. But /v2/<name>/manifests/<reference> and /v2/<name>/tags/list are mutable and name-addressable, so you need some other wording to justify them.

Naming just maps to these objects, the objects themselves are not mutable.

So add a sentence that includes this naming in the scope? As long as any mentions of “naming” are clearly distinct from the out-of-scope “… DNS based naming…” entry.

It is clearly listed up above that these mapping endpoints are in scope and will be supported.

Right, but that text is just informative background that won't end up in the scope table. I think we want normative language that will end up in the scope table to explain why tag listing and name-addressable references are in scope.

[dmcgowan]

So add a sentence that includes this naming in the scope?

Can you propose a sentence to add and we can discuss. I don't want to go back and forth on the correctness of the description. I am not sure how the scope table is generated, sounds like the previous sentences are just there for the proposal but unrelated to the scope table at https://www.opencontainers.org/about/oci-scope-table

[wking]

Can you propose a sentence to add and we can discuss.

My previous attempt is what you're removing here ;). I was trying to lean on image-spec indexes and org.opencontainers.image.ref.name to show why tag-listing and tag -> manifest-descriptor resolution were in scope. It sounds like that's the basic idea, but @stevvooe wants phrasing that doesn't mention the image-spec primitives, and I don't have a rewording I like better yet.

[wking]

I am not sure how the scope table is generated...

I think @caniszczyk just updates WordPress based on the instructions here after the TOB approves the proposal.

[dmcgowan]

I was trying to lean on image-spec indexes and org.opencontainers.image.ref.name to show why tag-listing and tag -> manifest-descriptor resolution were in scope.

I get that, the problem is that image-spec indexes and that annotation have nothing to do with the distribution specification, so it was confusing. The annotation may be the same as what the registry has, but there is no relationship in the API. The reason the current tag endpoints are included are for backwards compatibility and there are discussions which need to be had about updating that interface and capabilities. However the point of this proposal is to take the specification as is and keep the scope narrowed down. The CAS part of the specification has already been widely agreed on and is good to have in the scope table, we can always broaden later as the discussion matures.

[wking]

The reason the current tag endpoints are included are for backwards compatibility and there are discussions which need to be had about updating that interface and capabilities.

So add a sentence saying which endpoints are needed for backwards compat? And making it clear that backwards-compat with those endpoints is in-scope while backwards-compat with /v2/_catalog is out-of-scope?

[dmcgowan]

This is why I was asking about the format of these entries, these sentences are already there. Are you saying we should just move the sentences from before the What to under Description? The previous version also had in/out of scope language in two places, it is really not clear what this section is trying to accomplish with the formatting.

[wking]

This is why I was asking about the format of these entries, these sentences are already there. Are you saying we should just move the sentences from before the What to under Description? The previous version also had in/out of scope language in two places, it is really not clear what this section is trying to accomplish with the formatting.

My intention with the formatting was:

* “{What entry}”.
    {Detailed informative discussion, intended for the proposal only}.

    * What: {What entry}
    * In/Out/Future: {Scope entry}
    * Status: {Status entry}
    * Description: {Normative definiton}
    * Why: {High-level informative motivation}

If you want to drop the detailed-informative-discussion section entirely because you find it not-very-informative, that's fine with me. For the “Description” entry, how about something like:

    * Description: Define a protocol for creating, retrieving, updating, and deleting content-addressable objects, such as those defined in the [image specification][image-spec].
        Also define, as an optional layer, the `/v2/<name>/tags/list` and `/v2/<name>/manifests/<reference>` endpoints needed for backwards compatibility with existing clients.

[crosbymichael]

I'll be reopening the vote for the distribution spec after we have these last issues resolved on the proposal. Please take some time to review this PR so we can move forward.

Thanks!

[jonjohnsonjr]

I don't understand this.

Through the context of this PR, I can guess that this refers to using an image index to approximate listing tags, but on its own this sentence would be totally meaningless to me. Can we just drop it?

[dmcgowan]

I can guess that this refers to using an image index to approximate listing tags

What do you mean by this? I was thinking an external method would completely replace the tag listing, rather than just approximate it. There is a gap in the existing API where manifests must be uploaded with a tag, but we can easily address that especially for cases where tagging is not used at all within a registry. Since something like garbage collection does not belong in this specification, having a requirement about the relationship between tags and content is not required, allowing tags to be completely external.

[jonjohnsonjr]

What do you mean by this?

This PR drops Retrieving image indexes covers the current “tag listing” and adds External methods for listing tags, so I assumed you replaced the former with the latter to be more general?

I don't really see the need to mention this. It seems irrelevant, but I might just not understand what you're trying to say.

There is a gap in the existing API where manifests must be uploaded with a tag

Not per the spec. IIRC, docker doesn't allow you to push by a digest (maybe DockerHub too?), but I know at least GCR allows it.

Since something like garbage collection does not belong in this specification, having a requirement about the relationship between tags and content is not required, allowing tags to be completely external.

I don't see how tags can be completely external if tag listing is part of the spec. Are you saying tags are (mostly) undefined behavior in order to give freedom to implementors?

[dmcgowan]

I don't really see the need to mention this. It seems irrelevant, but I might just not understand what you're trying to say.

I was mostly just trying to account for the language which was here before, it really doesn't need to be mentioned. I will remove it, the scope section below is already clear that specification supports managing content addressable objects, what clients end up doing with that doesn't need to be mentioned here.

Not per the spec

Fair enough, the spec is clear that references can be digests, the incompatibility was mostly related to the deprecated schema 1 I believe. We will clean up these examples in the specification once it is in OCI.

I don't see how tags can be completely external if tag listing is part of the spec

I am only saying that a registry should be usable as only a CAS, without tagging at all. In which case any sort of resolution of a string -> content address is the responsibility of the distribution client. This does provide more freedom to distribution clients.

[jonjohnsonjr]

just trying to account for the language which was here before

Makes sense! I appreciate your patience :)

I am only saying that a registry should be usable as only a CAS, without tagging at all.

Right, I totally agree. My worry is that a registry that only implements the CAS portions of the spec will be seen as non-conforming.

resolution of a string -> content address is the responsibility of the distribution client

I think this is my ideal. Tag listing and tag resolution seem to be more about discovery/naming than distribution, and registries that wish to be compatible with older clients could choose to implement these via the old tags list endpoint.

Regardless, I think this is a good starting point. I just hope tag listing being a requirement isn't a foregone conclusion.

[jonjohnsonjr]

I'm not totally convinced that tag listing should be included in distribution.

Does "backwards compatibility with clients" here refer to just docker pull -a?

[dmcgowan]

I'm not totally convinced that tag listing should be included in distribution.

Specifically tag listing or tag operations in general?

Does "backwards compatibility with clients" here refer to just docker pull -a?

Also skopeo inspect

[jonjohnsonjr]

I'd argue tags in general but I doubt that would get much traction :)

Reading through the distribution spec, it seems to me that tags are an implicitly-defined mutable layer on top of the CAS with some missing features (e.g. deleting a tag). It would be nice if tags were first class. Merging distribution/distribution#2169 would help with some of my concerns, but I'm not sure what the plan is for the PRs under Related GitHub Issues.

Happy to punt on this for now if it can be improved later, but the registry is currently a mix of a DAG/CAS/CMS and I'd like to scope down 'distribution' to be as narrow as we can get away with.

[dmcgowan]

I agree with everything you just said 😄

There is where we are with tags right now (what we need to include so clients can continue to work just using the OCI specification) and where we want to be with tagging, which I believe is an explicit tags endpoint like you listed. Ideally the CAS and tagging elements are distinct and the registry API can be used as CAS only. I am trying to communicate that here so we remain open to extension, backwards compatible, and still support just CAS with tagging elsewhere. I think maybe this isn't fully accomplishing that, other than dropping do you have any recommendations for making the point more clear?

[stevvooe]

@jonjohnsonjr The intent of "Related Github Issues" was that these would be merged as part of the specification process. I am not sure at all how my comment about "These should be included" became "Related Github Issues".

In general, for the registry to be useful, you need CAS plus name pinning, so I don't think we can go full CAS here.

[stevvooe]

LGTM

[crosbymichael]

LGTM

[wking]

Are we sure we wanted to land this with a CAS-only mandate? More on that in this sub-thread. And I'd understood @stevvooe to be on board with a broader mandate based on his:

In general, for the registry to be useful, you need CAS plus name pinning, so I don't think we can go full CAS here.