[Umbrella Issue] Setup a GCR Repository for projects to use

[dims]

Split from #153 (see that for some context)

cc @javier-b-perez @mkumatag @listx @thockin

[dims]

Related to #157

[javier-b-perez]

Do we have an agreement in the structure of the storage?

[dims]

@javier-b-perez we need to come up with a proposal

[javier-b-perez]

You mean a proposal for the structure, like in the doc?

Option 1)
We have a single staging area

staging-k8s.gcr.io/<TYPE>/[<PROJECT>/]<IMAGE_NAME>:<TAG>

TYPE

• kubernetes - core kubernetes images like pause, apiserver etc.
• kubernetes-sigs - Kubernetes SIG-related work
• tests - used for tests
• examples - example images used in the documentation

PROJECT: (optional) can be e.g. dashboard.
IMAGE_NAME: name of the container image

TAG: structure <VERSION>-<OS>-<ARCH>

• <TAG>: image version
• <OS>: the same as GOOS, e.g. linux or windows
• <ARCH>: the same as GOARCH, e.g. amd64 or ppc64le

Example
k8s.gcr.io/kubernetes/pause

• k8s.gcr.io/kubernetes/pause:3.1-linux-amd64
• k8s.gcr.io/kubernetes/pause:3.1-linux-arm
• k8s.gcr.io/kubernetes/pause:3.1-linux-ppc64le
• k8s.gcr.io/kubernetes/pause:3.1-linux-s390x

Option 2)
Tim mentioned an option to have per-project staging areas

gcr.io/<PROJECT>/[<TYPE>/]<IMAGE_NAME>:<TAG>

Example
k8s.gcr.io/kubernetes/pause

• gcr.io/k8s-kubernetes-staging/pause:3.1-linux-amd64
  ...

Then we promote images to the official registry k8s.gcr.io/<TYPE>/[<PROJECT>/]<IMAGE_NAME>:<TAG>

[marquiz]

Per-project staging (option 2) sounds safer to me. However, the URI scheme in option 1 looks better without the -staging appended to the project name.

I'd be very happy to see this issue solved as node-feature-discovery would need a new home for its images (kubernetes-sigs/node-feature-discovery#177) 😉

[BenTheElder]

cc

[pohly]

kubernetes-csi also needs an area to publish its sidecar container images.

[pohly]

gcr.io/kubernetes-csi/<image>:<tag> sounds good to me.

[marquiz]

Any progress on this issue?

[dims]

ON Friday Feb 8th Tim and i ended up setting up the following for GCR.

As a first step, we are setting up staging repositories.

• gcr.io/k8s-staging-csi
• gcr.io/k8s-staging-cluster-api
• gcr.io/k8s-staging-coredns

Folks in the following google groups will be able to push to the staging repository:

• k8s-infra-gcr-staging-csi@googlegroups.com
• k8s-infra-gcr-staging-cluster-api@googlegroups.com
• k8s-infra-gcr-staging-coredns@googlegroups.com

The following google group has the admin privileges over all the GCR repos.

• k8s-infra-gcr-admins@googlegroups.com

Notes:

• Only release artifacts should be uploaded here
• artifacts will be cleaned up periodically (older than 2 weeks?) to prevent the use of these repos beyond staging
• We will have additional workflow for promotion from here to the main k8s.gcr.io repo
• We will be adding sig leads to the google groups sometime next week to kick the tires

cc @thockin @hh

PS: next on the list is cloud-provider(s)

[dims]

related to #186

[thockin]

Note that my PR does not add an age-out for these repos. If we're interested in that, we can add it.

[dims]

@thockin i think we should age-out all the artifacts in the staging repos to prevent them from being used beyond testing. I'd say 2 weeks.

[jeefy]

Per @dims (Thanks!)

Piggy-backing on this thread to request a k8s.gcr.io repo for https://github.com/kubernetes-sigs/dashboard-metrics-scraper when things get sorted out. 😄

[pohly]

gcr.io/k8s-staging-csi

This is meant for the images published by Kubernetes-CSI, right? @BenTheElder pointed out that images built locally can be side-loaded into kind, so for CI testing purposes alone it might not be necessary to push images to this staging area.

We could publish release images in this staging area, but then the question becomes where the final destination will be (k8s.gcr.io/csi?) and how quickly they will get copied there.

Folks in the following google groups will be able to push to the staging repository:

• k8s-infra-gcr-staging-csi@googlegroups.com

How will that work from inside a Prow job?

[neolit123]

TAG: structure <VERSION>-<OS>-<ARCH>

• <TAG>: image version
• <OS>: the same as GOOS, e.g. linux or windows
• <ARCH>: the same as GOARCH, e.g. amd64 or ppc64le

just wanted to raise one point about GOOS tags and Windows.

originally we tough that this can apply cleanly so that we can have one manifest list (say for the pause container) that runs on all OSes, but on Windows there is a bit of a problem with sizes.

workloads on Windows can be based on a Windows Server XX and Windows Nanoserver XX.
according to my research, the nano server one is at least 100MB, while the regular server one is 2GB (both without hacks).

depending on what workloads the users run on the worker nodes they might decide to go for the nanoserver (to have a smaller download), which we don't handle with the above tags.

i started to think that we might have to do paths instead (and if possible):
gcr.io/.../windows-nano/SOMEIMAGE:VERSION-ARCH
gcr.io/.../windows/SOMEIMAGE:VERSION-ARCH

[BenTheElder]

How will that work from inside a Prow job?

we'd need a GCP service account / service account key with access to this.

[chuckha]

@dims I am not a sig lead but am working on the release pipelines for cluster-api and cluster-api-provider-aws. I'd like to have my work eventually end up in other providers, but am starting there. Would it be possible to be added to the cluster-api staging group? I'll request to join. No worries if it's a sig-leads-only policy.

[dims]

@chuckha we are still waiting for a couple of pieces to go into prow, so yes please hang on :)

[justaugustus]

@dims -- Hiya! We (capz) are also interested in getting access to push images to staging and production GCR buckets. :)

ref: kubernetes-sigs/cluster-api-provider-azure#118

[spiffxp]

There is a script that creates repos, not yet a nice yaml file, but editing a script means it create repos

[spiffxp]

Actually, we do want to give staging repos to subprojects.... can write into staging repo, promoter pushes into production repo

[spiffxp]

We are basically willing to say yes to people who ask for gcr repos. At the moment, we would point people to the PR's @nikhita did for publishing-bot. Consider refining into a README that people can follow, and that we have better coverage around reconciling once PRs reduces

[nikhita]

At the moment, we would point people to the PR's @nikhita did for publishing-bot.

For reference: #282

Consider refining into a README that people can follow

created #286

[thockin]

Final tasks here are in-progress. Once e2e and DR are done, we can bulk-import and flip the vanity URL

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[listx]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
/lifecycle frozen

Once the vanity domain flip (VDF) is complete and we have a little more documentation (and maybe automation) around how people can use / provision this, I think we're done here

[spiffxp]

/close
I'm going to call this done in favor of using #157 to track completion of the vanity domain flip and whatever else we need to feel confident about an image promotion process that uses this repo. The repo itself has been setup.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
I'm going to call this done in favor of using #157 to track completion of the vanity domain flip and whatever else we need to feel confident about an image promotion process that uses this repo. The repo itself has been setup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.