Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Docker image and push to GHCR #230

Open
wants to merge 21 commits into
base: unstable/v1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
eb50ad4
Build Docker image and push to GHCR
br3ndonland Apr 16, 2024
1e0ccc9
Fix pre-commit errors
br3ndonland Apr 19, 2024
6364a07
Separate `docker login` and `docker push`
br3ndonland Apr 26, 2024
a6e3290
Generate Docker container action with Python
br3ndonland May 27, 2024
e236daa
Reset pre-commit `files:` regex
br3ndonland May 27, 2024
b17295d
Use YAML block strip syntax (`>-`) where possible
br3ndonland May 27, 2024
9337018
Add `workflow_dispatch` trigger for Docker builds
br3ndonland May 27, 2024
59e3edf
Don't update `actions/checkout@v3`
br3ndonland May 27, 2024
90ff142
Reset smoke test path
br3ndonland May 27, 2024
61e875d
Checkout `github.head_ref` and repo for PRs
br3ndonland May 31, 2024
c962c33
Dump action as JSON
br3ndonland Jun 1, 2024
1bcf9d8
Check repo owner ID instead of repo name
br3ndonland Jun 9, 2024
80b172c
Check repo ID instead of repo owner ID
br3ndonland Jun 10, 2024
6eb8d30
Fail-fast in unsupported environments
br3ndonland Jun 11, 2024
8d071a8
Drop args from create-docker-action.py
br3ndonland Jun 11, 2024
1ce7f9d
Verify fail-fast in unsupported environments
br3ndonland Jul 10, 2024
5521a4f
Add Docker tags for major and minor versions
br3ndonland Sep 7, 2024
031df10
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 7, 2024
314a411
Move smoke test to reusable workflow
br3ndonland Sep 8, 2024
fa8a0e8
Make smoke test job depend on build job
br3ndonland Sep 14, 2024
580d9d0
Make `workflow_dispatch` Docker tag input required
br3ndonland Sep 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions .github/workflows/build-and-push-docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---

name: 🏗️

on: # yamllint disable-line rule:truthy
pull_request:
push:
branches: ["release/*", "unstable/*"]
workflow_dispatch:
inputs:
tag:
description: Docker image tag
Comment on lines +11 to +12
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

I don't understand what you're asking here. You want separate inputs for major, minor, and patch versions? What if you're building the Docker image from the release/v1 branch? What is the "3-segment version" in that case?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland I was talking about the workflow inputs. Everything else should be computed from them.

The current workflow for me is as follows:

  1. Tag v1.10.1 on top of unstable/v1
  2. Fast-forward release/v1 to v1.10.1
  3. Fast-forward release/v1.10 to v1.10.1 (or create it)
  4. git push --atomic everything in one go
  5. Create a GitHub Release from the UI

With this, my 3-segment “input” is 1.10.1 and v1+v1.10 are being extracted from that. Do you think we could bake containers into this flow with not many changes? This will involve tagging the same image with multiple tags, as I understand. And the end-users referencing versions by both tags and branches should be able to retrieve whatever container there is, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webknjaz thanks for explaining that. I think we're actually pretty close to the desired workflow. There's just a small update needed to the Docker build workflow so it pushes tags with the major and minor version numbers. I've pushed that change.

Your updated workflow would look like this:

  1. Tag v1.10.1 on top of unstable/v1
  2. Fast-forward release/v1 to v1.10.1
  3. Fast-forward release/v1.10 to v1.10.1 (or create it)
  4. git push --atomic everything in one go
  5. Trigger the 🏗️ workflow from the Git tag. The workflow will push the Docker image with three Docker tags:
    1. ghcr.io/pypa/gh-action-pypi-publish:v1.10.1
    2. ghcr.io/pypa/gh-action-pypi-publish:v1.10
    3. ghcr.io/pypa/gh-action-pypi-publish:v1
  6. Create a GitHub Release from the UI

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland so with #45, I wanted to move all these steps inside the automation. For publishing Python projects, I usually use a workflow with workflow_dispatch that allows me to type in the desired version to release and that workflow creates the git tag, the gh release, signs stuff and so on.

The idea is that tag+branches+gh-release represent the result of release automation being successful and aren't triggers or manual actions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland so with #45, I wanted to move all these steps inside the automation. For publishing Python projects, I usually use a workflow with workflow_dispatch that allows me to type in the desired version to release and that workflow creates the git tag, the gh release, signs stuff and so on.

The idea is that tag+branches+gh-release represent the result of release automation being successful and aren't triggers or manual actions.

So... you want the Docker build workflow to run automatically when a tag is created? Or when a GitHub Release is created?

Please be more specific here, and please limit the scope of your requests. The goal of this PR is "Build Docker image and push to GHCR." The goal is not to develop your entire release workflow.

PR #45 is currently a draft and I don't think I should have to make this PR dependent on some other draft PR.

required: true
type: string

jobs:
build-and-push:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: |
DOCKER_TAG="${DOCKER_TAG/'/'/'-'}"
DOCKER_TAG_MAJOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1)
DOCKER_TAG_MAJOR_MINOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1-2)
IMAGE="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG}"
IMAGE_MAJOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR}"
IMAGE_MAJOR_MINOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR_MINOR}"
echo "IMAGE=$IMAGE" >>"$GITHUB_ENV"
echo "IMAGE_MAJOR=$IMAGE_MAJOR" >>"$GITHUB_ENV"
echo "IMAGE_MAJOR_MINOR=$IMAGE_MAJOR_MINOR" >>"$GITHUB_ENV"
docker build . \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from $IMAGE \
--tag $IMAGE
docker tag $IMAGE $IMAGE_MAJOR
docker tag $IMAGE $IMAGE_MAJOR_MINOR
env:
DOCKER_TAG: ${{ inputs.tag || github.ref_name }}
- name: Log in to GHCR
if: github.event_name != 'pull_request'
run: >-
echo ${{ secrets.GITHUB_TOKEN }} |
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
- name: Push Docker image to GHCR
if: github.event_name != 'pull_request'
run: |
docker push $IMAGE
docker push $IMAGE_MAJOR
docker push $IMAGE_MAJOR_MINOR
smoke-test:
br3ndonland marked this conversation as resolved.
Show resolved Hide resolved
needs:
- build-and-push
uses: ./.github/workflows/reusable-smoke-test.yml
122 changes: 122 additions & 0 deletions .github/workflows/reusable-smoke-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
---

name: ♻️ 🧪

on: # yamllint disable-line rule:truthy
workflow_call:

env:
devpi-password: abcd1234
devpi-username: root
devpi-port: 3141

FORCE_COLOR: 1 # Request colored output from CLI tools supporting it
MYPY_FORCE_COLOR: 1 # MyPy's color enforcement
PIP_DISABLE_PIP_VERSION_CHECK: 1
PIP_NO_PYTHON_VERSION_WARNING: 1
PIP_NO_WARN_SCRIPT_LOCATION: 1
PY_COLORS: 1 # Recognized by the `py` package, dependency of `pytest`
TOX_PARALLEL_NO_SPINNER: 1
TOX_TESTENV_PASSENV: >- # Make tox-wrapped tools see color requests
FORCE_COLOR
MYPY_FORCE_COLOR
NO_COLOR
PY_COLORS
PYTEST_THEME
PYTEST_THEME_MODE

jobs:
fail-fast:

strategy:
matrix:
os: [macos-latest, windows-latest]

runs-on: ${{ matrix.os }}

timeout-minutes: 2

steps:
- name: Check out the action locally
uses: actions/checkout@v3
with:
path: test
- name: Fail-fast in unsupported environments
continue-on-error: true
id: fail-fast
uses: ./test
- name: Error if action did not fail-fast in unsupported environments
if: steps.fail-fast.outcome == 'success'
run: |
>&2 echo This action should fail-fast in unsupported environments.
exit 1

smoke-test:

runs-on: ubuntu-latest

services:
devpi:
image: muccg/devpi
env:
DEVPI_PASSWORD: ${{ env.devpi-password }}
ports:
- 3141

timeout-minutes: 2

steps:
- name: Check out the action locally
uses: actions/checkout@v3
with:
path: test
- name: Install the packaging-related tools
run: python3 -m pip install build twine
env:
PIP_CONSTRAINT: test/requirements/runtime.txt
- name: Create the stub package importable directory
run: mkdir -pv src/test_package
- name: Populate the stub package `__init__.py`
run: echo '__version__ = "0.1"' > src/test_package/__init__.py
- name: Populate the stub package `README.md`
run: echo "# Test Package" > README.md
- name: Populate the stub package `pyproject.toml`
run: echo "$CONTENTS" > pyproject.toml
env:
CONTENTS: |
[build-system]
requires = [
"setuptools == 65.6.3",
]
build-backend = "setuptools.build_meta"

[project]
name = "test-package"
version = "0.1"
readme = "README.md"
- name: Build the stub package sdist and wheel distributions
run: python3 -m build
- name: Register the stub package in devpi
run: twine register dist/*.tar.gz
env:
TWINE_USERNAME: ${{ env.devpi-username }}
TWINE_PASSWORD: ${{ env.devpi-password }}
TWINE_REPOSITORY_URL: >-
http://localhost:${{
job.services.devpi.ports[env.devpi-port]
}}/${{
env.devpi-username
}}/public/
- name: ✅ Smoke-test the locally checked out action
uses: ./test
env:
DEBUG: >-
true
PATH: utter-nonsense
with:
user: ${{ env.devpi-username }}
password: ${{ env.devpi-password }}
repository-url: >-
http://devpi:${{ env.devpi-port }}/${{ env.devpi-username }}/public/

...
89 changes: 1 addition & 88 deletions .github/workflows/self-smoke-test-action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,95 +3,8 @@
name: 🧪

on: # yamllint disable-line rule:truthy
push:
pull_request:

env:
devpi-password: abcd1234
devpi-username: root
devpi-port: 3141

FORCE_COLOR: 1 # Request colored output from CLI tools supporting it
MYPY_FORCE_COLOR: 1 # MyPy's color enforcement
PIP_DISABLE_PIP_VERSION_CHECK: 1
PIP_NO_PYTHON_VERSION_WARNING: 1
PIP_NO_WARN_SCRIPT_LOCATION: 1
PY_COLORS: 1 # Recognized by the `py` package, dependency of `pytest`
TOX_PARALLEL_NO_SPINNER: 1
TOX_TESTENV_PASSENV: >- # Make tox-wrapped tools see color requests
FORCE_COLOR
MYPY_FORCE_COLOR
NO_COLOR
PY_COLORS
PYTEST_THEME
PYTEST_THEME_MODE

jobs:
smoke-test:
runs-on: ubuntu-latest

services:
devpi:
image: muccg/devpi
env:
DEVPI_PASSWORD: ${{ env.devpi-password }}
ports:
- 3141

timeout-minutes: 2

steps:
- name: Check out the action locally
uses: actions/checkout@v3
with:
path: test
- name: Install the packaging-related tools
run: python3 -m pip install build twine
env:
PIP_CONSTRAINT: test/requirements/runtime.txt
- name: Create the stub package importable directory
run: mkdir -pv src/test_package
- name: Populate the stub package `__init__.py`
run: echo '__version__ = "0.1"' > src/test_package/__init__.py
- name: Populate the stub package `README.md`
run: echo "# Test Package" > README.md
- name: Populate the stub package `pyproject.toml`
run: echo "$CONTENTS" > pyproject.toml
env:
CONTENTS: |
[build-system]
requires = [
"setuptools == 65.6.3",
]
build-backend = "setuptools.build_meta"

[project]
name = "test-package"
version = "0.1"
readme = "README.md"
- name: Build the stub package sdist and wheel distributions
run: python3 -m build
- name: Register the stub package in devpi
run: twine register dist/*.tar.gz
env:
TWINE_USERNAME: ${{ env.devpi-username }}
TWINE_PASSWORD: ${{ env.devpi-password }}
TWINE_REPOSITORY_URL: >-
http://localhost:${{
job.services.devpi.ports[env.devpi-port]
}}/${{
env.devpi-username
}}/public/
- name: ✅ Smoke-test the locally checked out action
uses: ./test
env:
DEBUG: >-
true
PATH: utter-nonsense
with:
user: ${{ env.devpi-username }}
password: ${{ env.devpi-password }}
repository-url: >-
http://devpi:${{ env.devpi-port }}/${{ env.devpi-username }}/public/

...
uses: ./.github/workflows/reusable-smoke-test.yml
80 changes: 68 additions & 12 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,71 @@ branding:
color: yellow
icon: upload-cloud
runs:
using: docker
image: Dockerfile
args:
- ${{ inputs.user }}
- ${{ inputs.password }}
- ${{ inputs.repository-url }}
- ${{ inputs.packages-dir }}
- ${{ inputs.verify-metadata }}
- ${{ inputs.skip-existing }}
- ${{ inputs.verbose }}
- ${{ inputs.print-hash }}
- ${{ inputs.attestations }}
using: composite
steps:
br3ndonland marked this conversation as resolved.
Show resolved Hide resolved
- name: Fail-fast in unsupported environments
if: runner.os != 'Linux'
run: |
>&2 echo This action is only able to run under GNU/Linux environments
exit 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add CI jobs testing that this fails under corresponding envs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added CI jobs that test failure on macOS and Windows.

shell: bash -eEuo pipefail {0}
- name: Reset path if needed
run: |
# Reset path if needed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this be needed outside the container?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because you've set up a test that modifies the $PATH (#112, 1350b8b).

- name: ✅ Smoke-test the locally checked out action
uses: ./test
env:
DEBUG: >-
true
PATH: utter-nonsense

# https://github.com/pypa/gh-action-pypi-publish/issues/112
if [[ $PATH != *"/usr/bin"* ]]; then
echo "\$PATH=$PATH. Resetting \$PATH for GitHub Actions."
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
echo "PATH=$PATH" >>"$GITHUB_ENV"
echo "$PATH" >>"$GITHUB_PATH"
echo "\$PATH reset. \$PATH=$PATH"
fi
shell: bash
- name: Set repo and ref from which to run Docker container action
id: set-repo-and-ref
run: |
# Set repo and ref from which to run Docker container action
# to handle cases in which `github.action_` context is not set
# https://github.com/actions/runner/issues/2473
REF=${{ env.ACTION_REF || env.PR_REF || github.ref_name }}
REPO=${{ env.ACTION_REPO || env.PR_REPO || github.repository }}
REPO_ID=${{ env.PR_REPO_ID || github.repository_id }}
echo "ref=$REF" >>"$GITHUB_OUTPUT"
echo "repo=$REPO" >>"$GITHUB_OUTPUT"
echo "repo-id=$REPO_ID" >>"$GITHUB_OUTPUT"
shell: bash
env:
ACTION_REF: ${{ github.action_ref }}
ACTION_REPO: ${{ github.action_repository }}
PR_REF: ${{ github.event.pull_request.head.ref }}
PR_REPO: ${{ github.event.pull_request.head.repo.full_name }}
PR_REPO_ID: ${{ github.event.pull_request.base.repo.id }}
- name: Check out action repo
uses: actions/checkout@v4
with:
path: action-repo
ref: ${{ steps.set-repo-and-ref.outputs.ref }}
repository: ${{ steps.set-repo-and-ref.outputs.repo }}
- name: Create Docker container action
run: |
# Create Docker container action
python create-docker-action.py
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if I'm missing something here: why is creating the container action done dynamically like this? Is there a reason it can't be a static file?

Copy link
Contributor Author

@br3ndonland br3ndonland Jun 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the reason is that otherwise there's no way to specify the correct Docker tag.

Docker actions support pulling in pre-built Docker images by supplying a registry address to the image: key. The downside to this syntax is that there's no way to specify the correct Docker tag because the GitHub Actions image: and uses: keys don't accept any context. For example, if a user's workflow has uses: pypa/gh-action-pypi-publish@release/v1.8, then the action should pull in a Docker image built from the release/v1.8 ref, something like ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 (Docker tags can't have /).

# this works but the image tag can't be customized
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8
# this doesn't work because `image:` doesn't support context
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }}

The workaround is to switch the top-level action.yml to a composite action that then creates the Docker container action, substituting the correct image name and tag.

Originally this PR proposed to create the Docker container action as a YAML file with a single placeholder field, like image: {{image}}, then replace the {{image}} placeholder with the image tag using sed. The maintainer preferred to have the entire Docker container action generated with a Python script instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@woodruffw I requested that. I don't want to have another file with hard-coded contents in the repo. There was a template-like thing before. The published container version is being generated dynamically.

env:
EVENT: ${{ github.event_name }}
REF: ${{ steps.set-repo-and-ref.outputs.ref }}
REPO: ${{ steps.set-repo-and-ref.outputs.repo }}
REPO_ID: ${{ steps.set-repo-and-ref.outputs.repo-id }}
shell: bash
working-directory: action-repo
- name: Run Docker container
uses: ./action-repo/.github/actions/run-docker-container
with:
user: ${{ inputs.user }}
password: ${{ inputs.password }}
repository-url: ${{ inputs.repository-url || inputs.repository_url }}
packages-dir: ${{ inputs.packages-dir || inputs.packages_dir }}
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
verify-metadata: ${{ inputs.verify-metadata || inputs.verify_metadata }}
skip-existing: ${{ inputs.skip-existing || inputs.skip_existing }}
verbose: ${{ inputs.verbose }}
print-hash: ${{ inputs.print-hash || inputs.print_hash }}
attestations: ${{ inputs.attestations }}
Loading
Loading