Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autobuild: Create Debian repository on release #3013

Merged
merged 2 commits into from
Jun 12, 2023
Merged

Autobuild: Create Debian repository on release #3013

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c11714b
Autobuild: Create Debian repository on release
ann0see Feb 14, 2023
09a2e17
Linux: Add script to install Debian repository automatically
hoffie Feb 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .github/workflows/autobuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,3 +445,98 @@ jobs:
- name: Perform CodeQL Analysis
if: matrix.config.run_codeql
uses: github/codeql-action/analyze@v2

create_deb_repo:
name: Create files for .deb repository (if requested)
runs-on: ubuntu-22.04
needs: [create_release, release_assets]
if: needs.create_release.outputs.publish_to_release == 'true'
# Set permissions to allow uploading artifact, uploading to release
permissions:
checks: write
contents: write
steps:
- name: Import GPG key
env:
GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
run: |
set -eu

[[ "${GPG_PRIVATE_KEY:-}" ]] || {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forks which don't have a GPG key set, will have a failed build.
I'm not sure if a silent failure would be better. Skipping the repo build should be possible.

There are multiple options:

  • Remove the whole job (easy)
  • Introduce a variable enabling/disabling the repo creation
  • If GPG_PRIVATE_KEY is not present, skip the job. We still need to spin up the container, I'm afraid. GitHub dosn't support checking for present secrets in the .yaml file directly as far as I understand

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will needs.create_release.outputs.publish_to_release == 'true' be true? Is that what you mean by "failed build"?

Copy link
Member Author

@ann0see ann0see Feb 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. If they publish a release it will be true and the job will fail.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Maybe my new approach works.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work: https://github.com/jamulussoftware/jamulus/actions/runs/4494259255/jobs/7906934988

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That run demonstrates that skipping the job works if the GPG_PRIVATE_KEY is not set. I can't see a run that demonstrates the action is working properly when the private key IS set.

I'm assuming that the main jamulus repo has a private key, but the ann0see repo does not. But to test this before merging, I guess either ann0see's repo needs a (temporary?) private key, or perhaps we need to merge these changes into a branch on the jamulus repo, so that it can use the jamulus repo secrets?

I don't profess to be an expert (or even moderately knowledgeable) in Debian packaging or repos. Most of my packaging experience is with RPMs on RH-based systems.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/ann0see/jamulus/releases/tag/r3_9_1devrepo
That's a working (?) repo

echo "Missing Github secret GPG_PRIVATE_KEY. Please set it on GitHub to enable deb repository releases. Skipping step..."
echo "GPG_REPO_KEY_MISSING=true" >> ${GITHUB_ENV}
exit 0
}

echo "GPG_REPO_KEY_MISSING=false" >> ${GITHUB_ENV}
mkdir -p gpghome
chmod 700 gpghome
echo "${GPG_PRIVATE_KEY}" | gpg --homedir gpghome --import -
# Unfortunately download-artifact action doesn't support wild card downloads. Thus downloading all artifacts
Copy link
Member Author

@ann0see ann0see Feb 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this approach is not perfect, I think it's easier to download all artifacts instead of one step per deb file. In future wildcards for downloading only .deb files might be supported.

Alternative: use gh command in CLI

- name: Download all artifacts
if: env.GPG_REPO_KEY_MISSING == 'false'
uses: actions/download-artifact@v3
with:
path: releasedl/
- name: Create Debian repository
if: env.GPG_REPO_KEY_MISSING == 'false'
run: |
set -eu
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be in a separate .sh file. Hoffie once said he's in favor of inlining.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's easier to review in context although it makes the autobuild.yml grow as it becomes more feature-rich. Maybe we need to have a think about it.


# Create and cd into repo directory
mkdir repo
mv releasedl/*.deb/*.deb repo/
pushd repo

# create repo files
apt-ftparchive packages . > Packages
apt-ftparchive release . > Release
gpg --homedir "../gpghome" --armor --yes --clearsign --output InRelease --detach-sign Release
gpg --homedir "../gpghome" --armor --export > "key.asc"

popd

- name: Upload Packages file to release
if: env.GPG_REPO_KEY_MISSING == 'false'
id: deb-upload-packagesfile
uses: actions/upload-release-asset@v1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

release-asset is outdated. Changing to the other action might need some work.

env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ needs.create_release.outputs.upload_url }}
asset_path: repo/Packages
asset_name: Packages
asset_content_type: text/plain
- name: Upload Release file to release
if: env.GPG_REPO_KEY_MISSING == 'false'
id: deb-upload-releasefile
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ needs.create_release.outputs.upload_url }}
asset_path: repo/Release
asset_name: Release
asset_content_type: message/rfc822
- name: Upload InRelease file to release
if: env.GPG_REPO_KEY_MISSING == 'false'
id: deb-upload-inreleasefile
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ needs.create_release.outputs.upload_url }}
asset_path: repo/InRelease
asset_name: InRelease
asset_content_type: text/PGP
- name: Upload Key file to release
if: env.GPG_REPO_KEY_MISSING == 'false'
id: deb-upload-keyascfile
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ needs.create_release.outputs.upload_url }}
asset_path: repo/key.asc
asset_name: key.asc
asset_content_type: application/pgp-keys
32 changes: 32 additions & 0 deletions linux/setup_repo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where should we host this file? Is this repo good or should it be on the website?


# This script installs a Jamulus repository to Debian based systems

if [[ ${EUID} -ne 0 ]]; then
echo "Error: This script must be run as root."
exit 1
fi

REPO_FILE=/etc/apt/sources.list.d/jamulus.list
KEY_FILE=/etc/apt/trusted.gpg.d/jamulus.asc
GITHUB_REPOSITORY="jamulussoftware/jamulus"

echo "Setting up Jamulus repo at ${REPO_FILE}..."
echo "deb https://github.com/${GITHUB_REPOSITORY}/releases/latest/download/ ./" > ${REPO_FILE}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uses a trick: latest points to the last release tagged as "stable" by GitHub. Thus, the link remains unchanged, but a redirect guarantees the correct source.

echo "Installing Jamulus GPG key at ${KEY_FILE}..."
curl --fail --show-error -sLo "${KEY_FILE}" https://github.com/${GITHUB_REPOSITORY}/releases/latest/download/key.asc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just trying this script locally, and this step fails with a 404 for key.asc.

Is this script intended for use by a Github workflow, or by an end user? I can't see it referred to anywhere else in the tree.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this script intended for use by a Github workflow, or by an end user? I can't see it referred to anywhere else in the tree.

Purely by our git build to create a repo for apt to auto update from.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'd need to replace the variables. The release will generate a runnable script.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've currently checked out ann0see/debrepo, but I can't see anywhere that calls setup_repo.sh. I would have expected it to be referred to somewhere within autobuild.yml? Or within another script somewhere in the tree?

You can tell I'm a bit rusty!

Copy link
Member Author

@ann0see ann0see May 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setup_repo.sh is a script which the user should run on his machine to install the repository. The logic to create the repository is in autobuild.yml. I believe the content in autobuild.yaml correctly replaces the content of the setup_repo.sh script and uploads the modified version to the release.
Edit: I think I've removed the replacing logic. You'll need to change the GITHUB_REPOSITORY variable to ann0see/jamulus


CURL_EXITCODE=$?
if [[ ${CURL_EXITCODE} -ne 0 ]]; then
echo "Error: Download of gpg key failed. Please try again later."
exit ${CURL_EXITCODE}
fi

echo "Running apt update..."
apt -qq update
echo "You should now be able to install a full Jamulus package via"
echo " apt install jamulus"
echo "or a server-only, dependency-reduced build via"
echo " apt install jamulus-headless"
echo
echo "This package will automatically be updated when you perform system updates."