-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Containerized CI Pipeline #836
Changes from all commits
d394110
c01bfa8
a66d16f
24c4621
ac69545
558df2a
f491fe7
26dee8b
35a08ed
7a4a744
c52f705
cab17c4
57b88f0
6a9300b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
|
||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
FROM ubuntu:20.04 | ||
|
||
# Install OS packages | ||
RUN export DEBIAN_FRONTEND=noninteractive && \ | ||
apt-get update && \ | ||
apt-get install -y \ | ||
bash \ | ||
build-essential \ | ||
curl \ | ||
expat \ | ||
gcc \ | ||
git \ | ||
libbz2-dev \ | ||
libcurl4-openssl-dev \ | ||
libffi-dev \ | ||
libgmp-dev \ | ||
liblzma-dev \ | ||
libmpfr-dev \ | ||
libncurses-dev \ | ||
libpq-dev \ | ||
libreadline-dev \ | ||
libsqlite3-dev \ | ||
libssl-dev \ | ||
locales \ | ||
make \ | ||
openssl \ | ||
python2-dev \ | ||
python3-dev \ | ||
python3-pip \ | ||
odbc-postgresql \ | ||
unzip \ | ||
wget \ | ||
zip \ | ||
zlib1g \ | ||
zlib1g-dev && \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
# Setup ODBC config | ||
RUN sed -i 's/Driver=psqlodbca.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbca.so/g' /etc/odbcinst.ini && \ | ||
sed -i 's/Driver=psqlodbcw.so/Driver=\/usr\/lib\/x86_64-linux-gnu\/odbc\/psqlodbcw.so/g' /etc/odbcinst.ini && \ | ||
sed -i 's/Setup=libodbcpsqlS.so/Setup=\/usr\/lib\/x86_64-linux-gnu\/odbc\/libodbcpsqlS.so/g' /etc/odbcinst.ini | ||
|
||
# Set the locale | ||
RUN locale-gen --no-purge en_US.UTF-8 | ||
ENV LANG=en_US.UTF-8 \ LANGUAGE=en_US:en \ LC_ALL=en_US.UTF-8 | ||
|
||
# Set user to non-root | ||
ENV HOME /home/github | ||
RUN groupadd -g 1000 github && \ | ||
useradd -m -d "${HOME}" -s /bin/bash -u 1000 -g 1000 github | ||
USER 1000:1000 | ||
WORKDIR "${HOME}" | ||
|
||
# Install pyenv | ||
ENV PYENV_ROOT="${HOME}/.pyenv" | ||
RUN curl https://pyenv.run/ | /bin/bash | ||
ENV PATH="$PYENV_ROOT/bin:$PYENV_ROOT/shims:${PATH}" | ||
RUN echo 'eval "$(pyenv init -)"' >>$HOME/.bashrc && \ | ||
pyenv update | ||
|
||
# Install Python | ||
ARG PYTHON_VERSIONS="3.10 3.9 3.8 3.7 3.11 2.7 pypy2.7 pypy3.7" | ||
COPY --chown=1000:1000 --chmod=+x ./install-python.sh /tmp/install-python.sh | ||
COPY ./requirements.txt /requirements.txt | ||
RUN /tmp/install-python.sh && \ | ||
rm /tmp/install-python.sh |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
# Repository root for mounting into container. | ||
REPO_ROOT:=$(realpath $(dir $(realpath $(firstword $(MAKEFILE_LIST))))../../) | ||
|
||
.PHONY: default | ||
default: test | ||
|
||
.PHONY: build | ||
build: | ||
@# Perform a shortened build for testing | ||
@docker build --build-arg='PYTHON_VERSIONS=3.10 2.7' . -t ghcr.io/newrelic/python-agent-ci:local | ||
|
||
.PHONY: test | ||
test: build | ||
@# Ensure python versions are usable | ||
@docker run --rm ghcr.io/newrelic/python-agent-ci:local /bin/bash -c '\ | ||
python3.10 --version && \ | ||
python2.7 --version && \ | ||
touch tox.ini && tox --version && \ | ||
echo "Success! Python versions installed."' | ||
|
||
.PHONY: run | ||
run: build | ||
@docker run --rm -it \ | ||
--mount type=bind,source="$(REPO_ROOT)",target=/home/github/python-agent \ | ||
--workdir=/home/github/python-agent \ | ||
-e NEW_RELIC_HOST="${NEW_RELIC_HOST}" \ | ||
-e NEW_RELIC_LICENSE_KEY="${NEW_RELIC_LICENSE_KEY}" \ | ||
-e NEW_RELIC_DEVELOPER_MODE="${NEW_RELIC_DEVELOPER_MODE}" \ | ||
ghcr.io/newrelic/python-agent-ci:local /bin/bash |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
#!/bin/bash | ||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
set -e | ||
|
||
SED=$(which gsed || which sed) | ||
|
||
SCRIPT_DIR=$(dirname "$0") | ||
PIP_REQUIREMENTS=$(cat /requirements.txt) | ||
|
||
main() { | ||
# Coerce space separated string to array | ||
if [[ ${#PYTHON_VERSIONS[@]} -eq 1 ]]; then | ||
PYTHON_VERSIONS=($PYTHON_VERSIONS) | ||
fi | ||
|
||
if [[ -z "${PYTHON_VERSIONS[@]}" ]]; then | ||
echo "No python versions specified. Make sure PYTHON_VERSIONS is set." 1>&2 | ||
exit 1 | ||
fi | ||
|
||
# Find all latest pyenv supported versions for requested python versions | ||
PYENV_VERSIONS=() | ||
for v in "${PYTHON_VERSIONS[@]}"; do | ||
LATEST=$(pyenv latest -k "$v" || get_latest_patch_version "$v") | ||
if [[ -z "$LATEST" ]]; then | ||
echo "Latest version could not be found for ${v}." 1>&2 | ||
exit 1 | ||
fi | ||
PYENV_VERSIONS+=($LATEST) | ||
done | ||
|
||
# Install each specific version | ||
for v in "${PYENV_VERSIONS[@]}"; do | ||
pyenv install "$v" & | ||
done | ||
wait | ||
|
||
# Set all installed versions as globally accessible | ||
pyenv global ${PYENV_VERSIONS[@]} | ||
|
||
# Install dependencies for main python installation | ||
pyenv exec pip install --upgrade $PIP_REQUIREMENTS | ||
} | ||
|
||
get_latest_patch_version() { | ||
pyenv install --list | # Get all python versions | ||
$SED 's/^ *//g' | # Remove leading whitespace | ||
grep -E "^$1" | # Find specified version by matching start of line | ||
grep -v -- "-c-jit-latest" | # Filter out pypy JIT versions | ||
$SED -E '/(-[a-zA-Z]+$)|(a[0-9]+)|(b[0-9]+)|(rc[0-9]+)/!{s/$/_/}' | # Append trailing _ to any non development versions to place them lower when sorted | ||
sort -V | # Sort using version sorting | ||
$SED 's/_$//' | # Remove any added trailing underscores to correct version names | ||
tail -1 # Grab last result as latest version | ||
} | ||
|
||
main |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
pip | ||
setuptools | ||
wheel | ||
virtualenv<20.22.1 | ||
tox |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
name: Build CI Image | ||
|
||
on: | ||
workflow_dispatch: # Allow manual trigger | ||
|
||
concurrency: | ||
group: ${{ github.ref || github.run_id }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
with: | ||
persist-credentials: false | ||
fetch-depth: 0 | ||
|
||
- name: Set up Docker Buildx | ||
id: buildx | ||
uses: docker/setup-buildx-action@v2 | ||
|
||
- name: Generate Docker Metadata (Tags and Labels) | ||
id: meta | ||
uses: docker/metadata-action@v4 | ||
with: | ||
images: ghcr.io/${{ github.repository }} | ||
flavor: | | ||
prefix= | ||
suffix= | ||
latest=false | ||
tags: | | ||
type=raw,value=latest,enable={{is_default_branch}} | ||
type=schedule,pattern={{date 'YYYY-MM-DD'}} | ||
type=sha,format=short,prefix=sha- | ||
type=sha,format=long,prefix=sha- | ||
- name: Login to GitHub Container Registry | ||
if: github.event_name != 'pull_request' | ||
uses: docker/login-action@v2 | ||
with: | ||
registry: ghcr.io | ||
username: ${{ github.repository_owner }} | ||
password: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Build and Publish Image | ||
uses: docker/build-push-action@v3 | ||
with: | ||
push: ${{ github.event_name != 'pull_request' }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I thought we decided that we were going to pushing it on each pull request? Did I miss-understand? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We definitely don't want to push it on commits to open PRs. It should normally only push when a PR merges to main. I pulled all the other trigger logic for now and put it down to manual runs until we've got everything wired up. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah ok! Thanks for the explanation! |
||
context: .github/containers | ||
platforms: linux/amd64 | ||
tags: ${{ steps.meta.outputs.tags }} | ||
labels: ${{ steps.meta.outputs.labels }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this latest=false correct here? Seems like it should be true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that disables the auto latest tagging which has some logic we're avoiding