Skip to content

Commit

Permalink
fix: Roll back depot_tools, bypass vpython (shaka-project#1045)
Browse files Browse the repository at this point in the history
Using the latest depot_tools no longer works.  depot_tools also wants
to auto-update itself, which must now be disabled.

We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.

Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.

This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.

Closes shaka-project#1023
  • Loading branch information
joeyparrish authored Mar 9, 2022
1 parent 2f90653 commit 3fd538a
Show file tree
Hide file tree
Showing 12 changed files with 101 additions and 36 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/build_and_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,8 @@ jobs:
- name: Install depot tools
shell: bash
run: |
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
touch depot_tools/.disable_auto_update
echo "${GITHUB_WORKSPACE}/depot_tools" >> $GITHUB_PATH
- name: Setup gclient
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/custom-actions/build-packager/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,30 @@ runs:
echo "::endgroup::"
fi
- name: Force Python 2 to support ancient build system (non-Linux only)
if: runner.os != 'Linux'
uses: actions/setup-python@v2
with:
python-version: '2.x'

- name: Force Python 2 to support ancient build system (Linux only)
if: runner.os == 'Linux'
shell: bash
run: |
echo "::group::Install python2"
sudo apt install -y python2
sudo ln -sf python2 /usr/bin/python
echo "::endgroup::"
- name: Install depot tools
shell: bash
run: |
echo "::group::Install depot_tools"
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
touch depot_tools/.disable_auto_update
echo "${GITHUB_WORKSPACE}/depot_tools" >> $GITHUB_PATH
# Bypass VPYTHON included by depot_tools. Prefer the system installation.
echo "VPYTHON_BYPASS=manually managed python not supported by chrome operations" >> $GITHUB_ENV
echo "::endgroup::"
- name: Build ninja (arm only)
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/github_release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,8 @@ jobs:
- name: Install depot tools
shell: bash
run: |
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
touch depot_tools/.disable_auto_update
echo "${GITHUB_WORKSPACE}/depot_tools" >> $GITHUB_PATH
- name: Setup gclient
Expand Down
14 changes: 9 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,28 @@ FROM alpine:3.11 as builder
# Install utilities, libraries, and dev tools.
RUN apk add --no-cache \
bash curl \
bsd-compat-headers linux-headers \
bsd-compat-headers c-ares-dev linux-headers \
build-base git ninja python2 python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH $PATH:/depot_tools

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Alpine uses musl which does not have mallinfo defined in malloc.h. Define the
# structure to workaround a Chromium base bug.
RUN sed -i \
'/malloc_usable_size/a \\nstruct mallinfo {\n int arena;\n int hblkhd;\n int uordblks;\n};' \
/usr/include/malloc.h
ENV GYP_DEFINES='musl=1'

# Bypass VPYTHON included by depot_tools, which no longer works in Alpine.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build shaka-packager from the current directory, rather than what has been
# merged.
WORKDIR shaka_packager
Expand Down
23 changes: 9 additions & 14 deletions docs/source/build_instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,18 @@ GYP_MSVS_OVERRIDE_PATH="C:/Program Files (x86)/Microsoft Visual Studio/2019/Comm

## Install `depot_tools`

### Linux and Mac

Clone the `depot_tools` repository from Chromium:
Clone a particular branch of the `depot_tools` repository from Chromium:

```shell
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
touch depot_tools/.disable_auto_update
```

The latest version of depot_tools will not work, so please use that branch!


### Linux and Mac

Add `depot_tools` to the end of your PATH (you will probably want to put this
in your `~/.bashrc` or `~/.zshrc`). Assuming you cloned `depot_tools` to
`/path/to/depot_tools`:
Expand All @@ -74,17 +78,8 @@ export PATH="$PATH:/path/to/depot_tools"

### Windows

Download the
[depot_tools bundle](https://storage.googleapis.com/chrome-infra/depot_tools.zip)
and extract it somewhere.

**WARNING: DO NOT** use drag-n-drop or copy-n-paste extract from Explorer,
this will not extract the hidden “.git” folder which is necessary for
depot_tools to autoupdate itself. You can use “Extract all…” from the context
menu though.

Add depot_tools to the start of your PATH (must be ahead of any installs of
Python). Assuming you unzipped the bundle to C:\src\depot_tools, open:
Python). Assuming you cloned the repo to C:\src\depot_tools, open:

Control Panel → System and Security → System → Advanced system settings

Expand Down
12 changes: 8 additions & 4 deletions packager/testing/dockers/Alpine_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,24 @@ RUN apk add --no-cache \
bsd-compat-headers c-ares-dev linux-headers \
build-base git ninja python2 python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH $PATH:/depot_tools

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Alpine uses musl which does not have mallinfo defined in malloc.h. Define the
# structure to workaround a Chromium base bug.
RUN sed -i \
'/malloc_usable_size/a \\nstruct mallinfo {\n int arena;\n int hblkhd;\n int uordblks;\n};' \
/usr/include/malloc.h
ENV GYP_DEFINES='musl=1'

# Bypass VPYTHON included by depot_tools, which no longer works in Alpine.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
9 changes: 8 additions & 1 deletion packager/testing/dockers/ArchLinux_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,17 @@ RUN pacman -Sy --needed --noconfirm \
c-ares \
gcc git python2 python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
16 changes: 13 additions & 3 deletions packager/testing/dockers/CentOS_Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,28 @@
FROM centos:8

# Fix up CentOS repo info, which is outdated and not maintained in DockerHub.
# See https://stackoverflow.com/a/71309215
RUN cd /etc/yum.repos.d/
RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*

# Install utilities, libraries, and dev tools.
RUN yum install -y \
which \
c-ares-devel libatomic \
gcc-c++ git python2 python3

# Default to python3.
RUN alternatives --set python /usr/bin/python3
# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
9 changes: 8 additions & 1 deletion packager/testing/dockers/Debian_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,16 @@ RUN apt-get install -y \
libc-ares-dev \
build-essential git python python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
10 changes: 7 additions & 3 deletions packager/testing/dockers/Fedora_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,17 @@ RUN yum install -y \
c-ares-devel libatomic \
gcc-c++ git python2

# Default to python3.
RUN alternatives --install /usr/bin/python python /usr/bin/python3 3
# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
9 changes: 8 additions & 1 deletion packager/testing/dockers/OpenSUSE_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,17 @@ RUN zypper in -y \
c-ares-devel \
gcc-c++ git python python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
WORKDIR /
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".
9 changes: 8 additions & 1 deletion packager/testing/dockers/Ubuntu_Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,16 @@ RUN apt-get install -y \
libc-ares-dev \
build-essential git python python3

# Default to python2 because our build system is ancient.
RUN ln -sf python2 /usr/bin/python

# Install depot_tools.
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN git clone -b chrome/4147 https://chromium.googlesource.com/chromium/tools/depot_tools.git
RUN touch depot_tools/.disable_auto_update
ENV PATH /depot_tools:$PATH

# Bypass VPYTHON included by depot_tools. Prefer the system installation.
ENV VPYTHON_BYPASS="manually managed python not supported by chrome operations"

# Build and run this docker by mapping shaka-packager with
# -v "shaka-packager:/shaka-packager".

0 comments on commit 3fd538a

Please sign in to comment.