Use micromamba based container for CI with GitHub Actions

[sblauth]

Hello everyone,

I have an issue using the micromamba docker image in a CI workflow in github actions. To test this, I have also created a minimal working example in this repo: https://github.com/sblauth/docker_mwe

My goal is the following: I want to create a docker container with some dependencies of my software, which I want to install using micromamba. I want to push this image to a registry (say ghcr.io) and then use this in another workflow to install my software in the container directly and then run some tests.

I use the following dockerfile for the container

FROM mambaorg/micromamba:1-jammy AS test

USER root

RUN apt-get update --fix-missing && \
    apt-get install -y libgl1-mesa-dev ffmpeg libsm6 libxext6 curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

USER $MAMBA_USER

RUN micromamba install -y -n base -c conda-forge numpy

ARG MAMBA_DOCKERFILE_ACTIVATE=1

RUN python -c "import numpy"

RUN micromamba list

For this test, I simply use numpy to demonstrate the issue. Also, the final two RUN commands are used for debugging.

Next, I use the following workflow file to build and push the image to ghcr.io

name: ghcr.io images

on:
  workflow_dispatch:
  push:
  schedule:
    - cron: '17 21 * * *'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-push-image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3.5.3

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./docker/Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

These two steps work as expected, the image is built and numpy is installed.

Now, I try to use this image in another workflow. Normally, I would install my software here, but for this test case, I just check whether numpy is installed correctly. I use the following workflow

name: Tests

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '17 22 * * *'

env:
  OMP_NUM_THREADS: 2
    
jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/sblauth/docker_mwe:main

    steps:
    - name: Test installation
      run: |
        micromamba list
        micromamba info
        which micromamba
        python -c "import numpy"

And here is the issue: Somehow, the environment is either not activated correctly or there is some other issue, but I get the following output

Run micromamba list
  micromamba list
  micromamba info
  which micromamba
  python -c "import numpy"
  shell: sh -e {0}
  env:
    OMP_NUM_THREADS: 2
List of packages in environment: ""


       libmamba version : 1.4.9
     micromamba version : 1.4.9
           curl version : libcurl/7.88.1 OpenSSL/3.1.1 zlib/1.2.13 zstd/1.5.2 libssh2/1.11.0 nghttp2/1.52.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       envs directories : /opt/conda/envs
          package cache : /opt/conda/pkgs
                          /github/home/.mamba/pkgs
            environment : None (not found)
           env location : -
      user config files : /github/home/.mambarc
 populated config files : 
       virtual packages : __unix=0=0
                          __linux=5.15.0=0
                          __glibc=2.35=0
                          __archspec=1=x86_64
               channels : 
       base environment : /opt/conda
               platform : linux-64
/usr/bin/micromamba
/__w/_temp/b4c93c22-5fa0-48c8-8dee-d92be776c054.sh: 4: python: not found
Error: Process completed with exit code 127.

I have tried several ideas (using shell: bash or bash -l {0}) but to no success. Does anyone know how to use the image that one creates this way in a github actions workflow?

Thanks a lot in advance,
Sebastian

[wholtz]

Hi @sblauth thank you for your interest in micromamba and thank you for the very nice reproduction of your issue. I'm surprised it has taken this long for someone to report this.

Github Actions in containers do not execute entrypoint scripts, which is one of the two ways we attempt to automatically activate the base environment. The other way is via ~/.bashrc, but GitHub uses /github/home as the home directory and not the /home/mambauser that we have pre-configured. Therefore our activate script never is executed and you don't get automatic activation.

Two small changes are required in your test in order to get the environment activated:

    steps:
    - name: environment activation
      shell: bash
      run: |
        source /usr/local/bin/_entrypoint.sh
        micromamba list
        micromamba info
        which micromamba
        python -c "import numpy"

First, shell: bash as we need bash and GitHub Actions default to sh. Then the entrypoint script is explicitly sourced.

Please let me know if that works for you.

[wholtz]

I will make an addition to our documentation that explains how to use our container with GitHub Actions.....

[sblauth]

Hi @wholtz,
thanks a lot for your quick reply! I already figured that it had something to do with the entrypoint and your solution works great for the test repo I have set up.
I will keep the repository alive for future users to see and will now try this on the actual repo.

And yeah, I think the documentation would benefit from this ;)

Thanks a lot again!

[sblauth]

Just a quick note:
I had another issue that I could fix. When I used this approach, everything worked fine for the demo example, but I ran into trouble using the actions/checkout@v3.5.3 action to checkout my repository. I could, however, fix this by using the option
options: --user=root

So the workflow is defined as

name: Tests

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '17 22 * * *'

env:
  OMP_NUM_THREADS: 2
    
jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/sblauth/docker_mwe:main
      options: --user=root

    steps:
    - name: environment activation
      shell: bash
      run: |
        source /usr/local/bin/_entrypoint.sh
        micromamba list
        micromamba info
        which micromamba
        python -c "import numpy"
        micromamba run -n base python -c "import numpy"

If someone knows a "nicer" workaround, please let me know.

[maresb]

Thanks a lot @sblauth for the feedback.

but I ran into trouble

Could you be more specific about the issue you had?

@WHolz, I am wondering if it would be better to replace shell: bash with shell: /usr/local/bin/_entrypoint.sh? This seems to me more in line with how the entrypoint script actually works. (I have not tested it.)

[wholtz]

@maresb - I like your idea. Unfortunately my testing revealed it is a bit more complex than I expected.

This works:

jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/sblauth/docker_mwe:main
    steps:
    - name: environment activation
      shell: _entrypoint.sh /bin/bash --noprofile --norc -eo pipefail {0}
      run: |
        micromamba list
        micromamba info
        which micromamba
        python -c "import numpy"

shell: throws an error if given the full path to _entrypoint.sh, so this is slightly brittle. (There is something weird going on. You can give a full path to /bin/bash.) And then the script that gets passed to the shell isn't executable, so we need to add /bin/bash in there. If you use a non-standard shell, you need to add {0} (there is an automatic expansion of the standard shells to this form). And then I copied the bash flags that are also part of the automatic expansion when using shell: bash.

Reference: https://github.com/actions/runner/blob/main/docs/adrs/0277-run-action-shell-options.md

[sblauth]

Thanks a lot @sblauth for the feedback.

but I ran into trouble

Could you be more specific about the issue you had?

I think it's something along the lines of actions/checkout#47
My error message was the following

/usr/bin/docker exec  22dfc1a271c6a85094fda4c6f7cae8ae85d15c4df010169be4a19eac687359a9 sh -c "cat /etc/*release | grep ^ID"
node:internal/fs/utils:345
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_3b69fc4d-9fb0-4c8c-83e5-c6a241d2cb9e'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2170:35)
    at Object.appendFileSync (node:fs:2232:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:2945:8)
    at Object.saveState (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:2862:31)
    at Object.8647 (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:2321:10)
    at __nccwpck_require__ (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:18251:43)
    at Object.2565 (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:146:34)
    at __nccwpck_require__ (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:18251:43)
    at Object.9210 (/__w/_actions/actions/checkout/v3.5.3/dist/index.js:1141:36) {
  errno: -13,
  syscall: 'open',
  code: 'EACCES',
  path: '/__w/_temp/_runner_file_commands/save_state_3b69fc4d-9fb0-4c8c-83e5-c6a241d2cb9e'
}

[wholtz]

Added to FAQ in #357