Conda -> PyPI (#331)

* Conda -> PyPI

* Typo

* Make sure to run CI on PRs too

* Move deps to setup.py, add requirements-dev.txt

* Pin tabulate (see Azure/azure-cli#20887)

* Readd tests as part of linting

* Disable import-error

* Publishing: use skip_existing and limit to non-pull-request

* Re-disable logging-fstring-interpolation in linting

.github/workflows/hail_update.yaml

@@ -1,12 +1,12 @@
 # This workflow is triggered after a new version of Hail has been built, with a
-# corresponding conda package. It leads to new driver and server Docker images being
+# corresponding pip package. It leads to new driver and server Docker images being
 # built, followed by the deployment of the server.
 name: Deploy analysis-runner server after Hail update
 on:
   workflow_dispatch:
     inputs:
       hail_version:
-        description: 'Hail version (as uploaded to Anaconda)'
+        description: 'Hail version (as uploaded to PyPI)'
         required: true
 
 jobs:

.github/workflows/lint.yaml

@@ -1,4 +1,4 @@
-name: CI
+name: Lint
 on: [push, pull_request]
 
 jobs:
@@ -11,18 +11,18 @@ jobs:
     steps:
     - uses: actions/checkout@v2
 
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: actions/setup-python@v2
+      with:
+        python-version: '3.10'
+        cache: 'pip'
 
     - name: Install packages
       run: |
-        conda env create --name test-env -f environment-dev.yml
+        pip install .
+        pip install -r requirements-dev.txt
 
     - name: pre-commit
-      run: |
-        conda activate test-env
-        pre-commit run --all-files
+      run: pre-commit run --all-files
 
     - name: Run-tests
-      run: |
-        conda activate test-env
-        python -m unittest test/test_analysis_runner.py
+      run: python -m unittest test/test_analysis_runner.py

.github/workflows/package.yaml

@@ -0,0 +1,42 @@
+name: Package
+on:
+  # Building on pull-requests, manual dispatch, and pushes to main. But restricting
+  # publishing only to main pushes and manual dispatch with `if`s in specific steps
+  pull_request:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+
+jobs:
+  package:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@main
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.10'
+
+      - name: Build
+        run: python setup.py sdist
+
+      - name: Test install
+        run: pip install dist/*
+
+      - name: Run tests
+        run: python -m unittest test/test_analysis_runner.py
+
+      # `skip_existing: true` makes sure that the package will be published
+      # only when new version is created
+      - name: Publish the wheel to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        if: ${{ github.event_name != 'pull_request' }}
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_API_TOKEN }}
+          packages_dir: dist/
+          skip_existing: true

.pylintrc

@@ -2,20 +2,26 @@
 # We disable the following inspections:
 # 1. f-string-without-interpolation (we allow f-strings that don't do any
 #    formatting for consistent looks and for future safety)
-# 2. inherit-non-class ("Inheriting 'NamedTuple', which is not a class" false
+# 2. logging-fstring-interpolation ("Use lazy % formatting in logging functions")
+# 3. inherit-non-class ("Inheriting 'NamedTuple', which is not a class" false
 #    positive, see: https://github.com/PyCQA/pylint/issues/3876)
-# 3. too-few-public-methods (produces false positives)
-# 4. inspections incompatible with Black (see
+# 4. too-few-public-methods (produces false positives)
+# 5. inspections incompatible with Black (see
      https://github.com/psf/black/blob/master/docs/compatible_configs.md#why-those-options-above-2):
 #    C0330: Wrong hanging indentation before block (add 4 spaces)
 #    C0326: Bad whitespace
-# 5. fixme (left 'TODO' lines)
-# 6. line-too-long
-# 7. similar lines
+# 6. fixme (left 'TODO' lines)
+# 7. line-too-long
+# 8. too-many-statements
+# 9. too-many-lines
+# 10. R0801 (duplicate-code)
+# 11. similar lines
+# 12. import-error
 
-disable=f-string-without-interpolation,inherit-non-class,too-few-public-methods,C0330,
-        C0326,fixme,logging-fstring-interpolation,line-too-long,too-many-statements,
-        R0801,too-many-lines
+disable=f-string-without-interpolation,logging-fstring-interpolation,
+        inherit-non-class,too-few-public-methods,C0330,
+        C0326,fixme,line-too-long,too-many-statements,too-many-lines,
+        R0801,import-error
 
 # Overriding variable name patterns to allow short 1- or 2-letter variables
 attr-rgx=[a-z_][a-z0-9_]{0,30}$

README.md

@@ -31,10 +31,10 @@ The analysis-runner is also integrated with our Cromwell server to run WDL based
 ## CLI
 
 The analysis-runner CLI can be used to start pipelines based on a GitHub repository,
-commit, and command to run. To install it, use mamba:
+commit, and command to run. To install it, use pip:
 
 ```bash
-mamba install -c cpg -c conda-forge analysis-runner
+pip install analysis-runner
 ```
 
 Run `analysis-runner --help` to see usage information.
@@ -139,15 +139,13 @@ To bring up a stack corresponding to a dataset as described in the
 [storage policies](https://github.com/populationgenomics/team-docs/tree/main/storage_policies),
 see the [stack](stack) directory.
 
-To set up a development environment for the analysis runner using mamba, run
+To set up a development environment for the analysis runner using pip, run
 the following:
 
 ```bash
-mamba env create --file environment-dev.yml
+pip install -r requirements-dev.txt
 
-conda activate analysis-runner
-
-pre-commit install
+pre-commit install --install-hooks
 
 pip install --editable .
 ```
@@ -156,21 +154,21 @@ pip install --editable .
 
 1. Add a Hail Batch service account for all supported datasets.
 1. [Copy the Hail tokens](tokens) to the Secret Manager.
-1. Deploy the [server](server) by invoking the [`hail_update` workflow](https://github.com/populationgenomics/analysis-runner/blob/main/.github/workflows/hail_update.yaml) manually, specifying the Hail package version in conda.
+1. Deploy the [server](server) by invoking the [`hail_update` workflow](https://github.com/populationgenomics/analysis-runner/blob/main/.github/workflows/hail_update.yaml) manually, specifying the Hail package version.
 1. Deploy the [Airtable](airtable) publisher.
-1. Publish the [CLI tool and library](analysis_runner) to conda.
+1. Publish the [CLI tool and library](analysis_runner) to PyPI.
 
-Note that the [`hail_update` workflow](https://github.com/populationgenomics/analysis-runner/blob/main/.github/workflows/hail_update.yaml) gets invoked whenever a new Hail package is published to conda. You can test this manually as follows:
+Note that the [`hail_update` workflow](https://github.com/populationgenomics/analysis-runner/blob/main/.github/workflows/hail_update.yaml) gets invoked whenever a new Hail package is published to PyPI. You can test this manually as follows:
 
 ```bash
 curl \
   -X POST \
   -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" \
   https://api.github.com/repos/populationgenomics/analysis-runner/actions/workflows/6364059/dispatches \
-  -d '{"ref": "main", "inputs": {"hail_version": "0.2.63.deveb7251e548b1"}}'
+  -d '{"ref": "main", "inputs": {"hail_version": "0.2.84"}}'
 ```
 
-The CLI tool is shipped as a conda package. To build a new version,
+The CLI tool is shipped as a pip package. To build a new version,
 we use [bump2version](https://pypi.org/project/bump2version/).
 For example, to increment the patch section of the version tag 1.0.0 and make
 it 1.0.1, run:
@@ -185,5 +183,4 @@ open "https://github.com/populationgenomics/analysis-runner/pull/new/add-new-ver
 
 It's important the pull request name start with "Bump version:" (which should happen
 by default). Once this is merged into `main`, a GitHub action workflow will build a
-new conda package that will be uploaded to the conda [CPG
-channel](https://anaconda.org/cpg/), and become available to install with `mamba install -c cpg -c conda-forge ...`
+new package that will be uploaded to PyPI, and become available to install with `pip install`.

access_group_cache/Dockerfile

@@ -1,18 +1,12 @@
-FROM python:3.9-slim
+FROM python:3.10-slim
 
 ENV PYTHONUNBUFFERED True
 
-ENV MAMBA_ROOT_PREFIX /root/micromamba
-ENV PATH $MAMBA_ROOT_PREFIX/bin:$PATH
-
-RUN apt-get update && apt-get install -y wget bzip2 && \
-    rm -r /var/lib/apt/lists/* && \
-    rm -r /var/cache/apt/* && \
-    wget -qO- https://api.anaconda.org/download/conda-forge/micromamba/0.8.2/linux-64/micromamba-0.8.2-he9b6cbd_0.tar.bz2 | tar -xvj -C /usr/local bin/micromamba && \
-    mkdir $MAMBA_ROOT_PREFIX && \
-    micromamba install -y --prefix $MAMBA_ROOT_PREFIX -c cpg -c conda-forge \
-    cpg-utils==2.1.2 aiohttp flask gunicorn && \
-    rm -r /root/micromamba/pkgs
+RUN pip3 install  \
+    cpg-utils==2.1.2  \
+    aiohttp \
+    flask \
+    gunicorn
 
 COPY main.py ./
 

analysis_runner/util.py

@@ -150,7 +150,7 @@ def _perform_version_check():
                 f'{current_version} != {latest_version} (current vs latest).\n'
                 f'Your analysis will still be submitted, but may not work as expected.'
                 f' You can update the analysis-runner by running '
-                f'"conda install -c cpg analysis-runner={latest_version}".'
+                f'"pip install analysis-runner={latest_version}".'
             )
             logger.warning(message)
         return

driver/Dockerfile

@@ -14,19 +14,20 @@ RUN apt-get update && apt-get install -y git wget bash bzip2 zip && \
     mkdir $MAMBA_ROOT_PREFIX && \
     micromamba install -y --prefix $MAMBA_ROOT_PREFIX \
         -c cpg -c bioconda -c conda-forge \
-        analysis-runner \
         bokeh \
-        cpg-utils \
         phantomjs \
         r-argparser \
         r-base=4.1.1 \
         r-essentials \
         r-tidyverse \
-        sample-metadata=4.1.1 \
         selenium \
         skopeo \
         statsmodels && \
     rm -r /root/micromamba/pkgs && \
-    pip3 install cpg-hail=$HAIL_VERSION && \
+    pip3 install \
+        cpg-hail=$HAIL_VERSION \
+        cpg-utils=2.1.1 \
+        analysis-runner \
+        sample-metadata=4.1.2 && \
     # hailctl dataproc uses gcloud beta dataproc.
     gcloud -q components install beta

examples/dataproc/README.md

@@ -1,6 +1,6 @@
 # Dataproc example
 
-This example shows how to run a Hail query script in Dataproc using Hail Batch. After installing the conda package for the analysis runner, you can run this as follows:
+This example shows how to run a Hail query script in Dataproc using Hail Batch. After installing the pip package for the analysis runner, you can run this as follows:
 
 ```bash
 cd examples/dataproc

requirements-dev.txt

@@ -0,0 +1,4 @@
+black
+bump2version
+pre-commit
+pylint

server/Dockerfile

@@ -2,15 +2,13 @@
 ARG DRIVER_IMAGE
 FROM ${DRIVER_IMAGE}
 
-RUN micromamba install --prefix $MAMBA_ROOT_PREFIX -c cpg -c conda-forge \
-    cpg-utils==2.1.1 \
+RUN pip install \
     google-api-python-client==2.10.0 \
     google-cloud-secret-manager==2.2.0 \
     google-cloud-pubsub==2.3.0 \
     google-cloud-logging==2.7.0 \
     gunicorn \
-    requests && \
-    rm -r /root/micromamba/pkgs
+    requests
 
 # Allow statements and log messages to immediately appear in the Knative logs.
 ENV PYTHONUNBUFFERED 1