Docker backend

.github/workflows/main.sh

@@ -2,8 +2,6 @@
 set -eux
 export OPAMYES=true
 
-sudo chmod a+x /usr/local/bin/runc
-
 sudo sh -c "cat > /usr/local/bin/uname" << EOF
 #!/bin/sh
 
@@ -19,6 +17,8 @@ opam exec -- make
 
 case "$1" in
     btrfs)
+        sudo chmod a+x /usr/local/bin/runc
+
         dd if=/dev/zero of=/tmp/btrfs.img bs=100M count=50
         BTRFS_LOOP=$(sudo losetup -f)
         sudo losetup -P "$BTRFS_LOOP" /tmp/btrfs.img
@@ -43,6 +43,8 @@ case "$1" in
         ;;
 
     zfs)
+        sudo chmod a+x /usr/local/bin/runc
+
         dd if=/dev/zero of=/tmp/zfs.img bs=100M count=50
         ZFS_LOOP=$(sudo losetup -f)
         sudo losetup -P "$ZFS_LOOP" /tmp/zfs.img
@@ -83,6 +85,8 @@ case "$1" in
     #     ;;
 
     rsync_hardlink)
+        sudo chmod a+x /usr/local/bin/runc
+
         sudo mkdir /rsync
         sudo chown "$(whoami)" /rsync
 
@@ -99,7 +103,9 @@ case "$1" in
         sudo rm -rf /rsync
         ;;
 
-   rsync_copy)
+    rsync_copy)
+        sudo chmod a+x /usr/local/bin/runc
+
         sudo mkdir /rsync
         sudo chown "$(whoami)" /rsync
 
@@ -115,6 +121,23 @@ case "$1" in
 
         sudo rm -rf /rsync
         ;;
+
+    docker)
+       sudo mkdir /var/lib/obuilder
+       sudo chown "$(whoami)" /var/lib/obuilder
+
+       opam exec -- dune exec -- obuilder healthcheck --store=docker:/var/lib/obuilder
+
+       # Populate the caches from our own GitHub Actions cache
+       sudo mkdir -p /var/lib/obuilder/cache/c-opam-archives
+       sudo cp -r ~/.opam/download-cache/* /var/lib/obuilder/cache/c-opam-archives/
+       sudo chown -R 1000:1000 /var/lib/obuilder/cache/c-opam-archives
+
+       opam exec -- dune exec -- obuilder build -f example.spec . --store=docker:/var/lib/obuilder --color=always
+
+       sudo rm -rf /var/lib/obuilder
+       ;;
+
     *)
         printf "Usage: .run-gha-tests.sh [btrfs|rsync_hardlink|rsync_copy|zfs]" >&2
         exit 1

.github/workflows/main.yml

@@ -128,3 +128,32 @@ jobs:
       - run: opam install . --deps-only --with-test
 
       - run: opam exec -- dune runtest
+
+  docker:
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - ubuntu-latest
+        ocaml-compiler:
+          - 4.14.x
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Use OCaml ${{ matrix.ocaml-compiler }}
+        uses: ocaml/setup-ocaml@v2
+        with:
+          ocaml-compiler: ${{ matrix.ocaml-compiler }}
+
+      - run: opam install . --deps-only --with-test
+
+      # - name: Set up QEMU
+      #   uses: docker/setup-qemu-action@v2
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - run: $GITHUB_WORKSPACE/.github/workflows/main.sh docker

CHANGES.md

@@ -1,5 +1,7 @@
 ### v0.5.1
 
+- Add a Docker backend for Windows and Linux jobs.
+  (@MisterDA #127 #75, reviewed by @talex5, @tmcgilchrist)
 - Updates to address rsync and sandbox issues.
   (@mtelvers #139, reviewed by @tmcgilchrist and @MisterDA)
 - Add an obuilder clean command to clean all build results.

README.md

@@ -7,11 +7,14 @@
 
 OBuilder takes a build script (similar to a Dockerfile) and performs the steps in it in a sandboxed environment.
 
-After each step, OBuilder uses the snapshot feature of the filesystem (ZFS or Btrfs) to store the state of the build. There is also an Rsync backend that copies the build state.
+After each step, OBuilder uses the snapshot feature of the filesystem (ZFS or
+Btrfs) to store the state of the build. There is also an Rsync backend that
+copies the build state. On Linux, it uses `runc` to sandbox the build steps, but
+any system that can run a command safely in a chroot could be used.
 Repeating a build will reuse the cached results where possible.
 
-OBuilder aims to be portable, although currently only Linux support is present.
-On Linux, it uses `runc` to sandbox the build steps, but any system that can run a command safely in a chroot could be used.
+OBuilder can also use Docker as a backend (fully replacing of `runc` and the
+snapshotting filesystem) on any system supported by Docker (Linux, Windows, …).
 
 OBuilder stores the log output of each build step.
 This is useful for CI, where you may still want to see the output even if the result was cached from some other build.
@@ -105,8 +108,8 @@ The initial context is supplied by the user (see [build.mli](lib/build.mli) for
 By default:
 - The environment is taken from the Docker configuration of `BASE`.
 - The user is `(uid 0) (gid 0)` on Linux, `(name ContainerAdministrator)` on Windows.
-- The workdir is `/`.
-- The shell is `/bin/bash -c`.
+- The workdir is `/`, `C:/` on Windows.
+- The shell is `/bin/bash -c`, `C:\Windows\System32\cmd.exe /S /C` on Windows.
 
 ### Multi-stage builds
 
@@ -130,7 +133,6 @@ For example:
 
 At the moment, the `(build …)` items must appear before the `(from …)` line.
 
-
 ### workdir
 
 ```sexp
@@ -169,7 +171,6 @@ The command run will be this list of arguments followed by the single argument `
  (network NETWORK…)?
  (secrets SECRET…)?
  (shell COMMAND))
-
 ```
 
 Examples:
@@ -210,9 +211,9 @@ the image. Each `SECRET` entry is under the form `(ID (target PATH))`, where `ID
 `PATH` is the location of the mounted secret file within the container.
 The sandbox context API contains a `secrets` parameter to provide values to the runtime.
 If a requested secret isn't provided with a value, the runtime fails.
-With the command line interface `obuilder`, use the `--secret ID:PATH` option to provide the path of the file
-containing the secret for `ID`.
-When used with Docker, make sure to use the **buildkit** syntax, as only buildkit supports a `--secret` option.
+Use the `--secret ID:PATH` option to provide the path of the file containing the
+secret for `ID`.
+When used with Docker, make sure to use the **BuildKit** syntax, as only BuildKit supports a `--secret` option.
 (See https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information)
 
 ### copy
@@ -261,8 +262,14 @@ Notes:
 
 - Both `SRC` and `DST` use `/` as the directory separator on all platforms.
 
-- The copy is currently done by running `tar` inside the container to receive the files.
-  Therefore, the filesystem must have a working `tar` binary.
+- The copy is currently done by running `tar` inside the container to receive
+  the files. Therefore, the filesystem must have a working `tar` binary. On
+  Windows when using the Docker backend, OBuilder provides a `tar` binary.
+
+- On Windows, copying from a build step image based on [Nano Server][nanoserver]
+  isn't supported.
+
+[nanoserver]: https://hub.docker.com/_/microsoft-windows-nanoserver
 
 ### user
 
@@ -312,10 +319,10 @@ The dockerfile should work the same way as the spec file, except for these limit
 - All `(network …)` fields are ignored, as Docker does not allow per-step control of
   networking.
 
-## Experimental macOS Support
+## Experimental macOS and Windows Support
 
 OBuilder abstracts over a fetching mechanism for the Docker base image, the sandboxing for the execution of build steps and the store for the cache.
-This makes OBuilder extremely portable and there exists a (very) experimental [macOS][] backend.
+This makes OBuilder extremely portable and there exists experimental [macOS][] and [Windows][] backends. The Windows backend currently requires Docker for Windows installed.
 
 ## Licensing
 
@@ -326,6 +333,7 @@ See [LICENSE][] for the full license text.
 [OCluster]: https://github.com/ocurrent/ocluster
 [LICENSE]: ./LICENSE
 [macOS]: ./macOS.md
+[Windows]: ./windows.md
 
 [github-shield]: https://github.com/ocurrent/obuilder/actions/workflows/main.yml/badge.svg
 [github-ci]: https://github.com/ocurrent/obuilder/actions/workflows/main.yml

dune-project

@@ -37,7 +37,9 @@
   ppx_sexp_conv
   (sha (>= 1.15.4))
   sqlite3
+  (crunch (and (>= 3.3.1) :build))
   (obuilder-spec (= :version))
+  fpath
   (ocaml (>= 4.14.1))
   (alcotest-lwt (and (>= 1.7.0) :with-test))))