Introduce a Docker backend in OBuilder for Windows and Linux

.github/workflows/main.sh

@@ -2,8 +2,6 @@
 set -eux
 export OPAMYES=true
 
-sudo chmod a+x /usr/local/bin/runc
-
 sudo sh -c "cat > /usr/local/bin/uname" << EOF
 #!/bin/sh
 
@@ -19,6 +17,8 @@ opam exec -- make
 
 case "$1" in
     btrfs)
+        sudo chmod a+x /usr/local/bin/runc
+
         dd if=/dev/zero of=/tmp/btrfs.img bs=100M count=50
         BTRFS_LOOP=$(sudo losetup -f)
         sudo losetup -P "$BTRFS_LOOP" /tmp/btrfs.img
@@ -43,6 +43,8 @@ case "$1" in
         ;;
 
     zfs)
+        sudo chmod a+x /usr/local/bin/runc
+
         dd if=/dev/zero of=/tmp/zfs.img bs=100M count=50
         ZFS_LOOP=$(sudo losetup -f)
         sudo losetup -P "$ZFS_LOOP" /tmp/zfs.img
@@ -83,6 +85,8 @@ case "$1" in
     #     ;;
 
     rsync_hardlink)
+        sudo chmod a+x /usr/local/bin/runc
+
         sudo mkdir /rsync
         sudo chown "$(whoami)" /rsync
 
@@ -99,7 +103,9 @@ case "$1" in
         sudo rm -rf /rsync
         ;;
 
-   rsync_copy)
+    rsync_copy)
+        sudo chmod a+x /usr/local/bin/runc
+
         sudo mkdir /rsync
         sudo chown "$(whoami)" /rsync
 
@@ -115,6 +121,23 @@ case "$1" in
 
         sudo rm -rf /rsync
         ;;
+
+    docker)
+       sudo mkdir /var/lib/obuilder
+       sudo chown "$(whoami)" /var/lib/obuilder
+
+       opam exec -- dune exec -- obuilder healthcheck --store=docker:/var/lib/obuilder
+
+       # Populate the caches from our own GitHub Actions cache
+       sudo mkdir -p /var/lib/obuilder/cache/c-opam-archives
+       sudo cp -r ~/.opam/download-cache/* /var/lib/obuilder/cache/c-opam-archives/
+       sudo chown -R 1000:1000 /var/lib/obuilder/cache/c-opam-archives
+
+       opam exec -- dune exec -- obuilder build -f example.spec . --store=docker:/var/lib/obuilder --color=always
+
+       sudo rm -rf /var/lib/obuilder
+       ;;
+
     *)
         printf "Usage: .run-gha-tests.sh [btrfs|rsync_hardlink|rsync_copy|zfs]" >&2
         exit 1

.github/workflows/main.yml

@@ -128,3 +128,32 @@ jobs:
       - run: opam install . --deps-only --with-test
 
       - run: opam exec -- dune runtest
+
+  docker:
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - ubuntu-latest
+        ocaml-compiler:
+          - 4.14.x
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Use OCaml ${{ matrix.ocaml-compiler }}
+        uses: ocaml/setup-ocaml@v2
+        with:
+          ocaml-compiler: ${{ matrix.ocaml-compiler }}
+
+      - run: opam install . --deps-only --with-test
+
+      # - name: Set up QEMU
+      #   uses: docker/setup-qemu-action@v2
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - run: ./.run-gha-tests.sh docker

CHANGES.md

@@ -1,5 +1,7 @@
 ### v0.5.1
 
+- Add a Docker backend for Windows and Linux jobs.
+  (@MisterDA #127 #75, reviewed by @talex5, @tmcgilchrist)
 - Updates to address rsync and sandbox issues.
   (@mtelvers #139, reviewed by @tmcgilchrist and @MisterDA)
 - Add an obuilder clean command to clean all build results.

README.md

@@ -7,11 +7,14 @@
 
 OBuilder takes a build script (similar to a Dockerfile) and performs the steps in it in a sandboxed environment.
 
-After each step, OBuilder uses the snapshot feature of the filesystem (ZFS or Btrfs) to store the state of the build. There is also an Rsync backend that copies the build state.
+After each step, OBuilder uses the snapshot feature of the filesystem (ZFS or
+Btrfs) to store the state of the build. There is also an Rsync backend that
+copies the build state. On Linux, it uses `runc` to sandbox the build steps, but
+any system that can run a command safely in a chroot could be used.
 Repeating a build will reuse the cached results where possible.
 
-OBuilder aims to be portable, although currently only Linux support is present.
-On Linux, it uses `runc` to sandbox the build steps, but any system that can run a command safely in a chroot could be used.
+OBuilder can also use Docker as a backend (fully replacing of `runc` and the
+snapshotting filesystem) on any system supported by Docker (Linux, Windows, …).
 
 OBuilder stores the log output of each build step.
 This is useful for CI, where you may still want to see the output even if the result was cached from some other build.
@@ -73,6 +76,11 @@ pass the `--fast-sync` option, which installs a seccomp filter that skips all
 sync syscalls. However, if you attempt to use this with an earlier version of
 runc then sync operations will instead fail with `EPERM`.
 
+### Windows
+
+The user running OBuilder must have access to `%PROGRAMDATA%\Docker\volumes`,
+because copying caches and maintaining internal tools is done directly on the host.
+
 ## The build specification language
 
 The spec files are loosely based on the [Dockerfile][] format.
@@ -105,8 +113,8 @@ The initial context is supplied by the user (see [build.mli](lib/build.mli) for
 By default:
 - The environment is taken from the Docker configuration of `BASE`.
 - The user is `(uid 0) (gid 0)` on Linux, `(name ContainerAdministrator)` on Windows.
-- The workdir is `/`.
-- The shell is `/bin/bash -c`.
+- The workdir is `/`, `C:/` on Windows.
+- The shell is `/bin/bash -c`, `C:\Windows\System32\cmd.exe /S /C` on Windows.
 
 ### Multi-stage builds
 
@@ -130,7 +138,6 @@ For example:
 
 At the moment, the `(build …)` items must appear before the `(from …)` line.
 
-
 ### workdir
 
 ```sexp
@@ -169,7 +176,6 @@ The command run will be this list of arguments followed by the single argument `
  (network NETWORK…)?
  (secrets SECRET…)?
  (shell COMMAND))
-
 ```
 
 Examples:
@@ -210,9 +216,9 @@ the image. Each `SECRET` entry is under the form `(ID (target PATH))`, where `ID
 `PATH` is the location of the mounted secret file within the container.
 The sandbox context API contains a `secrets` parameter to provide values to the runtime.
 If a requested secret isn't provided with a value, the runtime fails.
-With the command line interface `obuilder`, use the `--secret ID:PATH` option to provide the path of the file
-containing the secret for `ID`.
-When used with Docker, make sure to use the **buildkit** syntax, as only buildkit supports a `--secret` option.
+Use the `--secret ID:PATH` option to provide the path of the file containing the
+secret for `ID`.
+When used with Docker, make sure to use the **BuildKit** syntax, as only BuildKit supports a `--secret` option.
 (See https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information)
 
 ### copy
@@ -261,8 +267,14 @@ Notes:
 
 - Both `SRC` and `DST` use `/` as the directory separator on all platforms.
 
-- The copy is currently done by running `tar` inside the container to receive the files.
-  Therefore, the filesystem must have a working `tar` binary.
+- The copy is currently done by running `tar` inside the container to receive
+  the files. Therefore, the filesystem must have a working `tar` binary. On
+  Windows when using the Docker backend, OBuilder provides a `tar` binary.
+
+- On Windows, copying from a build step image based on [Nano Server][nanoserver]
+  isn't supported.
+
+[nanoserver]: https://hub.docker.com/_/microsoft-windows-nanoserver
 
 ### user
 

dune-project

@@ -25,8 +25,10 @@
   ppx_sexp_conv
   (sha (>= 1.15.4))
   sqlite3
+  (crunch (and (>= 3.3.1) :build))
   (obuilder-spec (= :version))
   (ocaml (>= 4.14.0))
+  fpath
   (alcotest-lwt :with-test)))
 (package
  (name obuilder-spec)

example.spec

@@ -7,11 +7,11 @@
 ; The result can then be found in /tank/HASH/rootfs/ (where HASH is displayed at the end of the build).
 
 ((build dev
-	((from ocaml/opam@sha256:cff69c5fda1e83f65d5f970ccea7b3d53bb853cfdcb0a5a6c4cbfae8c86752ae)
+        ((from ocaml/opam@sha256:18fb2672085fbf0692232c604b3dbf0e93a6382b62726641b56eb65c5bc58cdc)
 	 (workdir /src)
 	 (user (uid 1000) (gid 1000))                           ; Build as the "opam" user
 	 (run (shell "sudo chown opam /src"))
-	 (env OPAM_HASH "1443067283649a4e09423a4a9a7b082c70e0227b")
+	 (env OPAM_HASH "ceb238f83253b0767811797f093b75114778ec66") ; Fix the version of opam-repository we want
 	 (run
 	  (network host)
 	  (shell "sudo apt-get --allow-releaseinfo-change update"))

example.windows.spec

@@ -33,6 +33,11 @@
 	  (network "nat")
 	  (cache (opam-archives (target /opam/.opam/download-cache)))
 	  (shell "ocaml-env exec --64 -- opam pin add -yn ."))
+	 ; Pin dependencies outdated in fdopen/opam-repository-mingw
+	 (run
+	  (network "nat")
+	  (cache (opam-archives (target /opam/.opam/download-cache)))
+	  (shell "ocaml-env exec --64 -- opam pin add -yn sha.1.15.4 https://github.com/djs55/ocaml-sha/releases/download/v1.15.4/sha-1.15.4.tbz"))
 	 ; Install OS package dependencies
 	 (run
 	  (network "nat")