Chroot builds are slow

[edolstra]

Chroot builds have a significant overhead. For instance, this expression:

with import <nixpkgs> {};
with lib;
let deps = map (n: runCommand "depM-${toString n}" {} "touch $out") (range 1 100);
in runCommand "foo" { x = deps; } "touch $out"

(i.e. a trivial build with 100 trivial dependencies) takes 4.7s to build on my laptop without chroot, but 39.6s with chroot.

The main overhead seems to be in setting up the chroot (i.e. all the bind mounts), but the various CLONE_* flags also have significant overhead.

Unfortunately, there is not much we can do about this since it's all in the kernel, but it does mean we can't enable chroot builds by default on NixOS.

This is on Linux 3.4.70.

[vcunat]

Oh, not good. Are there some other standard sandboxing options? (except for LD_PRELOADing some libc hooks)

[edolstra]

I can imagine a cheaper chroot that just bind-mounts the entire Nix store. Maybe we could even put an ACL on /nix/store to deny "r" but not "x" permission to nixbld users. That way builds can only access store paths that they already know.

Also, maybe it's faster on newer kernels.

[vcunat]

Well, I don't think packages try finding something by listing /nix/store. In general, maybe we could deny "r" on it for everyone, but I fail to see any significant gain.

Maybe providing some cheap variant of chroot by default could be a good compromise (with possibility to switch to stronger guarantees).

[peti]

The benefits of chrooted builds are far more significant than the performance cost. Chroot builds should be totally enabled on NixOS by default!!!

[alexanderkjeldaas]

Are the bind mounts done in parallel?

[edolstra]

No.

[domenkozar]

@edolstra I'd still prefer purity/determinism over performance and enable chroots on Linux by default.

[domenkozar]

On my machine (SDD, running kernel 3.14):

real 0m27.129s
user 0m0.139s
sys 0m0.038s

[vcunat]

@iElectric: are you sure your measurement is correct? It shows mostly waiting and no real work. Or is that because the work is in fact done in another process?

[wmertens]

👍 for a mini chroot that has all of nix store. This could be reused, no? Same chroot for all builds?

[domenkozar]

@vcunat i'd say most of the time it's waiting for nix-daemon IO

[aristidb]

Computers have become faster in the past 2 years. We should re-evaluate whether the speed is really worth the significant impurities.

Note that the fact that NixOS default Hydra not using chroot leads to packages "randomly" failing to build locally for those who do use it.

So at least Hydra should enable it.

[edolstra]

Hydra does use it. It's the other way around, users like me might not have it enabled and think that a package builds properly when it doesn't. (Happened today with a PHP update, which turns out to do a download during its build.)

[domenkozar]

Yes, leaving our deterministic promise aside for a sake of some small overhead.

[benley]

When nix sets up chroots, is most of the time spent setting up bind mounts? Or does it do a lot of file copying too? If the latter, have you considered using something like Cowdancer (http://www.netfort.gr.jp/~dancer/software/cowdancer.html.en) to get copy-on-write bind mounts? It's low-overhead and fast to set up. Debian uses it in cowbuilder/pbuilder, which makes for an excellent ephemeral-chrooted build system.

[vcunat]

@benley: COW isn't needed, as all accessible in the chroot is read-only anyway. From the comments it seems noone has analyzed precisely what's the main cost, but bind mounts are suspected (and they probably were never meant to be used so massively).

[copumpkin]

Has anyone looked into proot for this purpose?

[edolstra]

"PRoot uses the CPU emulator QEMU user-mode to execute transparently guest programs." I doubt that's faster than bind mounts :-)

[vcunat]

Yeah, PRoot might be faster to setup, but it sounds significantly slower to run longer builds (which happen a lot). Various other preloading solutions might also slow down system calls, although probably not so much.

[copumpkin]

@edolstra oh sorry, my understanding was that it only used QEMU when the guest was of a different architecture

[benley]

I believe proot only uses qemu when it's running binaries from a non-native architecture. The proot website is fairly clear about that, unless I'm badly misinterpreting it: http://proot.me/

[benley]

It does still intercept system calls in userland, and it's going to have some unavoidable speed overhead.

[alexanderkjeldaas]

Isn't it documented to use ptrace? If so it will signal the controlling
process and wait for a command on every syscall that is intercepted.

On Tue, Jan 20, 2015 at 12:47 AM, Benjamin Staffin <notifications@github.com

wrote:

It does still intercept system calls in userland, and it's going to have
some unavoidable speed overhead.

—
Reply to this email directly or view it on GitHub
#179 (comment).

[wmertens]

My sophisticated web searches (i.e. "proot benchmark") didn't show up
anything. Anybody tried it yet?

On Tue Jan 20 2015 at 4:17:27 AM Alexander Kjeldaas <
notifications@github.com> wrote:

Isn't it documented to use ptrace? If so it will signal the controlling
process and wait for a command on every syscall that is intercepted.

On Tue, Jan 20, 2015 at 12:47 AM, Benjamin Staffin <
notifications@github.com

wrote:

It does still intercept system calls in userland, and it's going to have
some unavoidable speed overhead.

—
Reply to this email directly or view it on GitHub
#179 (comment).

—
Reply to this email directly or view it on GitHub
#179 (comment).

[cedric-vincent]

Hello all,

I confirm that PRoot uses QEMU to run non-native binaries only, and
that it is currently based on ptrace; which is known to cause a
significant slowdown. However, in order to decrease this slowdown as
much as possible, PRoot uses process_vm_{readv/writev} (available on
Linux 3.2+) and seccomp mode 2 (available on Linux 3.5+). For
information, here follow figures I've published when I've enabled
seccomp mode 2 in PRoot:

https://github.com/cedric-vincent/PRoot/blob/v5.1.0/doc/proot/changelog.txt#L510

My suggestion is to give PRoot a try if your kernel version is equal
or greater than 3.5, and if it's not too difficult to replace in your
scripts calls to "chroot" and to "mount --bind" with a call to
"proot". If PRoot is not fast enough, this will be likely fixed in
the future using kernel namepaces (available on Linux 3.8+).

Regards,
Cedric.

[Ericson2314]

Seems like using Linux namespaces would dovetail with the pure Darwin stdenv work. All the better if they are faster than chroots.

[copumpkin]

They actually already use Linux namespaces. chroot is a bad name for them.

[benley]

Heh, in that case NixOS should call them Containers and pick up some buzzword publicity points. "Build all the things in containers!" containers containers containers containers containers. ;-)

[zimbatm]

It would be good to look at how Bazel does it as they are facing similar problems.

[nh2]

Thanks for the explanations!

Maybe we should use a selective approach until Linux namespaces are very fast. While sandboxing ver every derivation is certainly desirable, it would already be a huge benefit if we could, for starters, sandbox "the average build" of nixpkgs libraries and applications. For example, I'd be very happy to pay a 24 ms overhead if in turn my 5 hour Chromium build is guaranteed to be pure. But right now it's full sandboxing or no sandboxing.

Another point: The nsenter benchmark at #179 (comment) measures 4 ms mean time. However, we are already dangerously close to Linux's process startup overhead that this number probably is not very meaningful. For example, just running the help text with time nsenter --help > /dev/null takes anything between 1 and 4 ms on my computer.

We should probably benchmark whatever nsenter does in a loop in C to get meaningful numbers for that.

[ryantrinkle]

FWIW, here are the results on my machine (the same one I used for the prior benchmarks), for nsenter --help >/dev/null:

» nix run -f channel:nixos-unstable bench -c bench "nsenter --help >/dev/null" -o unshare.html 
[4 copied (3.8 MiB), 11.5 MiB DL]
benchmarking nsenter --help >/dev/null
time                 3.108 ms   (3.088 ms .. 3.126 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 3.222 ms   (3.188 ms .. 3.289 ms)
std dev              161.1 μs   (93.24 μs .. 275.8 μs)
variance introduced by outliers: 31% (moderately inflated)

And here it is for true:

benchmarking true
time                 2.470 ms   (2.452 ms .. 2.492 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 2.456 ms   (2.445 ms .. 2.469 ms)
std dev              37.67 μs   (31.60 μs .. 46.29 μs)

So sandboxing is about an order of magnitude slower than running a minimal command. I definitely agree that this amount of time is not important for most use cases today.

[zimbatm]

And building a stdenv.mkDerivation is also going to execute bash which stat(2) for rc and profile files all over the place.

[edolstra]

@zimbatm Yes, but Nix does not require the use of stdenv.mkDerivation.

BTW on Linux 4.17 I get a 37% slowdown in the test mentioned in #179 (comment). That's a big improvement over the 742% slowdown in 2013...

[copumpkin]

Any idea of a good threshold for acceptable? I doubt it'll ever be zero cost, but purity-by-default is a big win IMO and I'd be willing to pay a slight cost on it. Especially since the benchmark you're citing mostly affects tiny derivations and not big builds. One even smallish build will completely eclipse a ton of small slowdowns on unit files and NixOS configuration files.

[zimbatm]

Having sandboxing turned on by default would be great. It would reduce the number of issues with nixpkgs submissions that don't compile and user reports. We'll be able to trim the PR and Issue templates. That being said, if nix is running inside of a docker container it won't work as docker containers don't support cgroups by default.

Back on the subject of sandboxing, is it possible to re-use sandboxes between runs? if sandboxes could be re-used then they could also be pooled where the pool size = maxJobs.

[Ericson2314]

We must be sound now. We must compete with the likes of Bazel on granularity soon. That's how I see it.

[edolstra]

@copumpkin I think the 37% slowdown is okay-ish, though obviously not ideal.

@zimbatm No, I don't think sandboxes can be reused. The main overhead seems to be setting up the mount namespace, which is necessarily different for each build (since they have different input closures). Of course, you could bind-mount all of /nix/store, but that reduces security a bit (since even if it has -wx permission, builders would be able to access path contents if they know the store path).

[7c6f434c]

The main overhead seems to be setting up the mount namespace, which is necessarily different for each build (since they have different input closures).

Setting up the namespace or setting up the mounts? I.e., would reusing the namespace, and mounting/unmounting only the paths that differ (keeping the stdenv paths untouched) help in any appreciable way?

[ryantrinkle]

@7c6f434c Good question! I think we would need to benchmark bind mounting to see.

[zimbatm]

Another motivation to enforce the sandboxing is that we could get rid of the nixbld\d+ users. Each sandbox gets it's own pid namespace so they could all run with the same uid/gid. That would be great to limit the footprint nix has on non-nixos systems.

[7c6f434c]

We-ell, some namespace bugs (kernel-level) could be mitigated by the fact that the real UID outside of namespace is different, so for maximum isolation we might still want to have these users…

[edolstra]

In ad1c827 I implemented automatic UID allocation from a range. You would still like to ensure that the UID range doesn't clash with any existing accounts, though it's unlikely people have UIDs in the range 872415232+...

[7c6f434c]

Well, if the range is configurable it should be easy to move outside the ranges used by other tools; definitely simpler than listing eight build users in global passwd. Thanks.

[domenkozar]

@edolstra I wanted to implement sandboxing to be on for Docker after @garbas talk, but really that road leads back to Nix doing it by default for overall good experience. Given that we're at the okayish threshold now, and kernel 4.19 was released that will be next LTS, can we make sandboxing by default on? :)

[copumpkin]

Why docker? I missed the talk but intuitively it feels like a step backwards

[domenkozar]

@copumpkin just to enable sanboxing for https://github.com/NixOS/docker, since it helps sandbox networking during Nix builds.

[zimbatm]

Does the Nix sandboxing work inside of Docker now?

[dtzWill]

I believe it's worked for quite some time -- but requires permissions/capabilities that aren't added by default. Don't remember the appropriate flags, sorry, others might be able to say more.

…

On Fri, 02 Nov 2018 16:56:08 -0700, zimbatm ***@***.***> wrote: Does the Nix sandboxing work inside of Docker now? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #179 (comment) part: text/html

[copumpkin]

Oh sorry, I misread and thought you wanted to change our sandboxing mechanism to use docker, rather than get docker to work from inside one of our sandboxes 😄 sorry!

[dtzWill]

Based on the talk (IIRC) I think it's actually neither of those xD, but rather getting our sandbox (and the Nix "story") to work in Docker (which to some extent is how people expect to "get started" these days).

…

On Fri, 02 Nov 2018 18:54:38 -0700, Daniel Peebles ***@***.***> wrote: Oh sorry, I misread and thought you wanted to change our sandboxing mechanism to _use_ docker, rather than get docker to work from inside one of our sandboxes 😄 sorry! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #179 (comment) part: text/html

[copumpkin]

oh, I see, thanks!

[domenkozar]

🎉

[Ericson2314]

#2759 A wildly different idea for maybe-even-faster sandboxing.

[maisiliym]

The solution to this is eBPF-based sandboxing, which would be essentially 'free' if the builder doesnt 'try anything funny'.