nix.settings.max-jobs defaults to auto which doesn't match nix's default

[LunNova]

Since nix 2.3 the default for max-jobs is 1. See 2.3 release notes.

Our nix-daemon module defaults to auto instead of 1 which will use the number of CPU cores.

Defaulting to auto will often lead to OOMs on systems with many cores. A 32 core system will run 32 jobs, each of which will use up to 32 cores. If most of those 32 jobs are compiling large C++ projects you can end up with hundreds of C++ compilers running.

https://github.com/NixOS/nix/blob/927234cfb26305d9da2b3d15721ed3e76baf3b33/src/libstore/globals.hh#L116-L127

[ck3d]

And max cores defaults to 0, which allows make and ninja to start as many jobs as core available.
A default NixOS allows to start on a 32 core machine 1024 build processes.
The load guard was removed with #192447 .

I highly recommend to set max jobs to 1.

[LunNova]

Looks like this is the same issue @vcunat noted there: #192447 (comment)

[ck3d]

I created a PR to set default from auto to 1, see #199491

[vcunat]

Well, honestly I wouldn't want 1 on my machines. (downloads, config files, etc.) But it's just a default I guess, so it won't hinder me personally.

My proposal didn't get support, so let's see about this one... but in any case I believe these things should be designed as a package, all the parameters at once, instead of arguing about individual pieces. And in release notes (thanks).

[sg2002]

While updating my 32 bit machine I was doing a lot of rebuilding on a less obsolete 64 bit build machine and along the way tested the performance of different max-jobs options. Having it set to 2 increased the total rebuild speed by about 50% over it being set to 1.

It's extremely unlikely someone would be running Nixos on a single core machine these days and if someone is rebuilding it's safe to assume that they would want max performance. Thus I think that 2 is probably the good safe default and maybe you could argue for 4 these days.

[bjornfor]

IMHO, only auto or 1 makes sense as a default -- hardcoding 2 or more is something you should configure yourself, if it suits your machines.

[SuperSandro2000]

If we change the default to 1 then I already know that almost all people I know will need to change their configuration and will question this change.

[LunNova]

I'm not sure one is a great default either. Maybe a proper fix needs an auto option in nix for this which picks a reasonable default for both small systems and those with many cores.

[kevincox]

Maybe Nix needs a better scheduler. Something based on the load average may be a way to allow more than 1 without greatly overloading the machine with jobs that do parallel building. Something like allow spawning jobs while the load average is less than the core count. However it is more of an art than a science so it will be years of tuning to get it to work well.

[vcunat]

As linked from the second post here, load average regulation used to be there and that route was refused. Lots of past discussions can be found through that link. It's difficult to do scheduling well, as build systems of the package inside vary a lot and sometimes they're hard to control (even if you'd use stuff like cgroups).

[nh2]

At least one part of this should be improved with the upcoming NixOS 23.11 which will have nix 2.17, and nix 2.16 had this change:

Speed-up of downloads from binary caches. The number of parallel downloads (also known as substitutions) has been separated from the --max-jobs setting. The new setting is called max-substitution-jobs. The number of parallel downloads is now set to 16 by default (previously, the default was 1 due to the coupling to build jobs).