Add support for make jobservers

[michaelwu]

Make's jobserver is described in http://make.mad-scientist.net/papers/jobserver-implementation/

This allows us to prevent make builds from launching too many jobs.

[alexcrichton]

Note that this could also be used for the compiler's own internal parallel codegen and possible future parallel compilation, so this could be quite useful!

[larsbergstrom]

Our contributors with less powerful machines (e.g., VMs with <= 4GB of RAM) frequently fail servo builds due to the OOM killer because right now we can sometimes have N_CORES clangs running building SpiderMonkey at the same time that there are N_CORES-1 rustc jobs going. Our current workaround is:

10:56 AM <Manishearth> imperio:  ./mach build -j1 until you reach utils
10:56 AM <Manishearth> then ./mach build

[alexcrichton]

FWIW I've thought about this from time to time, and implementation wise the details I believe look like this:

• The make driver at the top level creates a Unix pipe, and with -jN it write N-1 bytes onto the pipe.
• When you want to do some parallel work, you read a byte from this pipe, then spawn the work.
• When a job is done, the byte is written back to the pipe.

There's a bunch of trickery in make dealing with spawned processes and such, but for us we'll just read/write a byte every time we want to run a compiler (except for the first one we run). The interface to this is that make arranges for us to inherit a few file descriptors which correspond to the read/write ends of the pipe. On Unix this seems to be communicated through either the MFLAGS or MAKEFLAGS environment variable in the form of --jobserver-fds=3,4.

For example:

fn main() {
    for (k, v) in std::env::vars() {
        println!("{}={}", k, v);
    }
}

all: foo
    ./foo

foo: foo.rs
    rustc foo.rs

$ make -j10 -f foo.mk | grep FLAGS
MAKEFLAGS= --jobserver-fds=3,4 -j
MFLAGS=- --jobserver-fds=3,4 -j

So, all in all, here's what I think needs to happen:

• For the first job spawned, we don't do anything
• For any other job spawned, we send work to a thread where the first operation is to acquire a token. Once done, it releases the token

I think that all subprocesses will also be able to use the jobserver because environment variables and the relevant file descriptors will be inherited, so we should be covered there.

[glandium]

There are two dimensions to this:

• cargo can be running make on its own for e.g. crates implementing bindings for a $lang library. In that case, cargo may already be running multiple compilations, and will fight for CPU resources with make. For this, cargo would need to have its own jobserver that make would then use, or to do a round-trip through make so that cargo can use make's jobserver. Either way is tricky.
• cargo can be run from make. I don't know if that's what happens with servo, but that's definitely something that happens with Firefox. In this case, cargo, and/or maybe arguably rustc itself would use make's jobserver.

I don't think both need to be addressed at the same time. In fact, the latter is significantly easier than the former, and would be meaningful to have for Firefox.

Now, wrt alex's comment about how make makes the jobserver available, it should be noted that the picture is a little more complicated:

all:
    @echo MAKEFLAGS=$$MAKEFLAGS
    @echo MFLAGS=$$MFLAGS
    @ls -l /proc/self/fd
    +@ls -l /proc/self/fd

$ make -f foo.mk
MAKEFLAGS=
MFLAGS=
total 0
lrwx------ 1 glandium glandium 64 Sep  2 08:45 0 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:45 1 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:45 2 -> /dev/pts/2
lr-x------ 1 glandium glandium 64 Sep  2 08:45 3 -> /proc/29768/fd
total 0
lrwx------ 1 glandium glandium 64 Sep  2 08:45 0 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:45 1 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:45 2 -> /dev/pts/2
lr-x------ 1 glandium glandium 64 Sep  2 08:45 3 -> /proc/29769/fd

$ make -j10 -f foo.mk
MAKEFLAGS= -j --jobserver-fds=3,4
MFLAGS=-j --jobserver-fds=3,4
total 0
lrwx------ 1 glandium glandium 64 Sep  2 08:46 0 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:46 1 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:46 2 -> /dev/pts/2
lr-x------ 1 glandium glandium 64 Sep  2 08:46 3 -> /proc/29808/fd
total 0
lrwx------ 1 glandium glandium 64 Sep  2 08:46 0 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:46 1 -> /dev/pts/2
lrwx------ 1 glandium glandium 64 Sep  2 08:46 2 -> /dev/pts/2
lr-x------ 1 glandium glandium 64 Sep  2 08:46 3 -> pipe:[6654731]
l-wx------ 1 glandium glandium 64 Sep  2 08:46 4 -> pipe:[6654731]
lr-x------ 1 glandium glandium 64 Sep  2 08:46 5 -> /proc/29809/fd

While the environment variables are always set, the pipes to the jobserver are only actually passed when the command is preceded with + in the Makefile. And since opening a file will likely pick one of the descriptors passed through the environment (like ls above opened /proc/self/fd as fd 3), checking the jobserver needs to happen first and foremost.

[alexcrichton]

I have an initial version of support for this but unfortunately the situation on Windows is baffling me. It appears there's two make implementations in my MSYS installation locally:

• mingw32-make - this uses semaphores as documented online, and the jobserver client works.
• make - this apparently doesn't use semaphores!

The latter make executable actually exports the env var --jobserver-auth=3,4, which looks like it's attempting to use file descriptors. I'm not sure how that works at all on Windows because it seems like those wouldn't be inherited across processes. Clearly make itself works, though!

Does anyone else know what's up with the situation here? Is there a way to interact with those "file descriptors" (or whatever they are?) from a separate process? Is this make implementation never actually used by anyone?

[glandium]

An msys installation comes with two copies of various binaries, one of which is for the MINGW environment and the other for the MSYS environment.

mingw32-make - this uses semaphores as documented online, and the jobserver client works.

This is the MINGW make. It uses native Win32 APIs.

make - this apparently doesn't use semaphores!

This is the MSYS make. It uses the POSIX APIs, which means it works the same as make on a POSIX system. MSYS has a layer (inherited from cygwin) to translate POSIX into native Win32. Which means this is what handles the fd inheritance. I don't know how it's actually implemented, though.

People will unfortunately end up using make simply because it's the simple name, but in practice, it's not a great idea to do so, because it's slower (the POSIX layer emulates fork(), and that's slow).

[alexcrichton]

Yeah that was the conclusion I reached as well but what does it even mean to use the POSIX api on Windows? Surely that means there's basically just some translation layer, but what's it translating to? If a file descriptor is in a child process what does that mean? Is there a handling floating around we can identify and use?

[glandium]

cf. what I wrote :) there's a translation layer inherited from cygwin. Following links around from the msys site, it seems to be https://github.com/alexpux/cygwin .

[alexcrichton]

I've sent a PR for this in #4110