Parallel

[casey]

Version of #1562 using rust's native scoped threads.

[syphar]

apart from style I have one remark, coming from what you said about early exit when one of the tasks fails.

[syphar]

nit: I found it confusing when reading the code to expect true for the error case, but that might just be me.

[syphar]

If what you wanted to achieve is an early exit when the first task fails, this won't work in this implementation.

A thread::scope (also crossbeam::scope) will wait for all spawned threads to finish before exiting the scope.

I also tested this manually.

[syphar]

see above, the early exit won't work like this.

[casey]

@syphar Ah, good point about early termination. We definitely want that, since presumably, if users are using parallel execution, they have some long-running recipes and want them to terminate early. Hmm, that's annoying. There's no way to terminate a thread from outside of a thread in Rust, so the only way we could do this is by having a channel, and then when spawning commands doing something like:

loop {
  match rx.recv_timeout(Duration::from_millis(10)) {
    Err(RecvTimeoutErr::Timeout) => {}
    Err(RecvTimout::Disconnected) => { child.kill(); return Ok(None); }
    Ok(()) => { child.kill(); return Ok(None); }
  }
  match child.try_wait() {
    Ok(None) => {},
    Ok(Some(status)) => return Ok(Some(status)),
    Err(err) => return Err(err),
  }
}

[casey]

Other issues, which I haven't thought about:

• I don't know if this handles dependencies correctly. Previously, everything was synchronous, so if an invocation wasn't in ran, you would dispatch the dependencies, which would add themselves to ran. However, now, the dependencies might already be running, so you could double-dispatch them.
• Recipe output and line echoing will happen in random order. Which is fine. However, they might be intermingled within a single line, which would be illegible.

[syphar]

@syphar Ah, good point about early termination. We definitely want that, since presumably, if users are using parallel execution, they have some long-running recipes and want them to terminate early. Hmm, that's annoying. There's no way to terminate a thread from outside of a thread in Rust, so the only way we could do this is by having a channel, and then when spawning commands doing something like:

loop {

  match rx.recv_timeout(Duration::from_millis(10)) {

    Err(RecvTimeoutErr::Timeout) => {}

    Err(RecvTimout::Disconnected) => { child.kill(); return Ok(None); }

    Ok(()) => { child.kill(); return Ok(None); }

  }

  match child.try_wait() {

    Ok(None) => {},

    Ok(Some(status)) => return Ok(Some(status)),

    Err(err) => return Err(err),

  }

}

Yeah, I imagine that would be a valid approach, killing threads is not really nice, but in case of just, killing the spawned process (or exiting between calling processes in the thread) is probably good enough.

[syphar]

• I don't know if this handles dependencies correctly. Previously, everything was synchronous, so if an invocation wasn't in ran, you would dispatch the dependencies, which would add themselves to ran. However, now, the dependencies might already be running, so you could double-dispatch them.

From what I remember when testing dependencies, the parallel threads where handled as a group, so all dependencies of these were handled before handling the first of them. And everything depending on any one of the parallel tasks were waiting for the whole group to finish. I might misremember this, definitely needs testing.

• Recipe output and line echoing will happen in random order. Which is fine. However, they might be intermingled within a single line, which would be illegible.

I don't know enough about stdout buffering to know how exactly that behaviour would be, we might need to pipe the outpt and then take care of intermingling the lines properly? Then we could even add prefixes.

[syphar]

Generally, I'm happy to send time on some of the topics here, which do you want to pick? Should I pick?