Making _build dir locking opt-in

[emillon]

In #6360 (dune 3.6.0) we added a lock to make sure that concurrent dune processes can not share a _build directory. We went from never locking from always locking, which can be a bit too restrictive and seems to break some workflows. For example dune exec does not need a lock if no build happens, which is almost always the case if a concurrent dune build -w is running.
There has been some efforts in relaxing when the lock happens, so these cases should get less common. But when it happens the drawback is important (the workflow becomes impossible).
So my suggestion is to put this feature behind a configuration option until we're more confident that locking is only done when necessary or that there are sufficient escape hatches that we can turn this on by default (of course this also means that dune developers and power users should be encouraged to enable this and report remaining issues).
What do you think?

[snowleopard]

It seems to me that preventing multiple build or clean commands from running in parallel should not be an opt-in, because doing that is clearly a mistake.

For other commands, I agree, the answer may be less straightforward.

[Alizter]

Don't we want to essentially do RPC for all these commands? The idea is to have an RPC server started always, and if there is one already running (with possession of the lock), we connect to it.

AFAIK for this to happen, we need a good way of relaying the display back to the user from the server. That was part of the motivation to refactor the display stuff.

@rgrinberg knows more about how this is going.

[emillon]

Yes, I agree that the RPC is probably going to solve some of these issues and that we're going to have good solutions for that in several minor releases.
What I'm specifically after with this issue is to determine what to do for 3.7: for our users, there are some things that used to work in 3.5 that don't in 3.6, so I'd like to give them solutions for the short term (either rolling back the feature behind a flag, or an escape hatch).

[Alizter]

Maybe we need a command that is dune exec minus dune build . Something like dune env? If you have a watch mode running, exec doesn't need to build, and we should really just be setting the correct environment.

Or better yet, make dune exec smarter so it doesn't have to possess the lock of _build if everything it needs is already built.

But at the very least, I am OK with an (opt-in) backdoor around the lockfile, since people were essentially relying on a bug, until we can provide a better workflow.

[rgrinberg]

It doesn't seem like the issue is big enough to warrant an unsafe escape hatch.

[emillon]

Here's a recent example of a user being stuck because dune build -w takes the lock. I agree that not locking is unsafe, but this is also how dune has been operating since the beginning. Which is why instead of proposing an escape hatch, I'm proposing to delay enabling the feature for all users until it's a bit more fine grained.

[ejgallego]

dune exec while a build is ongoing is IMHO a pretty common case, so maybe we could indeed tweak that case, however not sure how easy it is in practice.

[rgrinberg]

There's already a workaround with --no-build being passed to dune exec. I think that's enough for now.

[emillon]

ok, thanks. from what I'm reading the consensus is that we agree that situation is not great but that we have enough workarounds for the common cases and that we're going to improve the situation in further releases. In other words, nothing needs to be done for 3.7 and I'm closing this issue. Thanks everyone.

[Alizter]

@emillon Should we perhaps document the work arounds in the faqs or something?

[rgrinberg]

I agree with Etienne's conclusion. I don't see the point of documenting workarounds since these are all issues some of us are actively working on at the moment. Let's try to get a prototype of dune monitor going and then re-asses the necessity of workarounds.

[jchavarri]

Documenting here a couple of use cases where either the existing workarounds are lacking or missing.

1. Building OCaml and Melange apps in the same project. Let's say a web server in OCaml native and some frontend implemented with Melange in the same dune workspace. It is common in these cases to keep two terminals opened because the contexts of both sides are too different. With lock in, this is no longer possible and one has to build both sides under same terminal. The problem gets worse if you want to run the server in watch mode, as one might get server executable log messages intertwined with Melange frontend build output. I guess this issue applies to any project where one needs to build two sufficiently independent apps simultaneously.
2. Using dune@fmt alias. If there is any ongoing dune build going on, one can't call the fmt alias without stopping the build first.

In general, I feel the trend added by this _build dir locking is that the friction between dune builds, especially in watch mode, and other dune features like integration with OCaml ecosystem variety —jsoo, coq, melange,...— and useful integrations with tooling like ocamlformat, will keep increasing.

If there is some dune monitor tool in the works, maybe it could be reconsidered to remove the lock until that feature is available?