Detached run and runs manager

[hadim]

This PR is an attempt to replace #386.

I worked on it without pinging first, and it's also quite a large PR, so if you don't have time to review currently, it's all good and no rush, please.

---

I extended the first implementation from #386 to add a full "runs manager" with the below features:

• Save pid, out/err and infos files for every detached runs into ./pixi/runs/. The structure is very flat but happy to make it more nested.
• List all the runs (terminated or alive ones).
• Clear one or all the terminated runs: simply delete the associated files in ./pixi/runs/.
• Kill one or all the runs by sending the KILL signal.
• Print the stdout or the stderr logs for a given run.

---

Important 1 : so far, I haven't been able to "stream" the logs and so those are only written to the stdout/stderr files whenever the run terminates (which is not super, super nice xD). I tried to dive into deno task and also pixi commit's history (looking at you execute_with_pipes) but without much success.

Important 2: while this PR is large, it does not touch anything outside this particular and well scoped feature. That should make the reviewing and maintenance easier.

---

See below the current API.

Note: I added an example to facilitate the dev process. You can use --manifest-path examples/daemon/pixi.toml

• Successful starts

$ pixi run -d --name mytask start
✔ Starting the task in the background with the name 'mytask'.

$ cat .pixi/runs/mytask.pid
55478

$ cat .pixi/runs/mytask.infos.json
{
  "name": "mytask",
  "task": [
    "start"
  ],
  "start_date": "2024-01-17T18:53:49.648662Z"
}

$ pixi run -d start
✔ Starting the task in the background with the name 'rebel-berry'.

• Fail if an existing run already exists

$ pixi run -d --name mytask start
✔ Starting the task in the background with the name 'mytask'.
$ pixi run -d --name mytask start
  × Failed to create new run: A run with the same name already exists. You can call `pixi runs clear` to clear all the terminated runs.

• Clearing a run (or all of them with clear-all) will remove the pid, logs and out/err files from the runs dir. It only works for terminated runs.

# Clear a terminated run
$ pixi runs clear mytask
✔ Run called 'mytask' correctly cleared

# Fail because still running
$ pixi runs clear mytask
  × The run is still running. You can call `pixi runs kill` to kill it.

# Fails because it does not exist
$ pixi runs clear mytask
  × No run with name 'mytask' found.

# Clear all the runs
$ pixi runs clear-all
✔ Run called 'mytask' correctly cleared
✔ Run called 'mytask2' correctly cleared
✔ Run called 'mytask5' correctly cleared
✔ All the terminated runs correctly cleared

• Print logs. For now, those cannot be streamed, so you have to wait for the run to be done before seeing the logs.

$ pixi runs logs mytask
<stdout logs>

$ pixi runs logs --err mytask
<stderr logs>

• List of detached runs

$ pixi runs list
 Name     Status      PID    Start Date           Task        Stdout Size  Stderr Size 
 mytask7  Terminated  91663  2024-01-17 16:33:00  start       280          280         
 mytask4  Terminated  91584  2024-01-17 16:32:55  start       280          280         
 mytask6  Run         87117  2024-01-17 16:25:24  start_long  16359        282

$ pixi runs list --json-pretty # or --json
<runs as JSON>

• Kill running runs:

$ pixi runs kill mytask
✔ Run called 'mytasklong' correctly killed

$ pixi runs kill-all
✔ Run called 'mytasklong2' correctly killed
✔ Run called 'mytasklong1' correctly killed
✔ Run called 'mytasklong4' correctly killed
✔ Run called 'mytasklong5' correctly killed
✔ All the running runs correctly killed

[hadim]

Important 1 : so far, I haven't been able to "stream" the logs and so those are only written to the stdout/stderr files whenever the run terminates

Actually, it happens with examples/daemon/ but logs streaming with examples/jupyterlab/ and on stderr seems to work fine.

[wolfv]

This is very exciting! I am going to test drive it a bit today :) Awesome!

[baszalmstra]

This is amazing! Without actually testing it (Im on Windows) this all looks pretty good to me. I did leave some comments but they are mostly related to the use of unwrap/expect and documentation.

[wolfv]

I am trying to find the tricks that we used in micromamba for this.

Apparently we prefaced with exec to replace the parent process entirely (the wrapping bash). Not sure if that would help.

https://github.com/mamba-org/mamba/blob/81504b5614d54fa53677548c56714736e7344590/micromamba/src/run.cpp#L192C1-L197C1

I also think we are sending the SIGTERM signal (and if that doesn't work, SIGKILL): https://github.com/mamba-org/mamba/blob/81504b5614d54fa53677548c56714736e7344590/libmamba/src/core/run.cpp#L444-L463

[wolfv]

I just tried with your latest branch and it still doesn't kill jupyterlab for me. Maybe we need to send it to the child PID or something?

[hadim]

I am done with the rest and will focus on kill now.

I tried quick with TERM and indeed it does not work (neither the examples/daemon/ by the way). But I wasn't expecting this to work.

I will try retrieving the child processes.

[hadim]

I think I got it to work. It's simply about iterating over all the children and killing them as well.

It requires iterating over all the processes of the system, but it's working for both daemon and jupyterlab.

[hadim]

You can try with the latest commit of this branch.

[hadim]

I am done for the moment. Feel free to review.

The killing logic works well after a few tests. I don't know how much this is bulletproof, but if it works for you, I suggest we proceed and wait for user's feedbacks if there are some edge cases.

[wolfv]

Hey @hadim I <3 this feature – back then I also implemented something like this in micromamba and I still think it's something of an easter egg!

I think we're not 100% sure about the CLI design (e.g. should it be called runs?).
The other point of contention is that it's not available on Windows (correct me if I am wrong). I personally don't mind that, @ruben-arts has a bit of a stronger stance on this. IIRC I also never attempted to implement this in micromamba for Windows.

I can already see this playing really well with a future "sandboxing" feature for a "docker-lite" (for example on Linux we can use namespaces for some process isolation, which would be very cool). I was always planning to test something like bubblewrap (https://wiki.archlinux.org/title/Bubblewrap) but Rust also has a bunch of nice crates for this.

Lastly I'd like to send you some pixi / prefix swag! Very awesome PRs lately! 🚀

[hadim]

I think we're not 100% sure about the CLI design (e.g. should it be called runs?).

I went ahead without pinging anyone here, and I agree this API could be revisited.

The other point of contention is that it's not available on Windows (correct me if I am wrong). I personally don't mind that, @ruben-arts has a bit of a stronger stance on this. IIRC I also never attempted to implement this in micromamba for Windows.

Indeed, it's because daemonize is Unix-only. daemonize-me seems to support Windows, but I don't think this is enough and pixi would need to be started as a service somehow (I might be off here).

I, personally, think it's common for this kind of feature to be shipped for Unix-only machines and nothing prevents someone from implementing the Windows support in the future. That being said, I understand if you guys prefer to ship something feature-complete.

I can already see this playing really well with a future "sandboxing" feature for a "docker-lite" (for example on Linux we can use namespaces for some process isolation, which would be very cool). I was always planning to test something like bubblewrap (https://wiki.archlinux.org/title/Bubblewrap) but Rust also has a bunch of nice crates for this.

It looks nice - I guess it's a tradeoff between reinventing the wheel but making it also standalone, independent and 100%-pixi-fit versus reusing existing platform-dependent tools and services.

Lastly I'd like to send you some pixi / prefix swag! Very awesome PRs lately! 🚀

ahah thanks 🤗

---

All good here - I understand it's quite a large PR that also comes with a maintenance burden.

I am happy to go wherever you want with that PR. Either hold, close, rethink the CLI entirely, etc.

[hadim]

(the fail is due to main failing at https://github.com/prefix-dev/pixi/actions/runs/7582913789/job/20653461505)

[ruben-arts]

Hey @hadim,

I think this looks from a code perspective really good, the feature itself, the size of it and the possible maintenance this will take scares me a little to be honest. The fact that there is no solution in the close future to be run in Windows also doesn't feel that well to me. As we're putting our-selfs out there as a cross platform tool but already show discrepancies between the platforms with this feature.

We've got some people that want to test it more thoroughly and show some more use cases. I would also like this to be tested in CI.

I hope this explains why we are a little slow on working on this branch.

[hadim]

No problem at all. It was fun coding and I understand your concerns.

You are welcome to take over that PR.

It's too bad that feature does not support Windows. I haven't really tried, but after looking a bit into it, I don't actually know how much feasible it is.

[hadim]

Closing here as it's not really active anymore. Feel free to re-open as needed!