Make commands in dev-dependencies available to run

[casey]

I have a rust project which uses a few shell scripts that need commands from other crates. I added them as dev-dependencies, but I couldn't find the binaries anywhere. It would be nice if the binaries were available to run from the command line.

As an example, npm makes binaries available in node_modules/.bin, so you can put them in devDependencies and use them in scripts.

[alexcrichton]

This sounds like it may be more of a use case for cargo install? We typically don't have binaries from any dependencies available to the main application

[casey]

The issues that I can see with cargo install for this use case are:

1. No record of dependencies. Let's say your project needs various binary crates for development, for example because they're called by non-rust glue and utility scripts. This creates the need to record somewhere that you'll need to cargo install xyz, which complicates getting started on a project, or starting work on it on a new machine.
2. No versioning. Although you can use cargo install --vers, dependency on a specific version of a crate isn't recorded.
3. Global install by default. Since cargo install installs globally by default, it isn't possible out of the box to keep separate installations of tools for different projects, in case they depend on different versions of the same binary crate. It's possible to get around this by doing cargo install --root or using the install.root configuration key and specifying a local location, but I suspect that this is a common enough use-case to warrant special handling.

My current motivation for raising this issue is that I'm developing bindings for a C-library using github.com/crabtw/rust-bindgen. rust-bindgen has a library crate which provides a rust api, as well as a binary crate which provides a command line API. I need to use the binary crate because the library API won't work for my use case. So, rust-bindgen needs to be installed and available as a runnable binary for doing routine development, and it would be nice to formally record that somewhere, and have a simple 'cargo update' or 'cargo install' that would fetch crates with the commands that I need.

[alexcrichton]

For that use case, wouldn't a build script suffice? In theory rust-bindgen would also have a library API and then your build script would take care of what while building.

[casey]

Rust-bindgen does indeed have a library API, but it is not exactly the same as the CLI. Not to get into too much detail, but the library API is a macro that generates bindings at compile time, and the CLI writes the bindings to a .rs file. I'm wrapping a library with safe rust bindings, so writing the bindings explictly to a .rs file where I'll see in git if they've changed and can review the changes before committing them is preferable.

I suspect that many crates are like this. The library api and the CLI are slightly different use cases, and so will be designed slightly differently and be good for different things. Or, the library will be a command-line oriented tool with no library API, but still used in the development of a particular rust project.

[casey]

Unless I'm mistaken, this was accidentally closed.

[alexcrichton]

Oh dear I did indeed fat finger that, sorry!

[jansegre]

I have a use case that also requires commands available from dev-dependencies.

My project has to run arbitrary user script/commands (sort of like codingame), those scripts are run as a child process by the library, which pipes and interact with stdin, stdout and stderr of the child. I want to test that, and the most sensible way I thought would be having in-project dev-dependencies with executables to act as user scripts. But then I found out that cargo does not compile dev-depenencies commands.

Although somewhat different than the original issue this also involves having binaries from dev-dependencies compiled. I'm open to alternatives, my strongest motivation for having this done in cargo is the continuous integration testing of the child process piping functionality on multiple platforms.

Is there a strong reasoning to oppose binaries in dependencies in general and in dev-dependencies in particular?

Edit: fiddle a bit and found #1581, which could also be used to compile my test user scripts.

[alexcrichton]

There's not really any particularly strong reason to not do this, it basically just needs a principled design. For example we can't really "just compile" all dev-dependency binaries and put them somewhere. Some concerns I have with that are:

• What if I don't actually need all those binaries? Will this just clutter up build times?
• When are those binaries actually compiled? Does cargo build do that? cargo test?
• How do we resolve binary name conflicts between dependencies?

In general build scripts tend to solve most of these problems naturally, so I personally like to push on those as hard as possible, but I can definitely see that for CLI tools this may fall down from time to time (but cargo install can often fill in)

[jansegre]

In my case I miss the ability to compile binaries for use inside tests, not running commands from dev-dependencies outside tests.

The more I think about it dev-build scripts seem more reasonable as in the general case it could be used to generate anything for tests and benches.

For now I'll resort to a Makefile + PS script (windows) for compiling a binary and making it available to tests via some environment variable. But I'm willing to write my first RFC for #1581.

[alexcrichton]

Out of curiosity, isn't it easier to link to a library and call a function than it is to execute a binary? Shouldn't the dev-dependencies ship libraries that you can call in tests?

I do have a feeling though, yeah, that dev build scripts will likely be a thing at some point.

[jansegre]

Well, the point is to specifically test integration on running a child
process, I guess in theory I could fork and set up pipes and call
functions, but I don't think it's in std, cross-platform, nor could use
the interface that would be tested.

On Fri, Feb 26, 2016, 03:25 Alex Crichton notifications@github.com wrote:

Out of curiosity, isn't it easier to link to a library and call a function
than it is to execute a binary? Shouldn't the dev-dependencies ship
libraries that you can call in tests?

I do have a feeling though, yeah, that dev build scripts will likely be a
thing at some point.

—
Reply to this email directly or view it on GitHub
#2267 (comment).

[brendanzab]

Are there any workarounds for this right now? I was getting flaky builds due to the problem of having to use cargo install mdbook on cached build servers. I'm now adding a hacky script as a workaround in brendanzab/pikelet#20. I have to use a hard version number though, otherwise if mdbook updates, I will always be forcing the rebuild. What I'd love is just to be able to add mdbook to my dependencies:

[dev-dependencies]
mdbook="0.1.5"

And then somehow be able to call that specific binary.

[Kinrany]

Perhaps instead of having separate main dependencies and devDependencies, cargo should make it possible to specify a list of environments for each dependency?

Same as features: by default the crate would only be available at runtime, but specifying "dev"/"bin"/"ci" would mean that the dependency should be installed only during development/should be installed with binaries/should be installed in ci pipelines

[ry]

Just want to chime in that this is also a problem for Deno. We have a rust http server that we test Deno programs against. It needs to be a standalone binary so we can use it outside of “cargo test” for debugging and whatnot. But the binary is a dependency of the tests as well. There is no way to express this dependency in cargo currently (as far as I can tell).

[kaimast]

I am trying to write a database system in Rust and would love to have this feature too. It's a distributed system where I need to run a couple of processes for integration testing.

Currently I have build script that re-runs cargo install whenever one of the binary projects (node and coordinator) changes. This is obviously ugly and can mess with other projects on the system, but see my code below for the somewhat expected behaviour of that feature.

use std::process::Command;

fn visit_dirs(dir: std::path::PathBuf) {
    println!("cargo:rerun-if-changed={}", dir.as_os_str().to_str().unwrap());

    for entry in std::fs::read_dir(dir).unwrap() {
        let entry = entry.unwrap();
        let path = entry.path();

        if path.is_dir() {
            visit_dirs(path);
        } else {
            println!("cargo:rerun-if-changed={}", path.as_os_str().to_str().unwrap());
        }
    }
}

fn main() {
    let mdir = std::env::var("CARGO_MANIFEST_DIR").unwrap();
    let path = std::path::Path::new(&mdir).join("..");

    Command::new("cargo").args(&["install", "--path=coordinator"])
        .current_dir(path.clone())
        .spawn().unwrap().wait().unwrap();
    Command::new("cargo").args(&["install", "--path=node"])
        .current_dir(path.clone())
        .spawn().unwrap().wait().unwrap();

    visit_dirs(path.join("coordinator"));
    visit_dirs(path.join("node"));

    println!("cargo:rerun-if-changed=build.rs");
}

[tchernobog]

Given the description, and that we want:

• the ability to select which version of the package whose binary we want to run
• we don't want a global installation since it might clash with other projects (mdbook and its preprocessors are a good example, as they often break if compiled against different versions of mdbook than what is being run)
• to run the binary from inside the target/<profile>/build/... folder

Wouldn't it be enough to teach cargo run --package to also operate on dependencies not part of the workspace and as specified by pkgid, if they are already downloaded? It already has the right granularity for the job:

• cargo knows how to build a missing executable, so it can be done lazily
• it knows where to find the file inside the target folder
• with pkgid we can specify a version
• with --bin we can specify which executable

[Eh2406]

Sorry, I listen to triage advice without reading to carefully.

[Ericson2314]

Perhaps I am missing something, but artifact deps do allow depending on an exe, right? Maybe the PATH isn't extended, but one can fish the binaries out of the new env vars in e.g. a build.rs, right?

[ehuss]

Yes, build.rs can run them. But that is relevant to build-dependencies. This issue is about dev-dependencies, and providing the user a means to run them manually on the CLI. I think this issue is more about having a tool of some kind (like diesel_cli) where the developer is intended to run commands manually, and you want that tool scoped and versioned to the local project.

[Ericson2314]

@ehuss My guess is specifying dev-dependencies in the title was a bit of an x-y problem thing. The compiled build.rs executable is still run regardless of the component being built, and it would invoke the command then not at build-time of the build.rs executable itself.

[mcandre]

Yeah, I really would have expected the cargo dev dependencies section to support CLI tools, similar to how NPM dev dependencies work.

Oddly, there isn't even necessarily a way to tell the cargo program to go ahead and install dev dependency CLI tools, especially if the downstream project is a library without much actionable cargo install ability.

One workaround mentioned previously involves a third party cargo-run-bin plugin. This may work for some uses. But it messes up the flow of my custom build tool, tinyrick. I don't want to require the user to laboriously type out cargo run tinyrick each time. The whole point of tinyrick is to save the user time.

I don't want to have to resort to shell aliases or shell functions that would expand tinyrick into cargo run tinyrick. Aliases and shell functions would break for many Windows users. That means fragile shell code. That means an interpreter and a set of linters, all growing the tech stack unnecessarily. And any shell setup would occur entirely outside of the Rust Cargo system. Which brings us back to the original point: We just need a first party way to install and to run Rust CLI tools using built-in Cargo capabilities.

WSL is nice to have, but I do not like to depend on it unless absolutely necessary.

While the user may manually run cargo install <tool>[@version], that fails to take advantage of any pinning that should have been in place in the Cargo.toml file. And when projects involve multiple CLI tools, then it becomes a hassle to enter the install commands one by one. The user is tempted to script them, such as in a sh/bat/PowerShell/fish/batsh file. But that is fragile, and tends to break for native, COMSPEC Windows software developers.

I am taking notes towards an accio equivalent in Rust. Pity that both Rust and Go lack robust support for pinning dev tools.

[sksat]

I would like this feature very much.
In embedded use cases, there is a need for tools to flash to the board (e.g. prube-run) and simulators (e.g. QEMU) to run in cargo run. There are also cases where you want to use a custom linker such as flip-link.

Naturally, these external tools have to run executable, but there is no way to use these tools on a per-project, fixed version basis.
Also, artifact deps are currently not a good solution because there is no way to get CARGO_BIN_* env directly from the custom runner (I guess there is a possibility to pass these env to the custom runner as well).

[epage]

See also #5120 for more use cases.

[ajmedeio]

I also believe this feature is greatly needed. I was shocked coming from an npm and python world. Currently, my local dev instructions require my users to cargo install several global dependencies, an arguably bad practice. We also have a version pinning problem.

In the meantime, I'll try the cargo-run-bin project.

[mightyiam]

Workaround: a different approach would be to use Nix.

[abhillman]

Here is a related proposal that could be adapted for this purpose #13359.