Target specification?

[ehuss]

This issue is for tracking any possible work needed for custom target specifications.

The current target JSON specification was introduced in RFC 131, implemented in 16156, updated in 49019. Although it can be used on stable rustc, in practice you likely need nightly-only features to use it.

It is unlikely that Cargo will need to add any special handling for the format (except if the extension changes). Cargo already supports JSON target files. However, when std-aware Cargo becomes stable, this suddenly offers an opportunity for the target specification to be more usable on stable. The teams will need to decide if they are comfortable with proceeding with the current format, or if stabilizing std-aware cargo should be gated on changing it.

There have been a few issues brought up with the current format:

• Doubts about using JSON. Some have suggested switching to TOML.
• The format is somewhat tied to LLVM. There are some concerns that may make using other backends more difficult. Here is a suggestion for making the backend explicit.
• See the A-target-specs issues.

Please leave any comments about ideas, possible changes, whether or not this should gate anything, or if the current ("stable") syntax is sufficient.

[ids1024]

My personal opinion is that json is unsuited for hand written files of this sort, mainly due to the lack of comments. Looking at librustc_target/spec, there are a fair number of comments, suggesting they are indeed important to have in target specifications.

And it just seems oddly inconsistent when Rust otherwise uses toml. When I first saw the json spec files, I assumed that was a format handled by LLVM rather than Rust-specific, since it didn't really match the conventions of the Rust ecosystem.

Keeping json wouldn't be the end of the world, but it does seem overall worse to me.

And although not specifically related to std aware cargo, I personally would like it if librustc_target used the spec format internally; it could be parsed at build time assuming it's still desirable to compile them into the binary. This way there would only be one format for target spec files, instead of both Rust and json. This would also provide a great test for the format, since a good spec format should be suitable for replacing the current Rust code in librustc_target/spec.

[SimonSapin]

Although it can be used on stable rustc, in practice you likely need nightly-only features to use it.

Could you say more about this? What exactly is stable? (And so we’re likely stuck with) What unstable features are required?

Is it "only" that custom targets do not have a standard library available, and the source code of libcore and libstd use unstable features?

[ehuss]

Is it "only" that custom targets do not have a standard library available, and the source code of libcore and libstd use unstable features?

Correct. JSON targets are stable, but to use it you need a standard library. Or #![no_core]. I think it would be theoretically possible to build a custom standard library that can be used with a stable compiler, but it is quite difficult. If stable cargo did something like pass RUSTC_BOOTSTRAP to build the standard library, then this would be the first situation where .json files would be officially usable on stable. I'd like that to be an explicit decision to stabilize the format rather than a side effect.

[SimonSapin]

I think it would be theoretically possible to build a custom standard library that can be used with a stable compiler

I believe this is not possible. Doing anything beyond exporting a macro_rules! macro requires a few language items, and defining those is unstable.

So I think we can consider JSON targets to be de-facto unstable for now, and fair game for any breaking change?

I'd like that to be an explicit decision to stabilize the format rather than a side effect.

👍

[Ericson2314]

Cargo can treat the file as a black box and just hash it's contents in order to keep binaries for different platforms apart. There's no reason to force it's stabilization.

[ehuss]

Cargo can treat the file as a black box and just hash it's contents in order to keep binaries for different platforms apart. There's no reason to force it's stabilization.

If some version of std-aware cargo is stabilized, then it would as a consequence stabilize the format of the target file. That is, cargo +stable build --target=foo.json would suddenly be possible, and thus the json format would de facto be stabilized. We don't want to do that without the compiler team being explicitly OK with whatever the format is.

I'm not sure what the stability guarantees there are about rustc command-line flags, but I assume it is desired that they stay as compatible as possible.

Also, Cargo currently triggers off the file extension (.json currently). If it changes to .toml or whatever, then it would need to know about that as well.

It may be possible that the current format is fine, and any enhancements will be done independently in the future in a backwards-compatible fashion. This issue is just to make that decision explicit.

[Ericson2314]

If some version of std-aware cargo is stabilized, then it would as a consequence stabilize the format of the target file

Nope!

cargo +stable build --target=foo.json

All cargo needs to do is cache binaries, and pass the contents of that file to rustc. We do stabilize that rustc --target <file> works but do not stabilize what the file contains. Cargo is like the postal service, and rustc and address with the mailbox. Rustc has a "stable" mailbox, but the postman doesn't look at your mail.

Also, Cargo currently triggers off the file extension (.json currently). If it changes to .toml or whatever, then it would need to know about that as well.

That's a fair point, we could require a leading ./ or / for an "explicitly" relative or absolute path, and thereby not assume any particular extension.

[SimonSapin]

It sounds like you have a different idea of what “stable” means. A goal of the Rust project is to have every feature, API, or command-line argument be either stable with a promise not to make breaking changes, or unstable where it can’t be used on the Stable or Beta release channel. Anything else is considered a bug. Documentation is not enough to warn users that they should not rely on something not changing.

[Ericson2314]

I don't think that is in conflict. If we consider this de facto unstable we should make it de jure unstable in rustc. Now suppose Cargo stabilizes the flag as a black box as I say. The "end to end" interaction (what you correctly point is a core part of Rust's guarantee) is still unstable because rustc will complain about the use of the flag being unstable.

[ehuss]

Adding a tangentially related item here: The standard library currently isn't configured to support JSON files very well. In particular, in some cases, it looks at the TARGET name to determine what to do, instead of looking at cfg values. This sometimes requires naming the .json file with a substring that matches the desired target. Preferably it wouldn't be sensitive to the filename, so some places that need to be fixed to use CARGO_CFG_TARGET_* values instead of TARGET:

• https://github.com/rust-lang/rust/blob/master/library/std/build.rs
• https://github.com/rust-lang/rust/blob/master/library/profiler_builtins/build.rs
• https://github.com/rust-lang/backtrace-rs/blob/master/crates/backtrace-sys/build.rs
• https://github.com/rust-lang/rust/blob/master/library/unwind/build.rs
• https://github.com/rust-lang/compiler-builtins/blob/master/build.rs

[ids1024]

So:

• Use .toml for target specs, instead of json
• Clear up any details that are too closely tied to LLVM
• Allow a specification to inherit from another (Add an "inherits" key or similar to JSON target specs rust#32819)
• If the target spec format is working well, it should be suitable for use in librustc_target/spec instead of Rust code (though it can still end up compiled into the binary)
• Fix any cases where standard library is using target name rather than spec

I suppose making progress on this will require a formal RFC and/or some experimentation with an implementation.

The embedded workgroup and portability workgroup would likely also have interest in improvement here.

Edit: Added last bullet point.

[raphaelcohn]

@ids1024 Absolutely agree with using .toml (or any other simple, line-orientated format with comments that's source control friendly - JSON isn't), and that rustc should no longer have a 'special' list. I'd also like target inheritance (this has come up over the years) - most of the time, I'm tweaking an existing target.

There also needs to be interaction withe Cargo's config. I've always found it really confusing that I can define target properties in .cargo/config files, eg a list of -C target-feature. These belong to a target "JSON" file, and would would much better if targets supported inheritance.

An additional feature that I'd like targets to support is the version of the system's libc in use. This comes up with freebsd all the time (versions can be breaking) and even musl (with the change to 64-bit time in musl 1.20). Some libc's versions implicitly version the OS (freebsd, macos) and some don't (Linux + musl can have separate combinations). It would be nice to be able to specify this in a target, so it 'seeds' cfg values.

Features that might not belong in a target file but are needed for reproducible builds:-

• The value of the CROSS_COMPILE environment variable;
• The names of the linker and archiver (ld and ar);
• The search path for native libs (Cargo currently uses one suitable for the current platform, which is fugly, eg DYLD_FALLBACK_LIBRARY_PATH is not appropriate)

These do, though, need to be linked to a target file but may differ by build platform, eg they'd be different on my mac and my linux box.

[roblabla]

Another feature I often need: The ability to bundle target json with other files, and reference those other files from within the target JSON in a relative way. For instance, to pass linker scripts to the linker.

[raphaelcohn]

@roblabla For me, I'd like the whole build system to be relative. As someone's whose expended considerable time creating reproducible, cross-compiler toolchains, nothing is more valuable. Cargo's current design of searching upwards parent folders is insane - and a genuine security risk (/tmp, anyone)? It makes it impossible to encapsulate and be assured of reproducible builds.