Opt-in for running installCheckPhase on cross builds

[Ma27]

It's not possible to run the installCheckPhase on cross-builds with stdenv.buildPlatform.canExecute stdenv.hostPlatform being false. However, in some cases that's still doable with stdenv.hostPlatform.emulator buildPackages (i.e. "${stdenv.hostPlatform.emulator buildPackages} $out/bin/rg").

See for instance #271072.

The only workaround I found so far is something like

{
  postInstall = ''
    export doInstallCheck=${toString (stdenv.hostPlatform.emulatorAvailable buildPackages)}
  '';
}

because whenever doInstallCheck gets passed to mkDerivation (also via overrideAttrs) it gets forcefully set to false if ! (stdenv.buildPlatform.canExecute stdenv.hostPlatform):

nixpkgs/pkgs/stdenv/generic/make-derivation.nix

Line 220 in 515794d

doInstallCheck' = doInstallCheck && stdenv.buildPlatform.canExecute stdenv.hostPlatform;

I understand that we can't just drop the behavior in here because we have a lot of packages that simply set doInstallCheck to true without taking cross into account (and all of that would need to be fixed now). On the other hand I'd like to not throw away basic checks on whether the build output works fine just because of that.

Some thoughts on how to tackle that issue:

• (long-term) if we had a module-system for drvs, we could allow derivations where install-check also runs on cross to set e.g. doInstallCheck = mkForce (stdenv.hostPlatform.emulatorAvailable buildPackages);.
• Add a forceDoInstallCheck argument to mkDerivation (ugly IMHO, yet another attribute and just a hack to "simulate" our module-system)
• deprecate the current behavior of doInstallCheck and throw a warning if the value changes from true to false because of canExecute for a transition period.

Also, would you consider the workaround I provided above acceptable in these cases?

CC

• people who do far more work on cross building in nixpkgs: @alyssais @amjoseph-nixpkgs @Ericson2314
• people from the nixpkgs architecture team because I'd be interested in their opinion: @DavHau @infinisil @roberth @tomberek @aakropotkin

---

Add a 👍 reaction to issues you find important.

[roberth]

What you really want is to put an override between mkDerivation and derivation.
Adding yet another overriding method would be crazy, so I agree with your view that we should flatten the sequence of functions into something that more resembles the module system. We don't necessarily have to use the module system for that.
Here's how that might look with overlay-like composition, with very simplistic merge logic that only handles the top attrset.
It's not implemented, but I know that it's feasible.

A first step should be to factor out the non-overriding logic of mkDerivation, as described here, so that these approaches can coexist without massive tech debt.

[alyssais]

IMO emulation is a fragile hack, and we should only be using it where it's absolutely necessary. I don't think install checks rise to that bar.

[Ma27]

IMO emulation is a fragile hack, and we should only be using it where it's absolutely necessary. I don't think install checks rise to that bar.

Sounds reasonable and understandable, but wanted to bring it up before deciding things on my own there because I didn't feel qualified to judge that on my own given that I'm not too experienced in tinkering with CPU arch emulators.

For @roberth's suggestion there's #273815 for discussion already, so I'd close this now, thanks!

[ghost]

IMO emulation is a fragile hack, and we should only be using it where it's absolutely necessary. I don't think install checks rise to that bar.

👍

Some thoughts on how to tackle that issue:

• Use EBBB derivations to put the installCheckPhase in a separate derivation with system set to the hostPlatform so it is dispatched to a builder that can run it natively.

Most people cross-compile not because they have zero machines of the hostPlatform type, but rather because they have lots of compute power of some other type.

[roberth]

@amjoseph-nixpkgs What is EBBB? I've been involved with dynamic derivations, but I haven't yet heard of this term, and can't find it despite some searches.

I don't think this kind of dynamism is needed, but I wouldn't dismiss it either, if it helps some use case or if it's more convenient.
What RFC 92 does not do, is allow you to build that "install check" derivation as part of the installPhase; it is less general than recursive-nix in that regard.

Something like the tests attribute (ie passthru.tests) can already run such checks natively in an extra derivation, but two problems remain before such a derivation can be applied at Nixpkgs scale.

• Not convenient enough. Nixpkgs has to
  • facilitate the construction of the test derivation
    • injection of dependencies (more splicing?)
    • invoke a nativePackages.stdenv.mkDerivation
    • provide access to the package sources or a copy of the build directory, if any of that is needed
  • still won't be as simple as a normal installPhase, so individual packages have to be migrated to support native install checks
• Unclear how the derivation should be wired into Nix invocations. tests attribute does not automatically become a build dependency. If it should happen automatically...
  • Could be done today with a "wrapper" derivation, but this is problematic
    • Wrapper is used to break the package -> test -> package cycle. Heavy handed solution that adds complexity and some cost at the store layer.
    • For the final "wrapper" package to match expectations (and not rebuild everything) it has to copy the untested derivation outputs. Need to rewrite self references, multi-output references, etc. Also fragile.
  • Could be tracked in string context to make that happen anyway, and do so concurrently (Track test derivations and parallelize building and testing nix#7662).
  • Evaluation performance might be prohibitive anyway

[ghost]

@amjoseph-nixpkgs What is EBBB? I

NixOS/rfcs#109 (comment)

EBEB = eval build eval build

EBEB = IFD = bad. EBB(B...) = RFC 92 = good.

What RFC 92 does not do, is allow you to build that "install check" derivation as part of the installPhase

Right, it has to be a "tail call" of the derivation, which is why you can't use it for the checkPhase. It has to be the last thing that the derivation does, which is (drum roll...):

the installCheckPhase

---

Something like the tests attribute

• Not convenient enough

I would say, more specifically: opt-in rather than opt-out. The checkPhase is so successful because you have to go out of your way to make it not happen.

The other problem is that most software doesn't install the test vectors into the --prefix. A lot of software even generates the test vectors as part of the build process. So you really need a snapshot of $NIX_BUILD_TOP at the end of the buildPhase. That snapshot would be an inputSrc of the builder-generated derivation (the "second B" in EBB).

Unclear how the derivation should be wired into Nix invocations. tests attribute does not automatically become a build dependency. If it should happen automatically...

NixOS/nix#7662 (comment)

[roberth]

which is (drum roll...):

the installCheckPhase

Not always but I suppose distPhase and/or custom phases sequences are quite rare, and probably not hard to get users to switch to an alternative.

EBEB = IFD

"IFD" sucks, but next thing you know, people start using RFR: Read From Realisation.
Can we please use IFD until an actual RFC main text establishes a new term to agree on?

So you really need a snapshot of $NIX_BUILD_TOP at the end of the buildPhase.

This is going to be a new source of impurities, as well as errors. Nix scans its outputs for references to the build directory. I think it's questionable whether it should, but fact is that it currently does, fwiw.