Pop, object system with multiple-inheritance for nix

[fare]

Add a new experimental library: POP, a prototype object system with multiple inheritance.

I use POP a lot to customize the Gerbil packages, as defined in
#114449

Motivation for this change

I needed a prototype object system to customize my configurations. As compared to the many unassuming object systems already in Nix, I tweaked the API so the types are slightly simpler, then I added multiple inheritance to it because it enabled more modular configurations.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[fare]

@siraben @7c6f434c @edolstra @thoughtpolice

[fare]

Ping?

[infinisil]

As already mentioned on IRC, this is something that should probably be done through an RFC. This also helps with getting discussions and feedback going.

[edolstra]

I agree with @infinisil, this is a major change that should be done via an RFC. In general I think we should be very careful about adding something like this, since Nixpkgs already has way too many composition/overriding mechanisms, making Nix quite hard to learn (see Lisp Curse).

Alternatively you could make it available as a flake, since that way people can use it easily without having to add it to Nixpkgs.

[7c6f434c]

I agree with @infinisil, this is a major change that should be done via an RFC. In general I think we should be very careful about adding something like this, since Nixpkgs already has way too many composition/overriding mechanisms, making Nix quite hard to learn (see [Lisp Curse](http://www.winestockwebdesign.com/Essays/Lisp_Curse.html)).

An annoying part is that module system is really heavy (but we need a comparison there) and also pretty unusually designed compared to anything outside Nix. And the rest are used for being lighter (but weak).

Alternatively you could make it available as a flake, since that way people can use it easily without having to add it to Nixpkgs.

Flakes as an alternative to RFCs sounds a bit ironic… But the point of POP in the first place is to make maintenance of a language ecosystem inside Nixpkgs easier.

[nbp]

POP attempts to establish existing object model on top of the hacks which piled up on Nix. While similarities are nice to notice and might help reasoning, we should not forget the specificity of the problem we are attempting to solve.

This makes me wonder, would there be any equivalent to #10851 description in object literature?

[fare]

I created RFC 0091 for POP in Nixpkgs.

[fare]

Note regarding #10851 that multiple inheritance would allow you to define dependencies between your security patches, so that they are applied in order and never redo effects twice or undo each other's effects by being included twice.

[7c6f434c]

Yeah, applying patches correctly with proper dependencies computed in Nix code is exactly the kind of things that gets rejected because people pretend Nix can avoid being a general purpose computing language (and well, its evaluator is inefficient)

[7c6f434c]

(I would probably just merge it with POP inside Gambit directory for now, but with top-level lib people will whine too loudly — and it definitely needs a comparison with modules if it is on the top level)

[fare]

I could move the code to be under the gerbil directory, but so as not to cause chicken-and-egg issues, it has to be addressed from a name outside of the gerbil hierarchy, because the gerbil hierarchy depends on it. Thus, I could register, e.g. POP directly under pkgs, but that wouldn't make it nicer than being under pkgs.lib.

[7c6f434c]

What's the chicken-and-egg issue? You could have a local default.nix that takes newScope from the outer scope of pkgs, imports pop.nix, then constructs a new callPackage that has pop and calls gerbil-stable.nix etc. using it. And probably exposes pop somewhere…

[fare]

I suppose the chicken-and-egg issue is that I was trying to extract pop from the self value of gerbil-support, rather than from its super value. That could be solved that way. OR, just like I have a top-level entry for gerbil-support, I could have a top-level entry for POP, or a lib entry for it, pointing to the gerbil directory.

[fare]

Of all the modules out there, which do something non-trivial with respect to combination with other modules?

[fare]

I read the module documentation at https://nixos.org/manual/nixos/stable/index.html#sec-writing-modules and while it is very interesting indeed, it's a very different mechanism from the extension system / object system that POP extends.

• The module system has an imports system the effect of which is not very well defined. That's one case where POP's multiple inheritance could give a good meaning to defining dependencies between module definitions such that each is combined only once in dependency order into the mix.
• The "NixOS module system" actually defines three kinds of entities: modules, configurations, types, each with its own semantics, merging algorithm, etc. (Though, if you squint, modules could a special case of types, and configurations be the type being define by the module system.) The definition for a replacement of each of these entities would require a different use of POP's meta-object protocol.
• The merging algorithms used for modules, configs and types are much more elaborate than the one used by POP by default (mergeAttrset), but POP's meta-object protocol (instantiateMeta) supports specifying an alternate mergeInstance mechanism (though the current "toplevel" functions pop, basePop, kPop, etc., all use the default mechanism and won't let you override it for now), so in this sense modules or something module-like could be done in POP and benefit from POP's features and approach.
• Similarly, the warning mechanism could be achieved in a POP setting by using the topProto feature of instantiateMeta to handle warnings at the end of config merging. The same mechanism could more generally "finalize" the values being defined from the result of the merges to something "user-visible".
• POP has a system of dependencies and defaults that is a partial replacement for the priority system of module options: the defaults in POP replace low-priority settings, and the multiple inheritance of POP ensures that settings by children objects (importing modules) will override those of parent objects (imported modules), without having to assign priority numbers.

So I'm not sure what the question or challenge exactly are with respect to "comparing POP and the NixOS module system". On the one hand, POP is a much simpler mechanism meant to replace the Nix extension system and not the NixOS module system. On the other hand, POP does have extension mechanisms with which the more elaborate aspects of the module system could be plugged in, yielding a variant of the module system that deals with imports as dependencies to be combined each a single time in a proper order.

[fare]

I wrote an article about the principles underlying POP, that will be published at the Scheme and Functional Programming Workshop 2021: http://fare.tunes.org/files/cs/poof.pdf

[stale]

I marked this as stale due to inactivity. → More info

[blaggacao]

Alternatively you could make it available as a flake, since that way people can use it easily without having to add it to Nixpkgs.

This just happened: https://github.com/divnix/POP

(planned as a dependency for https://github.com/divnix/grafonnix)

[bb010g]

@fare Could you explain the POP-native approaches to the following Nixpkgs extension patterns?

First, many objects in Nixpkgs exhibit multiple "axes" of extensibility. Your standard package derivation object pkgs.hello (from effective proto finalPkgs: prevPkgs: prevPkgs // { hello = finalPkgs.callPackage ../by-name/he/hello/package.nix { }; }) exhibits, through an internal call to lib.customisation.makeOverridable:

1. extensibility on the overriding attrset argument to callPackage (.override (internally, overrideArgs)
   • provides a light alternative to instantiating an inherited package set with a "shallow" overlay per package
2. extensibility on the argument to stdenv.mkDerivation (.overrideAttrs) (if the attribute .overrideAttrs is present on the base result)
3. extensibility on the derivation arguments (.overrideDerivation), accounting for splicing.

The effective "shallow" overlay in overrideArgs only effects direct references to the package set in the function args, leaving both transitive package set references & references through machinery like callPackage or the pkgs instance that are passed in through the overridden arguments alone. The overrideAttrs and overrideDerivation extensibilities utilize makeOverridable's internal helper overrideResult, as they both rely on evaluation of the base object after function arguments are overridden.

I would figure that ideally, all this extensibility can & should be (able to be) handled in the same inheritance DAG, to avoid overly constraining the order of prototype composition. If a package function within a package set acts like a normal proto for the package set, then overrideResult is no longer necessary for the overrideAttrs and overrideDerivation extensibilities, but, if extensibilities at the package level, along with the package set level, are implemented via POP, the problem of multiple interfaces being targeted for extensibility still remain.

Additionally, you can have functions that wrap stdenv.mkDerivation, like mkPythonDerivation, that are inextensible by default (the case for mkPythonDerivation) unless they override .overrideAttrs on the result of the function they're overriding, probably via similar machinery to stdenv.mkDerivation's makeDerivationExtensible, making stdenv.mkDerivation's arguments inaccessible for further extension in the process. Ideally, allowing extension of one wouldn't disallow extension of the other.

My guess is that with POPs, mkPythonDerivation would be implemented via a proto that has the proto for stdenv.mkDerivation in its supers. Similarly, stdenv.mkDerivation would have a base derivation POP with native derivation arguments in its "interface", but when overriding at that level you'd want to ensure that you're before stdenv.mkDerivation's proto. AFAIK, you can only specify protos that need to come before you in the proto precedence list, and not protos that need to come after you in the proto precedence list. For comparison, consider that systemd units can specify both Before= and After= options.

Also, if I want to write a robust transformer function for package POPs that can be relied upon to transform the package object before the transformations of stdenv.mkDerivation but after any interface-affecting transformations, such as those by mkPythonDerivation (e.g. consider that mkPythonDerivation defaults .doInstallCheck to true and my transformations here want to act on that computed data, not see the raw undefaulted value of .doInstallCheck), i.e. can be relied upon to be sorted after interface-affecting protos but before stdenv.mkDerivation's protos in computed inheritance lists, how would I go about that?

Second, how would POPs handle the sort of detached extensibility present in objects like pythonInterpreters / python3 = pythonInterpreters.python3; / python3Packages = python3.pkgs;? The shared python-packages-base.nix Python package set "belongs" to pythonInterpreters, and it would make sense to allow applying extensions to all Python package set instantiations through that object, but extension of the instantiated Python package set needs to occur at python3 and/or python3Packages. Even if pythonInterpreters exposed no Python interpreter attributes that instantiate Python package sets, the uninstantiated shared Python packages set should be extensible via protos. (I've accomplished this pattern with the lib.customisation package scope machinery before.)

Also, while fiddling with my own mini POP implementation, I realized that lib/pop.nix currently stores & works with __meta__.extension. Why is an extension stored over the actual proto (__meta__.proto)? I don't know of anything that currently hard relies on __meta__.extension, and __meta__.proto better reflects what POPs actually care about.

[bb010g]

Also, if I want to write a robust transformer function for package POPs that can be relied upon to transform the package object before the transformations of stdenv.mkDerivation but after any interface-affecting transformations, such as those by mkPythonDerivation (e.g. consider that mkPythonDerivation defaults .doInstallCheck to true and my transformations here want to act on that computed data, not see the raw undefaulted value of .doInstallCheck), i.e. can be relied upon to be sorted after interface-affecting protos but before stdenv.mkDerivation's protos in computed inheritance lists, how would I go about that?

Reading through the “Prototypes: Object Orientation, Functionally” paper, it says that the C3 Linearization algorithm used for sorting the inheritance DAG into an inheritance list «ensures that the precedence list of an object always contains as ordered sub-lists (though not necessarily with consecutive elements) the precedence list of each of the object’s super-objects, as well as the list of direct supers» and that «It also favors direct supers appearing as early as possible in the precedence list». I believe my concern here is that, if a package POP object has a prototype of mkPythonDerivation, where the prototype mkPythonDerivation has the prototype in its stdenv.mkDerivation supers (i.e. as a direct super), and I extend that package POP object with a prototype P that has stdenv.mkDerivation as a direct super, I cannot guarantee that the resulting, extended POP object will use a computed precedence list in which mkPythonDerivation occurs after P.

In a sense, I want to automatically introduce a new, possibly virtual node to the inheritance DAG, one that represents the interface provided by the proto stdenv.mkDerivation and not necessarily stdenv.mkDerivation itself. The proto P has stdenv.mkDerivation as a direct super, and the proto(?) interface(stdenv.mkDerivation) has the proto P as an (implicit?) super. The proto mkPythonDerivation has at least interface(stdenv.mkDerivation) as a direct super, and it has stdenv.mkDerivation as at least an indirect super, because it transforms the interface. Alternatively, the proto P expresses that it maintains the interface of its direct super stdenv.mkDerivation, and this is used to generate the implicit interface(stdenv.mkDerivation) DAG node that has P as a direct super. The proto mkPythonDerivation could then not express that it maintains the interface of its direct super stdenv.mkDerivation, and consequently would implicitly also have interface(stdenv.mkDerivation) as a direct super.

I think the idea of (virtual) "interface" protos / inheritance DAG nodes are important, as both the protos stdenv.mkDerivation and mkPythonDerivation implement some Nixpkgs package derivation interface, and this interface possibly could be associated with a real proto (e.g. a fundamental package proto is the one that converts the pname & version attr pair into name). Actually, thinking about (pname×version→name) as a proto, wouldn't that be one that occurs late in the computed precedence list? It's (effectively) one of the final transformations on a package object before it's consumed by the Nix derivation machinery, and something you don't necessarily want to provide publicly in new interfaces. If that transformation applied after the transformations of stdenv.mkDerivation, we'd have a bit more formal of a guarantee that packages are using .pname and .version instead of .name. We can think of earlier prototypes as providing interfaces, and later prototypes as consuming them. For backwards compatibility, you want want to provide attr .name as a consistently updated result computed from (i.e. consuming) .pname and .version, but after a deprecation period .name could be moved to something that's not provided alongside .pname and .version. A fundamental proto early in the precedence list "provides" .pname and .version (even if just conceptually), and a different fundamental proto late in the precedence list consumes .pname and .version and provides .name. That latter proto could explicitly be providing data for, could advertise as a sub-proto, an interface-style proto for a native Nix derivation attrset.

Unfortunately, C3 linearization only natively supports supers, so something would need to be figured out to support nodes advertising subs. Either a DAG where nodes map to both subs and supers would need to be converted into an equivalent DAG where nodes map to only supers, or a different algorithm would be required.

If the patterns described here can be sanely handled without equipping protos with sub lists (in addition to their existing super lists), please let me know. Otherwise, I'd consider the subs issue a blocker for robust use of POP throughout Nixpkgs.

[fare]

If you want to ensure that P occurs after Q,
(1) you can have Q inherit from P, or
(2) you can have some other class R inherit from Q and P in this order.
Problem solved, thanks to C3! R is your "virtual node" in this case. That's exactly what C3 is for.