Lazy, async instantiation and fetching (tracking)

[roberth]

A description of the solution space from an instantiation angle, having commonalities with eval-time fetching.

Is your feature request related to a problem? Please describe.

• Instantiation should be quick.
• Instantiation process forms a DAG, which allows some parallel processing.
• Instantiation isn't I/O but O; fundamentally only output, so most processing can happen asynchronously after being initiated from the evaluator; only IFD and the evaluator's callers may need to await.
• Fetching of sources has significant latent parallelism.

Ideally the I/O is non-blocking, so as not to need unnecessary thread blocking / switching / syncing, but I don't think this should be the focus yet. My use of "asynchronous" here mostly describes the computational nature of the domain; not necessarily its I/O control flow implementation.

Describe the solution you'd like

• Virtual file system overlaid onto the store, to implement write-back caching
  • writes immediately visible in the overlaid store FSO accessor
  • the underlying store receives the data asynchronously
  • has sync method to wait until a certain closure is written
    • ideally prioritizes it
• A thunk type that refers to a hash that hasn't been computed yet
• A thunk type that evaluates to a string concatenation
  • optimize certain string operations to avoid performing the concatenation
  • make derivationStrict return output path thunks of the form ConcatStrings(storeDir, "/", LazyHash(...), "-", name)
• Parallelism for strict data arguments, see GHC sparks
  • maybe this can also serve as the queue for all the hashing work
• Lazy fetchTree outPath path values #10252
• Virtual path semantics that are deterministic, lazy and suitable for performing deduplication of values by a path value key. #10689

Describe alternatives you've considered

Additional context

• The proposed items solve the non-determinism problems in Lazy trees #6530

• Multithreaded evaluator #10938

Priorities

Add 👍 to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-07-nix-team-meeting-minutes-167/50287/1