Deterministic solver output

[ezyang]

CC @grayjay

I wanted to add expect tests for the dependency solver but unfortunately the output wobbles depending on the state of built in packages. For example:

@@ -7,7 +7,6 @@
 cabal: Could not resolve dependencies:
 trying: custom-setup-1.0 (user goal)
 next goal: custom-setup:setup.Cabal (dependency of custom-setup-1.0)
-rejecting: custom-setup:setup.Cabal-1.24.0.0/installed-1.2... (conflict: custom-setup => custom-setup:setup.Cabal<1.20) 
-rejecting: custom-setup:setup.Cabal-1.18.0.0 (constraint from minimum version of Cabal used by Setup.hs requires >=1.20) 
+rejecting: custom-setup:setup.Cabal-1.18.1.5/installed-inp..., custom-setup:setup.Cabal-1.18.0.0 (constraint from minimum version of Cabal used by Setup.hs requires >=1.20) 
 fail (backjumping, conflict set: custom-setup, custom-setup:setup.Cabal)
 After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: custom-setup, custom-setup:setup.Cabal

Is there anything we could do to make this totally deterministic, even when the global packages change?

[grayjay]

I think we would need to somehow create a test package database that can be used with all versions of GHC. Is there anything specific that you want to test in the solver's output, or do you want to reduce the amount of output that must be excluded by the expect tests? The solver's output is also unstable because it can easily change after small code changes, especially changes to goal order.

[ezyang]

I think the benefit of putting solver errors into expect tests is that when we start improving solver error messages, we can see how the messages improve over time, and we can also see when a change regresses a message. Even if a solver change causes the messages to wobble, it's good to make sure that the wobble isn't significant.

I think it's reasonable for solver output tests to be run on an entirely synthetic test package database which is invariant over all versions of GHC. Will require a little engineering though.