Tests fail without network access

[xeals]

I'm updating the Numbat package for NixOS from 1.12.0 to 1.13.0 and encountered an issue where the example tests were failing that I struggled to reproduce -- it appeared the hard-coded values for the XKCD 2812 example were being evaluated as NaN (!).

For context, Nix builds all of its packages in a strict sandbox, which includes restricting network access. I was able to reproduce this build error by disabling my networking manually while building:

cargo test --profile release

Test log

failures:

---- examples_can_be_parsed_and_interpreted stdout ----
Testing example "../examples/acidity.nbt"
5.30103Testing example "../examples/air_resistance.nbt"
Testing example "../examples/barometric_formula.nbt"
Air pressure 1500 m above sea level: 845.586 hPaTesting example "../examples/binomial_coefficient.nbt"
Testing example "../examples/body_mass_index.nbt"
22.8571 BMITesting example "../examples/custom_dimensions.nbt"
Testing example "../examples/earth_mass.nbt"
Testing example "../examples/euler_approximation.nbt"
Testing example "../examples/factorial.nbt"
Testing example "../examples/flops.nbt"
672 GFLOPSTesting example "../examples/generics.nbt"
Testing example "../examples/hello_world.nbt"
hello worldTesting example "../examples/inter_dot_spacing.nbt"
Testing example "../examples/kinetic_energy.nbt"
Testing example "../examples/medication_dosage.nbt"
Total daily dose: 4500 mg/daySingle dose:      1500 mg/takingTesting example "../examples/molarity.nbt"
154.004 mmol/lTesting example "../examples/musical_note_frequency.nbt"
A5: 880 HzE4: 659.255 HzC4: 369.994 HzTesting example "../examples/nested_function_calls.nbt"
Testing example "../examples/numbat_syntax.nbt"
2 kwhhello worldvalue of pi = 3.14159sqrt(10) = 3.16228value of π ≈ 3.142= Length / TimeTesting example "../examples/paper_size.nbt"
Name    Width     Height        Area  ----   -------   --------   ----------A0      841 mm ×  1189 mm   9999.5 cm²
A1      594 mm ×   841 mm   4995.5 cm²
A2      420 mm ×   594 mm   2494.8 cm²
A3      297 mm ×   420 mm   1247.4 cm²
A4      210 mm ×   297 mm    623.7 cm²
A5      148 mm ×   210 mm    310.8 cm²
A6      105 mm ×   148 mm    155.4 cm²
A7       74 mm ×   105 mm     77.7 cm²
A8       52 mm ×    74 mm     38.5 cm²
A9       37 mm ×    52 mm     19.2 cm²
A10      26 mm ×    37 mm      9.6 cm²Testing example "../examples/pendulum.nbt"
Testing example "../examples/pipe_flow_rate.nbt"
Flow rate: 3.92699 l/sTesting example "../examples/population_growth.nbt"
Population in  20 years: 74591 personPopulation in 100 years: 369_453 personTesting example "../examples/print.nbt"
12 mhello worldpi = 3.141591 + 2 = 3Testing example "../examples/projectile_motion.nbt"
Testing example "../examples/readme-demo.nbt"
Energy of red photon: 1.87855 eVTesting example "../examples/recipe.nbt"
Milk:          750 mlFlour:         375 gSugar:         3 cupBaking powder: 6 tablespoonTesting example "../examples/stopping_distance.nbt"
Testing example "../examples/thermal_conductivity.nbt"
552 WTesting example "../examples/unicode.nbt"
Testing example "../examples/voyager.nbt"
Voyager sends data at a rate of 160 bps with 23 W.At a frequency of 8.3 GHz, this amounts to 4e24 photons/s.A 70 m dish on Earth will receive 1.3 aW of power.This corresponds to 240188 photons/s.Which means 1501 photons/bit.Testing example "../examples/what_if_11.nbt"
It happens once every ~193.96 yrsTesting example "../examples/what_if_158.nbt"
Radioactivity of potassium: 30.9526 Bq/gRadioactivity of a banana: 13.9596 Bq/bananaPower per banana: 2.95676 pW/bananaBananas per household: 1.15748e+14 banana/householdTesting example "../examples/workhours.nbt"
500 workday300 workday5.0 FTETesting example "../examples/xkcd_2585.nbt"
I can ride my bike at 45 mph.If you round.Testing example "../examples/xkcd_2812.nbt"
Option A: On the roof, south facingNaN $/yrthread 'examples_can_be_parsed_and_interpreted' panicked at numbat/tests/prelude_and_examples.rs:15:5:
Failed with: Err(
    RuntimeError(
        AssertEq3Failed(
            Span {
                start: SourceCodePositition {
                    byte: 605,
                    line: 24,
                    position: 11,
                },
                end: SourceCodePositition {
                    byte: 614,
                    line: 24,
                    position: 20,
                },
                code_source_id: 43,
            },
            Quantity {
                value: Number(
                    NaN,
                ),
                unit: Product {
                    factors: [
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "dollar",
                                canonical_name: CanonicalName {
                                    name: "$",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        NaN,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "euro",
                                                    canonical_name: CanonicalName {
                                                        name: "€",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Base,
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: 1,
                                denom: 1,
                            },
                        },
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "year",
                                canonical_name: CanonicalName {
                                    name: "yr",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        365.2421881,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "day",
                                                    canonical_name: CanonicalName {
                                                        name: "day",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Derived(
                                                        Number(
                                                            24.0,
                                                        ),
                                                        Product {
                                                            factors: [
                                                                UnitFactor {
                                                                    unit_id: UnitIdentifier {
                                                                        name: "hour",
                                                                        canonical_name: CanonicalName {
                                                                            name: "h",
                                                                            accepts_prefix: AcceptsPrefix {
                                                                                short: true,
                                                                                long: false,
                                                                            },
                                                                        },
                                                                        kind: Derived(
                                                                            Number(
                                                                                60.0,
                                                                            ),
                                                                            Product {
                                                                                factors: [
                                                                                    UnitFactor {
                                                                                        unit_id: UnitIdentifier {
                                                                                            name: "minute",
                                                                                            canonical_name: CanonicalName {
                                                                                                name: "min",
                                                                                                accepts_prefix: AcceptsPrefix {
                                                                                                    short: true,
                                                                                                    long: false,
                                                                                                },
                                                                                            },
                                                                                            kind: Derived(
                                                                                                Number(
                                                                                                    60.0,
                                                                                                ),
                                                                                                Product {
                                                                                                    factors: [
                                                                                                        UnitFactor {
                                                                                                            unit_id: UnitIdentifier{
                                                                                                                name: "second",
                                                                                                                canonical_name: CanonicalName {
                                                                                                                    name: "s",
                                                                                                                    accepts_prefix:AcceptsPrefix {
                                                                                                                        short: true,
                                                                                                                        long: false,
                                                                                                                    },
                                                                                                                },
                                                                                                                kind: Base,
                                                                                                            },
                                                                                                            prefix: Metric(
                                                                                                                0,
                                                                                                            ),
                                                                                                            exponent: Ratio {
                                                                                                                numer: 1,
                                                                                                                denom: 1,
                                                                                                            },
                                                                                                        },
                                                                                                    ],
                                                                                                },
                                                                                            ),
                                                                                        },
                                                                                        prefix: Metric(
                                                                                            0,
                                                                                        ),
                                                                                        exponent: Ratio {
                                                                                            numer: 1,
                                                                                            denom: 1,
                                                                                        },
                                                                                    },
                                                                                ],
                                                                            },
                                                                        ),
                                                                    },
                                                                    prefix: Metric(
                                                                        0,
                                                                    ),
                                                                    exponent: Ratio {
                                                                        numer: 1,
                                                                        denom: 1,
                                                                    },
                                                                },
                                                            ],
                                                        },
                                                    ),
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: -1,
                                denom: 1,
                            },
                        },
                    ],
                },
            },
            Span {
                start: SourceCodePositition {
                    byte: 616,
                    line: 24,
                    position: 22,
                },
                end: SourceCodePositition {
                    byte: 625,
                    line: 24,
                    position: 31,
                },
                code_source_id: 43,
            },
            Quantity {
                value: Number(
                    NaN,
                ),
                unit: Product {
                    factors: [
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "dollar",
                                canonical_name: CanonicalName {
                                    name: "$",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        NaN,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "euro",
                                                    canonical_name: CanonicalName {
                                                        name: "€",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Base,
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: 1,
                                denom: 1,
                            },
                        },
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "year",
                                canonical_name: CanonicalName {
                                    name: "yr",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        365.2421881,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "day",
                                                    canonical_name: CanonicalName {
                                                        name: "day",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Derived(
                                                        Number(
                                                            24.0,
                                                        ),
                                                        Product {
                                                            factors: [
                                                                UnitFactor {
                                                                    unit_id: UnitIdentifier {
                                                                        name: "hour",
                                                                        canonical_name: CanonicalName {
                                                                            name: "h",
                                                                            accepts_prefix: AcceptsPrefix {
                                                                                short: true,
                                                                                long: false,
                                                                            },
                                                                        },
                                                                        kind: Derived(
                                                                            Number(
                                                                                60.0,
                                                                            ),
                                                                            Product {
                                                                                factors: [
                                                                                    UnitFactor {
                                                                                        unit_id: UnitIdentifier {
                                                                                            name: "minute",
                                                                                            canonical_name: CanonicalName {
                                                                                                name: "min",
                                                                                                accepts_prefix: AcceptsPrefix {
                                                                                                    short: true,
                                                                                                    long: false,
                                                                                                },
                                                                                            },
                                                                                            kind: Derived(
                                                                                                Number(
                                                                                                    60.0,
                                                                                                ),
                                                                                                Product {
                                                                                                    factors: [
                                                                                                        UnitFactor {
                                                                                                            unit_id: UnitIdentifier{
                                                                                                                name: "second",
                                                                                                                canonical_name: CanonicalName {
                                                                                                                    name: "s",
                                                                                                                    accepts_prefix:AcceptsPrefix {
                                                                                                                        short: true,
                                                                                                                        long: false,
                                                                                                                    },
                                                                                                                },
                                                                                                                kind: Base,
                                                                                                            },
                                                                                                            prefix: Metric(
                                                                                                                0,
                                                                                                            ),
                                                                                                            exponent: Ratio {
                                                                                                                numer: 1,
                                                                                                                denom: 1,
                                                                                                            },
                                                                                                        },
                                                                                                    ],
                                                                                                },
                                                                                            ),
                                                                                        },
                                                                                        prefix: Metric(
                                                                                            0,
                                                                                        ),
                                                                                        exponent: Ratio {
                                                                                            numer: 1,
                                                                                            denom: 1,
                                                                                        },
                                                                                    },
                                                                                ],
                                                                            },
                                                                        ),
                                                                    },
                                                                    prefix: Metric(
                                                                        0,
                                                                    ),
                                                                    exponent: Ratio {
                                                                        numer: 1,
                                                                        denom: 1,
                                                                    },
                                                                },
                                                            ],
                                                        },
                                                    ),
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: -1,
                                denom: 1,
                            },
                        },
                    ],
                },
            },
            Quantity {
                value: Number(
                    NaN,
                ),
                unit: Product {
                    factors: [
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "dollar",
                                canonical_name: CanonicalName {
                                    name: "$",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        NaN,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "euro",
                                                    canonical_name: CanonicalName {
                                                        name: "€",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Base,
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: 1,
                                denom: 1,
                            },
                        },
                        UnitFactor {
                            unit_id: UnitIdentifier {
                                name: "year",
                                canonical_name: CanonicalName {
                                    name: "yr",
                                    accepts_prefix: AcceptsPrefix {
                                        short: true,
                                        long: false,
                                    },
                                },
                                kind: Derived(
                                    Number(
                                        365.2421881,
                                    ),
                                    Product {
                                        factors: [
                                            UnitFactor {
                                                unit_id: UnitIdentifier {
                                                    name: "day",
                                                    canonical_name: CanonicalName {
                                                        name: "day",
                                                        accepts_prefix: AcceptsPrefix {
                                                            short: true,
                                                            long: false,
                                                        },
                                                    },
                                                    kind: Derived(
                                                        Number(
                                                            24.0,
                                                        ),
                                                        Product {
                                                            factors: [
                                                                UnitFactor {
                                                                    unit_id: UnitIdentifier {
                                                                        name: "hour",
                                                                        canonical_name: CanonicalName {
                                                                            name: "h",
                                                                            accepts_prefix: AcceptsPrefix {
                                                                                short: true,
                                                                                long: false,
                                                                            },
                                                                        },
                                                                        kind: Derived(
                                                                            Number(
                                                                                60.0,
                                                                            ),
                                                                            Product {
                                                                                factors: [
                                                                                    UnitFactor {
                                                                                        unit_id: UnitIdentifier {
                                                                                            name: "minute",
                                                                                            canonical_name: CanonicalName {
                                                                                                name: "min",
                                                                                                accepts_prefix: AcceptsPrefix {
                                                                                                    short: true,
                                                                                                    long: false,
                                                                                                },
                                                                                            },
                                                                                            kind: Derived(
                                                                                                Number(
                                                                                                    60.0,
                                                                                                ),
                                                                                                Product {
                                                                                                    factors: [
                                                                                                        UnitFactor {
                                                                                                            unit_id: UnitIdentifier{
                                                                                                                name: "second",
                                                                                                                canonical_name: CanonicalName {
                                                                                                                    name: "s",
                                                                                                                    accepts_prefix:AcceptsPrefix {
                                                                                                                        short: true,
                                                                                                                        long: false,
                                                                                                                    },
                                                                                                                },
                                                                                                                kind: Base,
                                                                                                            },
                                                                                                            prefix: Metric(
                                                                                                                0,
                                                                                                            ),
                                                                                                            exponent: Ratio {
                                                                                                                numer: 1,
                                                                                                                denom: 1,
                                                                                                            },
                                                                                                        },
                                                                                                    ],
                                                                                                },
                                                                                            ),
                                                                                        },
                                                                                        prefix: Metric(
                                                                                            0,
                                                                                        ),
                                                                                        exponent: Ratio {
                                                                                            numer: 1,
                                                                                            denom: 1,
                                                                                        },
                                                                                    },
                                                                                ],
                                                                            },
                                                                        ),
                                                                    },
                                                                    prefix: Metric(
                                                                        0,
                                                                    ),
                                                                    exponent: Ratio {
                                                                        numer: 1,
                                                                        denom: 1,
                                                                    },
                                                                },
                                                            ],
                                                        },
                                                    ),
                                                },
                                                prefix: Metric(
                                                    0,
                                                ),
                                                exponent: Ratio {
                                                    numer: 1,
                                                    denom: 1,
                                                },
                                            },
                                        ],
                                    },
                                ),
                            },
                            prefix: Metric(
                                0,
                            ),
                            exponent: Ratio {
                                numer: -1,
                                denom: 1,
                            },
                        },
                    ],
                },
            },
        ),
    ),
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    examples_can_be_parsed_and_interpreted

test result: FAILED. 6 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.51s

This was mildly misleading as the error manifests at runtime either as a runtime error ("Could not load exchange rates") or an unknown identifier while typechecking (with the config option exchange-rates.fetching-policy = "never").

[xeals]

Bisected to 9073f19, which modifies the test under question to remove the $ stub and auto-import the currencies module into the test context.

[sharkdp]

I'm updating the Numbat package for NixOS from 1.12.0 to 1.13.0 and encountered an issue where the example tests were failing that I struggled to reproduce -- it appeared the hard-coded values for the XKCD 2812 example were being evaluated as NaN (!).

Thank you for reporting this. This is really good to know. The tests should work without having access to the internet, of course. The reason that you're seeing NaN values is a deliberate choice. When exchange rates are not available, we still want to provide the units for currencies. We just can't give them any conversion rates w.r.t. to the "base" currency EUR. Setting those values to NaN was the best I could come up with so far, but maybe it's time to rethink this behavior. Another option would be to provide each currency as a separate "base" unit. This way, we still couldn't convert them, but we wouldn't have to pick a number for the conversion factor.

For context, Nix builds all of its packages in a strict sandbox, which includes restricting network access.

Cool!

This was mildly misleading as the error manifests at runtime either as a runtime error ("Could not load exchange rates") or an unknown identifier while typechecking (with the config option exchange-rates.fetching-policy = "never").

It's actually the same in the test output log (look for RuntimeError or AssertEq3Failed). It's just the developer view (debug print) instead of the user-facing view of that runtime error. I agree that it look hideous.

Bisected to 9073f19, which modifies the test under question to remove the $ stub and auto-import the currencies module into the test context.

Thank you for the analysis. Now I remember why I used unit $ originally — thanks! 😄. I'll look into this.

The problem is that I don't want to encourage users to write unit $ in their scripts, as that is usually not needed. $ is already available. And this example is part of the docs (https://numbat.dev/doc/example-xkcd_2812.html). I think maybe I'll just remove those assertions (we remove them anyway when rendering the docs).