Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable custom messages for test failures while still rendering equality messages #234

Open
edkelly303 opened this issue Jun 24, 2024 · 0 comments

Comments

@edkelly303
Copy link

(Here is an idea that I discussed with @Janiczek at Elm Camp 2024)

I've been working on edkelly303/elm-exhaustive-generators, a package that exhaustively generates Elm values, which might potentially be useful in tests.

As part of the API, I've included an Exhaustive.test function that works much like Test.fuzz from elm-test, except that it takes an Exhaustive.Generator a instead of a Fuzz.Fuzzer a.

Here's an example of a test that will fail:

module Example exposing (..)

import Exhaustive
import Expect
import Test exposing (..)


zeroToTen : Exhaustive.Generator Int
zeroToTen =
    Exhaustive.values (List.range 0 11) -- oops, this should be `List.range 0 10`


suite : Test
suite =
    only <|
        Exhaustive.test zeroToTen "Input multiplied by 10 should never be larger than 100" <|
            \input ->
                input * 10 |> Expect.atMost 100

The output we get is:

Compiling > Starting tests

elm-test 0.19.1-revision12
--------------------------

Running 1 test. To reproduce these results, run: elm-test --fuzz 100 --seed 38488375388147

> Example
> Input multiplied by 10 should never be larger than 100

    110
    ╷
    │ Expect.atMost
    ╵
    100



TEST RUN FAILED

Duration: 186 ms
Passed:   0
Failed:   1                                                                                         

What I'd really like is to be able to insert some text into the test output to log the generated value that was passed into the test, and the index of this value in the sequence of terms generated by the Exhaustive.Generator. For example, something like this:

Compiling > Starting tests

elm-test 0.19.1-revision12
--------------------------

Running 1 test. To reproduce these results, run: elm-test --fuzz 100 --seed 38488375388147

> Example
> Input multiplied by 10 should never be larger than 100

    Test failed with generated value `11` (n = 11)

    110
    ╷
    │ Expect.atMost
    ╵
    100



TEST RUN FAILED

Duration: 186 ms
Passed:   0
Failed:   1                                                                                         

In the current elm-explorations/test API, Expect.onFail almost does what I'm looking for, but unfortunately the String that I provide as my custom error message completely replaces the default message, which means it no longer prints out this (very useful) bit of output:

    110
    ╷
    │ Expect.atMost
    ╵
    100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant