Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate sparkline tests #1224

Open
vks opened this issue Mar 29, 2022 · 1 comment
Open

Investigate sparkline tests #1224

vks opened this issue Mar 29, 2022 · 1 comment

Comments

@vks
Copy link
Collaborator

vks commented Mar 29, 2022

The sparkline tests are much more likely to fail than the statistical assumptions of the test suggest. This should be better understood by studying the error distribution.

I modified the test by feeding all the numbers in the range 0..100 to the seed_from_u64 and these seeds failed at least one expected error bucket.

1 2 3 8 10 12 14 15 25 38 44 46 48 49 54 63 66 73 81 82 87 93

This means that 22/100 random seeds failed the unit test with error more than 3 standard deviations from expected.
If I allow 4 standard deviations, only a single seed fails 87
I get similar results if I use from_entropy or switch back to the pcg32 implementation.

The other solution would ditch the truly obvious seed of 1 and pick a fair random number like 4 to move on.

Originally posted by @ironhaven in #1203 (comment)

@dhardy
Copy link
Member

dhardy commented Mar 30, 2022

Worth noting that this is using a million samples. It would be useful to calculate the chance of accidental failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants