-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decrease reporting threshold for snapshot comparisons #21486
Comments
Note, as of this writing, it looks like all of the tests that compare images are disabled. They would be in |
Pinging @elastic/kibana-reporting-services (Team:Reporting Services) |
Tests are re-enabled now as of #46076 I think I prefer keeping the threshhold as high as it needs to be, to keep them on. Even with a high threshhold, we still have a lot of benefits of automation:
@joelgriffith thoughts? |
Pinging @elastic/kibana-app-services (Team:AppServices) |
The threshold has been lowered in #87600 so that the percentage of difference should be lower than 0.09 |
Due to #21485 we had to bump the reporting threshold comparison higher.
In order to turn reporting snapshot tests on asap, we bumped the threshold higher to account for it. This will lead to more false passes, so we should either fix the bug (probably not going to happen), take phantom specific baselines (more effort than it's worth?) or just wait till we deprecate phantom, then drop our phantom reporting tests and drop the threshold down once we are only comparing on chromium.
Given time and priorities, I'm guessing we are going to end up with the last option.
The text was updated successfully, but these errors were encountered: