Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decrease reporting threshold for snapshot comparisons #21486

Closed
stacey-gammon opened this issue Jul 31, 2018 · 5 comments
Closed

Decrease reporting threshold for snapshot comparisons #21486

stacey-gammon opened this issue Jul 31, 2018 · 5 comments
Labels
chore (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Feature:Reporting:Framework Reporting issues pertaining to the overall framework needs-team Issues missing a team label

Comments

@stacey-gammon
Copy link
Contributor

Due to #21485 we had to bump the reporting threshold comparison higher.

In order to turn reporting snapshot tests on asap, we bumped the threshold higher to account for it. This will lead to more false passes, so we should either fix the bug (probably not going to happen), take phantom specific baselines (more effort than it's worth?) or just wait till we deprecate phantom, then drop our phantom reporting tests and drop the threshold down once we are only comparing on chromium.

Given time and priorities, I'm guessing we are going to end up with the last option.

@stacey-gammon stacey-gammon added chore :Sharing (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead labels Jul 31, 2018
@timroes timroes added Team:Visualizations Visualization editors, elastic-charts and infrastructure and removed Team:Visualizations Visualization editors, elastic-charts and infrastructure :Sharing labels Sep 13, 2018
@tsullivan tsullivan self-assigned this Dec 13, 2018
@tsullivan
Copy link
Member

Note, as of this writing, it looks like all of the tests that compare images are disabled. They would be in test/reporting/functional/reporting.js and use the checkIfPdfsMatch, checkIfPngsMatch functions.

@tsullivan tsullivan removed their assignment Feb 11, 2019
@timroes timroes added Team:Stack Services and removed Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Jul 18, 2019
@bmcconaghy bmcconaghy added Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) and removed Team:Stack Services labels Dec 12, 2019
@bmcconaghy bmcconaghy added Team:Reporting Services and removed Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Dec 20, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

@tsullivan
Copy link
Member

Tests are re-enabled now as of #46076

I think I prefer keeping the threshhold as high as it needs to be, to keep them on. Even with a high threshhold, we still have a lot of benefits of automation:

  • make sure that Reporting integration didn't suddenty break because of DOM refactoring
  • make sure the PDF ends up with the expected number of pages
  • make sure the page is showing an actual Kibana App, not something such as an error screen

@joelgriffith thoughts?

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-services (Team:AppServices)

@tsullivan
Copy link
Member

The threshold has been lowered in #87600 so that the percentage of difference should be lower than 0.09

@sophiec20 sophiec20 added the Feature:Reporting:Framework Reporting issues pertaining to the overall framework label Aug 21, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Feature:Reporting:Framework Reporting issues pertaining to the overall framework needs-team Issues missing a team label
Projects
None yet
Development

No branches or pull requests

6 participants