Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Applitools Cypress workflow #19956

Merged
merged 43 commits into from
May 9, 2022
Merged

Conversation

geido
Copy link
Member

@geido geido commented May 5, 2022

SUMMARY

This PR adds a new Github workflow to enable visual regression testing against Cypress.

This is meant to be an initial test of the functionality and it only checks for a few main sections, including:

  • Explore
  • SqlLab
  • Dashboard view
  • Dashboard edit mode
  • Dashboard and Chart lists

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@adam-stasiak
Copy link
Contributor

Not familiar with applitools for cypress but I have general concern how it would work with randomized css-xxx items in DOM which can be changed between builds. Would this DOM verification be stable and reliable?

@codecov
Copy link

codecov bot commented May 5, 2022

Codecov Report

Merging #19956 (187f9a7) into master (f144de4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #19956   +/-   ##
=======================================
  Coverage   66.29%   66.29%           
=======================================
  Files        1712     1712           
  Lines       63962    63962           
  Branches     6731     6731           
=======================================
  Hits        42404    42404           
  Misses      19847    19847           
  Partials     1711     1711           
Flag Coverage Δ
hive 53.65% <ø> (ø)
javascript 51.28% <ø> (ø)
mysql 81.99% <ø> (ø)
postgres 82.04% <ø> (ø)
presto 53.51% <ø> (ø)
python 82.47% <ø> (ø)
sqlite 81.78% <ø> (ø)
unit 48.63% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f144de4...187f9a7. Read the comment docs.

Copy link
Member

@rusackas rusackas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Exciting times!

@rusackas
Copy link
Member

rusackas commented May 5, 2022

Not familiar with applitools for cypress but I have general concern how it would work with randomized css-xxx items in DOM which can be changed between builds. Would this DOM verification be stable and reliable?

It doesn't even look at the dom... it literally takes screenshots of the UI and compares them to prior runs. in theory, you could change ALL SORTS of styling code, but if it looks the same, the tests will pass.

cy.eyesOpen({
testName: 'Dashboards card-view',
});
cy.eyesCheckWindow('Dashboards loaded');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the significance of the name that you pass to eyesCheckWindow and do these names have to be unique? Looking at how these two tests here have the same string passed in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently these are enclosed within each test... they don't need to be unique it seems.

Copy link
Member

@eschutho eschutho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cool @geido!

@geido geido merged commit d0b8b1e into master May 9, 2022
hughhhh pushed a commit to hve-labs/superset that referenced this pull request May 11, 2022
* WIP

* Attempt Github Actions integration

* Add completion notification to workflow

* Update env

* Add new line

* Add license

* Fix whitespaces

* Fix Yaml indentation

* Fix afterEach

* Add initial tests

* Update config

* Use test secret

* Clean up

* Add batchName

* Disable logs - add secret

* Create separate workflow

* Clean up

* Update workflow

* Exclude applitools tests

* Update jobs name

* Run applitools tests separetely

* Fix path pattern

* Run once

* Add more initial tests

* Enhance tests

* Add dashboard edit mode test

* Move env

* Exclude applitools from sqllab test run

* Attempt pull_request_target

* Catch Applitools failures

* Clean up tests

* Add test step

* Add test step

* Remove step

* Fix SqlLab test

* Update CURL request

* Fix Yaml

* Add empty data to batch completion
philipher29 pushed a commit to ValtechMobility/superset that referenced this pull request Jun 9, 2022
* WIP

* Attempt Github Actions integration

* Add completion notification to workflow

* Update env

* Add new line

* Add license

* Fix whitespaces

* Fix Yaml indentation

* Fix afterEach

* Add initial tests

* Update config

* Use test secret

* Clean up

* Add batchName

* Disable logs - add secret

* Create separate workflow

* Clean up

* Update workflow

* Exclude applitools tests

* Update jobs name

* Run applitools tests separetely

* Fix path pattern

* Run once

* Add more initial tests

* Enhance tests

* Add dashboard edit mode test

* Move env

* Exclude applitools from sqllab test run

* Attempt pull_request_target

* Catch Applitools failures

* Clean up tests

* Add test step

* Add test step

* Remove step

* Fix SqlLab test

* Update CURL request

* Fix Yaml

* Add empty data to batch completion
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.0.0 labels Mar 13, 2024
@mistercrunch mistercrunch deleted the feat/applitools-cypress-workflow branch March 26, 2024 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 2.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants