Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase default Cypress command timeout #2263

Merged

Conversation

williamjallen
Copy link
Collaborator

One of our Cypress tests has been particularly flaky lately. This appears to be due to the CI runners executing slowly, with API wait times exceeding the default 4-second timeout Cypress provides. This PR increases the default command timeout to 30 seconds.

A recent flaky test result: https://open.cdash.org/tests/1583856484

Full output:
DevTools listening on ws://127.0.0.1:35925/devtools/browser/54698314-e37e-4b52-ab9b-92bf86d406aa

tput: No value for $TERM and no -T specified
================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.7.1                                                                         │
  │ Browser:        Electron 118 (headless)                                                        │
  │ Node Version:   v20.14.0 (/usr/bin/node)                                                       │
  │ Specs:          1 found (tests.cy.js)                                                          │
  │ Searched:       tests/cypress/e2e/tests.cy.js                                                  │
  │ Experiments:    experimentalStudio=true                                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  tests.cy.js                                                                     (1 of 1)


  the test page
    ✓ can be reached from the viewTest page (2577ms)
    ✓ can be reached from the queryTests page (904ms)
    ✓ can be reached from the testSummary page (862ms)
    (Attempt 1 of 3) loads the right navigation links
    (Attempt 2 of 3) loads the right navigation links
    1) loads the right navigation links
    (Attempt 1 of 3) displays information about the test
    (Attempt 2 of 3) displays information about the test
    2) displays information about the test
    ✓ loads the "Test Time" and "Failing/Passing" graphs (1682ms)


  4 passing (41s)
  2 failing

  1) the test page
       loads the right navigation links:
     AssertionError: Timed out retrying after 4000ms: expected '<li#header-nav-previous-btn.btn-enabled>' to have class 'btn-disabled'
      at Context.eval (webpack://cdash/./tests/cypress/e2e/tests.cy.js:44:41)

  2) the test page
       displays information about the test:
     AssertionError: Timed out retrying after 4000ms: expected '<div#main_content>' to contain 'on 2018-01-25 17:25:19'
      at Context.eval (webpack://cdash/./tests/cypress/e2e/tests.cy.js:84:7)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        6                                                                                │
  │ Passing:      4                                                                                │
  │ Failing:      2                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  6                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     41 seconds                                                                       │
  │ Spec Ran:     tests.cy.js                                                                      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- loads the right na     (1280x720)
     vigation links (failed).png                                                                    
  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- loads the right na     (1280x720)
     vigation links (failed) (attempt 2).png                                                        
  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- loads the right na     (1280x720)
     vigation links (failed) (attempt 3).png                                                        
  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- displays informati     (1280x720)
     on about the test (failed).png                                                                 
  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- displays informati     (1280x720)
     on about the test (failed) (attempt 2).png                                                     
  -  /cdash/tests/cypress/screenshots/tests.cy.js/the test page -- displays informati     (1280x720)
     on about the test (failed) (attempt 3).png                                                     


================================================================================
tput: No value for $TERM and no -T specified

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  tests.cy.js                              00:41        6        4        2        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     00:41        6        4        2        -        - 

@williamjallen williamjallen added this to the v3.5 milestone Jun 10, 2024
@zackgalbreath zackgalbreath added this pull request to the merge queue Jun 10, 2024
Merged via the queue into Kitware:master with commit 168c831 Jun 10, 2024
6 checks passed
@williamjallen williamjallen deleted the extend-cypress-api-load-time branch June 11, 2024 00:55
github-merge-queue bot pushed a commit that referenced this pull request Sep 10, 2024
#2263 bumped the default Cypress
timeout, which reduced the test flakiness, but did not eliminate it
entirely. This PR bumps the limit again in an attempt to resolve the
issue. In the future, I plan to rewrite the flaky tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants