feat: Improve UX when Test Replay upload fails due to slow upload #30235

cacieprins · 2024-09-12T18:28:08Z

Closes

Additional details

Cypress monitors the upload of a Test Replay recording. If the upload speed drops below 102.4kbps over a period of five seconds, Cypress terminates the upload so that CI pipelines are not unnecessarily slowed. This value is determined by:

The default chunk size for an fs readStream is 64 kilobytes
The interval used to determine if an upload stream has stalled is 5 seconds
64 kB * 8 = 512kb; 512kbp / 5s = 102.4kbps

This PR changes that behavior such that:

A non-cryptic error message is displayed, providing recommended workarounds
App Capture code can provide a default interval value, so this duration can be tweaked without updating the Cypress app
Users can override the value with an environment variable
Uncategorized errors that occur when uploading a Test Replay recording will display a formatted error message.

Steps to test

limit machine bandwidth to lower than ~102kbps and increase the sampling interval to an appropriate amount by setting the CYPRESS_PROTOCOL_UPLOAD_SAMPLING_INTERVAL environment variable. If your bandwidth is set to 56kbps, for example, you should set the sampling interval to at least ten seconds (at 56kbps, it should take just over 9 seconds to transfer 512kb, or 64 kiloBytes)
the upload will not be cancelled

How has the user experience changed?

A new stream stall error message:

A new uncategorized error message:

PR Tasks

Have tests been added/updated?
Has a PR for user-facing changes been opened in cypress-documentation?
Have API changes been updated in the type definitions?

…consolidatd args for stream stall msg

…ocolArtifact

…hat with env var

jennifer-shehane · 2024-09-12T18:40:22Z

packages/errors/src/errors.ts

+        Warning: We encountered an error while upload the Test Replay recording of this spec.
+
+        These reults will not display Test Replay recordings.


@cacieprins typos

Suggested change

Warning: We encountered an error while upload the Test Replay recording of this spec.

These reults will not display Test Replay recordings.

Warning: We encountered an error while uploading the Test Replay recording of this spec.

These results will not display Test Replay recordings.

cypress · 2024-09-12T19:08:12Z

cypress Run #57146

Run Properties: Passed #57146 • b93916fcbc: Merge branch 'develop' into cacie/9143/stream-stall-enhancements

Project	`cypress`
Branch Review	`cacie/9143/stream-stall-enhancements`
Run status	`Passed #57146`
Run duration	`23m 53s`
Commit	`b93916fcbc: Merge branch 'develop' into cacie/9143/stream-stall-enhancements`
Committer	`Jennifer Shehane`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`8`
Pending	`1326`
Skipped	`0`
Passing	`29395`
View all changes introduced in this branch ↗︎

UI Coverage `44.89%`
Untested elements	`200`
Tested elements	`167`

Accessibility `91.3%`
Failed rules	`5 critical` `10 serious` `2 moderate` `2 minor`
Failed elements	`944`

MikeMcC399 · 2024-09-13T13:38:51Z

system-tests/__snapshots__/record_spec.js

@@ -3049,6 +3051,15 @@ exports['e2e record capture-protocol enabled protocol runtime errors db size too
  - Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
  - Test Replay - Failed Uploading after Xm, Ys ZZ.ZZms 2/2 - Spec recording too large: artifact is 1024 bytes, limit is 200 bytes

+Warning: We encountered an error while upload the Test Replay recording of this spec.
+


These lines still have the typos in them

Same below also

ryanthemanuel · 2024-09-16T20:07:58Z

packages/errors/src/errors.ts

+
+        If this error occurs often, the sampling period may be configured by setting the ${fmt.highlightSecondary('CYPRESS_PROTOCOL_UPLOAD_SAMPLING_INTERVAL')} environment variable to a higher value than ${fmt.stringify(error.maxActivityDwellTime)}.
+
+        To prevent long CI executing durations, this Test Replay recording will not be uploaded.


Suggested change

To prevent long CI executing durations, this Test Replay recording will not be uploaded.

To prevent long CI execution durations, this Test Replay recording will not be uploaded.

@cacieprins This still needs addressing.

ryanthemanuel · 2024-09-16T20:09:58Z

packages/errors/src/errors.ts

+
+        The upload transfer rate fell below ${fmt.highlightSecondary(`${kbpsThreshold}kbps`)} over a sampling period of ${fmt.highlightSecondary(`${error.maxActivityDwellTime}ms`)}.
+
+        If this error occurs often, the sampling period may be configured by setting the ${fmt.highlightSecondary('CYPRESS_PROTOCOL_UPLOAD_SAMPLING_INTERVAL')} environment variable to a higher value than ${fmt.stringify(error.maxActivityDwellTime)}.


I feel like this might flow better if we rearranged the paragraphs.

Warning... The upload... To prevent... The results for this... If this error occurs...

jennifer-shehane · 2024-09-18T17:30:33Z

packages/errors/test/unit/visualSnapshotErrors_spec.ts

@@ -719,6 +730,13 @@ describe('visual error templates', () => {
        withSystemError: [aggregateErrorWithSystemError],
      }
    },
+    CLOUD_PROTOCOL_UPLOAD_UNKNOWN_ERROR: () => {
+      const error = new Error('some message')


Will this be updated to be a more specific error message?

jennifer-shehane · 2024-09-18T17:31:32Z

packages/server/lib/cloud/artifacts/upload_artifacts.ts

-        console.log('')
-        errors.warning('CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE', error)
-      }
+      printProtocolUploadError(error)


jennifer-shehane · 2024-09-18T17:34:04Z

packages/server/lib/cloud/protocol.ts

@@ -320,7 +322,11 @@ export class ProtocolManager implements ProtocolManagerShape {
    debug(`uploading %s to %s with a file size of %s`, filePath, uploadUrl, fileSize)

    try {
-      await putProtocolArtifact(filePath, dbSizeLimit(), uploadUrl)
+      const samplingInterval = process.env.CYPRESS_TEST_REPLAY_UPLOAD_SAMPLING_INTERVAL ?


What is a user passes 'foo' to this env var, it'd evaluate to NaN, then what?

cacieprins and others added 8 commits September 11, 2024 13:06

new error message for stream stall upload failures

fa8f03f

new unknown error msg, extract printProtocolUploadError for testing, …

c494768

…consolidatd args for stream stall msg

rename env more appropriately, add sampling interval param to putProt…

0e519cf

…ocolArtifact

default to 5000, override with app capture protocol value, override t…

e27fe72

…hat with env var

update snapshots

a6529e1

fix math in upload stall error msg

819ab5f

changelog

c41d9ed

Merge branch 'develop' into cacie/9143/stream-stall-enhancements

23f1684

jennifer-shehane assigned cacieprins Sep 12, 2024

jennifer-shehane reviewed Sep 12, 2024

View reviewed changes

cacieprins added 4 commits September 12, 2024 15:26

update gql schema w/ new error msg, fix protocol tscheck

9807903

update test to use fn for app capture supplied interval

397af46

increase default stream stall interval to 10 seconds

7c76042

typos

331512f

MikeMcC399 reviewed Sep 13, 2024

View reviewed changes

cacieprins and others added 2 commits September 13, 2024 10:06

snapshots

334a035

Merge branch 'develop' into cacie/9143/stream-stall-enhancements

e6632ee

cacieprins marked this pull request as ready for review September 13, 2024 15:18

cacieprins requested review from AtofStryker, ryanjwilke and ryanthemanuel and removed request for ryanjwilke September 13, 2024 17:26

ryanthemanuel reviewed Sep 16, 2024

View reviewed changes

rename env var

e1fafcc

jennifer-shehane requested a review from ryanthemanuel September 18, 2024 17:27

Merge branch 'develop' into cacie/9143/stream-stall-enhancements

b93916f

jennifer-shehane requested changes Sep 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Improve UX when Test Replay upload fails due to slow upload #30235

feat: Improve UX when Test Replay upload fails due to slow upload #30235

cacieprins commented Sep 12, 2024 •

edited

Loading

jennifer-shehane Sep 12, 2024

cypress bot commented Sep 12, 2024 •

edited

Loading

MikeMcC399 Sep 13, 2024

ryanthemanuel Sep 16, 2024

jennifer-shehane Sep 18, 2024

ryanthemanuel Sep 16, 2024

jennifer-shehane Sep 18, 2024

jennifer-shehane Sep 18, 2024

jennifer-shehane Sep 18, 2024

		Warning: We encountered an error while upload the Test Replay recording of this spec.

		These reults will not display Test Replay recordings.


		If this error occurs often, the sampling period may be configured by setting the ${fmt.highlightSecondary('CYPRESS_PROTOCOL_UPLOAD_SAMPLING_INTERVAL')} environment variable to a higher value than ${fmt.stringify(error.maxActivityDwellTime)}.

		To prevent long CI executing durations, this Test Replay recording will not be uploaded.

	To prevent long CI executing durations, this Test Replay recording will not be uploaded.
	To prevent long CI execution durations, this Test Replay recording will not be uploaded.


		The upload transfer rate fell below ${fmt.highlightSecondary(`${kbpsThreshold}kbps`)} over a sampling period of ${fmt.highlightSecondary(`${error.maxActivityDwellTime}ms`)}.

		If this error occurs often, the sampling period may be configured by setting the ${fmt.highlightSecondary('CYPRESS_PROTOCOL_UPLOAD_SAMPLING_INTERVAL')} environment variable to a higher value than ${fmt.stringify(error.maxActivityDwellTime)}.

feat: Improve UX when Test Replay upload fails due to slow upload #30235

Are you sure you want to change the base?

feat: Improve UX when Test Replay upload fails due to slow upload #30235

Conversation

cacieprins commented Sep 12, 2024 • edited Loading

Additional details

Steps to test

How has the user experience changed?

PR Tasks

Choose a reason for hiding this comment

cypress bot commented Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cacieprins commented Sep 12, 2024 •

edited

Loading

cypress bot commented Sep 12, 2024 •

edited

Loading