[WIP] Addon-Test: Implement Addon Test TestProvider Backend #29171

valentinpalkovic · 2024-09-20T13:25:23Z

What I did

This PR introduces changes aimed at improving Storybook's Vitest integration by enabling the user to trigger a test run from Storybook's UI. The key changes include the addition of Vitest runner management functionality, integration with Storybook’s UI, and dependency updates to support improved functionality.

Vitest Configuration Updates

Modified Storybook UI's vitest.config.ts to include all stories (core, addons, lib) for testing. This was done temporarily to trigger tests for all stories, ensuring comprehensive testing of all components. This change has to be reverted because, currently, not all stories are running successfully in a Vitest Test environment.

Dependency Updates

Upgraded @vitest/browser and vitest from ^2.0.0 to ^2.1.1 to leverage newer features and stability improvements. We have to make sure, though, to remain compatible with < 2.1.1 versions. Unfortunately, a lot of types and internal API's of Vitest have changed significantly between 2.0 and 2.1.

Introduction of boot-test-runner.ts

A new file, boot-test-runner.ts, was added to manage child processes responsible for running Vitest tests. It restarts the Vitest runner upon failure, ensuring reliability. This process monitors Storybook core events such as TESTING_MODULE_RUN_REQUEST and TESTING_MODULE_CANCEL_TEST_RUN_REQUEST for running or canceling tests.

Custom Test Reporter (StorybookReporter)

Introduced StorybookReporter, a custom Vitest reporter that integrates Vitest's test reporting with Storybook's core events. It captures test progress and emits detailed results, including test assertions and failures, which are communicated to Storybook via the established channel.

Test Manager (test-manager.ts)

A new module to handle the orchestration of Vitest runs. It manages the Vitest lifecycle, including starting and stopping the test process, handling requests from the Storybook UI, and emitting progress updates to the UI.

Vitest Manager (vitest-manager.ts)

The Vitest Manager is a key component introduced to handle the orchestration and lifecycle of Vitest processes. It ensures proper initialization, execution, and termination of test runs, both for individual test cases and all tests at once.

The Vitest Manager communicates with the Test manager. It also ensures that Storybook test-specific settings, such as "watch mode," are honored during test runs.

The manager dynamically identifies the tests to run, either based on Storybook story imports or requests from the UI, ensuring that only the relevant stories are tested.

Sidebar Integration

Integrated the Vitest test runner into the Storybook sidebar. A new "Run test for all Stories" button was added, allowing users to trigger tests directly from the Storybook UI. A play button for the root Story groups was added as well to run tests for a specific set of stories. The sidebar now also includes a checkbox to toggle "Watch Mode," enabling continuous test watching for real-time feedback. All this change was introduced ONLY for testing purposes and doesn't fulfill or has anything to do with a proper solution, which implements event emitting/listening in a more generic fashion.

New Constants and Events

Added new core events (TESTING_MODULE_RUN_REQUEST, TESTING_MODULE_CANCEL_TEST_RUN_REQUEST, TESTING_MODULE_RUN_ALL_REQUEST, and more) to facilitate communication between the Storybook UI and the test runner.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

cd ./code
yarn
yarn nx run-many -t build --parallel=9
yarn storybook:ui
Visit Storybook and open your developer console
For testing purposes, new UI elements were added to the sidebar to trigger test runs. The play button for the story groups triggers a test run for all containing stories. The "Run test for all stories" button triggers a test run for all stories and the watch mode checkbox triggers "watch mode" requests for the backend. When watchmode is triggered, tests are not automatically running. A new test run has to be triggered.

Go to the "Network" tab and take a look at the Websocket Events. Notice how the proper events are sent and are received.

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

nx-cloud · 2024-09-20T13:28:00Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 998b8e1. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 1 target

nx run-many -t build --parallel=3

Sent with 💌 from NxCloud.

greptile-apps

_{15 file(s) reviewed, 12 comment(s)}
_{Edit PR Review Bot Settings}

code/addons/test/src/logger.ts

code/addons/test/src/node/boot-test-runner.ts

greptile-apps · 2024-09-20T13:27:48Z

code/addons/test/src/node/reporter.ts

+      // It is not simply (numPassedTests + numFailedTests) / numTotalTests
+      // because numTotalTests is dyanmic and can change during the run
+      // We need to calculate the progress based on the number of tests that have been run
+      progress: 0,


logic: Implement progress calculation based on completed tests

greptile-apps · 2024-09-20T13:27:49Z

code/addons/test/src/node/reporter.ts

+            return {
+              status,
+              duration: t.result?.duration || 0,
+              storyId: (t.meta as any).storyId,


style: Use type assertion or ensure storyId exists on t.meta

code/addons/test/src/node/test-manager.ts

greptile-apps · 2024-09-20T13:30:56Z

code/addons/test/src/node/vitest-manager.ts

+    }
+
+    await this.cancelCurrentRun();
+    await this.vitest!.runFiles(testList, true);


style: use optional chaining instead of non-null assertion

greptile-apps · 2024-09-20T13:31:25Z

code/addons/test/src/node/vitest.ts

+
+process.env.TEST = 'true';
+process.env.VITEST = 'true';
+process.env.NODE_ENV ??= 'test';


style: Consider using process.env.NODE_ENV = 'test' instead of ??= for consistency with other environment variable assignments

greptile-apps · 2024-09-20T13:31:26Z

code/addons/test/src/node/vitest.ts

+process.on('unhandledRejection', (reason) => {
+  throw new Error(`Unhandled Rejection: ${reason}`);
+});


logic: Throwing an error in unhandledRejection may lead to process termination. Consider logging and graceful handling

greptile-apps · 2024-09-20T13:32:24Z

code/core/src/core-events/data/testing-module.ts

+      storyId: string;
+    };
+
+export type Status = 'success' | 'failed' | 'pending';


style: This Status type is redundant as it's already defined in other payload types. Consider removing or using it consistently throughout the file

greptile-apps · 2024-09-20T13:33:16Z

code/core/src/manager/components/sidebar/Sidebar.tsx

+          <label>
+            <input type="checkbox" checked={watchMode} onChange={() => setWatchMode(!watchMode)} />{' '}
+            Watch Mode
+          </label>


style: Consider using a styled checkbox component for consistency with Storybook's UI.

…f things

…testprovider-backend

ghengeveld added 3 commits September 12, 2024 16:59

Add experimental_TEST_PROVIDER addon type

40ca30b

Test for experimental_TEST_PROVIDER addon type

b423331

Register Test addon as TEST_PROVIDER addon type

cea8021

valentinpalkovic added the bug label Sep 20, 2024

valentinpalkovic self-assigned this Sep 20, 2024

greptile-apps bot reviewed Sep 20, 2024

View reviewed changes

Addon-Test: Implement Addon Test TestProvider Backend

cb619ae

valentinpalkovic force-pushed the valentin/implement-addon-test-testprovider-backend branch from 9414580 to cb619ae Compare September 20, 2024 14:38

valentinpalkovic added the ci: do not merge label Sep 20, 2024

valentinpalkovic changed the title ~~Addon-Test: Implement Addon Test TestProvider Backend~~ [WIP] Addon-Test: Implement Addon Test TestProvider Backend Sep 20, 2024

ghengeveld added 2 commits September 23, 2024 23:19

Lazy boot for Vitest runner

123c6d2

Replace Node's process.fork with Execa and refactor/cleanup a bunch o…

72bd862

…f things

ghengeveld force-pushed the valentin/implement-addon-test-testprovider-backend branch from 9dca3bc to 72bd862 Compare September 24, 2024 12:01

yannbf and others added 2 commits September 24, 2024 14:27

make test run succeed

840fa04

Add unit tests and fix timeout logic

76c1c4e

yannbf changed the base branch from register-test-provider to unified-ui-testing September 24, 2024 14:20

Merge branch 'unified-ui-testing' into valentin/implement-addon-test-…

b71e2f2

…testprovider-backend

yannbf requested review from kylegach and jonniebigodes as code owners September 24, 2024 14:23

fix remaining tests

998b8e1

yannbf added ci:normal addon: test and removed ci: do not merge labels Sep 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Addon-Test: Implement Addon Test TestProvider Backend #29171

[WIP] Addon-Test: Implement Addon Test TestProvider Backend #29171

valentinpalkovic commented Sep 20, 2024 •

edited

Loading

nx-cloud bot commented Sep 20, 2024 •

edited

Loading

greptile-apps bot left a comment

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

greptile-apps bot Sep 20, 2024

[WIP] Addon-Test: Implement Addon Test TestProvider Backend #29171

Are you sure you want to change the base?

[WIP] Addon-Test: Implement Addon Test TestProvider Backend #29171

Conversation

valentinpalkovic commented Sep 20, 2024 • edited Loading

What I did

Vitest Configuration Updates

Dependency Updates

Introduction of boot-test-runner.ts

Custom Test Reporter (StorybookReporter)

Test Manager (test-manager.ts)

Vitest Manager (vitest-manager.ts)

Sidebar Integration

New Constants and Events

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

nx-cloud bot commented Sep 20, 2024 • edited Loading

☁️ Nx Cloud Report

greptile-apps bot left a comment

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 20, 2024

Choose a reason for hiding this comment

valentinpalkovic commented Sep 20, 2024 •

edited

Loading

nx-cloud bot commented Sep 20, 2024 •

edited

Loading