Experimenting Chromatic

[sainthkh]

One of the topics of Core JS meeting at May 5th was using Chromatic in Gutenberg. I signed up and tested it with my forked Gutenberg and am sharing my experience and thoughts.

How it works.

1. Setup

As it advertises, the setup process is really simple.

1. Sign up

2. Execute commands

npm install --save-dev chromatic
npx chromatic --project-token=<my-token> --build-script-name=storybook:build

We need --build-script-name option here because our build script name isn't build-storybook.

3. Wait.

Then, we can use chromatic.

2. UI and Workflow

You can see the storybook build results by builds and PRs.

If there are visual changes, the list is provided:

And you can discuss the change on PR page. (It's impossible in discuss page.):

3. CI and Github checks.

I've used Github actions and tested 3 cases:

• Changing the primary button color with CSS
• No change in components
• Only changing component id text

As the color of the button has been changed, the status of case 1 is unfinished. The result is interesting because this change cannot be detected with storyshot because CSS code is changed. But chromatic detected it.

If there is no change in components, it simply returns pass.

And the final case solves the problem in #21485. When a component uses useInstanceId(), it changes ids when new storybook tests are added. And it breaks a lot of tests. That's why storyshot has been removed. But chromatic solves the problem. Even when the React/HTML code is changed, the visual result is the same, chromatic considers it's OK.

Pros and Cons

Pros

• We can be sure PRs don't include unintended visual changes in components.

Cons

• Chromatic does visual test. In other words, it cannot detect functional problems. It only alarms when IE cannot render some components correctly. But it doesn't alarm when something went wrong when a button is clicked in IE. For the latter case, we need the help of BrowserStack (e2e test with selenium + browserstack for legacy browsers like IE. #21528)

Things to Discuss

How should we set this up?

As mentioned above, I used Github Actions to test this. But in Github Actions, "Secrets are not passed to workflows that are triggered by a pull request from a fork." In other words, any pull requests from forked repos don't trigger Chromatic.

One of the solutions is to make the token public.

Interestingly, it is one of the recommended way of setting up CI according to the documentation. I also thought about it and it's not a bad idea, because there's nothing abusers can get with the token. They cannot use Chromatic for free because they need to approve the changes first to compare their changes but they cannot approve without joining the team.

[gziolo]

Awesome work exploring how we could take advantage of Chromatic ❤️

We need --build-script-name option here because our build script name isn't build-storybook.

I wasn't aware that they have a convention for it, but it seems fine as is.

Cross-linking the previous exploration percy.io that is built around Storybook as well: #18797.