Add end to end tests for the storybook playground block editor

[talldan]

Is your feature request related to a problem? Please describe.
A couple of times recently bugs have been introduced in the block editor package that could've been prevented by a small suite of end to end tests:

• Fix inserter expecting experimental settings to exist in the context #24554
• Fix block editor example in storybook #25976

Both issues were related to some configuration that shouldn't need to be specified but was present in the post editor, an easy mistake to make, but also easy for automated testing to catch.

We've often relied on user reports to solve these bugs, but they could be caught before introduction using end to end tests

The tests would include some basic interactions, adding/removing blocks, changing settings etc...

Potentially some of these tests could even run across multiple editors (post editor, standalone editor, FSE editor, widget, navigation) where the feature is consistent across those editors.

Describe the solution you'd like
Add e2e tests for the storybook example of the Block Editor:
https://wordpress.github.io/gutenberg/?path=/story/playground-block-editor--default

Describe alternatives you've considered
Alternatively, add a basic example within the plugin itself, similar to the demo page which has e2e tests.

[gziolo]

Actually, we discussed it several times that a subset of tests could work with Storybook instance only. In particular, the block editor seems to be a perfect fit as it by definition should work more or less the same on WordPress instance, in the mobile app, in the 3rd party app like Storybook. I'm 100% on board with the idea. It should be way much easier to set everything up than for WordPress instance as you don't need Docker 😃

[talldan]

On other projects I've worked on similar has been achieved by setting up test suites, which have some basic configuration, something like:

{
  name: 'storybook-tests',
  tests: [
    'editor/various/adding-blocks.test.js',
    // ...,
  ],
  config: {
    EDITOR_URL: 'https://localhost:12345/?path=/story/playground-block-editor--default'
  }
}

Some challenges:

• the storybook would need to be built and run per branch
• Lots of assertions we've written use expect( await getEditedPostContent() ).toMatchSnapshot();, which relies on a selector that I believe is specific to the post editor. I wonder what a good alternative could be? Maybe something wp.blocks.serialize( wp.data.select( 'core/block-editor' ).getBlocks() )?

[gziolo]

I personally don't like the idea of sharing test suites between different platforms. If the test can be run with Storybook only, then there is no need to repeat it using the full WordPress instance.

[gziolo]

I just wanted to reshare some previous work in the same area to give a better overview of what was already explored.

@ellatrix did some very promising explorations to write e2e like tests that use Storybook to run tests related to RichText functionality:

• RichText: add faster, separate e2e tests #17303 - it uses HTML file for testing
• Add playground e2e tests #17065 - it uses the playground (now in Storybook) for testing

There were also explorations to use visual testing:

• Chromatic that it's based on Storybook as it's developed by the same team: Experimenting Chromatic #22155
• percy.io that is built around Storybook as well: Added Visual Regression Testing via percy.io #18797.

We have been playing with snapshot testing for components as well, You can check #18031 where Storyshots integration with Storybook was enabled. It was removed a few months later.