Should we improve cell toolbar logic or disable cell toolbar in tests?

[krassowski]

Description

A bottleneck in the PR review are often the visual test failures. In addition to a few existing test cases which are flaky, a very common problem is the cell toolbar randomly showing up when it should not or randomly hiding when it should be visible for a split second. It is not a problem in normal usage when it is mostly unnoticeable, but it leads to many spurious failed tests, wasting hundreds of hours in CI time and in the time of maintainers, for example:

Actual	Expected	Diff

Actual	Expected	Diff

Reproduce

Screenshots taken from https://github.com/jupyterlab/jupyterlab/actions/runs/6640451043/job/18051878571?pr=15296 but every other PR has this issue. If you seen green status it likely means that one of the maintainers manually restarted the job after checking the failures manually (so they did that instead of improving the codebase or merging a PR).

[JasonWeill]

This may be related to known bugs #13115 and #13857 regarding the cell toolbar's display. In addition, there's #12223, an enhancement request to make the cell toolbar visible all the time, perhaps in collapsed form.

[krassowski]

Also, there is #15314 which might help with things.

[krassowski]

It continues to be a drag on maintenance and development (for example see incorrect snapshot update of notebook-run.test.ts-snapshots/notebook-panel-1-jupyterlab-linux.png in 1549442).

[krassowski]

In discussion on #15422 with @brichet we note that after refreshing the page the toolbar often does not hide even though it should. I can reliably reproduce this on main:

This looks like a bug in the logic which has implications on the maintainers ability to move forward because these tests are always flaky and often fail the run, adding about 15 to 30 minutes wasted to each PR review (we need to download the artifact, unpack, analyze, re-run a few times).

[krassowski]

This is also a hurdle for new contributors as seen in #16284.