Jest axe a11y testing #127185

yuliacech · 2022-03-08T18:22:19Z

Summary

This PR adds jest-axe library to run axe automated a11y testing in jest. An axe_helpers file in kbn/test-jest-helpers exports a expectToBeAccessible function that can be used directly in tests. As an example, I added a11y tests for Indices tab in Index Management. To reuse a Kibana wide axe configuration, I moved a file with constants into kbn-test package.

Motivation

We can already run axe tests in functional and Cypress tests. Those tests are great for e2e testing as they work with real ES resources in a test cluster. The downside is that those tests can be slow to re-run, because often we need to spin up a new ES test cluster for a "clean slate" (for example, if a test creates an index and fails before clean up has finished). Also it's more difficult to setup an error state in these tests, for example, if a connection to ES is lost. And some conditions can't be changed from inside the test, for example when testing Cloud vs on-prem UI differences.

Because of these reasons, I personally prefer to use CITs - component integration tests. In those tests, we mount a high-order component, for example, Index Management component with tabs for indices, data streams and index templates. We mock Kibana dependencies and API requests and can test all UI interactions including loading and error states. This PR adds a possibility to also test for a11y violations on those UI states.

The downside of a11y testing in CITs that I noticed is that mocked versions of child components can cause false positives or false negatives. For example, EUI components are replaced by their mocked version defined in testenv files. This can be updated in the EUI repo (see elastic/eui#5702 and elastic/eui#5709)

…aration of concerns. Also added tests for indices tab in Index Management.

elastichelix · 2022-03-08T21:21:24Z

packages/kbn-test-jest-helpers/src/axe_helpers.ts

+  impactLevel?: ImpactValue
+): Promise<void> => {
+  const violations = await getA11yViolations(component, impactLevel);
+  expect(violations).toHaveLength(0);


Will the output for this include all of the violations?

That is a great question! I think I need to update the logic here, because it will only get 'critical' violations by default. I don't even think I need a parameter for impactLevel here. I'll ask in the a11y channel which level we want to test for.

Great call removing the impactLevel. Our goal is to find and remove all accessibility issues, regardless of their impact. It has been my experience that serious and moderate issues can cause significant UX problems in large enough quantities. They're not breaking UX issues per se, but each one causes friction and adds to the accumulated cognitive load.

elastichelix · 2022-03-08T21:23:42Z

x-pack/plugins/index_management/__jest__/client_integration/helpers/http_requests.ts

+
+    server.respondWith('GET', `${API_BASE_PATH}/mapping/:name`, [
+      status,
+      { 'Content-Type': 'application/json' },


would it make sense to represent this content type block as a constant HEADERS or something to avoid repeating or is there an expectation these might be different across tests?

yes, this file is very repetitive. I'll see how all the same lines can be refactored into a function or similar.

…es that are now in kbn-test package

…actored http requests helper file.

elasticmachine · 2022-03-10T18:00:23Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

elasticmachine · 2022-03-10T18:00:23Z

Pinging @elastic/kibana-accessibility (Project:Accessibility)

scottybollinger

Enterprise Search changes LGTM

elasticmachine · 2022-03-10T18:00:46Z

Pinging @elastic/apm-ui (Team:apm)

sabarasaba

nice work @yuliacech! Changes lgtm! Would love to see an issue created to patch the tests up once that eui#5709 lands in kibana 🚀

sabarasaba · 2022-03-14T09:12:57Z

x-pack/plugins/index_management/__jest__/a11y/indices_tab.a11y.test.ts

+    });
+    const { component } = testBed;
+    component.update();
+    // this is expected to fail and needs to be updated when eui#5674 is available in Kibana


👍🏼 Wondering if we should perhaps create a new issue to update this bit of code when this eui issue is fixed

I changed the comment to mention that the problem will be resolved with EUI 52.0.0 update (probably in this PR). I think we don't need an extra issue for that because when EUI version is updated, the tests will start to fail and will need to be updated as well.

sabarasaba · 2022-03-14T09:31:49Z

x-pack/plugins/index_management/__jest__/client_integration/helpers/http_requests.ts

-    ]);
-  };
+export const init = () => {
+  const server = sinon.fakeServer.create();


Slightly unrelated, but there's another pr that is trying to fix the incorrect usage of axios+sinon that we have throughout our apps in order to use HttpSetup instead. There's a few very nice abstractions that are created there that I can see you also thought about here! I'll create an issue to patch this up later on when the other PR is merged

spalger

kbn-test changes LGTM

1Copenut

Thank you for adding these tests @yuliacech. I added one comment affirming your decision to remove the impactLevel argument from the expectToBeAccessible() calls.

1Copenut · 2022-03-28T14:11:29Z

packages/kbn-test-jest-helpers/src/axe_helpers.ts

+  impactLevel?: ImpactValue
+): Promise<void> => {
+  const violations = await getA11yViolations(component, impactLevel);
+  expect(violations).toHaveLength(0);


Great call removing the impactLevel. Our goal is to find and remove all accessibility issues, regardless of their impact. It has been my experience that serious and moderate issues can cause significant UX problems in large enough quantities. They're not breaking UX issues per se, but each one causes friction and adds to the accumulated cognitive load.

kibana-ci · 2022-03-28T18:51:28Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/test`	214	224	+10
`@kbn/test-jest-helpers`	101	103	+2
total			+12

Unknown metric groups

API count

id	before	after	diff
`@kbn/test`	252	262	+10
`@kbn/test-jest-helpers`	128	132	+4
total			+14

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	7	9	+2

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	7	9	+2

History

💔 Build #34132 failed f7f9905
💔 Build #34024 failed 09fa429
💔 Build #34001 failed df62d6d
💔 Build #33963 failed f1748ff
💔 Build #29545 failed f8c0bc8
💔 Build #29542 failed 8c263df

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

yuliacech added 2 commits March 7, 2022 18:33

[A11y] Added axe testing to testBed to be used in jest tests (CITs)

72753f5

[A11y] Refactored axe helpers to be outside of testBed for better sep…

490a221

…aration of concerns. Also added tests for indices tab in Index Management.

elastichelix reviewed Mar 8, 2022

View reviewed changes

Merge branch 'main' into jest_axe_a11y_testing

00f248a

yuliacech mentioned this pull request Mar 9, 2022

[A11y] Added aria-label to the flyout testenv file elastic/eui#5702

Merged

1 task

yuliacech added 2 commits March 9, 2022 19:37

[A11y] Switched axe_helpers to use a Kibana wide axe config and rul…

3637886

…es that are now in kbn-test package

[A11y] Switched axe_helpers to use a Kibana wide axe config and rul…

a08d801

…es that are now in kbn-test package

yuliacech mentioned this pull request Mar 10, 2022

[A11y] Render aria-label as text in the icon testenv file elastic/eui#5709

Merged

1 task

yuliacech and others added 3 commits March 10, 2022 18:42

[A11y] Completed a11y tests in Index Management Indices tab. Also ref…

04be538

…actored http requests helper file.

Removed import comment

d89efe6

Merge branch 'main' into jest_axe_a11y_testing

c690915

yuliacech marked this pull request as ready for review March 10, 2022 17:57

yuliacech requested a review from a team as a code owner March 10, 2022 17:57

yuliacech requested a review from a team March 10, 2022 17:57

yuliacech requested review from a team as code owners March 10, 2022 17:57

yuliacech added Feature:Index Management Index and index templates UI Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v8.2.0 labels Mar 10, 2022

scottybollinger approved these changes Mar 10, 2022

View reviewed changes

botelastic bot added the Team:APM All issues that need APM UI Team support label Mar 10, 2022

yuliacech added 2 commits March 10, 2022 19:03

Reverted yarn.lock change

8c263df

Correct yarn.lock changes

f8c0bc8

sabarasaba approved these changes Mar 14, 2022

View reviewed changes

spalger approved these changes Mar 14, 2022

View reviewed changes

dgieselaar approved these changes Mar 25, 2022

View reviewed changes

yuliacech and others added 3 commits March 28, 2022 14:08

Merge branch 'main' into jest_axe_a11y_testing

f1748ff

[IM] Updated the branch after merging main

df62d6d

[IM] Reverted unrelated ResponseError interface changes

09fa429

1Copenut reviewed Mar 28, 2022

View reviewed changes

[IM] Fixed eslint issue with kbn/test import

f7f9905

yuliacech enabled auto-merge (squash) March 28, 2022 16:42

Merge branch 'main' into jest_axe_a11y_testing

e47dd65

yuliacech disabled auto-merge March 28, 2022 18:46

yuliacech added the backport:skip This commit does not require backporting label Mar 29, 2022

yuliacech merged commit bef90a5 into elastic:main Mar 29, 2022

alisonelizabeth mentioned this pull request Mar 31, 2022

Failing test: Jest Tests.x-pack/plugins/index_management/__jest__/a11y - A11y Indices tab index details flyout settings tab #128833

Closed

yuliacech deleted the jest_axe_a11y_testing branch November 22, 2022 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jest axe a11y testing #127185

Jest axe a11y testing #127185

yuliacech commented Mar 8, 2022 •

edited

Loading

elastichelix Mar 8, 2022

yuliacech Mar 9, 2022

1Copenut Mar 28, 2022

elastichelix Mar 8, 2022

yuliacech Mar 9, 2022

elasticmachine commented Mar 10, 2022

elasticmachine commented Mar 10, 2022

scottybollinger left a comment

elasticmachine commented Mar 10, 2022

sabarasaba left a comment

sabarasaba Mar 14, 2022

yuliacech Mar 28, 2022

sabarasaba Mar 14, 2022

spalger left a comment

1Copenut left a comment

1Copenut Mar 28, 2022

kibana-ci commented Mar 28, 2022

API count

ESLint disabled line counts

Total ESLint disabled count

Jest axe a11y testing #127185

Jest axe a11y testing #127185

Conversation

yuliacech commented Mar 8, 2022 • edited Loading

Summary

Motivation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 10, 2022

elasticmachine commented Mar 10, 2022

scottybollinger left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 10, 2022

sabarasaba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

1Copenut left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Mar 28, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled line counts

Total ESLint disabled count

History

yuliacech commented Mar 8, 2022 •

edited

Loading