-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into timeline-error-marker-avoid-negative-offset
- Loading branch information
Showing
273 changed files
with
3,918 additions
and
2,678 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
101 changes: 91 additions & 10 deletions
101
docs/developer/contributing/development-accessibility-tests.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,104 @@ | ||
| [[development-accessibility-tests]] | ||
| == Automated Accessibility Testing | ||
|
|
||
|
|
||
| To write an accessibility test, use the provided accessibility service `getService('a11y')`. Accessibility tests are fairly straightforward to write as https://github.com/dequelabs/axe-core[axe] does most of the heavy lifting. Navigate to the UI that you need to test, then call `testAppSnapshot();` from the service imported earlier to make sure axe finds no failures. Navigate through every portion of the UI for the best coverage. | ||
|
|
||
| An example test might look like this: | ||
| [source,js] | ||
| ---- | ||
| export default function ({ getService, getPageObjects }) { | ||
| const { common, home } = getPageObjects(['common', 'home']); | ||
| const a11y = getService('a11y'); /* this is the wrapping service around axe */ | ||
| describe('Kibana Home', () => { | ||
| before(async () => { | ||
| await common.navigateToApp('home'); /* navigates to the page we want to test */ | ||
| }); | ||
| it('Kibana Home view', async () => { | ||
| await retry.waitFor( | ||
| 'home page visible', | ||
| async () => await testSubjects.exists('homeApp') | ||
| ); /* confirm you're on the correct page and that it's loaded */ | ||
| await a11y.testAppSnapshot(); /* this expects that there are no failures found by axe */ | ||
| }); | ||
| /** | ||
| * If these tests were added by our QA team, tests that fail that require significant app code | ||
| * changes to be fixed will be skipped with a corresponding issue label with more info | ||
| */ | ||
| // Skipped due to https://github.com/elastic/kibana/issues/99999 | ||
| it.skip('all plugins view page meets a11y requirements', async () => { | ||
| await home.clickAllKibanaPlugins(); | ||
| await a11y.testAppSnapshot(); | ||
| }); | ||
| /** | ||
| * Testing all the versions and different views of of a page is important to get good | ||
| * coverage. Things like empty states, different license levels, different permissions, and | ||
| * loaded data can all significantly change the UI which necessitates their own test. | ||
| */ | ||
| it('Add Kibana sample data page', async () => { | ||
| await common.navigateToUrl('home', '/tutorial_directory/sampleData', { | ||
| useActualUrl: true, | ||
| }); | ||
| await a11y.testAppSnapshot(); | ||
| }); | ||
| }); | ||
| } | ||
| ---- | ||
|
|
||
| === Running tests | ||
| To run the tests locally: | ||
|
|
||
| [arabic] | ||
| . In one terminal window run | ||
| `node scripts/functional_tests_server --config test/accessibility/config.ts` | ||
| . In another terminal window run | ||
| `node scripts/functional_test_runner.js --config test/accessibility/config.ts` | ||
| . In one terminal window run: | ||
| + | ||
| [source,shell] | ||
| ----------- | ||
| node scripts/functional_tests_server --config test/accessibility/config.ts | ||
| ----------- | ||
|
|
||
| . When the server prints that it is ready, in another terminal window run: | ||
| + | ||
| [source,shell] | ||
| ----------- | ||
| node scripts/functional_test_runner.js --config test/accessibility/config.ts | ||
| ----------- | ||
|
|
||
| To run the x-pack tests, swap the config file out for | ||
| `x-pack/test/accessibility/config.ts`. | ||
|
|
||
| After the server is up, you can go to this instance of {kib} at | ||
| `localhost:5620`. | ||
|
|
||
| The testing is done using https://github.com/dequelabs/axe-core[axe]. | ||
| The same thing that runs in CI, can be run locally using their browser | ||
| plugins: | ||
| You can run the same thing that runs CI using browser plugins: | ||
|
|
||
| * https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US[Chrome] | ||
| * https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox] | ||
| * https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox] | ||
|
|
||
| === Anatomy of a failure | ||
|
|
||
| Failures can seem confusing if you've never seen one before. Here is a breakdown of what a failure coming from CI might look like: | ||
| [source,bash] | ||
| ---- | ||
| 1) Dashboard | ||
| create dashboard button: | ||
| Error: a11y report: | ||
| VIOLATION | ||
| [aria-hidden-focus]: Ensures aria-hidden elements do not contain focusable elements | ||
| Help: https://dequeuniversity.com/rules/axe/3.5/aria-hidden-focus?application=axeAPI | ||
| Elements: | ||
| - #example | ||
| at Accessibility.testAxeReport (test/accessibility/services/a11y/a11y.ts:90:15) | ||
| at Accessibility.testAppSnapshot (test/accessibility/services/a11y/a11y.ts:58:18) | ||
| at process._tickCallback (internal/process/next_tick.js:68:7) | ||
| ---- | ||
|
|
||
|
|
||
| * "Dashboard" and "create dashboard button" are the names of the test suite and specific test that failed. | ||
| * Always in brackets, "[aria-hidden-focus]" is the name of the axe rule that failed, followed by a short description. | ||
| * "Help: <url>" links to the axe documentation for that rule, including severity, remediation tips, and good and bad code examples. | ||
| * "Elements:" points to where in the DOM the failure originated (using CSS selector syntax). In this example, the problem came from an element with the ID `example`. If the selector is too complicated to find the source of the problem, use the browser plugins mentioned earlier to locate it. If you have a general idea where the issue is coming from, you can also try adding unique IDs to the page to narrow down the location. | ||
| * The stack trace points to the internals of axe. The stack trace is there in case the test failure is a bug in axe and not in your code, although this is unlikely. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.