-
Notifications
You must be signed in to change notification settings - Fork 28
Test Format Definition V2
NOTE:
- This page provides specifications useful for both test writing and test file processing. Test Format V2 replaces Test Format V1 Definition for new versions of test plans starting September 2023.
- The ARIA-AT web site will continue to support the display of test plans and reports in the V1 format.
- Development of V2 of the test format is documented in ARIA-AT issue 974.
Each test plan is defined by a set of files in its own directory inside the tests directory in this repository. The name of the test plan subdirectory is named to reflect the test case, e.g, "alert" or "horizontal-slider". In the below list of files, the directory for each plan is represented by the string PLAN_DIRECTORY.
The test build scripts that generate files for the ARIA-AT web site (App) extract and translate content from the following files:
tests/PLAN_DIRECTORY/data/tests.csv
tests/PLAN_DIRECTORY/data/assertions.csv
- One file for each covered AT with a file name format of
tests/PLAN_DIRECTORY/data/AT_KEY-commands.csv
whereAT_KEY
is the key for an AT defined insupport.json
. Example:jaws-commands.csv
. tests/PLAN_DIRECTORY/data/scripts.csv
tests/PLAN_DIRECTORY/data/references.csv
tests/commands.json
tests/support.json
Specifications for the content of each CSV file are in the following sections. Usage of each JSON file is defined where applicable to the CSV file content.
This file specifies information about the tests included in a test plan, including their names, the order in which they are presented, and the assertions covered by each test.
Note that:
- It is located in the directory
tests/PLAN_DIRECTORY/data
. - The first row is assumed to contain column titles and will be ignored.
- Otherwise, the order of rows in the file is inconsequential.
Specifications for the information contained in each column are defined in the following sections.
Camel case string that identifies the test.
Allowed characters are a-z
, 0-9
, and dash ("-").
String that names the test. This string appears in headings that title test pages and report sections and in the row headers of report tables.
The title string should:
- Be written with sentence capitalization, i.e., the first word is capitalized.
- Should not have a period at the end.
- describe the user task the test is evaluating in straightforward and brief terms that enable people reading reports to understand what users of this AT can and cannot do when they encounter an element represented by the test case.
Examples:
- Navigate forwards to a slider
- Navigate backwards to a slider
- Request information about a slider
- Increment a slider by one step
Preferred verbs and objects for screen reader tests include the following.
Phrase | Usage Notes |
---|---|
Navigate forwards | Use for tasks where users will move a reading cursor or application focus in a forwards direction. |
Navigate backwards | Use for tasks where users will move a reading cursor or application focus in a backwards direction. |
Request information | Use for tasks where users will acquire information about the current element. "Request" is preferable to "Read" because "Read" could be interpreted as involving movement of a reading cursor. |
Activate | Use for tasks where the user will perform the default action on an element. |
Increment or Decrement | Use for tasks where users will increase or decrease the value of an element. |
Input value | Use for tasks where users will type a value. |
The above guidance is not intended to provide an exclusive or exhaustive list.
A positive integer that controls the order of presentation of tests in the test runner and reports. By default, tests will be presented in ascending order using the values in this column.
The values in this column allow the test author to specify any order for the tests. Thus, when a new test is added, it can be placed anywhere in the sequence.
The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js
directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv
file for command-button
refers to file tests/command-button/data/js/focusOnFirstLink.js
.
This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.
This is a sentence that describes the user task that is being evaluated by the test. It is presented as one of the elements in the list of instructions given to testers when executing a test.
The sentence:
- Includes a verb that is synonymous with the verb used in the test name.
- It may start with a prepositional phrase that provides information about the starting point of the task being performed.
- Note: If a test will be performed with both reading cursor commands and application commands, be careful to avoid wording that infers a particular cursor or screen reader mode.
Examples:
- Starting at the 'Navigate forwards from here' link, navigate to the 'Print Page' button.
- With focus on the 'Red' slider, increment the value 10 steps to a value of 138.
a space-separated list of assertionId values that specify which assertions to include in this test and in what order they should be presented. The assertionIds are specified in assertions.csv
.
Example:
role name value128 orientation
This specifies that four assertions with assertionId
values of role
, name
, value128
, and orientation
should be included in this test, and that they should be presented in that order.
It is possible to override the assertion priority that is specified in assertions.csv
with the syntax:
new_assertion_priority_integer:assertionId
For example, to specify that the orientation
assertion in the above example should be reduced to a "MAY-have behavior" for a given test, it would be written as:
role name value128 3:orientation
Generally speaking, the priority of an assertion should be the same for all tests, so overriding assertion priority in this file should be unusual. It is not common that an assertion should have a given priority in one test and a different priority in another test.
More details about assertion priority syntax are provided below in the description of assertions.csv
.
This file provides wording, priority, and ARIA spec reference information for the assertions included in the test plan.
Note that:
- It is located in the directory
tests/PLAN_DIRECTORY/data
. - The first row is assumed to contain column titles and will be ignored.
- Otherwise, the order of rows in the file is inconsequential.
Specifications for the information contained in each column are defined in the following sections.
Camel case string that identifies the assertion.
Allowed characters are a-z
, 0-9
, and dash ("-").
An integer that specifies the priority of an assertion.
Assertions can have three priorities specified with the integer 1, 2, or 3.
Priority ID | Priority Name |
---|---|
1 | MUST-have behavior |
2 | SHOULD-have behavior |
3 | MAY-have behavior |
Note about the precedence of assertion priority specifications:
- Assertion priorities specified in
assertions.csv
are the default for all tests run with any covered AT using any command specified for a test. - If the assertion priority 'P' is specified for testId 'T' in
tests.csv
, 'P' takes precedence over the assertion priority value specified inassertions.csv
for all AT only when running and reporting on 'T'. - If assertion priority 'P' is specified in
AT_KEY-commands.csv
for testId 'T' and command 'C' of AT 'A', 'P' takes precedence over assertion priority values specified in bothtests.csv
andassertions.csv
only when running and reporting on performance of 'T' with 'C' of AT 'A'.
Each assertion is a human-readable string that describes a single expected behavior of the assistive technology. The complete set of assertions included in a given test answers the question: "What information should be communicated to users by the assistive technology as a result of this interaction?"
NOTE: The assertion may be written with tokens that are replaced with AT-specific values during the test build process. The method of token definition and substitution is described below.
Assertions should be written such that:
- Each assertion can fail or pass independently of other assertions.
- Most are written in the form "attribute-name 'attribute-value' is conveyed". Some may be written in form "AT verb action".
- The first word is capitalized and there is no punctuation at the end.
- If a value is specified, the value is enclosed in single-quote characters.
- The assertion provides precise expectations using terms from the example test case. For example, if asserting the name of a checkbox is conveyed, and the accessible name of the checkbox is "Lettuce", then the assertion should be written as: "Name 'Lettuce' is conveyed".
Example assertions:
- Role 'menubar' is conveyed
- Menubar name 'Text Formatting' is conveyed
- Menu item name 'Font' is conveyed
- Availability of a submenu is conveyed
- Position '1' of the item in the menu is conveyed
- Menu contains a total of '3' items is conveyed
It is possible to write assertions that are rendered with AT-specific language. This reduces how often people reading reports need to learn ARIA-AT terms that are used to describe AT features in a generic way, e.g., "reading cursor" or "application focus". It also helps avoid misunderstandings when communicating with AT developers.
For example, to test for mode switching in Windows screen readers, a generic assertion might be: "AT switched from reading mode to interaction mode". Reports would be more clear if a report about JAWS listed the assertion as "JAWS changed from virtual cursor active to PC cursor active, and a report about NVDA listed the assertion as "NVDA switched from browse mode to focus mode.
This can be done by writing the assertion using tokens that enable the build script to pull AT-specific values from tests/support.json
. The format of the assertion is "generic-assertion-wording|tokenized-assertion-wording". The generic wording allows for a fall-back presentation of the assertion in the event that an AT doesn't have the necessary values specified in tests/support.json
.
Token names are specified in curly braces. For example, the above assertion could be written as:
AT switched from reading mode to interaction mode|{at} switched from {READING_MODE} to {INTERACTION_MODE}
The token names and their screen text equivalence are defined in an array named assertionTokens
that is defined for specific ATs in the ats
array in the file tests/support.json
. The build script looks up the AT using the AT key from the name of the at_key-commands.csv
file. In the object for that AT, it looks for an array named assertionTokens
.
The assertionTokens
array contains objects with properties name
and screenText
. The build script substitutes the token in the assertion with the string in the screenText
property. The build script can render the above example assertion for JAWS and NVDA as follows:
JAWS switched from virtual cursor active to PC cursor active
NVDA switched from browse mode to focus mode
To get that result, support.json
would need to contain the following:
"ats": [
{
"name": "JAWS",
"key": "jaws",
"assertionTokens": [
{"name": "AT", "screenText": "JAWS"},
{"name": "READING_MODE", "screenText": "virtual cursor active"},
{"name": "INTERACTION_MODE", "screenText": "PC cursor active"}
]
},
{
"name": "NVDA",
"key": "nvda",
"assertionTokens": [
{"name": "AT", "screenText": "NVDA"},
{"name": "READING_MODE", "screenText": "browse mode"},
{"name": "INTERACTION_MODE", "screenText": "focus mode"}
]
}
]
If the build script cannot find the necessary entries in support.json
for a specific AT, it will use the fallback generic wording of the assertion.
This column specifies a shorter phrase form of the assertion wording to be used in reports.
The format is typically "convey attribute-name 'attribute-value'" or "convey 'attribute-value' attribute-name".
Some examples are:
assertionStatement | assertionPhrase |
---|---|
Role 'slider' is conveyed | convey role 'slider' |
Name 'Red' is conveyed | convey name 'Red' |
Value '128' is conveyed | convey value '128' |
Orientation 'horizontal' is conveyed | convey 'horizontal' orientation |
Minimum value '0' is conveyed | convey minimum value '0' |
Maximum value '255' is conveyed | convey maximum value '255' |
AT switched from reading mode to interaction mode|{AT} switched from {READING_MODE} to {INTERACTION_MODE} | switch from reading mode to interaction mode|switch from {READING_MODE} to {INTERACTION_MODE} |
The assertion phrases:
- start with a lower case letter.
- Don't contain ending punctuation.
They should fit into a sentence of form:
AT-name MUST assertionPhrase
For example:
JAWS MUST convey role 'slider'
A space-separated list of the names of ARIA roles, states, or properties covered by the assertion. Typically, an assertion should cover only one ARIA attribute. Some assertions will not have a value in this column, e.g, an assertion related to screen readers automatically switching from reading mode to interaction mode.
These files specify what commands are to be performed for each test in the test plan. There is one file of commands for each AT covered by a plan. The name of the file starts with the key string defined for the AT in tests/support.json
. For example, a test plan that covers JAWS, NVDA, and VoiceOver for macOS contains the following files in its data directory:
jaws-commands.csv
nvda-commands.csv
-
voiceover_macos-commands.csv
.
Note that:
- These files are located in the directory
tests/PLAN_DIRECTORY/data
. - The first row is assumed to contain column titles and will be ignored.
- Otherwise, the order of rows in the file is inconsequential.
Specifications for the information contained in each column are defined in the following sections.
A string that matches a value in the testId
column in the tests.csv
file. The testId indicates a test in which the command on this line of the file will be used.
If a screen reader provides multiple commands for executing the task represented by a test, this file will have multiple rows with the same testId value -- one for each command.
This column specifies the command or command sequence the tester executes in order to perform the interaction being tested.
Each command is a string composed of a set of one or more tokens defined in tests/commands.json
. When a command includes simultaneous activation of multiple key board keys, multiple tokens are concatenated by the plus ("+") symbol.
Examples:
- down
- ctrl+opt+right
- ins+up
If the interaction requires a sequence of commands, such as two consecutive presses of an arrow key, each command in the sequence is separated by a space.
Examples:
- down down
- ctrl+opt+right ctrl+opt+right
Some tokens or combinations of tokens have aliases. For example, "vo" is an alias for "ctrl+opt". So, shorthand for "ctrl+opt+right" is "vo+right".
The build script uses the content of commands.json
to translate the token strings into strings for display in test plans and reports. The translation algorithm is defined in a separate section below.
There are 4 types of tokens defined in commands.json
:
- modifiers, e.g.,
ctrl
,ins
- modifier aliases, e.g.,
vo
,nvda
,jaws
- keys, e.g.,
a
,b
,c
,0
,1
,2
,dash
,delete
- key aliases, e.g.,
del
Each type of token has a dictionary in commands.json
. The tokens are the key
elements of the key:value
pairs. The values are the display strings.
The format of a command token string is:
- The string starts with zero or more modifier alias tokens that are joined by a plus.
- The next tokens in the string are zero or more modifier tokens that are joined by a plus.
- The final token in the string is one key token or key alias token.
A space-separated list of setting names.
A value is specified in this column if it is necessary for the tester to ensure the screen reader is in a specific state before executing the command. Note, all tests assume a screen reader is in its default state.
Settings specified in this column should be settings that screen reader users are expected to frequently change on the fly as they work. In some cases, they are changed automatically by the command being tested. The motivating use case for this parameter is to describe whether or not a screen reader reading cursor must be active before the command is executed.
The setting names must match the value of the name
property of one of the objects in the settings
array contained in the relevant object in the ats
array in tests/support.json
. For example, if a setting of "VIRTUAL_CURSOR" is specified in a file named jaws-commands.csv
, then tests/support.json
must contain:
{
"name": "JAWS",
"key": "jaws",
"settings": [
"name": "VIRTUAL_CURSOR",
"screenText": "virtual cursor active",
"instructions": ["Press Alt+Delete to determine which cursor is active.", "If the PC cursor is active, press Escape to activate the virtual cursor."]
]
}
A space-separated list of assertion priority exceptions in the form NEW_PRIORITY:assertionId
. An assertion can be removed from a command by setting its priority to 0.
For example, to specify that the priority of an assertion with assertionId ROLE
should be set to 2
and that assertion with assertionId INTERACTION_ON
should not be evaluated for a command, the value in this column for that command would be:
2:ROLE 0:INTERACTION_ON
This is particularly useful for testing automatic switching from reading to interaction for JAWS and NVDA. They are expected to automatically switch settings when certain navigation and interaction commands are executed and not switch settings when other commands are used. Similarly, VoiceOver does not automatically change settings while navigating, so such assertions do not apply to voiceOver commands.
If assertion INTERACTION_ON
stated that the screen reader switched mode from reading to interaction, this column would need to include the value 0:INTERACTION_ON
in the rows for tests that include that assertion with commands that would not be expected to cause the change in screen reader state.
An integer specifying the presentation order of commands for a test. For example:
- This controls the order in which commands are presented in the test plan and reports where under each test, there is a heading for each command within that test.
- This controls the order of commands in the test runner when manually running a test.
This file provides descriptions for the setup scripts specified in the setupScript
column in the tests.csv
file.
Note that:
- It is located in the directory
tests/PLAN_DIRECTORY/data
. - The first row is assumed to contain column titles and will be ignored.
- Otherwise, the order of rows in the file is inconsequential.
Specifications for the information contained in each column are defined in the following sections.
The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js
directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv
file for command-button
refers to file tests/command-button/data/js/focusOnFirstLink.js
.
This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.
A short description of what the setup script does. This is presented in the test plan review page.
Because this description will be embedded inside of a sentence, it should not start with a capital letter, and it should not have punctuation at the end.
Examples:
- sets focus on the 'Font' menu item
- sets focus on a link just before the menubar
This file provides values for various elements of reference information used when building the test. It contains two kinds of information:
- Metadata about the test plan, e.g., author, test case URI, etc.
- URIs for specifications of ARIA attributes specified in the
refIds
column in theassertions.csv
file.
Note that:
- It is located in the directory
tests/PLAN_DIRECTORY/data
. - The first row is assumed to contain column titles and will be ignored.
- Otherwise, the order of rows in the file is inconsequential.
Specifications for the information contained in each column are defined in the following sections.
A string identifying the type of reference information.
The string may include upper and lowercase letters, numbers, dash ("-") or underscore ("_").
The following is a list of refId
s that are often included:
author
authorEmail
-
reference
(note: this is a required refId) -
example
(note: this is a required refId): Refers to the example test case, typically an APG example page. -
designPattern
: Refers to an APG design pattern if relevant. -
developmentDocumentation
: used for the GitHub issue that documents development of a test plan. - Names of ARIA roles, states, and properties: Each
refId
inassertions.csv
must have a corresponding row in this file. Examples:button
oraria-pressed
The value for the corresponding refId
.
Following are the values that should be specified for the commonly used refIds listed above.
refId | Value Description | Example Value |
---|---|---|
author |
Name of the author of the test plan | John Doe |
authorEmail |
Email address for the test plan author | john.doe@example.com |
reference |
URI of the test case html contained in the reference subdirectory of the test plan directory. Skeleton HTML for this reference can be generated from the example using the update-reference script. The reference HTML file should be placed in a subdirectory of the reference directory that is named with the date of creation using format YYYY-MM-DD_HHmmss . When anything in this reference file needs to change in a way that could effect test results (i.e not just fixing a typo), a new reference subdirectory should be created and this file should be updated. |
reference/2020-11-23_175030/checkbox-1/checkbox-1.html |
example |
URI of APG page containing the specific example implementation covered by the test plan | https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/ |
designPattern |
URI of the APG design pattern related to the test plan example | https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/ |
developmentDocumentation |
URI of GitHub issue that documents development of the test plan | https://github.com/w3c/aria-at/issues/156 |
ARIA role, state, or property name, e.g., checkbox
|
Unversioned URI of the ARIA specification for the role, state, or property | https://www.w3.org/TR/wai-aria/#checkbox |
Specify when there is a URI in the value column. This is used when making a link from the URI in the value column.
The test plan build scripts use data in tests/commands.csv
to process values in the command
column of AT_KEY-commands.csv
files to create HTML content used in test plan review and report pages.
For example, if the voiceover_macos-commands.csv
file contains the following values, the result of the processing described below would yield the following HTML content.
command | HTML content |
---|---|
ctrl+opt+right |
<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Right Arrow</kbd> |
vo+shift+down vo-down |
<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> then <kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Down Arrow</kbd> |
The following terms are used to refer to values in the command
column of the CSV file and their constituent parts.
Term | Description | Example |
---|---|---|
token | A substring of the value that ends with a plus, space, or end of value. |
ctrl , opt , and down are the 3 tokens in ctrl+opt+down
|
command | One or more tokens joined by plus characters. | ctrl+opt+right |
command sequence | 2 or more commands separated by one or more elements of white space. | vo+shift+down vo+down |
Tokens are valid if they are a property name in one of the objects in commands.json
. Some validation rules for commands are included in the validation section below.
There are 4 possible types of tokens.
Token Type | Description |
---|---|
key | A property name in the keys object in commands.json
|
modifier | A property name in the modifiers object in commands.json
|
key alias | A property name in the keyAliases object in commands.json
|
modifier alias | A property name in the modifierAliases object in commands.json
|
To translate a value into HTML content:
- For each command:
- Replace any tokens that are key or modifier aliases with their values from
keyAliases
andmodifierAliases
. - Replace each modifier and key token with
"" + tokenValue + ""
where `tokenValue` is the value of the property for each token.
- Replace any tokens that are key or modifier aliases with their values from
- Replace the space separating each command in a command sequence with the string
" then "
.
For example, given the value vo+shift+down down
, the above steps are as follows.
Step | Output |
---|---|
Replace aliases in commands | ctrl+opt+shift+down down |
Replace key tokens with key values | <kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> <kbd>Down Arrow</kbd> |
Separate commands in a sequence with the word "then" | <kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> then <kbd>Down Arrow</kbd> |
- The following files must exist:
tests.csv
,assertions.csv
,scripts.csv
, and at least one*-commands.csv
- The portion of the file name before the
-
in any*-commands.csv
file must be a validkey
value in theats
array insupport.json
. - Allow
assertionId
, andtestId
values to include characters that are in the set[A-Za-z0-9_]
. - Allow
refId
values to include characters in the set[A-Za-z0-9_-]
. - The
assertionId
column ofassertions.csv
provides the list of valid values; they must be unique. All assertionId values in other files must exist in this column. - The
testId
column oftests.csv
provides the list of valid values; they must be unique. All testId values in other files must exist in this column. - The
refId
column ofreferences.csv
provides the list of valid values; they must be unique. All refId values in other files must exist in this column. - Values in the
setupScript
column inscripts.csv
must be unique and each must be equivalent to the name of a.js
file in the./js
directory. - Values in the
title
column intests.csv
must be unique after converting to lowercase and removing white space. - Values in the
presentationNumber
column intests.csv
must be unique integers. - The
setupScript
values intests.csv
must be present in thesetupScript
column ofscripts.csv
. -
assertionId
values with priority prefixes in the list of assertions in theassertions
column intests.csv
must match the pattern[0-4]:\w+
. - The list of assertions in a given cell of the
assertions
column intests.csv
must be unique. Comparison must be made after stripping away any priority prefixes, e.g.,1:role 2:role
is an error. - Values in the
priority
column of assertions.csvmust be in the set
[0-3]`. - Values in the
assertionStatement
andassertionPhrase
columns of assertions.csv` must be unique after converting to lowercase and removing white space.