Test Format Definition V2

ARIA-AT Test Format V2: Definition and Processing

Usage notes and change history

NOTE:

This page provides specifications useful for both test writing and test file processing.
Test Format V2 replaces Test Format V1 Definition for new versions of test plans starting September 2023.
The ARIA-AT web site will continue to support the display of test plans and reports in the V1 format.
Development of V2 of the test format is documented in ARIA-AT issue 974.
Mar 10, 2024: This page has been updated to document the enhancement described in issue 1039 and implemented in pull request 1042.

Overview of test files

Each test plan is defined by a set of files in its own directory inside the tests directory in this repository. The name of the test plan subdirectory is named to reflect the test case, e.g, "alert" or "horizontal-slider". In the below list of files, the directory for each plan is represented by the string PLAN_DIRECTORY.

The test build scripts that generate files for the ARIA-AT web site (App) extract and translate content from the following files:

tests/PLAN_DIRECTORY/data/tests.csv
tests/PLAN_DIRECTORY/data/assertions.csv
One file for each covered AT with a file name format of tests/PLAN_DIRECTORY/data/AT_KEY-commands.csv where AT_KEY is the key for an AT defined in support.json. Example: jaws-commands.csv.
tests/PLAN_DIRECTORY/data/scripts.csv
tests/PLAN_DIRECTORY/data/references.csv
tests/commands.json
tests/support.json

Specifications for the content of each CSV file are in the following sections. Usage of each JSON file is defined where applicable to the CSV file content.

`tests.csv`

This file specifies information about the tests included in a test plan, including their names, the order in which they are presented, and the assertions covered by each test.

Note that:

It is located in the directory tests/PLAN_DIRECTORY/data.
The first row is assumed to contain column titles and will be ignored.
Otherwise, the order of rows in the file is inconsequential.

Specifications for the information contained in each column are defined in the following sections.

`testId`

Camel case string that identifies the test.

The testId should:

Only include characters a-z, 0-9, and dash ("-").
Be generated from the testTitle by removing extraneous words, shortening some common words (e.g., navigate to nav), converting to lowercase, capitalizing the first letter of the second and subsequent words, and removing spaces and punctuation.

Examples:

navForwardsToSlider
navBackToSlider
reqInfoAboutSlider
incrementSliderOneStep

`title`

String that names the test. This string appears in headings that title test pages and report sections and in the row headers of report tables.

The title string should:

Be written with sentence capitalization, i.e., the first word is capitalized.
Should not have a period at the end.
describe the user task the test is evaluating in straightforward and brief terms that enable people reading reports to understand what users of this AT can and cannot do when they encounter an element represented by the test case.

Examples:

Navigate forwards to a slider
Navigate backwards to a slider
Request information about a slider
Increment a slider by one step

Preferred verbs and objects for screen reader tests include the following.

Phrase	Usage Notes
Navigate forwards	Use for tasks where users will move a reading cursor or application focus in a forwards direction.
Navigate backwards	Use for tasks where users will move a reading cursor or application focus in a backwards direction.
Request information	Use for tasks where users will acquire information about the current element. "Request" is preferable to "Read" because "Read" could be interpreted as involving movement of a reading cursor.
Activate	Use for tasks where the user will perform the default action on an element.
Increment or Decrement	Use for tasks where users will increase or decrease the value of an element.
Input value	Use for tasks where users will type a value.

The above guidance is not intended to provide an exclusive or exhaustive list.

`presentationNumber`

A positive number that controls the order of presentation of tests in the test runner and reports. By default, tests will be presented in numerical ascending order using the values in this column, i.e., the values are Number, not String, primatives when sorted.

The values in this column allow the test author to specify any order for the tests. Thus, when a new test is added, it can be placed anywhere in the sequence.

`setupScript`

The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv file for command-button refers to file tests/command-button/data/js/focusOnFirstLink.js.

This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.

`instructions`

This is a sentence that describes the user task that is being evaluated by the test. It is presented as one of the elements in the list of instructions given to testers when executing a test.

The sentence:

Includes a verb that is synonymous with the verb used in the test name.
It may start with a prepositional phrase that provides information about the starting point of the task being performed.
Note: If a test will be performed with both reading cursor commands and application commands, be careful to avoid wording that infers a particular cursor or screen reader mode.

Examples:

Starting at the 'Navigate forwards from here' link, navigate to the 'Print Page' button.
With focus on the 'Red' slider, increment the value 10 steps to a value of 138.

`assertions`

a space-separated list of assertionId values that specify which assertions to include in this test and in what order they should be presented. The assertionIds are specified in assertions.csv.

Example:

roleSlider nameRed value128 horizontalOrientation

This specifies that four assertions with assertionId values of roleSlider,nameRed, value128, and horizontalOrientation should be included in this test, and that they should be presented in the test runner and reports in that order.

It is possible to override the assertion priority that is specified in assertions.csv with the syntax:

new_assertion_priority_integer:assertionId

For example, to specify that the orientation assertion in the above example should be reduced to a "MAY-have behavior" for a given test, it would be written as:

roleSlider nameRed value128 3:horizontalOrientation

It is possible to set an assertion priority to 0 for a test, which means that the assertion will not be listed as an assertion for this test unless a commands file overrides the 0 priority.

roleSlider nameRed value128 0:horizontalOrientation

In this scenario, the horizontalOrientation assertion will not be included for any commands unless there is a command in a commands file that as an assertion exception that sets priority of 1, 2, or 3 for that command.

Specifying a 0-priority for an assertion in tests.csv is most useful in scenarios where nearly all commands are expected to provide behavior A1 but there is one command where behaviorA2 is expected. For example, when navigating backward into a radio group where none of the radio buttons are checked with a screen reader, most commands are expected to move the reading cursor to the last radio. However, Shift+Tab is expected to move the reading cursor to the first radio. In this case, we could have:

The assertions column in tests.csv with: roleGroup nameGroupPizzaCrust roleRadio nameThinCrust 0:nameRegularCrust stateRadioUnchecked positionRadio3 0:positionRadio1 numberRadioButtonsGroup3 0:interactionModeEnabled
The assertionExceptions column in *-commands.csv for the Shift+Tab command with: 0:nameThinCrust 1:nameRegularCrust 0:positionRadio3 2:positionRadio1

This enables the Shift+Tab command to have assertions in the same order as other commands but to have different values for the name and position assertions. The behaviors that are unique to Shift+Tab are specified as 0-priority in the tests.csv file. This puts them in the correct sequence when they are overridden in commands.csv. Then, in commands.csv, for the Shift+Tab command, the behaviors that are not expected for Shift+Tab are set to 0 priority and the behaviors that are expected are given a non-0 priority.

Generally speaking, the priority of an assertion should be the same for all tests, so overriding assertion priority in this file should be unusual. It is not common that an assertion should have a given priority in one test and a different priority in another test.

More details about assertion priority syntax are provided below in the description of assertions.csv.

`assertions.csv`

This file provides wording, priority, and ARIA spec reference information for the assertions included in the test plan.

Note that:

It is located in the directory tests/PLAN_DIRECTORY/data.
The first row is assumed to contain column titles and will be ignored.
Otherwise, the order of rows in the file is inconsequential.

Specifications for the information contained in each column are defined in the following sections.

`assertionId`

Camel case string that identifies the assertion.

the assertionId should:

Only include characters a-z, 0-9, and dash ("-").
Be generated from the assertionStatement by removing extraneous words (e.g., 'conveyed'), shortening some common words, converting to lowercase, capitalizing the first letter of the second and subsequent words, and removing spaces and punctuation.

Examples:

roleMenubar
menubarNameTextFormatting
menuItemNameFont
availabilityOfSubmenu
position1InMenu
menuContains3Items

`priority`

An integer that specifies the priority of an assertion.

Assertions can have three priorities specified with the integer 1, 2, or 3.

Priority ID	Priority Name
1	MUST-have behavior
2	SHOULD-have behavior
3	MAY-have behavior

Note about the precedence of assertion priority specifications:

Assertion priorities specified in assertions.csv are the default for all tests run with any covered AT using any command specified for a test.
If the assertion priority 'P' is specified for testId 'T' in tests.csv, 'P' takes precedence over the assertion priority value specified in assertions.csv for all AT only when running and reporting on 'T'.
If assertion priority 'P' is specified in AT_KEY-commands.csv for testId 'T' and command 'C' of AT 'A', 'P' takes precedence over assertion priority values specified in both tests.csv and assertions.csv only when running and reporting on performance of 'T' with 'C' of AT 'A'.

`assertionStatement`

Each assertion is a human-readable string that describes a single expected behavior of the assistive technology. The complete set of assertions included in a given test answers the question: "What information should be communicated to users by the assistive technology as a result of this interaction?"

NOTE: The assertion may be written with tokens that are replaced with AT-specific values during the test build process. The method of token definition and substitution is described below.

Assertions should be written such that:

Each assertion can fail or pass independently of other assertions.
Most are written in the form "attribute-name 'attribute-value' is conveyed". Some may be written in form "AT verb action".
The first word is capitalized and there is no punctuation at the end.
If a value is specified, the value is enclosed in single-quote characters, e.g., name 'Red', position '1'.
The assertion provides precise expectations using terms from the example test case. For example, if asserting the name of a checkbox is conveyed, and the accessible name of the checkbox is "Lettuce", then the assertion should be written as: "Name of the checkbox, 'Lettuce', is conveyed".

Example assertion statements:

Role, 'menubar', is conveyed
Name of the menubar, 'Text Formatting', is conveyed
Menu item name, 'Font', is conveyed
Availability of a submenu is conveyed
Position of the item in the menu, '1' is conveyed
Number of items in the menu, '3', is conveyed

It is possible to write assertions that are rendered with AT-specific language. This reduces how often people reading reports need to learn ARIA-AT terms that are used to describe AT features in a generic way, e.g., "reading cursor" or "application focus". It also helps avoid misunderstandings when communicating with AT developers.

For example, to test for mode switching in Windows screen readers, a generic assertion might be: "The screen reader switched from reading mode to interaction mode". Reports would be more clear if a report about JAWS listed the assertion as "JAWS changed from virtual cursor active to PC cursor active, and a report about NVDA listed the assertion as "NVDA switched from browse mode to focus mode.

This can be done by writing the assertion using tokens that enable the build script to pull AT-specific values from tests/support.json. The format of the assertion is:

generic-assertion-wording|tokenized-assertion-wording

The generic wording allows for a fall-back presentation of the assertion in the event that an AT doesn't have the necessary values specified in tests/support.json.

Token names are specified in curly braces. For example, the above assertion could be written as:

The screen reader switched from reading mode to interaction mode|{screenReader} switched from {readingMode} to {interactionMode}

The token names and their replacement values are defined in an object named assertionTokens that is defined within each of the relevant objects in the ats array in the file tests/support.json.

When the build script is creating the table of assertions for a given AT and command, it locates the relevant ats array entry using the AT key from the name of the at_key-commands.csv file. Each token name, e.g. readingMode, is a property name in the assertionTokens object within the ats array entry. The build script substitutes the token and its surrounding braces with the value of that property.

The build script would then render the above example assertion for any JAWS or NVDA command to which it applies as follows:

JAWS switched from virtual cursor active to PC cursor active

NVDA switched from browse mode to focus mode

To get that result, support.json would need to contain the following:

"ats": [
  {
    "name": "JAWS",
    "key": "jaws",
    "assertionTokens": {
      "screenReader": "JAWS",
      "readingMode": "virtual cursor active",
      "interactionMode": "PC cursor active"
    }
  },
  {
    "name": "NVDA",
    "key": "nvda",
    "assertionTokens": {
      "screenReader": "NVDA",
      "readingMode": "browse mode",
      "interactionMode": "focus mode"
    }
  }
]

If the build script cannot find the necessary entries in support.json for a specific AT, it will use the fallback generic wording of the assertion.

`assertionPhrase`

This column specifies a shorter phrase form of the assertion wording to be used in reports.

The format is typically "convey attribute-name 'attribute-value'" or "convey 'attribute-value' attribute-name".

Some examples are:

assertionStatement	assertionPhrase
Role 'radio button' is conveyed	convey role 'radio button'
Name of the radio button, 'Deep dish', is conveyed	convey name of the radio button, 'Deep dish'
Number of radio buttons in the group, 3, is conveyed	convey number of radio buttons in the group, 3
Position of the radio button, 1, is conveyed	convey position of the radio button, 1
Change in state of the radio button to 'checked' is conveyed	convey change in state of the radio button to 'checked'

The assertion phrases:

start with a lower case letter.
Don't contain ending punctuation.

They should fit into a sentence of form:

AT-name MUST assertionPhrase

For example:

JAWS MUST convey role 'slider'

`refIds`

A space-separated list of refId values defined in references.csv.

The refId values designate references to the ARIA or HTML-AAM specifications that are related to the asserted behavior. Some assertions will cover a behavior that is not described by any specification, so this cell will be blank in those rows.

The convention for ARIA attributes is that the refId is equivalent to the ARIA attribute, e.g., slider or aria-orientation. For HTML-AAM mappings, it is a good idea to avoid ambiguity by using a refId that includes the string html, e.g., htmlButton, or htmlLink.

`AT_KEY-commands.csv` (Example: `voiceover_macos-commands.csv`)

These files specify what commands are to be performed for each test in the test plan. There is one file of commands for each AT covered by a plan. The name of the file starts with the key string defined for the AT in tests/support.json. For example, a test plan that covers JAWS, NVDA, and VoiceOver for macOS contains the following files in its data directory:

jaws-commands.csv
nvda-commands.csv
voiceover_macos-commands.csv.

Note that:

These files are located in the directory tests/PLAN_DIRECTORY/data.
The first row is assumed to contain column titles and will be ignored.
Otherwise, the order of rows in the file is inconsequential.

Specifications for the information contained in each column are defined in the following sections.

`testId`

A string that matches a value in the testId column in the tests.csv file. The testId indicates a test in which the command on this line of the file will be used.

There must be at least one row for every testId in tests.csv. If a screen reader provides multiple commands for executing the task represented by a test, there will be multiple rows with the same testId value -- one for each command.

`command`

This column specifies the command or command sequence the tester executes in order to perform the interaction being tested.

Each command is a string composed of a set of one or more tokens defined in tests/commands.json. When a command includes simultaneous activation of multiple key board keys, multiple tokens are concatenated by the plus ("+") symbol.

Examples:

down
ctrl+opt+right
ins+up

If the interaction requires a sequence of commands, such as two consecutive presses of an arrow key, each command in the sequence is separated by a space.

Examples:

down down
ctrl+opt+right ctrl+opt+right

Some tokens or combinations of tokens have aliases. For example, "vo" is an alias for "ctrl+opt". So, shorthand for "ctrl+opt+right" is "vo+right".

The build script uses the content of commands.json to translate the token strings into strings for display in test plans and reports. The translation algorithm is defined in a separate section below.

There are 4 types of tokens defined in commands.json:

modifiers, e.g., ctrl, ins
modifier aliases, e.g., vo, nvda, jaws
keys, e.g., a, b, c, 0, 1, 2, dash, delete
key aliases, e.g., del

There as an object in commands.json for each type of token. The tokens need to be equivalent to a property name of one of the properties in one of those four objects. The values of those properties are the display strings for the commands.

The format of a command token string is:

The string starts with zero or more modifier alias tokens that are joined by a plus.
The next tokens in the string are zero or more modifier tokens that are joined by a plus.
The final token in the string is one key token or key alias token that is preceded by a plus if it is not the first token.

`settings`

A space-separated list of setting names.

A value is specified in this column if it is necessary for the tester to ensure the screen reader is in a specific state before executing the command. Note, all tests assume a screen reader is in its default state.

The only type of settings that should be specified in this column are settings where it is typical for users to frequently change them while working. In some cases, they are changed automatically by the command being tested. The motivating use case for this parameter is to describe whether or not a screen reader reading cursor must be active before the command is executed.

Each setting specified in this column must correspond to the name of a property in the settings object contained in the relevant object in the ats array in tests/support.json. For example, if the setting virtualCursor is specified in a file named jaws-commands.csv, then tests/support.json must contain:

    {
      "name": "JAWS",
      "key": "jaws",
      "settings": {
        "virtualCursor": {
          "screenText": "virtual cursor active",
          "instructions": [
            "Press &lt;kbd&gt;Alt&lt;/kbd&gt;+&lt;kbd&gt;Delete &lt;/kbd&gt; to determine which cursor is active.",
            "If the PC cursor is active, press &lt;kbd&gt;Escape&lt;/kbd&gt; to activate the virtual cursor."
          ]
        },
        "pcCursor": {
          "screenText": "PC cursor active",
          "instructions": [
            "Press &lt;kbd&gt;Alt&lt;/kbd&gt;+&lt;kbd&gt;Delete&lt;/kbd&gt; to determine which cursor is active.",
            "If the virtual cursor is active, press &lt;kbd&gt;Insert&lt;/kbd&gt;+&lt;kbd&gt;z&lt;/kbd&gt; to disable the virtual cursor."
          ]
        }
      }
    }

`assertionExceptions`

A space-separated list of assertion priority exceptions in the form NEW_PRIORITY:assertionId. An assertion can be removed from a command by setting its priority to 0.

For example, to specify that the priority of an assertion with assertionId groupBoundary should be reduced to 3 for a particular command, the value in this column for that command would be:

3:groupBoundary

This is particularly useful for testing automatic switching from reading to interaction for JAWS and NVDA. They are expected to automatically switch settings when certain navigation and interaction commands are executed and not switch settings when other commands are used. Similarly, VoiceOver does not automatically change settings while navigating, so such assertions do not apply to voiceOver commands.

To specify that an assertion with id interactionModeEnabled should have a priority of 2 for a given command, the command could have an assertionException of 2:interactionModeEnabled. If the priority of the interactionModeEnabled assertion is set to 0 in tests.csv, then this assertion will only be listed for commands that override the 0 priority.

`presentationNumber`

A positive number that controls the order of presentation of commands in the test runner and reports. By default, commands within a test will be presented in numerical ascending order using the values in this column, i.e., the values are Number, not String, primatives when sorted.

This value controls:

The order in which commands are presented in the test plan and reports where under each test, there is a heading for each command within that test.
The order of commands in the test runner when manually running a test.

`scripts.csv`

This file provides descriptions for the setup scripts specified in the setupScript column in the tests.csv file.

Note that:

It is located in the directory tests/PLAN_DIRECTORY/data.
The first row is assumed to contain column titles and will be ignored.
Otherwise, the order of rows in the file is inconsequential.

Specifications for the information contained in each column are defined in the following sections.

`setupScript`

The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv file for command-button refers to file tests/command-button/data/js/focusOnFirstLink.js.

This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.

`setupScriptDescription`

A short description of what the setup script does. This is presented in the test plan review page.

Because this description will be embedded inside of a sentence, it should not start with a capital letter, and it should not have punctuation at the end.

Examples:

sets focus on the 'Font' menu item
sets focus on a link just before the menubar

`references.csv`

This file provides values for various elements of reference information used when building the test. It contains two kinds of information:

Metadata about the test plan, e.g., author, test case URI, etc.
URIs for specifications of ARIA attributes and HTML API accessibility mappings specified in the refIds column in the assertions.csv file.

Note that:

It is located in the directory tests/PLAN_DIRECTORY/data.
The first row is assumed to contain column titles and will be ignored.
Otherwise, the order of rows in the file is inconsequential.

Specifications for the information contained in each column are defined in the following sections.

refId

A string identifying the type of reference information.

The string may include upper and lowercase letters, numbers, dash ("-") or underscore ("_").

The following is a list of refIds that are often included:

author
authorEmail
reference (note: this is a required refId)
example (note: this is a required refId): Refers to the example test case, typically an APG example page.
designPattern: Refers to an APG design pattern if relevant.
developmentDocumentation: used for the GitHub issue that documents development of a test plan.
Names of ARIA roles, states, and properties: Each refId in assertions.csv must have a corresponding row in this file. Examples: button or aria-pressed
htmlElement where Element is the name of an HTML element that has an HTML-AAM specification covered by an assertion, htmlButton, htmlLink. Note: Link is used instead of A for eadability in this example. The refId does not need to specify the exact name. The value and linkText columns control the lookup of the specification URL and presentation of the link.

`type`

A string indicating the type of reference. It may have one of three values:

metadata: Indicates the refId is for information that applies to all tests in the plan.
aria: Indicates that the refId is for an ARIA attribute specification.
htmlAam: Indicates the refId is for an HTML element mapping specification.

This column controls:

Where the reference information is displayed in the test plan review page.
Where the test plan builder will look for the URL to a specification.

Specification URL values are defined in the references object in tests/support.json for the aria and htmlAam types. Requirements are described below in the description of the value column.

value

The value for the corresponding refId.

Following are the values that should be specified for the commonly used refIds listed above.

refId	type	Value Description	Example Value
`author`	`metadata`	Name of the author of the test plan	`John Doe`
`authorEmail`	`metadata`	Email address for the test plan author	`john.doe@example.com`
`reference`	`metadata`	URI of the test case html contained in the reference subdirectory of the test plan directory. Skeleton HTML for this reference can be generated from the `example` using the `update-reference` script. The reference HTML file should be placed in a subdirectory of the `reference` directory that is named with the date of creation using format `YYYY-MM-DD_HHmmss`. When anything in this reference file needs to change in a way that could effect test results (i.e not just fixing a typo), a new reference subdirectory should be created and this file should be updated.	`reference/2020-11-23_175030/checkbox-1/checkbox-1.html`
`example`	`metadata`	URI of APG page containing the specific example implementation covered by the test plan	https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/
`designPattern`	`metadata`	URI of the APG design pattern related to the test plan example	https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/
`developmentDocumentation`	`metadata`	URI of GitHub issue that documents development of the test plan	https://github.com/w3c/aria-at/issues/156
ARIA attribute reference, e.g., `checkbox`	`aria`	Property name in the `fragmentIds` object in the `aria` object in the `references` object in `tests/support.json`	`checkbox`
HTML-AAM specification reference, e.g., htmlButton	htmlAam	Property name in the `fragmentIds` object in the `htmlAam` object in the `references` object in `tests/support.json`	`button`

`linkText`

Text used in the calculation of link text for references that generate links, e.g., example, designPattern, developmentDocumentation, and specifications.

If the type is metadata, the text in this column is the complete link text for the link. If the type is aria or htmlAam the link text is a combination of text from this column and the corresponding linkText property in support.json. Calculation of link text and href values for links is specified in the following section.

Conventions for metadata link text are:

For APG design patterns, the link text format is "APG pattern: PATTERN_TITLE" where PATTERN_TITLE is the text of the link to the pattern on the APG patterns page.
For APG Examples, the link text format is "APG Example: EXAMPLE_TITLE" where EXAMPLE_TITLE is the text of the link to the example on the APG Index page.
For refId of reference, the link text format is "Test Case Page for PLAN_TITLE" where PLAN_TITLE is the text in the value column for the title refId.
For refId of developmentDocumentation, the link text format is "Development documentation for PLAN_TITLE test plan" where PLAN_TITLE is the text in the value column for the title refId.

The convention for aria type references is to repeat the refId in the linkText column. Because the tests/support.json file specifies ARIA Specification for the second part of the link text, the complete link text will be, for example, checkbox ARIA Specification.

The convention for htmlAam type references is to specify linkText of "HTML ELEMENT_NAME", e.g., "HTML button". Because tests/support.json specifies "Accessibility API Mapping" for the second portion of the link text, the complete link text would be, for example, "HTML button Accessibility API Mapping".

Reference link calculation

The way the href values and text for links to references is derived from the references.csv data depends on the reference type.

If the type is metadata, the value column provides the href value and the linkText column provides the complete text. Note that for refId = reference, the value is a a partial relative URL to content that wil be hosted by the app, but for all other metadata references, the value is a complete URL.

Example:

refId = example
type = metadata
value = https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/
linkText = APG Example: Checkbox (Two State)

Result:

<a href="https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/">APG Example: Checkbox (Two State)</a>

If the reference type is aria:

href = supportJSON.references.aria.baseUrl + supportJSON.references.aria.fragmentIds[referencesValue]

text = `${referenceLinkText} ${supportJSON.references.aria.linkText}`

Where:

supportJSON refers to the object in tests/support.json.
referenceValue refers to the value in the value column of the references.csv file.
referenceLinkText refers to the value in the linkText column of the references.csv file.

Example:

refId = checkbox
type = aria
value = checkbox
linkText = checkbox

Result:

<a href="https://www.w3.org/TR/wai-aria/#checkbox">checkbox ARIA Specification</a>

If the reference type is htmlAam:

href = supportJSON.references.htmlAam.baseUrl + supportJSON.references.htmlAam.fragmentIds[referencesValue]

text = `${referenceLinkText} ${supportJSON.references.htmlAam.linkText}`

Where:

supportJSON refers to the object in tests/support.json.
referenceValue refers to the value in the value column of the references.csv file.
referenceLinkText refers to the value in the linkText column of the references.csv file.

Example:

refId = htmlLink
type = htmlAam
value = a
linkText = HTML Hyperlink

Result:

<a href="https://www.w3.org/TR/html-aam-1.0/#el-a">HTML Hyperlink Accessibility API Mapping</a>

`commands.json` processing: Building strings for command display

The test plan build scripts use data in tests/commands.csv to process values in the command column of AT_KEY-commands.csv files to create HTML content used in test plan review and report pages.

For example, if the voiceover_macos-commands.csv file contains the following values, the result of the processing described below would yield the following HTML content.

command	HTML content
`ctrl+opt+right`	`<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Right Arrow</kbd>`
`vo+shift+down down`	`<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> then <kbd>Down Arrow</kbd>`

The following terms are used to refer to values in the command column of the CSV file and their constituent parts.

Term	Description	Example
token	A substring of the value that ends with a plus, space, or end of value.	`ctrl`, `opt`, and `down` are the 3 tokens in `ctrl+opt+down`
command	One or more tokens joined by plus characters.	`ctrl+opt+right`
command sequence	2 or more commands separated by one or more elements of white space.	`vo+shift+down vo+down`

Tokens are valid if they are a property name in one of the objects in commands.json. Some validation rules for commands are included in the validation section below.

There are 4 possible types of tokens.

Token Type	Description
key	A property name in the `keys` object in `commands.json`
modifier	A property name in the `modifiers` object in `commands.json`
key alias	A property name in the `keyAliases` object in `commands.json`
modifier alias	A property name in the `modifierAliases` object in `commands.json`

To translate a value into HTML content:

For each command:
1. Replace any tokens that are key or modifier aliases with their values from keyAliases and modifierAliases.
2. Replace each modifier and key token with <kbd> + tokenValue + </kbd> where tokenValue is the value of the property for each token.
3. Throw an error for any token that does not have a matching key name in any object.
Replace the space separating each command in a command sequence with the string " then ".

For example, given the value vo+shift+down down, the above steps are as follows.

Step	Output
Replace aliases in commands	`ctrl+opt+shift+down down`
Replace key tokens with key values	`<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> <kbd>Down Arrow</kbd>`
Separate commands in a sequence with the word "then"	`<kbd>Control</kbd>+<kbd>Option</kbd>+<kbd>Shift</kbd>+<kbd>Down Arrow</kbd> then <kbd>Down Arrow</kbd>`

Validation

Here are some examples of conditions for potential validation checks to ensure that the CSV file content in a test plan is well formed.

The following files must exist: tests.csv, assertions.csv, scripts.csv, and at least one *-commands.csv
The portion of the file name before the - in any *-commands.csv file must be a valid key value in the ats array in support.json.
Values in the testId column of tests.csv:
- Must contain only characters that are in the set [A-Za-z0-9-].
- Must contain at least one alphanumeric character.
- Must not start or end with punctuation.
- Must be unique.
- Provide the list of valid values; all testId values in other files must exist in this column.
Values in the title column in tests.csv:
- Must contain at least 3 words.
- Must start with an uppercase letter.
- Must end with a period.
- Must be unique after converting to lowercase and removing white space.
Values in the presentationNumber column in tests.csv must be unique numbers that are >=0.
The setupScript values in tests.csv must be present in the setupScript column of scripts.csv.
The values in the assertions column in tests.csv:
- Must not be empty after removing white space.
- Each space-separated string must match the pattern ([0-3]:)?(\w+). The first capturing group represents an optional priority prefix.
- Each space-separated token must exist in the assertionId column of assertions.csv. Comparison must be made after stripping away any priority prefixes.
- Each space-separated token must be unique. Comparison must be made after stripping away any priority prefixes, e.g., 1:role 2:role is an error.
Values in the assertionId column of assertions.csv:
- Must contain only characters that are in the set [A-Za-z0-9-].
- Must contain at least one alphanumeric character.
- Must not start or end with punctuation.
- Must be unique.
- Provide the list of valid values; all assertionId values in other files must exist in this column.
Values in the priority column of assertions.csv must be in the set [1-3].
Values in the assertionStatement column of assertions.csv:
- Must contain at least 3 words.
- Must be unique after converting to lowercase and removing white space.
- Must start with an uppercase letter.
- Must NOT end with any punctuation character except apostrophe.
Values in the assertionPhrase column of assertions.csv:
- Must contain at least 3 words.
- Must be unique after converting to lowercase and removing white space.
- Must start with a lowercase letter.
- Must NOT end with any punctuation character except apostrophe.
Values in the testId column of *-commands.sv files:
- Must exist in the testId column of tests.csv.
- Must contain at least one instance of every testId in tests.csv. If not, throw an error stating that the file must contain at least one command for every test and provide testId values for the missing tests.
Values in the command column
- Must contain only characters in the set [A-Za-z0-9\+ ].
- Must contain at least one alphanumeric character
- Plus characters must have an alphanumeric character on both sides.
Each space-separated Value in the settings column of *-commands.csv files:
- Must contain only characters in the set [A-Za-z0-9-].
- Must match a property name in the corresponding settings object in support.json.
Values in the assertionExceptions column of *-commands.csv files:
- May be NULL after removing white space.
- If not NULL, each space-separated string must match the pattern ([0-3]:)?(\w+). The first capturing group represents an optional priority prefix.
- Each space-separated token must exist in the assertionId column of assertions.csv. Comparison must be made after stripping away any priority prefixes.
- Each space-separated token must be unique. Comparison must be made after stripping away any priority prefixes, e.g., 1:role 2:role is an error.
Values in the presentationNumber column of *-commands.csv files:
- Must be a number that is >=0.
- Values do not have to be unique. Ideally, values are unique within a set of rows that all start with the same testId.
Values in the setupScript column in scripts.csv:
- Must be unique.
- Must be equivalent to the name of a .js file in the ./js directory.
Values in the setupScriptDescription column in scripts.csv:
- Must contain at least 3 words.
- Must start with a lowercase letter.
- Must NOT end with any punctuation character except apostrophe.
The values in the refId column of references.csv:
- Must contain only characters that are in the set [A-Za-z0-9-].
- Must contain at least one alphanumeric character.
- Must not start or end with punctuation.
- Must be unique.
- Provide the list of valid values; all refId values in other files must exist in this column.
- Two of the values must be reference and example. Throw an error if either are missing.
The values in the type column of references.csv:
- Must not be null.
- Must be one of metadata, aria or htmlAam.
The values in the value column of references.csv:
- Must not be null.
- If the type is aria, then the value must be a property name in the fragmentIds object in the aria object in the references object in support.json.
- If the type is htmlAam, then the value must be a property name in the fragmentIds object in the htmlAam object in the references object in support.json.
The values in the linkText column of references.csv:
- Must not be null if the type is metadata and the value column contains a URI.
- Must not be null if the type is aria or htmlAam.
- If not null, must not start or end with punctuation characters.

Test Format Definition V2

ARIA-AT Test Format V2: Definition and Processing

Usage notes and change history

Overview of test files

tests.csv

testId

title

presentationNumber

setupScript

instructions

assertions

assertions.csv

assertionId

priority

assertionStatement

assertionPhrase

refIds

AT_KEY-commands.csv (Example: voiceover_macos-commands.csv)

testId

command

settings

assertionExceptions

presentationNumber

scripts.csv

setupScript

setupScriptDescription

references.csv

refId

type

value

linkText

Reference link calculation

commands.json processing: Building strings for command display

Validation

ARIA AT Wiki Pages

Getting Started

Running and Developing AT Tests

Projects

Clone this wiki locally

`tests.csv`

`testId`

`title`

`presentationNumber`

`setupScript`

`instructions`

`assertions`

`assertions.csv`

`assertionId`

`priority`

`assertionStatement`

`assertionPhrase`

`refIds`

`AT_KEY-commands.csv` (Example: `voiceover_macos-commands.csv`)

`testId`

`command`

`settings`

`assertionExceptions`

`presentationNumber`

`scripts.csv`

`setupScript`

`setupScriptDescription`

`references.csv`

`type`

`linkText`

`commands.json` processing: Building strings for command display