Update report status dialog for recommended test plans to include information about required AT/Browser versions

[mcking65]

The following requirement is listed in #648 under the heading "Test plan report status dialog Description":

TODO: If the phase is recommended, the algorithm for determining what rows to put in the table will be described in the next update of this specification.

This issue will complete that specification and track development of the requisite changes to the report status dialog.

ARIA-AT goal

One goal of the ARIA-AT working mode specific to test plans in the recommended phase is to keep their interop reports current. That means, for each AT/Browser combo that has a report, generate a new report when a new version of an AT or browser is released. In addition to keeping data current, over time, this will also create data that can be used to publish trend charts.

As of September 27, 2023, the CG plans to update recommended reports for each public release of a supported screen reader; see Minutes for ARIA and Assistive Technologies Community Group Weekly Teleconference – 27 September 2023. Until the ARIA-AT project has sufficient automation and compute resources, new browser releases will not trigger report updates. The browser requirement will be "Any Recent Version". In the event of report run conflicts that are attributable to browser version differences, data from the more recent browser version will be published.

Report status dialog design update objective

When viewing the report status for a recommended test plan, make it easy to determine which AT/Browser version combinations are missing reports and add them to the test queue.

Test queue dependency

The test queue does not currently support version-specific test plan runs. Issue #791 must be resolved at the same time or before this issue.

Primary test plan run dependency

The meet the data processing requirements of this solution, the system will need to enable admins to mark one of the test plan runs as "primary" when finalizing a report. The requirements for updates to the report finalization process are defined in issue ...

Definition of "First Required AT Versions" for a test plan

Incomplete DRAFT CONTENT

The goal is to update reports for all AT versions that are newer than the versions used to generate the data that supported the candidate review process of the test plan. In other words, if new AT versions were released during the review process for which reports were not generated, reports using those versions will need to be generated after the test plan becomes recommended. This prevents important gaps in the history of the AT support levels.

Thus, each recommended test plan will have its own set of "First Required AT Versions" or "First Required AT Version Dates". Whether date or version is used is an implementation detail; either will do. For the purpose of the following definition, version number is used.

A test plan has a "First Required AT Version" for each AT that meets the following two criteria:

• The AT is covered by the test plan
• the AT is designated as requiring reports for recommended test plans

The "First Required AT Version" for each AT that meets the above criteria is the earliest version used to generate any of the primary test plan results for any of the reports that are final on the date the test plan is advanced to recommended.

For example, consider a scenario where the command button test plan is advanced to recommended. At the time it is advanced to recommended, It has a report for NVDA and Chrome that was generated during draft review using NVDA 2022.4. This data was used during the review process It also has a report for NVDA and Firefox generated during candidate review using NVDA 2023.1. In this case, the first required NVDA version would be 2022.4. That means that
that was used to generate the primary test plan results shown in is

Report status data changes

When a test plan is advanced to the recommended phase, make the following changes to the data in the table in the report status dialog.

• In any row in the table that was present prior to the date the test plan entered the recommended phase, in cells in the AT column, append the version number of the AT that was used to generate the results for the primary test plan run.
• If that AT is marked as required for recommended phase in the database, then for this test plan, that version is the "first required version" of that AT.
• Add a new row to the table for each AT/Browser combination that meets all the following criteria:

[ccanash]

Hi @mcking65 would you be able to add the criterias you mentioned at the end of the issue description that outlines the criteria for adding new row to the table for each AT/Browser combination.

[mcking65]

@ccanash

This issue should be part of the trend reporting project. I tried to find the project and could not. Have you created it? If so, it is not associated with either the aria-at or aria-at-app repos.

This is a pretty complex issue and it raises some questions that need discussion. It might be better if I were to consider how to break it out into multiple issues.

[howard-e]

@mcking65 to facilitate future discussions, I’ve summarized some key points of our recent discussions that may have not yet been captured by the top comment, captured elsewhere or just in need of further clarifications:

Key points

• For 2024, we won’t be concerned with capturing specific versions for browsers, only ATs
• Admins should be able to create a test plan report and indicate that the tester can use "Any recent" version OR explicitly specify which version of AT the testers should be using to capture results.
  • If the test plan version is already in recommended, setting a specific version of the AT is required
• When upgrading a report to recommended, the first required AT version will be what’s displayed. Otherwise it will be what’s specified by the test admin when adding new reports to the test queue for a test plan that has already been recommended.
• When a test plan version is promoted to recommended, the AT version should be included in the title (eg. from "Alert with NVDA and Firefox" to "Alert with NVDA 2023.3 and Firefox")
• When a new AT Version is added to the system, for a test plan that is currently recommended, should now show that required reports are missing. The required reports dialog would display the following as an example; the AT versions used are what were recently included in the system:
  • NVDA 2024 (or similar) with Chrome (any version) is missing required report, with an add button to immediately add it to the test queue
  • JAWS 2025 (or similar) with Chrome (any version) is missing required report with similar UI
  • And so on …

Example

• Using Alert V23.12.06 Example with JAWS and Chrome as an example, version history is only shown under the Run History section.
  • This is fine for reports in draft and candidate today.
  • But when a test plan is promoted to recommended, the report needs to show that a specific AT Version is being used.
  • If the current report is promoted to recommended, JAWS 2024.2310.70 shown under the run history section would be set as the first required report.
  • If the test plan is promoted to recommended, the report’s title would now read as "Alert with JAWS 2024.2310.70 and Chrome".
  • If a new AT Version, JAWS 2025 for example, is added to the system, when looking at the Data Management page for the Alert row, the required reports should now say "required reports missing". The dialog would now show that JAWS 2025 (or similar) with Chrome (any version) is required and missing, with a button to immediately add it to the test queue.

[howard-e]

• Admins should be able to create a test plan report and indicate that the tester can use "Any recent" version OR explicitly specify which version of AT the testers should be using to capture results.
  • If the test plan version is already in recommended, setting a specific version of the AT is required

@mcking65 In addition to the current restriction applied to the main point, there was a brief discussion on an additional restriction that if the test plan was in a draft phase, it can only be added with "Any recent" version. Could you confirm if that was understood correctly? And if so what was the decision there?

[howard-e]

• Admins should be able to create a test plan report and indicate that the tester can use "Any recent" version OR explicitly specify which version of AT the testers should be using to capture results.

  • If the test plan version is already in recommended, setting a specific version of the AT is required

... In addition to the current restriction applied to the main point, there was a brief discussion on an additional restriction that if the test plan was in a draft phase, it can only be added with "Any recent" version. Could you confirm if that was understood correctly? And if so what was the decision there?

Discussed outside this issue - the decision is not to add this additional constraint so give admins and testers can have greater flexibility in results which can be collected.

The existing constraint remains if the test plan is in recommended.