Support UI Automation in Microsoft Edge Chromium and Google Chrome

[codeofdusk]

Is your feature request related to a problem? Please describe.

When starting Chrome with the --enable-experimental-ui-automation flag, browse mode is not functional.

Describe the solution you'd like

Support for UIA Chrome, including browse mode and full functionality of all widgets.

Describe alternatives you've considered

Continue to use the IA2 implementation and forcibly ignore UIA (similar to #7497) if UIA becomes no longer behind a feature flag. However, UIA would likely improve performance on large pages.

[josephsl]

Hi, I think there is an issue on Chromium Edge/UIA, but I think we should use this one. CC @LeonarddeR

[LeonarddeR]

I closed the other issue.

[Nikita34196]

The native UI Automation is enabled by default in microsoft edge starting with version 80.0.355.0 (Official Build) canary.

[LeonarddeR]

Even in the stable build of edge, UIA now turns out to be the default.

[Nikita34196]

I reported it to Microsoft. they said there should be UIA support on the nvda side. they also offer to turn it off manually, but not everyone knows how to do it. I don't think a lot of people will be able to turn it off in edge.

[LeonarddeR]

The problem is that NVDA doesn't create a tree interceptor to activate browse mode. This is not something on Microsofts end.

[Adriani90]

In Edge Cromium 81 which is the last stable version, UIA is enabled by default and browse mode seems to work as expected.
@codeofdusk can you confirm?

[codeofdusk]

On my machine (Chrome 83) with the --enable-experimental-ui-automation flag, NVDA now seems to force IAccessible 2.

[josephsl]

Hi, this is because UIA will be ignored from document window itself. Other controls will be recognized as UIA if they are UIA elements. Thanks.

[codeofdusk]

Makes sense, so this issue is still not resolved (we still need to implement support for UIA in Chrome).

[Adriani90]

CC: @feerrenrut after our private discussion I think this is maybe nice to keep an eye on for the future.

[LeonarddeR]

Personally, I think a first pr should address two things:

1. Ad a flag to NVDA's advanced settings to enforce IA2 in Edge and Chrome, which is enabled by default
2. Implement a basic browse mode tree interceptor for UIA in Edge/Chrome that's active when UIA is used.

[codeofdusk]

Ad a flag to NVDA's advanced settings to enforce IA2 in Edge and Chrome, which is enabled by default

Why enforce IA2? Wouldn't it make sense to make the flag be for enabling UIA, similar to Word and Windows Console?

[feerrenrut]

For these feature flags its important to additionally include an explicit "use default" option, so for a binary decision, it becomes ternary:

• The current NVDA default (most users should / will stay on this option) the outcome of this option can change in future. It may result in X in this release and Y in the next.
• Explicitly X
• Explicitly Y

This is particularly important if developers can foresee that the default setting may need to change. It allows developers to take the users explicit preferences into account during an upgrade procedure.

[codeofdusk]

Yes, but like UIA console, don't expose this option in the GUI until it becomes default (compare the current checkbox to the combo box in #10964).

[feerrenrut]

Yes, but like UIA console, don't expose this option in the GUI until it becomes default.

No. This does not solve the problem I outlined. It must be exposed in the GUI otherwise the user is missing one of the explicit choices. They are left with only:

• Explicit X
• Current default Y

If the user specifically wants to use Y until some amount of time after X has become the default doing this will break their config. To provide an effective update path you need three options for a binary setting:

• Explicit X
• Explicit Y
• Current default (Y now, X later)

This allows those who just want the recommended settings to get them. Those who want to try the latest thing to do so, and those who want to stay on the "tried and true" as long as possible to do so also.

[k-kolev1985]

I don't know how important it is, but recently I was looking for the setting "Native UI Automation" in "edge://flags" in 64-bit Microsoft Edge Canary version 84.0.499.0 and did not find it by searching for "Native UI Automation" from within the provided search field. Could it be that the setting was removed and now UIA is the default and could not be changed as we could in the past? I don't know if this matters for this discussion, but I decided to share it with you just in case it does matter.

[LeonarddeR]

I will do so in #11079, though now it is still binary, and really broken if enabled. I actually followed @codeofdusk's suggestion and worded it positively towards UIA instead.

[LeonarddeR]

I implemented a prototype in #11079 that works somehow. However, it looks like browse mode is pretty broken in that it keeps bouncing all over the page in complexer documents. Therefore, I don't consider it good enough to go into a release build.

May be either @feerrenrut or @michaelDCurran could elaborate on what the current plans are with Chromium and UIA. I see several solution.

1. Abandon this ticket for now and close Add advanced setting to allow UIA in chromium based browsers #11079. It could serve as a reference for future attempts
2. Only keep the Add advanced setting to allow UIA in chromium based browsers #11079 part that adds the flag to advanced settings, but clearly document that browse mode is not available. While being useless for most people, I think this might serve developers to rapidly test the state of the UIA implementation in test builds of Edge.
3. Keep this as is, of course with a code quality review, but clearly document that browse mode is far from usable yet and that this implementation only serves development of Edge/Chromium. It might be suitable for Edge and Chromium accessibility devs to test their implementation based on what NVDA expects from it.

[codeofdusk]

I vote for option 3 if the freeze can be fixed.

[feerrenrut]

Just to document it, and it might be handy to know about for UIA in Chrome is the docs for "Runtime two way IAccessible2 to UI Automation elements look up via unique id": https://chromium.googlesource.com/chromium/src/+/master/docs/accessibility/ia2_to_uia.md

[feerrenrut]

Should have been closed with PR #12025