Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add advanced setting to allow UIA in chromium based browsers #11079

Closed
wants to merge 27 commits into from

Conversation

LeonarddeR
Copy link
Collaborator

@LeonarddeR LeonarddeR commented May 1, 2020

Link to issue number:

Related to #10555

Summary of the issue:

UIA is now forcefully disabled in chrome based browsers. While this makes perfect sense, it could be beneficial for developers and/or development purpose to circumvent this without hacking.

Description of how this pull request fixes the issue:

  1. Adds an advanced setting that allows using UIA in Chromium based browsers, particularly in Edge. This option defaults to only when necessary, which is when UIA is exposed by the browser and we are unable to inject. If yes, UIA is used, but still only when UIA is exposed by the browser (not by default in Chrome). If no, there's just no browse mode when we can't inject.
  2. Added NVDAObjects.UIA.web and NVDAObjects.UIA.chromium. The chromium module is meant to contain (Edge) Chromium specific code. The web module is meant to contain code shared between Edge Chromium and Edge Legacy. The Edge module contains Edge Legacy specific code. The code I moved to the web module was moved mainly by guessing what we'd need and what not, so some code in there can still be considered Edge legacy specific. It is very hard to assess this, though. I haven't considered renaming edge to edgeLegacy.

Known issues (Todo)

  • Update documentation
  • Headings announcements are duplicated, probably because Edge Chromium both exposes them as control and format field. We'd have to decide what to use and whether the EdgeHeadingQuicknavIterator should be moved to the Web module.
  • The makeTextInfo implementation of the Edge tree interceptor has been moved to the web tree interceptor, so is also used for Chromium. Without this, I"m getting lots of errors indicating that a range couldn't be cloned. This is either a bug in Edgium, or we need another workaround, as navigation in browse mode is still pretty buggy and bouncy.
  • It looks like navigation by line is also broken, as I've seen several cases (e.g. at the top of babbage.com) where lines are broken into words.

Testing performed:

  • Tested in Edge that the config flag works.

Change log entry:

t.b.d.

@nvaccess nvaccess deleted a comment from AppVeyorBot May 1, 2020
@nvaccess nvaccess deleted a comment from AppVeyorBot May 1, 2020
@nvaccess nvaccess deleted a comment from AppVeyorBot May 1, 2020
@codeofdusk
Copy link
Contributor

On my machine, NVDA completely freezes with this PR in use:

  1. Download the Appvayor build, and opt to continue running.
  2. Set UIA in Chrome to "yes".
  3. Start chrome (at the run dialog "chrome --enable-experimental-ui-automation").
  4. Tab into the document.

@LeonarddeR
Copy link
Collaborator Author

What version of chrome was this? I think it is likely that Microsoft works on the UIA implementation separately from the chromium project and that it's only merged into chromium when a major milestone has been reached. Therefore, we couldn't be sure that the UIA implementation in Edge 81 is the same as in Chrome 81, etc.

@codeofdusk
Copy link
Contributor

Running Chrome 83.0.4103.34.

@codeofdusk
Copy link
Contributor

codeofdusk commented May 2, 2020

I've just installed Chrome Canary (version 84) and retested. I opened Chrome, NVDA hung and I closed it. I get the following in the log:

DEBUGWARNING - _UIAHandler.UIAHandler.IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent (03:12:56.765) - Dummy-3 (15196):
HandleFocusChangedEvent: Ignoring duplicate focus event
DEBUGWARNING - watchdog._watcher (03:13:07.013) - watchdog (16652):
Trying to recover from freeze, core stack:
  File "nvda.pyw", line 215, in <module>
  File "core.pyc", line 545, in main
  File "wx\core.pyc", line 2134, in MainLoop
  File "gui\__init__.pyc", line 1032, in Notify
  File "core.pyc", line 515, in run
  File "queueHandler.pyc", line 83, in pumpAll
  File "queueHandler.pyc", line 50, in flushQueue
  File "sayAllHandler.pyc", line 172, in nextLine
  File "speech\types.pyc", line 49, in _flattenNestedSequences
  File "speech\types.pyc", line 49, in <genexpr>
  File "speech\__init__.pyc", line 1116, in getTextInfoSpeech
  File "treeInterceptorHandler.pyc", line 231, in getTextWithFields
  File "NVDAObjects\UIA\__init__.pyc", line 716, in getTextWithFields
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  [Previous line repeated 263 more times]
  File "NVDAObjects\UIA\__init__.pyc", line 594, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 428, in _getControlFieldForObject
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 145, in _getPropertyViaCache
  File "NVDAObjects\UIA\__init__.pyc", line 1533, in _get_positionInfo
  File "NVDAObjects\UIA\__init__.pyc", line 811, in _getUIACacheablePropertyValue
  File "comtypesMonkeyPatches.pyc", line 26, in __call__

DEBUGWARNING - queueHandler.flushQueue (03:13:14.991) - MainThread (6988):
Error in func nextLine from eventQueue
Traceback (most recent call last):
  File "queueHandler.pyc", line 50, in flushQueue
  File "sayAllHandler.pyc", line 172, in nextLine
  File "speech\types.pyc", line 49, in _flattenNestedSequences
  File "speech\types.pyc", line 49, in <genexpr>
  File "speech\__init__.pyc", line 1116, in getTextInfoSpeech
  File "treeInterceptorHandler.pyc", line 231, in getTextWithFields
  File "NVDAObjects\UIA\__init__.pyc", line 716, in getTextWithFields
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 686, in _getTextWithFieldsForUIARange
  [Previous line repeated 456 more times]
  File "NVDAObjects\UIA\__init__.pyc", line 704, in _getTextWithFieldsForUIARange
  File "NVDAObjects\UIA\__init__.pyc", line 489, in _getTextWithFields_text
  File "NVDAObjects\UIA\__init__.pyc", line 455, in _getTextFromUIARange
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.IAccessible.IAccessible._get_IAccessibleRole (03:13:16.347) - MainThread (6988):
accRole failed: (-2147024809, 'The parameter is incorrect.', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.UIA.UIATextInfo.__init__ (03:13:16.361) - MainThread (6988):
Could not clone range
Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 296, in __init__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.UIA.web.UIAWebTreeInterceptor.makeTextInfo (03:13:16.361) - MainThread (6988):
selection died. Using first position instead
ERROR - scriptHandler.executeScript (03:13:16.362) - MainThread (6988):
error executing script: <bound method CursorManager.script_moveByLine_back of <NVDAObjects.UIA.chromium.ChromiumUIATreeInterceptor object at 0x093FA950>> with gesture 'up arrow'
Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 296, in __init__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NVDAObjects\UIA\web.pyc", line 392, in makeTextInfo
  File "cursorManager.pyc", line 448, in makeTextInfo
  File "treeInterceptorHandler.pyc", line 179, in copy
  File "NVDAObjects\UIA\__init__.pyc", line 759, in copy
  File "NVDAObjects\UIA\__init__.pyc", line 300, in __init__
RuntimeError: Could not clone range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 316, in __init__
  File "comtypes\__init__.pyc", line 279, in __getattr__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scriptHandler.pyc", line 205, in executeScript
  File "cursorManager.pyc", line 214, in script_moveByLine_back
  File "cursorManager.pyc", line 114, in _caretMovementScriptHelper
  File "NVDAObjects\UIA\web.pyc", line 399, in makeTextInfo
  File "NVDAObjects\UIA\web.pyc", line 402, in makeTextInfo
  File "NVDAObjects\UIA\web.pyc", line 392, in makeTextInfo
  File "cursorManager.pyc", line 453, in makeTextInfo
  File "documentBase.pyc", line 24, in makeTextInfo
  File "treeInterceptorHandler.pyc", line 173, in __init__
  File "NVDAObjects\UIA\__init__.pyc", line 319, in __init__
RuntimeError
DEBUGWARNING - NVDAObjects.UIA.UIATextInfo.__init__ (03:13:16.372) - MainThread (6988):
Could not clone range
Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 296, in __init__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.UIA.web.UIAWebTreeInterceptor.makeTextInfo (03:13:16.373) - MainThread (6988):
selection died. Using first position instead
ERROR - scriptHandler.executeScript (03:13:16.373) - MainThread (6988):
error executing script: <bound method CursorManager.script_moveByLine_forward of <NVDAObjects.UIA.chromium.ChromiumUIATreeInterceptor object at 0x093FA950>> with gesture 'down arrow'
Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 296, in __init__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NVDAObjects\UIA\web.pyc", line 392, in makeTextInfo
  File "cursorManager.pyc", line 448, in makeTextInfo
  File "treeInterceptorHandler.pyc", line 179, in copy
  File "NVDAObjects\UIA\__init__.pyc", line 759, in copy
  File "NVDAObjects\UIA\__init__.pyc", line 300, in __init__
RuntimeError: Could not clone range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 316, in __init__
  File "comtypes\__init__.pyc", line 279, in __getattr__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scriptHandler.pyc", line 205, in executeScript
  File "cursorManager.pyc", line 218, in script_moveByLine_forward
  File "cursorManager.pyc", line 114, in _caretMovementScriptHelper
  File "NVDAObjects\UIA\web.pyc", line 399, in makeTextInfo
  File "NVDAObjects\UIA\web.pyc", line 402, in makeTextInfo
  File "NVDAObjects\UIA\web.pyc", line 392, in makeTextInfo
  File "cursorManager.pyc", line 453, in makeTextInfo
  File "documentBase.pyc", line 24, in makeTextInfo
  File "treeInterceptorHandler.pyc", line 173, in __init__
  File "NVDAObjects\UIA\__init__.pyc", line 319, in __init__
RuntimeError
DEBUGWARNING - NVDAObjects.UIA.UIA._prefetchUIACacheForPropertyIDs (03:13:16.395) - MainThread (6988):
IUIAutomationElement.buildUpdatedCache failed given IDs of {30019, 30086, 30022, 30025, 30036, 30070, 30103, 30008, 30009, 30010, 30046, 30079}
DEBUGWARNING - eventHandler.executeEvent (03:13:16.396) - MainThread (6988):
error executing event: typedCharacter on <NVDAObjects.UIA.chromium.ChromiumUIA object at 0x0928BFD0> with extra args of {'ch': '\x17'}
Traceback (most recent call last):
  File "eventHandler.pyc", line 155, in executeEvent
  File "eventHandler.pyc", line 92, in __init__
  File "eventHandler.pyc", line 100, in next
  File "NVDAObjects\__init__.pyc", line 1047, in event_typedCharacter
  File "speech\__init__.pyc", line 966, in speakTypedCharacters
  File "api.pyc", line 264, in isTypingProtected
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 145, in _getPropertyViaCache
  File "NVDAObjects\__init__.pyc", line 921, in _get_isProtected
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 145, in _getPropertyViaCache
  File "NVDAObjects\UIA\web.pyc", line 321, in _get_states
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 145, in _getPropertyViaCache
  File "NVDAObjects\UIA\__init__.pyc", line 1234, in _get_states
  File "NVDAObjects\UIA\__init__.pyc", line 814, in _getUIACacheablePropertyValue
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

@LeonarddeR
Copy link
Collaborator Author

I also installed Chrome Canary. The point is that, when using alpha and force UIA so it will at least work on the address bar, the address bar itself isn't very accessible. Therefore I don't think these UIA implementations are in sync.

@michaelDCurran
Copy link
Member

NV access is interested in this pr, specifically to support Edge on Windows 10 S with the NVDA store app (which cannot inject in-process). Edge 81 does not have the UIA freezing bugs last time I tested. I think we should only enable UIA for msEdge. Not all of Chromium, if this is possible.

@codeofdusk
Copy link
Contributor

I think we should only enable UIA for msEdge.

If the improved UIA implementation makes it to Chrome, wouldn't it significantly improve performance to enable it there as well?

@michaelDCurran
Copy link
Member

michaelDCurran commented May 7, 2020 via email

@josephsl
Copy link
Collaborator

josephsl commented May 7, 2020 via email

@LeonarddeR
Copy link
Collaborator Author

LeonarddeR commented May 7, 2020

I'm happy to change this into something that only works in Edge, but it might help to know how Chrome and Edge UIA implementations relate to each other. @michaelDCurran, would this be something Microsoft and/or Google could inform us about?

Note that in Chrome, the UIA support is still considered to be experimental and still needs a command line parameter to work. If they are the same implementations though, I don't see a reason to block UIA from working in Chrome if someone explicitly passes the experimental uiautomation cmd parameter.

@michaelDCurran
Copy link
Member

michaelDCurran commented May 7, 2020 via email

@LeonarddeR
Copy link
Collaborator Author

But we would not want to publish the idea of UIA with Chromium too heavily here.

Agreed. So either the documentation should contain a huge disclaimer, or we should advertise the option as Edge specific, yet being chromium specific under the hood. I'd go for the former. We can reword the option in such a way that it mentions Edge and supported Chromium based browsers, for example.

@josephsl
Copy link
Collaborator

josephsl commented May 7, 2020 via email

@LeonarddeR
Copy link
Collaborator Author

@michaelDCurran I have updated the initial description to reflect the current state of the pr. While I started this work, I"m afraid I neither have the time nor the skills to work further on this. You really know UIA much better than I do. May be @josephsl might be interested to take a look as well?

Note that the new UIA.web module contains code from the edge module, all moved around, so that makes reviewing this code a bit easier.

@josephsl
Copy link
Collaborator

josephsl commented May 7, 2020 via email

…e them in format fields, as headings were being reported twice.
…completely before the start of the parent textRange. Stops a freeze on some checkboxes in Edgium and is generally safer.
@AppVeyorBot

This comment has been minimized.

@LeonarddeR
Copy link
Collaborator Author

@michaelDCurran Thanks for taking this further!
Just to make sure, I assume you are aware that every change done to UIA.web has to be tested in legacy Edge as well, at least on older versions of Windows 10? Even on current versions, the legacy Edge implementation is still used when searching in the start menu, for example.

@michaelDCurran
Copy link
Member

michaelDCurran commented Dec 14, 2020

In order for existing logic around when NVDA should and should not report the accessible name for a control, Chromium / Edge needs to expose aria-label in the UIA element's AriaProperties property. Edge classic did this previously, but Chromium's UIA implementation does not.
See Chromium bug 1147700:
https://bugs.chromium.org/p/chromium/issues/detail?id=1147700

@Adriani90
Copy link
Collaborator

cc: @aleventhal, @ObjectInSpace

@michaelDCurran michaelDCurran marked this pull request as ready for review January 5, 2021 05:25
@michaelDCurran
Copy link
Member

@codeofdusk Would you mind testing this branch again, and let me know if you see any freezes. Rememeber to turn on Use UIA in Edge and other Chromium-based brwosers in NvDA's advanced panel.

@AppVeyorBot

This comment has been minimized.

@feerrenrut
Copy link
Contributor

@michaelDCurran Could you double check the PR description to make sure it is still accurate please?

@dpy013

This comment has been minimized.

@codeofdusk
Copy link
Contributor

codeofdusk commented Jan 5, 2021

Running Chrome with the --enable-experimental-ui-automation flag now uses UIA as expected (confirmed in Python console) and documents are browsable.

Opening this page in IA2 Chrome causes a hang of several seconds. The same page in UIA Chrome loads almost instantly, but attempting to move by heading causes the browser to crash:

NVDA traceback

ERROR - scriptHandler.executeScript (05:10:01.927) - MainThread (15344):
error executing script: <bound method CursorManager.script_moveByLine_forward of <NVDAObjects.UIA.chromium.ChromiumUIATreeInterceptor object at 0x072F9630>> with gesture 'down arrow'
Traceback (most recent call last):
  File "NVDAObjects\UIA\__init__.pyc", line 297, in __init__
  File "comtypesMonkeyPatches.pyc", line 26, in __call__
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "NVDAObjects\UIA\web.pyc", line 411, in makeTextInfo
File "cursorManager.pyc", line 481, in makeTextInfo
File "treeInterceptorHandler.pyc", line 180, in copy
File "NVDAObjects\UIA_init_.pyc", line 780, in copy
File "NVDAObjects\UIA_init_.pyc", line 301, in init
RuntimeError: Could not clone range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "NVDAObjects\UIA_init_.pyc", line 317, in init
File "comtypes_init_.pyc", line 279, in getattr
File "comtypesMonkeyPatches.pyc", line 26, in call
_ctypes.COMError: (-2147220991, 'An event was unable to invoke any of the subscribers', (None, None, None, 0, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "scriptHandler.pyc", line 208, in executeScript
File "cursorManager.pyc", line 255, in script_moveByLine_forward
File "cursorManager.pyc", line 129, in caretMovementScriptHelper
File "NVDAObjects\UIA\web.pyc", line 418, in makeTextInfo
File "NVDAObjects\UIA\web.pyc", line 421, in makeTextInfo
File "NVDAObjects\UIA\web.pyc", line 411, in makeTextInfo
File "cursorManager.pyc", line 486, in makeTextInfo
File "documentBase.pyc", line 24, in makeTextInfo
File "treeInterceptorHandler.pyc", line 174, in init
File "NVDAObjects\UIA_init
.pyc", line 320, in init
RuntimeError

@feerrenrut
Copy link
Contributor

I've been looking at this, but I'm not sure how to review it effectively. Without tests or further documentation about why the changes are made and the circumstances in which they are required I'd only be rubber stamping it.

I'd like to see a plan for making this area of the code easier to understand. How can we move away from large recursive functions with many pathways?

@josephsl
Copy link
Collaborator

josephsl commented Jan 13, 2021 via email

@feerrenrut
Copy link
Contributor

One option is to rebase these commits to clean up the history. So that each commit message can explain the need for the change. Move classes / functions around first (without further modification), then further commits to fix up the things that need fixing with full explanations of why it needs to change in the commit message (eg the scenario where there is an issue). This would be with the goal to merge this rather than squash merge.

@feerrenrut
Copy link
Contributor

Closing this PR as it is replaced by #12025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants