[Windows] Crash while enumerating opened files

[BoboTiG]

Platform

• { Windows 7 SP1 64-bits, Windows 10 64-bits }
• { psutil version: 5.7.0 }
• { 3.7.4, 3.7.7 }

Bug description
That simple code is making Python to crash:

from contextlib import suppress
import psutil

for proc in psutil.process_iter(attrs=["pid"]):
    with suppress(psutil.AccessDenied):
        for handler in proc.open_files():
            print(proc.pid, handler.path)

It will enumerate several proc and files but then it will simply crash. I know for sure that it did not happen on earlier versions of psutil.

Test results
I managed to run pytest into the tests folder and this is the output:

...
tests/test_connections.py::TestFilters::test_combos FAILED                                                       [  9%]
tests/test_connections.py::TestSystemWideConnections::test_multi_sockets_procs FAILED                            ...
tests/test_contracts.py::TestFetchAllProcesses::test_fetch_all Windows fatal exception: access violation

Current thread 0x00002110 (most recent call first):
  File "C:\...\psutil\_pswindows.py", line 993 in open_files
  File "C:\...\psutil\_pswindows.py", line 679 in wrapper
  File "C:\...\psutil\__init__.py", line 1154 in open_files
  File "C:\...\psutil\tests\test_contracts.py", line 357 in call_meth
  File "C:\...\psutil\tests\test_contracts.py", line 368 in test_fetch_all
  File "C:\Python377-32\lib\unittest\case.py", line 628 in run
  File "C:\Python377-32\lib\unittest\case.py", line 676 in __call__
  File "C:\...\_pytest\unittest.py", line 231 in runtest
  File "C:\...\_pytest\runner.py", line 135 in pytest_runtest_call
  File "C:\...\pluggy\callers.py", line 187 in _multicall
  File "C:\...\pluggy\manager.py", line 87 in <lambda>
  File "C:\...\pluggy\manager.py", line 93 in _hookexec
  File "C:\...\pluggy\hooks.py", line 286 in __call__
  File "C:\...\_pytest\runner.py", line 217 in <lambda>
  File "C:\...\_pytest\runner.py", line 244 in from_call
  File "C:\...\_pytest\runner.py", line 217 in call_runtest_hook
  File "C:\...\_pytest\runner.py", line 186 in call_and_report
  File "C:\...\_pytest\runner.py", line 100 in runtestprotocol
  File "C:\...\_pytest\runner.py", line 85 in pytest_runtest_protocol
  File "C:\...\pluggy\callers.py", line 187 in _multicall
  File "C:\...\pluggy\manager.py", line 87 in <lambda>
  File "C:\...\pluggy\manager.py", line 93 in _hookexec
  File "C:\...\pluggy\hooks.py", line 286 in __call__
  File "C:\...\_pytest\main.py", line 272 in pytest_runtestloop
  File "C:\...\pluggy\callers.py", line 187 in _multicall
  File "C:\...\pluggy\manager.py", line 87 in <lambda>
  File "C:\...\pluggy\manager.py", line 93 in _hookexec
  File "C:\...\pluggy\hooks.py", line 286 in __call__
  File "C:\...\_pytest\main.py", line 247 in _main
  File "C:\...\_pytest\main.py", line 191 in wrap_session
  File "C:\...\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\...\pluggy\callers.py", line 187 in _multicall
  File "C:\...\pluggy\manager.py", line 87 in <lambda>
  File "C:\...\pluggy\manager.py", line 93 in _hookexec
  File "C:\...\pluggy\hooks.py", line 286 in __call__
  File "C:\...\_pytest\config\__init__.py", line 125 in main
  File "C:\...\pytest\__main__.py", line 7 in <module>
  File "C:\Python377-32\lib\runpy.py", line 85 in _run_code
  File "C:\Python377-32\lib\runpy.py", line 193 in _run_module_as_main

Do you need more details? Let me know what I can do if so :)

[BoboTiG]

Sadly, our CI (GitHub Actions) did not catch it and I think yours has the same issue too :/

[giampaolo]

Well, this is definitively bad news, but at least now we know what the heck is wrong with Github Actions on Windows. =)
CC: @Czaki #1762

[Czaki]

@BoboTiG But this is 32 or 64 bits python? Because I meet unexpected quit for 32 bits python on 64 bits Windows.

[BoboTiG]

Python 32-bit.

[Czaki]

So thanks for information. I have no idea why test on github actions fail in my PR when 64 bits test pass.

I think that Your CI does not catch this because it works on 64 bits python.

[BoboTiG]

Arf, you are right :)
Thanks for the clue, I will change that right now!

[BoboTiG]

FTR I updated the workflow to use Python 32-bits and tests are green 🤔 (cf https://github.com/nuxeo/nuxeo-drive/pull/1677/checks?check_run_id=690301455, and it shouldn't ^^)

[BoboTiG]

GH is running Windows Server 2019, that could explan why.

[Czaki]

GH is running Windows Server 2019, that could explay why.

But in my PR there is fail for 32 bits on GH actions.

[Czaki]

Could you test your application on python 64 bits on physical machine?

[BoboTiG]

It crashes too.

[Czaki]

Argh...

[BoboTiG]

If it can help, here is the Windows error report:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name="Application Error" /> 
        <EventID Qualifiers="0">1000</EventID> 
        <Level>2</Level> 
        <Task>100</Task> 
        <Keywords>0x80000000000000</Keywords> 
        <TimeCreated SystemTime="2020-05-20T09:39:46.428630900Z" /> 
        <EventRecordID>2410</EventRecordID> 
        <Channel>Application</Channel> 
        <Computer>WM-10-Drive-Mika</Computer> 
        <Security /> 
    </System>
    <EventData>
        <Data>python.EXE</Data> 
        <Data>3.7.7150.1013</Data> 
        <Data>5e676f16</Data> 
        <Data>_psutil_windows.cp37-win_amd64.pyd</Data> 
        <Data>0.0.0.0</Data> 
        <Data>5e4c25a3</Data> 
        <Data>c0000005</Data> 
        <Data>00000000000073fd</Data> 
        <Data>2730</Data> 
        <Data>01d62e8a95f90bc0</Data> 
        <Data>C:\Users\Windows10\Desktop\nuxeo-drive\.tox\unit\Scripts\python.EXE</Data> 
        <Data>C:\Users\Windows10\Desktop\nuxeo-drive\.tox\unit\lib\site-packages\psutil\_psutil_windows.cp37-win_amd64.pyd</Data> 
        <Data>d9d7b8dc-2bf0-473c-96aa-a00cc8faf209</Data> 
        <Data /> 
        <Data /> 
    </EventData>
</Event>

And with Python 32-bit, it is _psutil_windows.cp37-win32.pyd.

Both are printing the "Windows fatal exception: access violation" sentence before crashing.

[Czaki]

Maybe you know some windows debugger witch can be run in batch mode?

Something like gdb --batch -ex run -ex bt --args python path/to/script.py

[giampaolo]

@Czaki I am between a laptop upgrade / drama (new hardware, reinstall of Ubuntu, nvidia drivers not working, a lot of cursing in Italian etc. =)) so I don't have a Windows box to test on right now, but if you want to try fixing this "blindly" you may try getting rid of this part (the one which spawns a thread):

psutil/psutil/arch/windows/process_handles.c

Line 151 in 33fea55

psutil_threaded_get_filename(HANDLE hFile) {

...and instead call this function directly (just change the function signature and pass it a file HANDLE):

psutil/psutil/arch/windows/process_handles.c

Line 85 in 33fea55

psutil_get_filename(LPVOID lpvParam) {

Maybe you're lucky, who knows.

open_files() code on Windows is probably the most complex part of psutil, so if that's the real origin, it sort of makes sense the problem emerged there.

Note: I refactored / rewritten open_files() not long ago, and I may have introduced some problems: another thing you can try is using the pre-refactored version, here (copy & paste should be enough I guess):
https://github.com/giampaolo/psutil/blob/release-5.6.0/psutil/arch/windows/process_handles.c

[giampaolo]

I should have fixed this. Please reopen if I didn't.