Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MNT] Windows issue on python 3.10-3.12 #1632

Open
fkiraly opened this issue Aug 30, 2024 · 5 comments
Open

[MNT] Windows issue on python 3.10-3.12 #1632

fkiraly opened this issue Aug 30, 2024 · 5 comments
Labels
maintenance Continuous integration, unit testing & package distribution

Comments

@fkiraly
Copy link
Collaborator

fkiraly commented Aug 30, 2024

There is another issue on python 3.10-3.12, tests crash with this:

Windows fatal exception: code 0xc0000374

Thread 0x000002c4 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 359 in wait
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 655 in wait
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\tqdm\_monitor.py", line 60 in run
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 1075 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 1032 in _bootstrap

Thread 0x000008c8 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 359 in wait
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 655 in wait
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\tqdm\_monitor.py", line 60 in run
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 1075 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\threading.py", line 1032 in _bootstrap

Current thread 0x00000c88 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\torch\autograd\graph.py", line 768 in _engine_run_backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\torch\autograd\__init__.py", line 289 in backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\torch\_tensor.py", line 521 in backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\lightning\pytorch\core\module.py", line 1101 in backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\lightning\pytorch\plugins\precision\precision.py", line 72 in backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\lightning\pytorch\strategies\strategy.py", line 212 in backward
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\runner.py", line 132 in runtestprotocol
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\main.py", line 337 in _main
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\config\__init__.py", line 175 in main
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\_pytest\config\__init__.py", line 201 in console_main
  File "C:\hostedtoolcache\windows\Python\3.12.5\x64\Lib\site-packages\pytest\__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
tests/test_models/test_temporal_fusion_transformer.py::test_hyperparameter_optimization_integration[True]
@fkiraly fkiraly added the maintenance Continuous integration, unit testing & package distribution label Aug 30, 2024
This was referenced Aug 30, 2024
Open
Merged
@benHeid
Copy link
Collaborator

benHeid commented Aug 30, 2024

This is also occurring in sktime with TFT https://github.com/sktime/sktime/actions/runs/10610051364/job/29406877935#step:11:32632

fkiraly added a commit that referenced this issue Aug 30, 2024
@fkiraly
[MNT] skip tests involved in windows failures until resolved (#1631)
6c85b8e 
This PR skips tests involved in the failures on windows listed in #1623 until the underlying issues are resolved, see #1632 and #1632
@benHeid
Copy link
Collaborator

benHeid commented Aug 30, 2024

According to different threads at GitHub. This is an error that multiple libraries are facing when using xdist.. See for example: coiled/benchmarks#404. There someone had the assumption that there is to much load on the worker and thus they are failing...

@benHeid
Copy link
Collaborator

benHeid commented Aug 30, 2024

Some people seem to be successful with reducing the number of workers:

@benHeid benHeid mentioned this issue Aug 30, 2024
Merged
@benHeid
Copy link
Collaborator

benHeid commented Aug 30, 2024

I would suggest to select smaller possible values in optimize_hyperparameters then the default ones. Hopefully, this is reducing the load so that these weird errors do not occur any more.. :/

@fkiraly
Copy link
Collaborator Author

fkiraly commented Aug 30, 2024 •
edited
Loading

We could also try turning off pytest-xdist? At least for diagnostic purposes?

fkiraly pushed a commit to sktime/sktime that referenced this issue Sep 7, 2024
@benHeid
[MNT] Try to reduce load for the runners (#7061)
262a9f2 
See also sktime/pytorch-forecasting#1632

I tried to reduce the load for the runner by reducing the batch size in
the PyTorch forecasting models and reduced the model size of the TFT.

In my opinion, this PR is successful if in multiple runs, the CI is not
crashing because of HEAP errors and crashed runners...
benHeid added a commit to Z-Fran/sktime that referenced this issue Oct 3, 2024
@benHeid
[MNT] Try to reduce load for the runners (sktime#7061)
5058dc8 
See also sktime/pytorch-forecasting#1632

I tried to reduce the load for the runner by reducing the batch size in
the PyTorch forecasting models and reduced the model size of the TFT.

In my opinion, this PR is successful if in multiple runs, the CI is not
crashing because of HEAP errors and crashed runners...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
maintenance Continuous integration, unit testing & package distribution
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fkiraly @benHeid