Test collection consumes too many resources in large workspaces

[koliyo]

Environment data

• VS Code version: Version 1.30.2 (1.30.2)
• Extension version (available under the Extensions sidebar): 2019.1.0
• OS and version: Mac os Mojave
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.2
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: XXX

Expected behaviour

Not taking over all my CPUs

Actual behaviour

Large amounts of pytest processes running, grabbing all my CPU cores.

NOTE: Not running actual tests, only gathering test info with --collect-only flag

Steps to reproduce:

Not sure at this point

I do have pytest functionality enabled

Python › Unit Test: Py Test Enabled

Logs

[d3r3kk]

Hi @koliyo, thanks for the bug report.

I've attempted to reproduce your issue using the pytest-dev/pytest repo on Github, on Windows 10 and Ubuntu 18.10, with no luck. I see you are running on macOS so that might be my issue (I don't have access to a macOS machine at the moment).

Question: What version of pytest are you using?

Also, could you try the following and let me know if it works for you?

Run from the command shell:

1. Open a command shell
2. Navigate to your workspace where you observed this behaviour.
3. Activate your Python interpreter (3.7.2)
4. Run pytest --collect-only and observe the process that occurs.
5. Copy/paste the results here

Run against a simplified workspace:

1. Create a new workspace
2. Create a python file called test_pytest.py in your workspace.
3. Add the following code to it:

def test_pytest_discovery():
    assert 1 + 1 == 2

4. Select your interpreter in the status bar (3.7.2)
5. Click on 'Run Tests' in the status bar, and select 'Discover Unit Tests' from the command menu that appears.

[koliyo]

Some new information. This is for a big C++ project were we have SWIG python bindings and python scripts.

The problem was pytest is recursing down all directories, which in my case includes a bunch of build directories inside this project root. I have

${root}/oos/build-config-a
${root}/oos/build-config-b
...

I have now added a pytest.ini file in the root with explicit listing of relevant python paths

[pytest]
testpaths = data/python

I also looked at the norecursedirs configuration option.

Now I no longer see this problem. But it makes it a little discouraging to enable pytest globally in my VS Code config. Seems better to enable it only for specific projects, which is what I am going to do.

[d3r3kk]

...The problem was pytest is recursing down all directories...

Ah I see, the problem is that our test discovery solution is burning through cycles in a relatively large project.

Problem Statement

On a very large project with many folders in the workspace, Python unit test discovery can take a lot of resources to complete. Detect when test discovery is taking too long and/or visiting too many folders and propose a solution to the user via some mechanism such as a popup.

Workaround

Make use of pytest.ini:

[pytest]
testPaths = relative/path/to/tests/root

or the something like "python.unitTest.pyTestArgs": [ "${workFolder}/relative/path/to/tests/root" ] in the .vscode/settings.json file.

Changing the name of this issue and proposing it as an enhancement to the team.

[DonJayamanne]

Closing as nothing needs to be done here.
Its up to the user to configure pytest.ini or if they don't have that, configure settings.json (done via extension UI prompts).
Looks like in this case the user had set the root of the tests to be the root directory that caused pytest to recurse through all of the project folders and that's out of our control.
I.e. if user used pytest from the command line the same thing would have happened.

[koliyo]

I solved this by adding a pytest.ini, but I do not agree that this is only user problem.

This is why

1. I enabled pytest functionality globally using Python › Unit Test: Py Test Enabled
2. I opened a project where we do not even use pytest, but we do have some python files
3. VS Code becomes unresponsive and locks up huge amounts of resources

Looks like in this case the user had set the root

I had not configured pytest at all. We are not even using pytest in this project, this is why I still think this is an issue

If the conclusion is that this is a user problem, then I think there should be a BIG warning in the global VS code setting for Python › Unit Test: Py Test Enabled to say that it may cause big problems.

[DonJayamanne]

I had not configured pytest at all. We are not even using pytest in this project, this is why I still think this is an issue

This is something the user chose to enable , see below

I enabled pytest functionality globally using Python › Unit Test: Py Test Enabled

I think there should be a BIG warning in the global VS code setting for Python › Unit Test: Py Test Enabled to say that it may cause big problems.

Again, user enabled this, and it is pytest consuming the resources as it had been enabled for this workspace implicitly. We can reopen this issue depending on the number of upvotes to this issue. I.e. if others come across similar issues, we will address it,