python language server uses too much memory

[DonJayamanne]

@lisnb commented on Fri May 31 2019

Issue Type: Bug

I'm using this amazing Remote-SSH extension, but the python language server on the remote uses a lot of memory up to 120G+, I try to unistall this extension or kill -9 the process, but it's not working. It seems this process starts itself once killed.

Extension version: 2019.5.17517
VS Code version: Code - Insiders 1.35.0-insider (e9c50663154c369a06355ce752b447af5b580dc3, 2019-05-30T05:18:06.502Z)
OS version: Darwin x64 18.2.0
Remote OS version: Linux x64 4.4.0-33.bm.1-amd64
Remote OS version: Linux x64 4.4.0-33.bm.1-amd64

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-7660U CPU @ 2.50GHz (4 x 2500)
GPU Status	2d_canvas: enabled checker_imaging: disabled_off flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: enabled rasterization: enabled surface_synchronization: enabled_on video_decode: enabled webgl: enabled webgl2: enabled
Load (avg)	2, 2, 2
Memory (System)	16.00GB (0.49GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: dev.29
OS	Linux x64 4.4.0-33.bm.1-amd64
CPUs	Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (48 x 2301)
Memory (System)	251.31GB (70.78GB free)
VM	0%

Item	Value
Remote	SSH: dev.29
OS	Linux x64 4.4.0-33.bm.1-amd64
CPUs	Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (48 x 2301)
Memory (System)	251.31GB (70.78GB free)
VM	0%

---

@lisnb commented on Fri May 31 2019

I noticed that this process was located in ~/.vscode-remote directory but as the remote-ssh extension keeps updating, the vscode is now under ~/.vscode-server-insider directory

[MikhailArkhipov]

@lisnb, please provide version of the LS (it is displayed at startup) and the list of libraries/packages you are importing.

[CMLL]

I ran into this issue today after a long time of the LS behaving pretty ok compared with the previous versions.

LS Version 0.2.98.0

Started VSCode and let it sit on a project that I open daily. Took no action on it, just let it sit. Memory started being consumed without stopping until the machine would become unresponsive, the language server process would be killed and restarted again and the process would repeat itself, keep consuming memory until it crashes again.

I noted also the the background analysis message would be stuck on the same number without decreasing during this whole time. Also noticed that an exception of what appeared to be the Intellicode extension caused a restart of the analysis which then would fire the whole memory consuming process.

I uninstalled the Intellicode extension to see if removing that would help but I see still that the analysis message keeps stuck in a high number, even though this time around there was no exception on the logs. However memory still kept growing, I killed it before it got to the point of crashing my machine.

I'm not sure if it has to do with anything but one thing I did different today was installing the new remote development extensions that got released with vscode 1.35, but I haven't gotten around to using them yet so I'm not sure if that's related or not.

Not sure if this is accurate but I enabled Trace logging into the extension and some of times when logs would stop at this point and never log anything else. Not sure if something in there the cause of the background analysis never finishing.

[MikhailArkhipov]

@CMLL - I can see in the log that modules are getting reloaded. This means LS decided that packages changed. LS needs to distinguish libraries from user code. @jakebailey worked in this.

Do you happen to have search paths pointing to your code so perhaps LS is confusing file save with package installation?

[CMLL]

Yeah, I do.
I have this in my workspace settings...

    "python.autoComplete.extraPaths": [
        "/home/cllamach/Panopta/classic/src/",
    ],

It points to the app that I am working on currently. For other reasons that path is also part of my local PYTHONPATH so the apps I'm running can detect inner model requirements, but this is separate from all the library dependencies which live in a .virtualenvs folder at $HOME.

[jakebailey]

That's outside of the folder you have open, correct? We currently only treat code inside the workspace (that isn't an interpreter file, for users who put their venvs in the workspace) as user code, so if it's outside, it'll be treated as library code, which I believe will also mean that it will cause a reload when it changes.

[CMLL]

Actually it isn't. The current workspace I have open is /home/cllamach/Panopta/classic. One question, where are the Configuration search paths being determined from? Currently is showing this on LS start.

Which are the values I had set on my PYTHONPATH variable in the shell, but I removed those from there and they still show up on the LS.

[jakebailey]

Do you see your log spammed with the "reloading modules" message, or just the two times shown in the screenshot? Are you also setting PYTHONPATH somewhere?

As an aside, you can provide paths relative to the workspace root in that option, so you can specify "./src" and not have to hardcode the entire path on your filesystem.

[jakebailey]

Configuration search paths are those given by PYTHONPATH and extraPaths, merged together into a single list (by the python extension before we start). After that logging, we go and classify what's in the workspace and what isn't to see what's user code (and I've been planning on moving those prints somewhere later to get a real picture).

[CMLL]

They are not being spammed constantly, however the background analysis still doesn't complete, or at least the message does not go away.

If configuration search path work that way, then I don't really need to set the extraPaths variables, but I've been trying to have the LS start without those 2 paths to discard something there and no luck. I must have a PYTHONPATH set somewhere.

Update:
Was able to start the LS without 2 configuration search paths included and the analysis still seems to stuck somewhere. Memory usage seems to keep increasing steadily, didn't let it crash my machine but it def looked like that was going to be the end result.

[jakebailey]

With your setup (a src directory below the workspace root), you'll have to specify ./src as a search path at least somewhere, otherwise we won't be able to resolve your imports. If your output isn't being spammed with the reloading, then it's probably not a file watch or directory classification problem, but is something unrelated. Is this project somewhere we can test it?

[CMLL]

Sadly no, is a private project for the company I work for. Removing the configuration search paths doesn't to fix the issue so it leads me to believe it's not a inner workspace code issue, but it could be a virtualenv issue with one of the installed libraries, which is pretty big list.

Which makes sense, other projects with different virtualenvs don't seem affected by the same issue, at least in some preliminary testing I did.

[CMLL]

Seems like is indeed some thirdparty library in the virtualenv folder that's causing the issue, moving the workspace python interpreter to the system python lets the analysis complete fine.

[CMLL]

Although that has lead me to another weird situation with the completion. Since I changed the python environment for the workspace the installed modules don't show as available and don't show autocomplete options, as expected.

But standard library options and modules in src/ which is part of my configuration path and extraPaths show as available but the completions results show all the available results when the completion is triggered on a an empty space instead of the ones pertaining the target modules, which is weird.

[jakebailey]

Since that's different, I'd appreciate it if you could file that separately with a screenshot. Note that completion works by us providing a big list, which the editor then filters based on the context, so if you're completing on "nothing", then the list will look pretty big.

If possible, could you reload and copy out the trace log, and provide a pip freeze list of the packages that are involved?

[CMLL]

Sure thing. Here is a copy of a trace log that stopped at that place I mentioned above along with a list of all the thirdparty libraries present in the virtualenv. I'll create a new issue with the second issue I'm running into. Thanks for all your help, really appreciate it. 😄

pyls_log_files.zip

[CMLL]

One extra piece of information. To be thorough I tried analyzing the same workspace with version 0.2.96.0 of the LS, the one not currently in the beta channel, and the analysis doesn't get stuck in that version. Memory usage seems stable in there too.

[CMLL]

I think it has to do with this library https://pypi.org/project/docutils/ Was able to reproduce the freezing of the analysis by creating a virtualenv with only that library installed and putting that as the python interpreter. Also noticed that after the analysis stops, changing the environment doesn't seem to move it along, only changing the environment and reloading the window allowed me to get past the message of analyzing.

[jakebailey]

I've tried reproducing this in a Python 2.7 virtualenv with docutils installed, in a file which imports every docutils module listed in your logs, but no luck. The analysis completes and sits at about 100MB, not changing if the file is changed.

Are you importing docutils directly in your project? (If you aren't, then we shouldn't find it when you're using an environment with only that package, since it wouldn't be analyzed at all.)

[jakebailey]

If you aren't already, you may want to try the daily build:

"python.analysis.downloadChannel": "daily"

0.3.0+ contain some large memory-impacting changes, though I can't say if they'd help in this specific scenario.

[CMLL]

I've been working with daily all day and it has been pretty good regarding memory management, but I haven't tried reinstalling that module in the virtualenv that I'm. Weirdly, I search for that module in my project and I don't see it directly so it could be that the module is a requirement from another one that is also installed.

I'm going to run another tests see if the behaviour persists with that module. Thanks for trying to reproduce this.

[jakebailey]

The trouble is that if you were able to reproduce it in a virtualenv with just docutils, but your code doesn't actually import it, then docutils shouldn't have been able to be loaded at all, since there wouldn't even be any indirect imports. That's the confusing bit.

[MikhailArkhipov]

Since daily appears to be better, closing. @jakebailey is working on improving our search paths handling, he can suggest which issue in that space may be related.