Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pylance latest release killing remote connections to notebooks #4983

Closed
ivirshup opened this issue Oct 19, 2023 · 20 comments
Closed

Pylance latest release killing remote connections to notebooks #4983

ivirshup opened this issue Oct 19, 2023 · 20 comments
Assignees
Labels
bug Something isn't working fixed in next version (main) A fix has been implemented and will appear in an upcoming version P0

Comments

@ivirshup
Copy link

ivirshup commented Oct 19, 2023

Most importantly, see multiple other reports of this issue over at:

Downgrading the pylance extension seems to fix that issue for most people.

Environment data

  • Language Server version:
    ms-python.python@2023.19.12921011
    ms-python.vscode-pylance@2023.10.31
  • OS and version: Ubuntu 22.04.3 LTS
  • Python version (& distribution if applicable, e.g. Anaconda): python 3.10.6, but mamba is also available

(more environment info related to this here: #4986

Code Snippet

NA

Repro Steps

  1. Open a notebook in a remote vscode window

Expected behavior

Able to work with that notebook

Actual behavior

Remote window disconnects and fails to reload.

Downgrading pylance to v2023.10.20 on remote fixes this.

I was able to downgrade, select a kernel, upgrade, and have that specific notebook work. Opening a new notebook would then kill the connection.

Logs

output
2023-10-19 15:49:32.128 [info] Experiment 'pythonPromptNewFormatterExt' is active
2023-10-19 15:49:32.128 [info] Experiment 'pythonPromptNewToolsExt' is active
2023-10-19 15:49:32.128 [info] Experiment 'pythonRecommendTensorboardExt' is active
2023-10-19 15:49:32.128 [info] Experiment 'pythonREPLSmartSend' is active
2023-10-19 15:49:32.128 [info] Experiment 'pythonTerminalEnvVarActivation' is active
2023-10-19 15:49:32.128 [info] Experiment 'pythonTestAdapter' is active
2023-10-19 15:49:32.128 [info] Default formatter is set to ms-python.black-formatter for workspace /mnt/workspace/notebooks
2023-10-19 15:49:32.128 [info] Test server listening.
2023-10-19 15:49:32.128 [info] > conda info --json
2023-10-19 15:49:32.128 [info] Python interpreter path: /usr/bin/python3
2023-10-19 15:49:32.281 [info] Found: /bin/python3 --> /bin/python3.10
2023-10-19 15:49:32.306 [info] Found: /bin/python3.10 --> /bin/python3.10
2023-10-19 15:49:32.817 [info] Found: /usr/bin/python3 --> /usr/bin/python3.10
2023-10-19 15:49:32.825 [info] Found: /usr/bin/python3.10 --> /usr/bin/python3.10
2023-10-19 15:49:32.905 [info] > /mnt/workspace/mambaforge/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.030 [info] > conda run -n kvikio-env --no-capture-output python /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/printEnvVariables.py
2023-10-19 15:49:33.030 [info] shell: bash
2023-10-19 15:49:33.030 [info] > conda run -n kvikio-env --no-capture-output python /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/printEnvVariables.py
2023-10-19 15:49:33.030 [info] shell: bash
2023-10-19 15:49:33.189 [info] > /mnt/workspace/mambaforge/envs/anndata-0.10.0rc1/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.227 [info] > /mnt/workspace/mambaforge/envs/anndata-dev/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.376 [info] > /mnt/workspace/mambaforge/envs/anndata-dev-3.10/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.444 [info] > /mnt/workspace/mambaforge/envs/anndata-dev-no-cupy/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.477 [info] > /mnt/workspace/mambaforge/envs/cupy-dev/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.527 [info] > /mnt/workspace/mambaforge/envs/don-niche/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.598 [info] > /mnt/workspace/mambaforge/envs/don-niche-cpu-v2/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.635 [info] > /mnt/workspace/mambaforge/envs/don-niche-gpu/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.737 [info] > /mnt/workspace/mambaforge/envs/jax/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.810 [info] > /mnt/workspace/mambaforge/envs/kvikio-env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.846 [info] > /mnt/workspace/mambaforge/envs/kvikio-nightly-env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.885 [info] > /mnt/workspace/mambaforge/envs/kvikio_env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.921 [info] > /mnt/workspace/mambaforge/envs/multithreaded-sparse/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.957 [info] > /mnt/workspace/mambaforge/envs/ncem-env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:33.993 [info] > /mnt/workspace/mambaforge/envs/ncem-gpu-env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.033 [info] > /mnt/workspace/mambaforge/envs/pypi/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.072 [info] > /mnt/workspace/mambaforge/envs/rapids-single-cell-anndata/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.115 [info] > /mnt/workspace/mambaforge/envs/scanpy/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.155 [info] > /mnt/workspace/mambaforge/envs/scipy-dev/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.210 [info] > /mnt/workspace/mambaforge/envs/sklearn-env/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.251 [info] > /mnt/workspace/mambaforge/envs/test-release/bin/python -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.340 [info] > /bin/python3 -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py
2023-10-19 15:49:34.363 [info] Starting Pylance language server.
2023-10-19 15:49:34.430 [info] > /usr/bin/python3 -I /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/get_output_via_markers.py /mnt/workspace/isaac/.vscode-server-insiders/extensions/ms-python.python-2023.19.12921011/pythonFiles/interpreterInfo.py

@github-actions github-actions bot added the needs repro Issue has not been reproduced yet label Oct 19, 2023
@ndgayan
Copy link

ndgayan commented Oct 19, 2023

I'm seeing the same issue. Version v2023.10.20 works.

@debonte debonte assigned rchiodo and unassigned heejaechang Oct 19, 2023
@debonte debonte added bug Something isn't working P0 and removed needs repro Issue has not been reproduced yet labels Oct 19, 2023
@debonte
Copy link
Contributor

debonte commented Oct 19, 2023

Thanks. We are able to repro this and are investigating.

@rchiodo
Copy link
Contributor

rchiodo commented Oct 19, 2023

I believe the culprit is this commit:
https://github.com/microsoft/pyrx/commit/18e1ffae9c63f7ece1f5737ea36e72cbfd83db91

Not sure how that's causing a crash, but I cannot repro the crash with the commit before this and it repros with this commit.

@rchiodo
Copy link
Contributor

rchiodo commented Oct 19, 2023

Okay now I'm not sure. The repro isn't very consistent for me.

@deepak1556, is there a way to get the callstack of the crashing node extension host? Hopefully that might tell us what code is causing the crash.

@rchiodo
Copy link
Contributor

rchiodo commented Oct 19, 2023

It seems I also cannot reproduce the crash with 1.81 of VS code (and 2023.10.30 for Pylance). 1.82 is where VS code moved to Electron 25/Node 18.5.

My guess is that's the cause of the problem. Javascript code doesn't normally crash Node, so this might be a bug in Electron/Node.

@rchiodo
Copy link
Contributor

rchiodo commented Oct 19, 2023

Oh it seems Deepak is OOF. @rzhao271 do you have any tips on collecting a callstack for when the extension host goes down? I can't reproduce this in 1.81 of VS code, so it might be a Node 18.5 specific problem.

@deepak1556
Copy link

@rchiodo if you are able to repro with local extension host then https://github.com/microsoft/vscode/wiki/Native-Crash-Issues#creating-a-crash-report would be the way to go. If you want to collect crash dumps for remote extension host then set ulimit -c unlimited to get core files https://manpages.ubuntu.com/manpages/jammy/en/man5/core.5.html. You can forward the core files to Deepak.Mohan@microsoft.com.

What are the steps to repro, I can give it a try and get you the callstacks.

@rchiodo
Copy link
Contributor

rchiodo commented Oct 20, 2023

To reproduce do what the original poster said.

VS code 1.83
Python extension stable
Jupyter extension stable
Pylance extension 2023.10.30

Open a notebook in WSL or SSH. It should crash the extension host pretty easily.

@luabud
Copy link
Member

luabud commented Oct 20, 2023

We believe we have figured out a workaround, and we'll do our best to ship a hot-fix release with the changes tomorrow morning (PST). Until then, you can workaround this issue by downgrading Pylance to the previous stable version (2023.10.20):

Click on the gear icon next to the Pylance extension in the extension view and select "install another version"

image

@debonte debonte added the fixed in next version (main) A fix has been implemented and will appear in an upcoming version label Oct 20, 2023
@deepak1556
Copy link

@rchiodo I was able to repro the issue, however there is no crash involved with remote agent process as well as extension host process (core dumps were not generated). I captured strace output for both these processes and turns out the extension host was killed with SIGTERM and the remote agent process received SIGHUP from the kernel causing the default behavior of Node.js to exit these processes. Just before receiving these signals, both the logs showed there was attempt to send SIGTERM to a pid and I expect that to be the python process (can validate with audit logs if needed). To validate this exit case, I did the following steps

  • Launch VSCode from wsl
  • Get the pid of remote agent process
  • kill -s SIGUSR1 <pid-remote-agent>
  • You should now be able to connect an inspector to the remote agent process
  • In the devtools console, add
process.on('SIGHUP', () => { console.log('control terminal destroyed'); })
  • Open a notebook
  • You should see the log printed and also the remote agent no longer exits since we prevented the default behavior (ofcouse extension host would still exit in this case)

Now the question is why does exiting the python process result in a SIGTERM for the extension host, based on your patch an uncaught exception seems to be the trigger. AFAIK, failure in the call should only emit an error event and there were no API changes between v16 -> v18. So it would be good to understand what kind of error is thrown when the kill call failed, if you are able to capture the error that would be helpful or if you can isolate a repro outside the extension for us to investigate further. Thanks!

@EdGaere
Copy link

EdGaere commented Oct 20, 2023

Solution from @luabud works for me

@dx44
Copy link

dx44 commented Oct 20, 2023

Solution from @luabud work for me BUT with a pretty trick otherwise as soon as pylance is up, my vscode server crash.

What I did :

  1. Remove ~/.vscode-server/extensions/ms-python.vscode-pylance-2023.10.30
  2. Disable your local pylance extension
  3. Install pylance in your remote wsl
  4. On your remote pylance extension, use 'Install Another Version' and select 2023.10.20
  5. Enable your remote pylance extension

@rchiodo
Copy link
Contributor

rchiodo commented Oct 20, 2023

@deepak1556 thanks for the analysis. That gives us greater confidence that the fix is the correct one. I'm not sure how the failure to kill a process in a subprocess is cascading up to the extension host though. And why notebooks have to be involved.

I'm going to try creating a simpler repro and I'll log a bug on VS code with it if I can get the same behavior.

@heejaechang
Copy link
Contributor

heejaechang commented Oct 20, 2023

This issue has been fixed in release version 2023.10.40 and prerelease version 2023.10.41, which we've just released. You can find the changelog here: CHANGELOG.md

@debonte
Copy link
Contributor

debonte commented Oct 20, 2023

And why notebooks have to be involved.

@rchiodo, assuming that #4982 is a dupe, this user says that they're seeing it without using notebooks -- #4982 (comment)

@luabud
Copy link
Member

luabud commented Oct 20, 2023

If the crash happens before the Pylance extension gets a chance to be automatically updated to 2023.10.40 (pr 2023.10.41), there are two workarounds:

  • Reload the window and then try to close the open editor as soon as possible so Pylance v2023.20.30/31 doesn't activate.
  • @dx44's comment above (i.e. deleting ~/.vscode-server/extensions/ms-python.vscode-pylance-2023.10.30)

@debonte
Copy link
Contributor

debonte commented Oct 20, 2023

there are two workarounds:

On Windows, I used the "New Window" option to create an instance of VS Code without loading any folders/documents. I'm assuming this is also available in some way on Linux/Mac.

image

@rchiodo
Copy link
Contributor

rchiodo commented Oct 20, 2023

Unfortunately, I can't seem to get a simple repro to cause the same problem. The child is just killed. It doesn't cause any ripple effects.

My attempt at a simpler repro was here:
https://github.com/rchiodo/test-childprocess-crash.git

@lgaitan
Copy link

lgaitan commented Oct 20, 2023

This issue has been fixed in prerelease version 2023.10.41, which we've just released. You can find the changelog here: CHANGELOG.md

I was facing this issue, and can confirm that upgrading to the 2023.10.41 pre-release version solves it.

Thank you!

@luabud
Copy link
Member

luabud commented Oct 20, 2023

On Windows, I used the "New Window" option to create an instance of VS Code without loading any folders/documents. I'm assuming this is also available in some way on Linux/Mac.

image

Yes, good point! And here's how to do this same thing when using Remote-SSH: microsoft/vscode#195617 (comment)

  • Open an empty Remote SSH window (Remote indicator -> Connect to Host ... -> Pick the remote)
  • Open extensions view and find Pylance, upgrade to latest v2023.10.40

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fixed in next version (main) A fix has been implemented and will appear in an upcoming version P0
Projects
None yet
Development

No branches or pull requests

10 participants