Pylance with a multi-root workspace

[heejaechang]

Discussed in #6303

^{Originally posted by splint33 August 27, 2024}
Hi everyone, I'm kinda new to Pylance, sorry if my question is dumb.
I have a VSCode workspace, containing multiple folders. Each folder is a GIT repository.
My code-workspace is looking like that at the moment:

{
    "folders": [
        {
            "name": "root",
            "path": "."
        },
        {
            "name": "lib_1",
            "path": "C:\\AutomatedTests\\lib_1"
        },
        {
            "name": "lib_2",
            "path": "C:\\AutomatedTests\\lib_2"
        },
        {
            "name": "lib_3",
            "path": "C:\\AutomatedTests\\lib_3"
        }
    ],
    "settings": {
        "python.analysis.extraPaths": [
            "${workspaceFolder:root}/scripts",
            "${workspaceFolder:lib_1}",
            "${workspaceFolder:lib_2}",
            "${workspaceFolder:lib_3}"
        ]
}

When I edit lib1, I must restart the language server in order to be able to see changes in the root folder.

I understand that Pylance is indexing each folder individually at the start of the language server, and seems to use a cached version of the index.

Changes made into lib1 are reflected into lib1 immediately, but not on other workspace folders.
Is there a way to keep this structure and make it work with pylance ?
Thanks

[heejaechang]

we assumed extraPath is part of workspace but it could be full path to outside of workspace. in that case, we need to set up file watcher for it.

[heejaechang]

I am finding multiple issues here. tackling one by one.

1. when extra path is used to see repro each other in mono repo (multi root workspace) set up, we are not propagating text changes properly to all workspaces.
2. when extra path is point to external path outside of any workspaces, we are not doing file watches on those folders.
3. persisting indices for packages outside of builtin stubs has a race for multi root workspace and sometime fail to persist affecting perf
4. there are room for improvement on indices sharing on multi root workspace (we already do, but certain modules we re-index them just to drop them since those modules don't have any symbols in them)

[erictraut]

Item 2 was an intentional design choice, but maybe it's worth revisiting. The extraPaths setting is intended to be used for project-relative paths only. It wouldn't make sense to use absolute paths here because such paths would be invalid for anyone else working on a project. That said, a project-relative path could point to a location above the workspace's root directory, so one could argue that there are legitimate use cases not covered by existing file watchers. Do we have evidence that this use case is something users want pylance to support?

[heejaechang]

We (Pylance) have already been supporting absolute paths (or relative paths outside of a root) in extra paths for multi-root workspace users. They can also use ${workspaceFolder:xxx} to point to other roots in the settings.json.

There are multiple ways users have been doing this, such as legacy editable installs, PYTHONPATH, etc., and extra paths are also one of the methods users have been utilizing.

They could also place all repositories under the same root and use other configurations (extra paths, execution environments, include/exclude, etc.) to set it up under one workspace.

However, we don't force them to use any particular method since each has its own pros and cons.

for example, if a user is the only one using it, multi root workspace that points to each repo with extra path with${workspaceFolder:xxx} in settings.json is the simplest way to do it.

[StellaHuang95]

This issue has been fixed in prerelease version 2024.9.100, which we've just released. You can find the changelog here: CHANGELOG.md