Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python file with more than one entry breaks module resolution #3727

Closed
jaredcm opened this issue Dec 17, 2018 · 6 comments
Closed

Python file with more than one entry breaks module resolution #3727

jaredcm opened this issue Dec 17, 2018 · 6 comments
Labels
area-intellisense LSP-related functionality: auto-complete, docstrings, navigation, refactoring, etc. bug Issue identified by VS Code Team member as probable bug

Comments

@jaredcm
Copy link

jaredcm commented Dec 17, 2018

Environment data

  • VS Code version: Version 1.30.0 (1.30.0)
  • Extension version (available under the Extensions sidebar): 2018.12.1
  • OS and version: Mac OS 10.14.1
  • Python version (& distribution if applicable, e.g. Anaconda): 3.7 cPython
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pyenv
  • Relevant/affected Python packages and their versions: None

Expected behaviour

Adding custom locations to .pth file should be resolved by the Python intellisense.

Actual behaviour

I have a custom package that is in a subdirectory of my project. I want Python to resolve it as if it were an installed module.

The project is setup as such.
os_path
-|+- project_dir
--|+- bin
---|+- test.py
--|+- lib
---|+- custom_module
----|+- __init__.py

In the python 3.7 site-packages directory I have a .pth file with something like the following contents.
{os_dir}/{project_dir}
{os_dir}/{project_dir}/lib

In Python this resolves without issue and so I can do.

Python 3.7.0 (default, Aug 21 2018, 11:16:24) 
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom_module
>>> custom_module.test()
Hello

If I create a test.py file in the project's bin directory like so:

"""This is a test."""
import custom_module
custom_module.test()

In VSCode I get unresolved import:

{
"resource": "/{os_path}/{project_dir}/bin/test.py",
"owner": "generated_diagnostic_collection_name#0",
"code": "unresolved-import",
"severity": 4,
"message": "unresolved import 'custom_module'",
"source": "Python",
"startLineNumber": 2,
"startColumn": 8,
"endLineNumber": 2,
"endColumn": 21
}

If I switch the import to lib.{custom_module} then the module resolution works. I'm guessing this is because VSCode adds the workspace to the PYTHONPATH.

Steps to reproduce:

  1. Setup a project with a module that is +1 level deep in the project. i.e. lib/custom_module/__init__.py
  2. Add a test function for resolution to __init__.py
"""This is a test."""


def test() -> None:
    """This is a test function."""
    print("Hello")

  1. In the Python site-packages create a .pth file and add the full OS path to the lib directory.
  2. Try to import the custom_module in a test file (see my test file above).

Logs

Output for Python in the Output panel (ViewOutput, change the drop-down the upper-right of the Output panel to Python)

##########Linting Output - pydocstyle##########
##########Linting Output - mypy##########
##########Linting Output - prospector##########
##########Linting Output - prospector##########
##########Linting Output - pylama##########

Log (Window)

[2018-12-17 11:35:21.724] [renderer1] [error] Tree element not found: [object Object]: Error: Tree element not found: [object Object]
    at e.getElementLocation (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:214:266)
    at e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:213:757)
    at i.e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:723)
    at i.e.isExpanded (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:786)
    at t.revealMarkersForCurrentActiveEditor (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:827)
    at t.autoReveal (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:647)
    at t.onActiveEditorChanged (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4707:826)
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:532)
    at t.doEmitActiveEditorChangeEvent (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:942)
    at t.handleActiveEditorChange (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:718)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4721:55
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:512)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:5170:861
    at Object.g [as _notify] (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:167:807)
    at Object.enter (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:171:85)
    at n.Class.derive._creator._run (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:932)
    at n.Class.derive._creator._completed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:373)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

log.ts:169   ERR Tree element not found: [object Object]: Error: Tree element not found: [object Object]
    at e.getElementLocation (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:214:266)
    at e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:213:757)
    at i.e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:723)
    at i.e.isExpanded (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:786)
    at t.revealMarkersForCurrentActiveEditor (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:827)
    at t.autoReveal (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:647)
    at t.onActiveEditorChanged (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4707:826)
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:532)
    at t.doEmitActiveEditorChangeEvent (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:942)
    at t.handleActiveEditorChange (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:718)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4721:55
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:512)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:5170:861
    at Object.g [as _notify] (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:167:807)
    at Object.enter (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:171:85)
    at n.Class.derive._creator._run (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:932)
    at n.Class.derive._creator._completed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:373)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
console.ts:134 [Extension Host] Python Extension: Failed to parse Prospector output SyntaxError: Unexpected end of JSON input
	at JSON.parse (<anonymous>)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585133)
	at Generator.next (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584675)
	at new Promise (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584452)
	at module.exports.t.Prospector.parseMessages (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585083)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:77469)
	at Generator.next (<anonymous>)
	at s (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:75269)
	at <anonymous>
t.log @ console.ts:134
t._logExtensionHostMessage @ extensionHost.ts:453
(anonymous) @ extensionHost.ts:244
emitTwo @ events.js:126
emit @ events.js:214
emit @ internal/child_process.js:772
_combinedTickCallback @ internal/process/next_tick.js:141
_tickCallback @ internal/process/next_tick.js:180
@jaredcm
Copy link
Author

jaredcm commented Dec 17, 2018

I tried to create a brand new project and recreate this. The custom module is resolved without issue. When I try and import modules from my initial project I still cannot resolve them. In both projects I get an error message in the workload console like:

/Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:303 [Extension Host] Python Extension: Failed to parse Prospector output SyntaxError: Unexpected end of JSON input
	at JSON.parse (<anonymous>)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585133)
	at Generator.next (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584675)
	at new Promise (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584452)
	at module.exports.t.Prospector.parseMessages (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585083)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:77469)
	at Generator.next (<anonymous>)
	at s (/Users/jared/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:75269)
	at <anonymous>
t.log @ /Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:303

@jaredcm jaredcm changed the title Python linter not able to resolve paths specified in pth file. Corrupted mypy cache prevents vscode from finding modules. Dec 17, 2018
@jaredcm
Copy link
Author

jaredcm commented Dec 17, 2018

Found the issue. Looks to be a corrupted .mypy_cache. I removed the cache directory from my initial project and the imports started working in both projects.

@jaredcm
Copy link
Author

jaredcm commented Dec 18, 2018

Looks like I was wrong. I restarted my computer and now my custom module cannot be resolved again. I remember I had been messing about with my pth file. Normally I have 4 paths in it which are all loaded into Python without issue. While filling out this bug report, I had removed all but 1, which was when VS Code started working. I had to add the other 3 paths back again later for my scripts to function properly. Which is why it stopped working and I didn't notice until VSCode was restarted.

project.pth =
os_path/project
os_path/project/lib
os_path/project/tests
os_path/project/unit_tests

I've done some testing:

  1. I added a project_lib.pth file which has 1 entry pointing to my lib directory and removed the entry from my project.pth file. The custom module is not resolvable.
  2. I change the name of project.pth to project.orig and restart VSCode. The module is resolvable.
  3. I change the name of project.orig back to project.pth and restart VSCode. The module is unresolvable again.
  4. I rename project.pth again, create a new file named project_root.pth, add the root directory for my project and restart VSCode. The module is resolvable as are modules in my project's root directory.

It looks like VSCode gets confused when pth files have more than one entry.

@jaredcm jaredcm changed the title Corrupted mypy cache prevents vscode from finding modules. PTh file with more than one entry break module resolution Dec 18, 2018
@jaredcm
Copy link
Author

jaredcm commented Dec 18, 2018

New issue:
In custom_module I have many sub_modules and they have other sub_modules. Within each of the sub_modules I use relative imports to load their child classes. The sub_module directory has a __init__,py with the import statement of "from .rest import Rest". This should resolve to a child directory named 'rest' that has a __init__.py file where the Rest class is defined. The full resolution from the root module to the imported class is 'custom_module.sub_module.rest.Rest'

The error message is:
{
"resource": "~/project/lib/custom_module/sub_module/__init__.py",
"owner": "generated_diagnostic_collection_name#0",
"code": "unresolved-import",
"severity": 4,
"message": "unresolved import 'lib.custom_modulerest'",
"source": "Python",
"startLineNumber": 6,
"startColumn": 6,
"endLineNumber": 6,
"endColumn": 11
}

Also notice that the message is not showing the correct module import path. The lib directory is not a module as it does not have a __init__.py file. If I change the path of the relative import to a full module import of 'from custom_module.sub_module.rest import Rest', then the error goes away. The path in the message is also missing the current module. It's almost like it the path it is looking for is shifted too far up the tree. Instead of looking in custom_module/sub_module/rest, I think it is looking for lib/custom_module/rest which doesn't exist.

Developer Console output

  ERR Tree element not found: [object Object]: Error: Tree element not found: [object Object]
    at e.getElementLocation (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:214:266)
    at e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:213:757)
    at i.e.isCollapsed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:723)
    at i.e.isExpanded (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:962:786)
    at t.revealMarkersForCurrentActiveEditor (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:827)
    at t.autoReveal (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4710:647)
    at t.onActiveEditorChanged (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4707:826)
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:532)
    at t.doEmitActiveEditorChangeEvent (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:942)
    at t.handleActiveEditorChange (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4720:718)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:4721:55
    at e.fire (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:189:512)
    at file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:5170:861
    at Object.g [as _notify] (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:167:807)
    at Object.enter (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:171:85)
    at n.Class.derive._creator._run (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:932)
    at n.Class.derive._creator._completed (file:///Applications/Design/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.main.js:172:373)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
console.ts:134 [Extension Host] Python Extension: Failed to parse Prospector output SyntaxError: Unexpected end of JSON input
	at JSON.parse (<anonymous>)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585133)
	at Generator.next (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584675)
	at new Promise (<anonymous>)
	at module.exports.r (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:584452)
	at module.exports.t.Prospector.parseMessages (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:83:585083)
	at module.exports.t.Prospector.<anonymous> (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:77469)
	at Generator.next (<anonymous>)
	at s (/~/.vscode/extensions/ms-python.python-2018.12.1/out/client/extension.js:1:75269)
	at <anonymous>
t.log @ console.ts:134
t._logExtensionHostMessage @ extensionHost.ts:453
(anonymous) @ extensionHost.ts:244
emitTwo @ events.js:126
emit @ events.js:214
emit @ internal/child_process.js:772
_combinedTickCallback @ internal/process/next_tick.js:141
_tickCallback @ internal/process/next_tick.js:180

@d3r3kk d3r3kk changed the title PTh file with more than one entry break module resolution Python file with more than one entry breaks module resolution Jan 17, 2019
@d3r3kk
Copy link

d3r3kk commented Jan 17, 2019

Thanks for bringing this up @jaredcm, an interesting problem!

I didn't know about .pth files until this issue popped up so thank you for that.

I can repro this issue with the latest stable VS Code (1.30.2) + latest extension (2018.12.1), using the Jedi language server ("python.jediEnabled": true").

Workaround: The Microsoft Python Language Server (as of version 0.1.72.0, or greater) seems to handle this case (see below). Have you attempted to use this instead of Jedi? See this issue for details on how to enable/update the Microsoft Python Language Server.

Adding more than one entry.

Actually, on second look I have found that this is not handled properly by the Microsoft Python Language Server as I'd originally stated (at least version 0.1.72.0 doesn't) as I'd left out the multiple lines in the .pth part after I got things up and running.

Indeed, if you add a .pth file to your site-packages with > 1 path, the issue does repro.

Here's the sample workspace:

3727_module_resolution.zip

Here's the sample .pth file contents that I placed in the site-packages. I created a .venv in my workspace and copied this file to ${workspaceFolder}/.venv/Lib/site-packages/test_3727.pth.

C:\\path\\to\\workspaceFolder\\3727_module_resolution
C:\\path\\to\\workspaceFolder\\3727_module_resolution\\lib

@d3r3kk d3r3kk added bug Issue identified by VS Code Team member as probable bug area-intellisense LSP-related functionality: auto-complete, docstrings, navigation, refactoring, etc. needs PR and removed triage labels Jan 17, 2019
@d3r3kk d3r3kk removed their assignment Jan 17, 2019
@luabud
Copy link
Member

luabud commented Sep 11, 2019

Closing in favour of microsoft/python-language-server#537. If you experience this with jedi, please file an issue on the jedi repo

@luabud luabud closed this as completed Sep 11, 2019
@ghost ghost removed the needs upstream fix label Sep 11, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Sep 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-intellisense LSP-related functionality: auto-complete, docstrings, navigation, refactoring, etc. bug Issue identified by VS Code Team member as probable bug
Projects
None yet
Development

No branches or pull requests

4 participants