Python import issue

[ysyyork]

I generate python file by this command (docker run --rm -v `pwd`:/defs namely/protoc-all:1.11 -d ../proto/ -o my_project/my_project -l python) to a folder and then I got the following file structure:

my_project
    my_project
        __init__.py
        a_pb2.py
        a_pb2_grpc.py
    __init__.py
    setup.py

In the a_pb2_grpc.py file, it import a_pb2 as a__pb2 which doesn't work cus in python if you want to import a module in the same folder, you need to do from . import a_pb2 as a__pb2. Not sure if this is a bug. By the way I'm using python 3.6.1

[echarrod]

You also have another option other than the one you stated:
Change the subfolder init.py file to include:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

[ido-namely]

Hi @ysyyork sorry for the late reply.
Is this still an issue for you?

[maxjakob]

I am also observing this issue. A workaround is to call the container with an explicit --grpc-out argument:

docker run --rm \
  -v `pwd`:/project \
  -w /project \
  namely/protoc-all:1.50 \
    -f project.proto \
    -l python \
    -o . \
    --grpc-out ./proto

Now all *.py files are inside proto/ and the imports work.

What I don't like is that __init__.py files are created recursively for each directory starting at ., so this workaround definitely has drawbacks.

@ido-namely Do you think this is something we can fix inside the container?

[ido-namely]

Hi @maxjakob ,
Thank you for your input. I'll try to take some time to better understand the issue.
I personally haven't used this project to generate protos for python.
if you can clarify in the meantime though - in your example, do project_pb2.py project_pb2_grpc.py still get generated in the same dir?
What is the import command that they are generated with?

[maxjakob]

@ido-namely After running

$ docker run --rm \
  -v `pwd`:/project \
  -w /project \
  namely/protoc-all:1.50 \
    -f project.proto \
    -l python \
    -o ./ \
    --grpc-out ./proto

I get the following files:

$ ls proto/
project_pb2_grpc.py  project_pb2_grpc.pyi  project_pb2.py  project_pb2.pyi  __init__.py

The import paths in these files are such that the issue reported by @ysyyork goes away, the imports work with this combination of parameters.

However, you now also get __init__.py files in the root and all subdirs recursively:

$ ls .
proto/  otherdir/  __init__.py
$ ls otherdir/
__init__.py

These excess __init__.py files are a problem in our git and linting workflows. Is there some way to turn off the generation of these? Or is there a combination of arguments that both fixes the issue described by @ysyyork and prevents these files from being generated everywhere?

[maxjakob]

For now, we are going to call protoc directly which also solves this issue:

EDIT: I spoke too soon. This does not solve the import issue.

[maxjakob]

I think it's actually this issue: grpc/grpc#29459, though I think this is even an issue without proto files importing other proto files. It's already issue with the generated gGRPC file importing the other generated Python file.

See also grpc/grpc#9575 (comment)

[maxjakob]

I figured out a solution now. If you have the directory structure

project/
  proto/
    project.proto

inside the project/ directory, you can run this command:

$ docker run --rm \
  -v `pwd`:/work \
  -w /work \
  --entrypoint="" \
  namely/protoc-all:1.50_0 \
    bash -c \
      "protoc --python_out=. --mypy_out=. --grpc_out=. --mypy_grpc_out=. --plugin=protoc-gen-grpc=/usr/local/bin/grpc_python_plugin ./proto/project.proto"

and add an __init__.py file yourself:

touch proto/__init__.py

to end up with these files:

project/
  proto/
    __init__.py
    project_pb2_grpc.py
    project_pb2.py
    project_pb2.pyi
    project.proto

The imports in project_pb2_grpc.py will have the form from proto import project_pb2 as ... and will therefore work as expected.

I know that the idea of entrypoint.sh is to abstract away the exact configuration for all the parameters. Maybe this gives somebody enough pointers to fix it there.