AttributeError: 'NoneType' object has no attribute 'message_types_by_name'

[bowenatbear]

AttributeError: 'NoneType' object has no attribute 'message_types_by_name'

Environment details:

Ubuntu 16.04, Python 3.7

Description

I am trying to use the google-cloud-pubsub in my codes, but when only testing the imports for all the binaries, the following error shows:

from google.cloud import pubsub_v1
  File "/home/me/.cache/bazel/_bazel/xx/sandbox/processwrapper-sandbox/3/execroot/bazel-out/k8-fastbuild/bin/binary_smoke_test.runfiles/deps/pypi__google_cloud_pubsub/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/home/me/.cache/bazel/_bazel/xx/sandbox/processwrapper-sandbox/3/execroot/bazel-out/k8-fastbuild/bin/binary_smoke_test.runfiles/deps/pypi__google_cloud_pubsub/google/cloud/pubsub_v1/types.py", line 26, in <module>
    from google.api import http_pb2  # type: ignore
  File "/home/me/.cache/bazel/_bazel/xx/sandbox/processwrapper-sandbox/3/execroot/bazel-out/k8-fastbuild/bin/binary_smoke_test.runfiles/deps/pypi__googleapis_common_protos/google/api/http_pb2.py", line 36, in <module>
    _HTTP = DESCRIPTOR.message_types_by_name["Http"]
AttributeError: 'NoneType' object has no attribute 'message_types_by_name'

Is this a regression?

I don't think so.

Minimal Reproduction

The Build rule for the file:

load("@deps//:requirements.bzl", "requirement")
py_library(
    name="hello",
    srcs=["hello.py"],
    deps=[
      requirement("google-cloud-pubsub"),
    ],
)

py_binary(
    name="main",
    srcs=["main.py"],
    legacy_create_init=False,
    deps=[
     ":hello",
    ],
)

##############
# Binary Tests
##############

sh_test(
    name="binary_smoke_test",
    size="small",
    srcs=["binary_smoke_test.sh"],
    data=[
        ":main",
    ],
)

requiremens.txt as follow:

google-cloud-pubsub==2.13.4

and in the hello.py there are some imports:

from google.auth import jwt
from google.cloud import pubsub_v1

publisher_batch_settings = pubsub_v1.types.BatchSettings(
    max_messages=100,
    max_bytes=10240, 
    max_latency=1,
)

service_account_info = json.load(open(Some_Credentials_File_Path))
publisher_audience = 'https://pubsub.googleapis.com/google.pubsub.v1.Publisher'
credentials_pub = jwt.Credentials.from_service_account_info(
    service_account_info, audience=publisher_audience)

publisher = pubsub_v1.PublisherClient(
    batch_settings=publisher_batch_settings,
    credentials=credentials_pub)

Found a few relevant issues but their fixes suggestions are not working

E.g. bazel-contrib/rules_jvm_external#240 and #65.

While I added all the transitive dependencies manually & explicitly as follow but still got the same error message above.

google-cloud-pubsub==2.13.4
googleapis-common-protos==1.56.4
google-api-core==2.8.2
google-auth==2.10.0
google-cloud-core==2.3.2
google-cloud-storage==2.5.0
protobuf==3.20.1

E.g. #93

legacy_create_init=False

is added to all the py_pytest_test and py_binary and the issue is still there.

[groodt]

That requirements.txt file doesn’t appear to be complete. It’s missing the full list of transitive dependencies.

Can you provide a “locked” set of dependencies? An easy way to produce one is with pip-tools https://github.com/jazzband/pip-tools#example-usage-for-pip-compile or export one from poetry. You can use the compile_pip_requirements repository rule as well https://github.com/bazelbuild/rules_python/blob/main/docs/pip.md#compile_pip_requirements

[bowenatbear]

@groodt Hey, thanks for the suggestion and I have run pip-compile and get a full list of transitive dependencies as follow. I update my requirement.txt and BUILD file according to the following, but I still see the exact same error message AttributeError: 'NoneType' object has no attribute 'message_types_by_name'.

Actually I don't see why this error is related with transitive dependencies and may I know do you have any further suggestion?

#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
#
#    pip-compile requirements.txt
#
cachetools==5.2.0
    # via google-auth
certifi==2022.6.15
    # via requests
charset-normalizer==2.1.0
    # via requests
google-api-core[grpc]==2.8.2
    # via google-cloud-pubsub
google-auth==2.10.0
    # via google-api-core
google-cloud-pubsub==2.13.4
    # via -r r.txt
googleapis-common-protos[grpc]==1.56.4
    # via
    #   google-api-core
    #   grpc-google-iam-v1
    #   grpcio-status
grpc-google-iam-v1==0.12.4
    # via google-cloud-pubsub
grpcio==1.47.0
    # via
    #   google-api-core
    #   google-cloud-pubsub
    #   googleapis-common-protos
    #   grpc-google-iam-v1
    #   grpcio-status
grpcio-status==1.47.0
    # via
    #   google-api-core
    #   google-cloud-pubsub
idna==3.3
    # via requests
proto-plus==1.20.6
    # via google-cloud-pubsub
protobuf==3.20.1
    # via
    #   google-api-core
    #   google-cloud-pubsub
    #   googleapis-common-protos
    #   grpcio-status
    #   proto-plus
pyasn1==0.4.8
    # via
    #   pyasn1-modules
    #   rsa
pyasn1-modules==0.2.8
    # via google-auth
requests==2.28.1
    # via google-api-core
rsa==4.9
    # via google-auth
six==1.16.0
    # via
    #   google-auth
    #   grpcio
urllib3==1.26.11
    # via requests

[VinnyPutty]

It appears that the new method of creating a DESCRIPTOR in the generated py proto files, by adding a serialized file to descriptor_pool.Default() instead of using descriptor.FileDescriptor, doesn't work properly (I don't know if this is a Bazel-specific issue, though). When I downgrade the relevant packages to the latest release version before these changes (and their dependencies accordingly), I no longer see that error message (googleapis-common-protos==1.56.0 google-api-core==2.8.0 google-cloud-pubsub==2.13.1). However, I then see essentially the same error in the generated py proto of the grpc-google-iam-v1 package, which I'm not able to downgrade, because there is only one release of the preview package. This is the traceback:

Traceback (most recent call last):
File "/home/vinay/.../hello.py", line 2, in <module>
    from google.cloud import pubsub_v1
  File "/home/vinay/.cache/bazel/.../main.runfiles/deps/pypi__google_cloud_pubsub/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/home/vinay/.cache/bazel/.../main.runfiles/deps/pypi__google_cloud_pubsub/google/cloud/pubsub_v1/types.py", line 28, in <module>
    from google.iam.v1 import iam_policy_pb2  # type: ignore
  File "/home/vinay/.cache/bazel/.../main.runfiles/deps/pypi__grpc_google_iam_v1/google/iam/v1/iam_policy_pb2.py", line 35, in <module>
    from google.iam.v1 import options_pb2 as google_dot_iam_dot_v1_dot_options__pb2
  File "/home/vinay/.cache/bazel/.../main.runfiles/deps/pypi__grpc_google_iam_v1/google/iam/v1/options_pb2.py", line 36, in <module>
    _GETPOLICYOPTIONS = DESCRIPTOR.message_types_by_name["GetPolicyOptions"]
AttributeError: 'NoneType' object has no attribute 'message_types_by_name'

[bowenatbear]

@groodt I double checked the codes and just create a very simple codes sample with only the imports.
It sees the error starts to occur when I add a proto dependency into the BUILD:

py_binary(
    name="main",
    srcs=["main.py"],
    legacy_create_init=False,
    deps=[
        ":test_py_proto",
        requirement("google-cloud-pubsub"),
    ],
)

py_proto_library(
    name="test_py_proto",
    srcs=["test.proto"],
    deps=["@com_google_protobuf//:protobuf_python"],
)

while the test.proto is as simple as

syntax = "proto3";

import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";

message Metrics {
  google.protobuf.Timestamp time = 1;
  google.protobuf.Any message = 2;
}

Without the proto dependency, everything works fine and I am not sure if this is a Bazel issue or I should ask this question somewhere else?

[bowenatbear]

Update. Can close. This is fixed by making

deps=[
    requirement("google-cloud-pubsub"),
    ":test_py_proto",
],

instead of

deps=[
    ":test_py_proto",
    requirement("google-cloud-pubsub"),
],