Importing gz.transport module throws an error message. #18

kkishore9891 · 2023-03-08T15:41:23Z

Hello! First here is the context:

I am trying to work on a project to control a camera gimbal for a drone using python in gazebo garden. I initially tried using the ros2_gz bridge to control the joints but that slows down the real time factor of the simulation by a large amount thus I am trying to use a python binding to directly interface with gz and hence I tried to install this package.

Now to get to the problem:
I have a binary installation of gz-garden and I followed the instructions given in the README.md and when I tried to import the transport submodule of gz I get the following error message:

>>> import gz.transport Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /home/kkishore9891/gz_ws/src/gz-python/build/python/gz/transport.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN17pybind11_protobuf21PyProtoDescriptorNameB5cxx11EN8pybind116handleE

I am not sure if I did something wrong by I followed the instructions several times and this issue wouldn't go away. Can you help fix this issue?

The text was updated successfully, but these errors were encountered:

srmainwaring · 2023-03-08T15:46:00Z

Hi @kkishore9891 - this similar to the issue discussed in #17. The problem is the version of the protobuf compiler used in the binary build is not compatible with the version required by pybind11_protobuf. At the moment you'll need a source build of gazebo garden to get everything consistent.

kkishore9891 · 2023-03-08T16:21:46Z

Got it. I will try that. I will close the issue myself once I get it working. Thank you.

srmainwaring · 2023-03-08T16:26:33Z

No prob. I should update the README stating only source builds are supported.

kkishore9891 · 2023-03-08T18:52:50Z

Hello @srmainwaring . I tried building gz-garden from source and I still got the same error:

>>> from gz.transport import AdvertiseMessageOptions Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /home/kkishore9891/workspace/src/gz-python/build/python/gz/transport.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN17pybind11_protobuf21PyProtoDescriptorNameB5cxx11EN8pybind116handleE

srmainwaring · 2023-03-08T19:37:50Z

What version of protoc do you have installed?

protoc --version

kkishore9891 · 2023-03-08T19:38:26Z

libprotoc 3.12.4

srmainwaring · 2023-03-08T19:43:46Z

Ok - it's expecting libprotoc 3.21.5:

gz-python/cmake/dependencies/CMakeLists.txt

Lines 1 to 14 in c38289b

    
           include(FetchContent) 
        
           #============================================================================ 
        
           # Declare all dependencies first 
        
           if(BUILD_protobuf) 
        
             set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "") 
        
             FetchContent_Declare( 
        
               protobuf 
        
               GIT_REPOSITORY "https://github.com/protocolbuffers/protobuf.git" 
        
               GIT_TAG "v21.5" 
        
               GIT_SUBMODULES "" 
        
             ) 
        
           endif()

If you read through #17 you'll see a section on updating cmake and protoc with instructions. Once these are updated you can rebuild gz-msgs and gz-transport (use the --cmake-clean-cache flag to force updating the cmake build)

kkishore9891 · 2023-03-08T20:29:35Z

Hello! I have updated cmake and protoc.
cmake --version returns:
`cmake version 3.18.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
whileprotoc --versionreturns:libprotoc 3.21.5`

Trying to build gz-python by running cmake .. and make commands the build directory throws the following error:

error_msg.txt

I am guessing maybe I should build garden once again from scratch using the new cmake and protoc versions?

srmainwaring · 2023-03-08T20:53:52Z

Yes - either a clean build, or remove the gz-msgs and gs-transport subfolder from ./build and ./install and rebuild. Looks like the original proto headers were not regenerated.

kkishore9891 · 2023-03-08T21:09:08Z

I couldn't even build gz-garden from scratch with the new cmake and protoc anymore. I am getting this error:

`--- stderr: gz-math7
CMake Warning at /home/kkishore9891/workspace/install/share/cmake/gz-cmake3/cmake3/GzConfigureBuild.cmake:69 (message):
CONFIGURATION WARNINGS:
-- Python is missing: Python interfaces are disabled.
Call Stack (most recent call first):
CMakeLists.txt:124 (gz_configure_build)

Finished <<< gz-math7 [28.0s]
Starting >>> gz-common5
Starting >>> gz-msgs9
Starting >>> sdformat13
--- stderr: sdformat13
CMake Error at /usr/local/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES
Development Development.Module Development.Embed)
Call Stack (most recent call first):
/usr/local/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
/usr/local/share/cmake-3.18/Modules/FindPython/Support.cmake:2971 (find_package_handle_standard_args)
/usr/local/share/cmake-3.18/Modules/FindPython3.cmake:389 (include)
CMakeLists.txt:148 (find_package)

Failed <<< sdformat13 [2.47s, exited with code 1]
Aborted <<< gz-msgs9 [3.62s]
Aborted <<< gz-common5 [4.08s]

Summary: 4 packages finished [37.0s]
1 package failed: sdformat13
2 packages aborted: gz-common5 gz-msgs9
2 packages had stderr output: gz-math7 sdformat13
9 packages not processed
`

srmainwaring · 2023-03-08T21:18:21Z

Odd - unsure why the build is missing Python headers.

What OS and version are you running?
Have you run sudo apt update and sudo apt upgrade recently to bring the machine up to date (assuming Ubunutu)?
Python install location and version

which python
python --version

which python3
python3 --version

kkishore9891 · 2023-03-08T21:25:12Z

1)I am using Ubuntu 22.04
2) I have run sudo apt update several times. I ran sudo apt upgrade just now (but the result is the same).
3) both python --version as well as python3 --version commands output 3.10.6

srmainwaring · 2023-03-09T10:39:03Z

@kkishore9891 I've added a GHW to build on Ubuntu 22.04 (#13). There is also an update in the dependency on pybind11_protobuf as the cmake PR was merged yesterday (#19).

The CI build is clean against a binary installation of Gazebo Garden, so I am not sure what is different with your environment that is preventing the project from building correctly. The GHW yaml file under .github/workflows and the build log under the project's Actions folder may help you to determine what is different on your system.

kkishore9891 · 2023-03-09T11:48:46Z

So I am trying to build gz-garden from scratch after updating the cmake and the protoc versions and I am getting error messages now especially while building the gz-msgs package. Here is the log file:
stdout_stderr.log

These builds used to work properly with the earlier cmake and protoc versions. So I am suspecting that the update had something to do with it.

kkishore9891 · 2023-03-09T12:01:46Z

I am trying to build the ros_gz_sim package and even that throws errors. It used to work perfectly fine before updating protoc.

srmainwaring · 2023-03-09T13:07:03Z

In which case you should roll back the changes to cmake and protoc. You may need to delete some of the files manually as the cmake builds do not have uninstallers (running make install will show you where all the files are placed). Then re-install cmake and protoc with apt.

Apologies that this is causing you difficulties - I'm going through the install on an Ubuntu 22.04 VM to see if I can replicate the first issue you saw. My initial assumption that this was the same issues as seen on Ubuntu 20.04 does not seem to be correct.

System

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

$ python --version
Python 3.10.6

$ python -m platform
Linux-5.19.0-35-generic-x86_64-with-glibc2.35

$ cmake --version
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Build

CMake configure - OK
Build - OK

C++ example

$ ./build/publisher 
Publishing hello on topic [/foo]
Publishing hello on topic [/foo]
Publishing hello on topic [/foo]
Publishing hello on topic [/foo]

C++ example - OK

Python example

$ ./python/publisher.py 
Traceback (most recent call last):
  File "/home/rhys/Code/robotics/gz_python_ws/src/gz-python/./python/publisher.py", line 21, in <module>
    from gz.transport import AdvertiseMessageOptions
ImportError: /home/rhys/Code/robotics/gz_python_ws/src/gz-python/build/python/gz/transport.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN17pybind11_protobuf21PyProtoDescriptorNameB5cxx11EN8pybind116handleE

Ok - so I can replicate the problem. Let me figure out why this is not working.

Update

It's a runtime dynamic library resolution problem - I'll need to put in a proper fix to install the libs correctly when using fetch content to build, but a quick workaround is to edit ./gz-python/build/_deps/pybind11_protobuf-src/CMakeLists.txt and replace SHARED with STATIC in:

# pybind11_native_proto_caster shared library
add_library(pybind11_native_proto_caster SHARED

with

# pybind11_native_proto_caster shared library
add_library(pybind11_native_proto_caster STATIC

kkishore9891 · 2023-03-09T14:53:27Z

Hello! Thank you so much. I was just about to comment that I installed the latest versions of protoc and cmake with working build and I still get the same error anyways.

Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /home/kkishore9891/workspace/src/gz-python/build/python/gz/transport.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN17pybind11_protobuf21PyProtoDescriptorNameB5cxx11EN8pybind116handleE
Thank you for your help. I am sure you'll figure out what's wrong.

srmainwaring · 2023-03-09T14:55:22Z

I am sure you'll figure out what's wrong

Yep - library path. Strange as this has been working so not sure what's changed re. dependent libraries. Also need to add some Python tests to the CI as this is a runtime rather than build issue.

A good check that things are working well is to run gz sim shapes.sdf then

$ ./python/gz_topic_echo.py -t world/shapes/clock
Subscribing to topic [world/shapes/clock]
system {
  sec: 1678374293
  nsec: 48634280
}
real {
  sec: 121
  nsec: 188231732
...

kkishore9891 · 2023-03-09T15:14:11Z

Exactly! It works now! Thank you! Also could you please guide me on how to call gazebo services using this package?

srmainwaring · 2023-03-09T15:28:09Z

For service requests take a look at https://github.com/srmainwaring/gz-python/blob/srmainwaring/service-request/python/pid_service.py

There are two examples: the non-blocking call-back example requires an upstream release of gz-transport (gazebosim/gz-transport#386), the non-blocking version works with the version you are using.

tblaha · 2023-04-01T08:26:18Z

Hello! Immense thank you for writing these bindings and supporting them!

I ran into this issue as well and (since I'm relatively new to cmake) I found that re-running cmake .. overwrites the modified /gz-python/build/_deps/pybind11_protobuf-src/CMakeLists.txt again and has no effect. What worked is modifying that file and then simply re-running make. Is this the way, or am I missing something?

Regards,
Till

srmainwaring · 2023-04-01T13:20:56Z

Hi @tblaha - yes you'll have to update it manually after running cmake (as it will detect the change and refetch). I either need to make a change upstream in pybind11_protobuf or add a patch which gets applied during configuration. I haven't got round to that yet.

tblaha · 2023-04-01T13:33:08Z

Not a problem! Thanks for confirming!

srmainwaring self-assigned this Mar 8, 2023

srmainwaring added duplicate This issue or pull request already exists support User requires help and removed duplicate This issue or pull request already exists labels Mar 8, 2023

kkishore9891 closed this as completed Mar 9, 2023

srmainwaring mentioned this issue Mar 9, 2023

Fix installation of dependency shared libraries #20

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing gz.transport module throws an error message. #18

Importing gz.transport module throws an error message. #18

kkishore9891 commented Mar 8, 2023 •

edited

Loading

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023 •

edited

Loading

kkishore9891 commented Mar 8, 2023 •

edited

Loading

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023 •

edited

Loading

srmainwaring commented Mar 9, 2023

kkishore9891 commented Mar 9, 2023

kkishore9891 commented Mar 9, 2023 •

edited

Loading

srmainwaring commented Mar 9, 2023 •

edited

Loading

kkishore9891 commented Mar 9, 2023

srmainwaring commented Mar 9, 2023 •

edited

Loading

kkishore9891 commented Mar 9, 2023

srmainwaring commented Mar 9, 2023

tblaha commented Apr 1, 2023

srmainwaring commented Apr 1, 2023

tblaha commented Apr 1, 2023

Importing gz.transport module throws an error message. #18

Importing gz.transport module throws an error message. #18

Comments

kkishore9891 commented Mar 8, 2023 • edited Loading

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023 • edited Loading

kkishore9891 commented Mar 8, 2023 • edited Loading

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023

srmainwaring commented Mar 8, 2023

kkishore9891 commented Mar 8, 2023 • edited Loading

srmainwaring commented Mar 9, 2023

kkishore9891 commented Mar 9, 2023

kkishore9891 commented Mar 9, 2023 • edited Loading

srmainwaring commented Mar 9, 2023 • edited Loading

System

Build

C++ example

Python example

kkishore9891 commented Mar 9, 2023

srmainwaring commented Mar 9, 2023 • edited Loading

kkishore9891 commented Mar 9, 2023

srmainwaring commented Mar 9, 2023

tblaha commented Apr 1, 2023

srmainwaring commented Apr 1, 2023

tblaha commented Apr 1, 2023

kkishore9891 commented Mar 8, 2023 •

edited

Loading

srmainwaring commented Mar 8, 2023 •

edited

Loading

kkishore9891 commented Mar 8, 2023 •

edited

Loading

kkishore9891 commented Mar 8, 2023 •

edited

Loading

kkishore9891 commented Mar 9, 2023 •

edited

Loading

srmainwaring commented Mar 9, 2023 •

edited

Loading

srmainwaring commented Mar 9, 2023 •

edited

Loading