Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protobuf: static libraries #136843

Closed
wants to merge 2 commits into from

Conversation

ilya-lavrenov
Copy link
Sponsor Contributor

@ilya-lavrenov ilya-lavrenov commented Jul 17, 2023

  • Have you followed the guidelines for contributing?
  • Have you ensured that your commits follow the commit style guide?
  • Have you checked that there aren't other open pull requests for the same formula update/change?
  • Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
  • Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
  • Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

Aligned with protobuf@21 and older versions (see

system "cmake", "-S", ".", "-B", "static",
"-Dprotobuf_BUILD_SHARED_LIBS=OFF",
"-DCMAKE_POSITION_INDEPENDENT_CODE=ON",
"-DWITH_PROTOC=#{bin}/protoc",
*cmake_args, *std_cmake_args
system "cmake", "--build", "static"
lib.install buildpath.glob("static/*.a")
)

@carlocab
Copy link
Member

Remind me: which formulae need the static protobuf libraries? openvino? Anything else?

@ilya-lavrenov
Copy link
Sponsor Contributor Author

At least, I'm not aware of others.

@carlocab
Copy link
Member

How about we just fix openvino to use the shared libraries?

We can build these again now, but these are much less useful since you still need to link to a whole bunch of abseil libraries (which we don't have static libraries for).

@ilya-lavrenov
Copy link
Sponsor Contributor Author

ilya-lavrenov commented Jul 19, 2023

How about we just fix openvino to use the shared libraries?

I would say the problem is with protobuf, because it has global storage for registered schemas.
In OpenVINO we load such schemes dynamically (via plugins), so loading / unloading the same plugin multiple times (which is valid scenario) leads to exception, that schema is already registered. Protobuf does not have API to unload already registered schema.

With static protobuf each plugin has its own storage and loading / unloading don't lead to errors.
I cannot find exact link right now, but there were recommendations to use static protobuf for cases when protobuf schemas are used in dynamically loaded plugins.

We can build these again now, but these are much less useful since you still need to link to a whole bunch of abseil libraries (which we don't have static libraries for).

It's not a problem I hope, because protobuf-config.cmake is:

# User options
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-options.cmake")

# Depend packages
if(NOT ZLIB_FOUND)
  find_package(ZLIB)
endif()
if(NOT TARGET absl::strings)
  find_package(absl CONFIG)
endif()
if(NOT TARGET utf8_range)
  find_package(utf8_range CONFIG)
endif()

# Imported targets
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-targets.cmake")

# protobuf-generate function
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-generate.cmake")

# CMake FindProtobuf module compatible file
if(protobuf_MODULE_COMPATIBLE)
  include("${CMAKE_CURRENT_LIST_DIR}/protobuf-module.cmake")
endif()

So, absl is found and linked as a transitive dependency of protobuf or protobuf-lite.

@carlocab
Copy link
Member

It's not a problem I hope, because protobuf-config.cmake is:

I suspect it will be, because, if I understand correctly, openvino relies on the built-in FindProtobuf.cmake module to link statically with the Protobuf libraries. The built-in CMake module will not attempt to link with Abseil as a transitive dependency.

Have you been able to build openvino against protobuf after this change?

@carlocab carlocab marked this pull request as draft July 21, 2023 10:02
@carlocab carlocab added the CI-skip-dependents Pass --skip-dependents to brew test-bot. label Jul 21, 2023
@carlocab
Copy link
Member

Actually, why don't we use CI to find out...

@ilya-lavrenov
Copy link
Sponsor Contributor Author

ilya-lavrenov commented Jul 21, 2023

It's not a problem I hope, because protobuf-config.cmake is:

I suspect it will be, because, if I understand correctly, openvino relies on the built-in FindProtobuf.cmake module to link statically with the Protobuf libraries. The built-in CMake module will not attempt to link with Abseil as a transitive dependency.

Right, by default OpenVINO (and all other projects, who does not set CONFIG / NO_MODULE into find_package call) uses cmake FindProtobuf.cmake module (which is cmake's default).
But with ABSL integration into protobuf we should use one of two approaches to properly use static protobuf:

  • -DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON to have properly configured targets and their dependencies.
  • Explicitly add CONFIG / NO_MODULE to find_package(Protobuf ...)

Have you been able to build openvino against protobuf after this change?

With the first option and static protobuf + dynamic absl libraries, the list of dependencies looks like:

(openvino-pre-release) sandye51@Ilyas-MacBook-Air openvino % otool -L bin/arm64/Release/libopenvino_onnx_frontend.dylib
bin/arm64/Release/libopenvino_onnx_frontend.dylib:
	@rpath/libopenvino_onnx_frontend.2310.dylib (compatibility version 2310.0.0, current version 2023.1.0)
	@rpath/libopenvino.2310.dylib (compatibility version 2310.0.0, current version 2023.1.0)
	/opt/homebrew/opt/protobuf/lib/libprotobuf-lite.23.4.0.dylib (compatibility version 0.0.0, current version 23.4.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_check_op.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_leak_check.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_die_if_null.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_conditions.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_message.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_nullguard.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_examine_stack.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_format.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_proto.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_log_sink_set.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_sink.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_entry.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_marshalling.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_reflection.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_config.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_program_name.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_private_handle_accessor.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_commandlineflag.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_flags_commandlineflag_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_initialize.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_globals.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_internal_globals.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_hash.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_city.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_low_level_hash.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_raw_hash_set.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_hashtablez_sampler.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_statusor.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_status.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_cord.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_cordz_info.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_cord_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_cordz_functions.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_exponential_biased.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_cordz_handle.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_crc_cord_state.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_crc32c.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_crc_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_crc_cpu_detect.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_bad_optional_access.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_str_format_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_strerror.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_synchronization.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_stacktrace.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_symbolize.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_debugging_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_demangle_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_graphcycles_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_malloc_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_time.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_strings.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_throw_delegate.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_strings_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_base.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_spinlock_wait.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_int128.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_civil_time.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_time_zone.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_bad_variant_access.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_raw_logging_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/opt/homebrew/opt/abseil/lib/libabsl_log_severity.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

With your change, OpenVINO build failed, because 2023.0.1 version does not support protobuf 4.x (higher CXX standard requirements), it's fixed in current master (commit, future 2023.1.0 release).

@carlocab
Copy link
Member

carlocab commented Jul 21, 2023

  • Explicitly add CONFIG / NO_MODULE to find_package(Protobuf ...)

This seems nicer and would be done upstream if possible.

With your change, OpenVINO build failed, because 2023.0.1 version does not support protobuf 4.x (higher CXX standard requirements), it's fixed in current master (commit, future 2023.1.0 release).

In that case, I suggest adding the static libraries when 2023.1.0 has been released (or, perhaps, just building them in the openvino formula directly using the code from this PR). The static libraries are now much more difficult to use, so it seems less valuable to ship them as part of the protobuf formula.

@ilya-lavrenov
Copy link
Sponsor Contributor Author

This seems nicer and would be done upstream if possible.

Looks like it's aligned with "official" suggestion from protobuf team protocolbuffers/protobuf#12637 (comment)

With requiring CONFIG don't you think brew formula still can contain static libraries?

or, perhaps, just building them in the openvino formula directly using the code from this PR

Do you mean compile own static version of protobuf and use only in OpenVINO formula?

@ilya-lavrenov
Copy link
Sponsor Contributor Author

BTW, cmake developers also recommend to use CONFIG option https://gitlab.kitware.com/cmake/cmake/-/issues/24321
And in future FindProtobuf.cmake module will be removed. It means, that even now we can start relying on in-package ProtobufConfig.cmake. And static libraries work for this case.

@ilya-lavrenov
Copy link
Sponsor Contributor Author

Ok, let's use static protobuf 4.x directly in openvino.rb

@ilya-lavrenov ilya-lavrenov deleted the protobuf-static branch August 1, 2023 12:03
@github-actions github-actions bot added the outdated PR was locked due to age label Sep 1, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CI-skip-dependents Pass --skip-dependents to brew test-bot. outdated PR was locked due to age
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants