ENH: equivalent of local_cc_dependency override in bzlmod

[h-vetinari]

Description of the feature request:

Hi!

I help maintain the conda-forge distribution, where we clash pretty fundamentally with bazel's philosophy of hermetic builds -- we absolutely cannot rebuild all dependencies in a single build, it MUST be possible to leverage libraries we have already built for the respective platform (with a given version, ABI, etc.), whether they're built by CMake, autotools, meson or whatever. I can explain the reasons for this imperative requirement in more detail if desired, but it is utterly non-negotiable. I believe that conda-forge is not a "special" distribution in this regard - other distributions also have requirements to build things incrementally (both architecturally and practically), prefer shared libraries built once against a particular ABI, etc.

This has historically been extremely painful with bazel, to the point that many packaging efforts for packages natively built by bazel have either stalled completely (e.g. tensorflow-addons), are buggy (ray vs. protobuf interaction), or are maintained with extreme effort by a handful of people who know how to hack a bazel build enough to inject local dependencies (e.g. tensorflow).

For tensorflow in particular, this works by generating a custom toolchain, with lots of manual patching, and so on. It is unworkable at scale.

I was very much looking forward to the bzlmod effort reducing the difficulty of doing this, but now that I'm taking a closer look, AFAICT all of the documented overrides point to a repo, or at least need some bazel metadata (same for the given examples) This is very likely still an improvement over the previous state of affairs, but still leaves us with manually having to write that metadata on the fly for every consuming library, which is similarly unrealistic at the scale of 100s-1000s of packages.

Based on having an existing library with headers, library artefacts (either shared or static), and potentially some binaries (like protoc), what would help us¹, in ascending order of awesomeness:

• Provide a way to generate the required bazel-metadata on the fly
• Provide an equivalent of local_cc_dependency in the bzlmod world that allows pointing to a given prefix resp. path.
  • It doesn't matter if we need to map each library/binary individually; though headers should only require one path.
• Provide a way to "use local build of this dependency from $PREFIX"; similar to of what tensorflow does with TF_SYSTEM_LIBS or protobuf with -Dprotobuf_<lib>_PROVIDER="package".
• Provide the ability to consume existing metadata from CMake / pkgconfig directly.

Note that this is not a new request, c.f. #5252 or on SO (1, 2, ...). I've tried to look for related issues but didn't find anything for this point specifically.

Which category does this issue belong to?

Configurability, External Dependency, Local Execution

What underlying problem are you trying to solve with this feature?

Distribute packages built by bazel, built against an existing (& well-controlled) set of prebuilt dependencies.

Which operating system are you running Bazel on?

Linux, MacOS, Windows

What is the output of bazel info release?

We'll generally build & use the last released bazel version

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

Footnotes

1. I only speak for myself; that said, I'm quite experienced in packaging, but I do not have the energy to deal with all the roadblocks that bazel builds put in my way. I think I can claim in good conscience that it would help many other people in conda-forge as well. ↩

[fmeum]

Sounds very much related to #20581, although that's more about switching between Bazel and system version of given deps, which requires them to have been Bazelified in the first place.

I don't know much about conda, but maybe its packages could be turned into repos by a module extension? That is the most important new bit of Bzlmod that allows for better integration with package managers.