feat(bzlmod): python.override for toolchain customization

[aignas]

Before the PR users would not be able to override various things configured by
rules_python when it comes to downloading toolchains. In the WORKSPACE world,
they would be able to do this and there could be users of rules_python who
might not be able to migrate to bzlmod due to the lack of such capability.

This PR refactors the internal register_all_versions tag class and adds a
generic way to override how python toolchains are registered and how they are
downloaded. In this way users can add extra toolchains, remove existing ones
or do anything they can do in WORKSPACE builds.

Fixes #1172
Work towards #2081

[rickeylev]

Overall, I think its pretty promising.

All the override APIs need is_root checks, though -- letting intermediate modules override would cause quite a headache.

Did you look at some of the other use cases in #2081 ?

Some cases that seem relevant to this api are:

• Using a local toolchain
• Using the free-thread interpreters
• Using one of the alternative LTO/debug builds
• Using a musl interpreter

Though, I'm not sure if it makes sense to override entirely vs register an additional toolchain with an additional constraint. I guess overriding provides an escape hatch. Part of me thinks that it might make more sense to push those cases into "register your own toolchain, here's some helpers to make it easier for you" -- it feels like, if you're overriding things such that they no longer resemble the python-build-standalone tars that get extracted and processed, then you're probably better off not trying to override the machinery that expects such a tar file.

[rickeylev]

Forgot one comment: in theory, can we use these APIs ourselves to register everything? i.e. replace versions.bzl with a series of calls in MODULE.bazel? I'm not suggesting we do that, but if we can, then that's pretty cool, and I think points to the API being in the right direction.

[aignas]

All the override APIs need is_root checks, though -- letting intermediate modules override would cause quite a headache.

Right now everything is under mod.is_root if statement, but I think the code needs a little bit of a refactor and test coverage, so that will be improved.

Did you look at some of the other use cases in #2081 ?

Some cases that seem relevant to this api are:

• Using a local toolchain

I haven't thought about the local toolchain yet.

• Using the free-thread interpreters
• Using one of the alternative LTO/debug builds
• Using a musl interpreter

Three of the above could be modelled as target platform with different config settings:

• free-threaded interpreters won't be available for all OS/Arches/versions initially, so asking the user to use a specific set of flag_values may be a good idea. The PLATFORMS right now is static, so actually doing that requires them to change rules_python, but I do think we could make it more ergonomic.
• LTO/debug builds probably should have //python/config_settings:lto=[auto|enabled|disabled], debug builds - similarly? If we allow users to override the available platforms, that could be more doable I think.
• MUSL interpreter should be handled by flag_values, this is already the case for glibc toolchain builds:

  rules_python/python/versions.bzl

  Line 529 in 6361057

  Label("//python/config_settings:py_linux_libc"): "glibc",

Though, I'm not sure if it makes sense to override entirely vs register an additional toolchain with an additional constraint. I guess overriding provides an escape hatch. Part of me thinks that it might make more sense to push those cases into "register your own toolchain, here's some helpers to make it easier for you" -- it feels like, if you're overriding things such that they no longer resemble the python-build-standalone tars that get extracted and processed, then you're probably better off not trying to override the machinery that expects such a tar file.

Exactly my thoughts.

Forgot one comment: in theory, can we use these APIs ourselves to register everything? i.e. replace versions.bzl with a series of calls in MODULE.bazel? I'm not suggesting we do that, but if we can, then that's pretty cool, and I think points to the API being in the right direction.

Yeah, I think we should do that. The refactor I mentioned above is to allow for that. This idea came to me when I was looking how I was constructing and overriding the tool_versions and I think we should be able to dog food the API.

[aignas]

Created #2189 and #2188 so that it is easier to review this change later.

[aignas]

This is a bit too big to easily review, will split it up.