Third party dependencies are incorrect when using RBE because host != exec

[groodt]

Context

This is a tracking issue to recognise the lack of support for Remote Execution (RBE) when using the support for third party dependencies from PyPI (or similar indexes/mirrors).

A reproduction of the issue has been developed in this rules_python_rbe_exec_exploration repository.

The issue occurs when there are native dependencies in the transitive closure of third party dependencies, which is very common in the Python ecosystem.

The root cause of the issue is that the current implementation that imports third party dependencies as targets of the bazel build assumes that the host platform is the same as the execution platform (host == exec). However, a very common configuration of remote execution would be macos or windows host platforms executing remote builds on linux platforms (host != exec).

Known workarounds

1. Only use pure python third party dependencies (this is not really practical)
2. Try the experimental support for multi-platform wheel repositories using experimental_index_url (follow "Cross compilation" of py_binary/py_image/py_library targets #260 for when this becomes default)
3. Use alternative rules such as rules_pycross that can import third party dependencies where host != exec

Related

• "Cross compilation" of py_binary/py_image/py_library targets #260