rules_python and protobuf

[gfrankliu]

I am upgrading rules_python 0.38.0 to 1.4.1, which implicitly upgraded protobuf to protobuf@29.0-rc3.

Our Python codes use tensorflow 2.15 which has a dependency on the old protobuf 3.2x, as listed in the requirements.txt

Now python bazel test fails with

ImportError: cannot import name 'runtime_version' from 'google.protobuf' 

Can we make python use the old protobuf package from requirements.txt while letting MODULE.bazel to have a newer protobuf version? If this is not possible, is there a workaround or can we make it a feature request?

[rickeylev]

can we have a different version at runtime vs what is in the module?

In theory, yes. The two should be separable and separated.

In practice, if I remember correctly, the protobuf rules have an implicit dependency on protobuf runtimes hard-coded into their rules. So, no matter what, you end up with a dependency on whatever runtime libraries the proto rules are pointing to themselves.

What you should be able to do is add your own dependency on the proto runtimes (the ones that come via requirements.txt). So long as it comes earlier in the sys.path ordering, it'll be used.

In any case, there isn't much rules_python itself can do here, not directly. The main issue is the proto rules having a reference to a particular implementation of the runtime libraries without the user having a way to control that. Until that reference is removed (or implemented a different way), then we're stuck with the proto rules doing this.

Because the problem is mostly on the proto rules side, closing this issue.

[groodt]

I think you may be able to do this via something like:

proto_lang_toolchain(
    name = "protoc_python_toolchain",
    runtime = "@some_external_repo//:protobuf_python_runtime", # Replace with actual runtime dependency
    toolchain_type = "@rules_proto//proto:toolchain_type",
)

I think where the generated deps are created is here:

https://github.com/protocolbuffers/protobuf/blob/1016f1b1a1a3db9425ceaba620e2dd667f0f4de4/bazel/py_proto_library.bzl#L61

I think that's where the deps are injected and I think it brings along a default toolchain with their dependencies. If you ensure your runtime deps are compatible and inject the right ones it will probably work.

[gfrankliu]

The main issue is the protoc compiler. The runtime doesn't like the protos compiled from higher version of protoc. I was hoping toolchains_protoc could solve this, but it seems it doesn't support multiple versions of protoc. I opened aspect-build/toolchains_protoc#76 and hopefully upstream can solve this ultimately.

For now, I used below hack by passing my custom protoc script via bazel build argument --proto_compiler

1. I downloaded precompiled protoc binaries from https://github.com/protocolbuffers/protobuf/releases for both the old version that matches my python requirements.txt and the latest version that everything else use. They are unzipped on my host under /usr/local/protoc-24.4 and /usr/local/protoc-31.1

2. Add build --proto_compiler=//third_party/toolchain:protoc to my .bazelrc

3. Create directory third_party/toolchain and create script protoc.sh:

#!/usr/bin/env bash

if [[ -n "$1" && "$@" == *"--python_out="* ]]
then
  /usr/local/protoc-24.4/bin/protoc "$@"
else
  /usr/local/protoc-31.1/bin/protoc "$@"
fi

4. Create third_party/toolchain/BUILD

package(default_visibility = ["//visibility:public"])

sh_binary(
    name = "protoc",
    srcs = ["protoc.sh"],
)