`evaluate_markers_py()` does not work with `bazel vendor`

[armandomontanez]

🐞 bug report

Affected Rule

//python/private/pypi:evaluate_markers.bzl --> evaluate_markers_py

Is this a regression?

No (AFAIK), this likely never worked.

Description

Because of how runfiles are managed, evaluate_markers_py() doesn't work with bazel vendor. This appears to be due to how things are symlinked. Example:

amontanez:~/development/projects/bazel/rules_python/examples/pip_parse$ /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python++python+python_3_9_13_host/python -c 'import runpy; print("ok")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'runpy'
amontanez:~/development/projects/bazel/rules_python/examples/pip_parse$ $(realpath /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python++python+python_3_9_13_host/python) -c 'import runpy; print("ok")'
ok

🔬 Minimal Reproduction

• Add an environment marker to the pip_parse lockfile.

diff --git a/examples/pip_parse/requirements_lock.txt b/examples/pip_parse/requirements_lock.txt
index dc34b45a..1cac5b37 100644
--- a/examples/pip_parse/requirements_lock.txt
+++ b/examples/pip_parse/requirements_lock.txt
@@ -6,7 +6,8 @@
 #
 alabaster==0.7.13 \
     --hash=sha256:1ee19aca801bbabb5ba3f5f258e4422dfa86f82f3e9cefb0859b283cdd7f62a3 \
-    --hash=sha256:a27a4a084d5e690e16e01e03ad2b2e552c61a65469419b907243193de1a84ae2
+    --hash=sha256:a27a4a084d5e690e16e01e03ad2b2e552c61a65469419b907243193de1a84ae2 \
+    ; python_version == '3.9' or python_version == '3.10'
     # via sphinx
 babel==2.13.1 \
     --hash=sha256:33e0952d7dd6374af8dbf6768cc4ddf3ccfefc244f9986d4074704f2fbd18900 \

• $ cd examples/pip_parse
• $ bazel vendor --vendor_dir=../vendor //... && bazel build --vendor_dir=../vendor //...

🔥 Exception or Error

$ bazel clean && rm -rf ../vendor && bazel vendor --vendor_dir=../vendor //... && bazel build --vendor_dir=../vendor //...
Starting local Bazel server (8.2.1) and connecting to it...
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.
WARNING: /usr/local/google/home/amontanez/development/projects/bazel/rules_python/examples/pip_parse/BUILD.bazel:57:25: target '//:requirements_test' is deprecated: Use 'requirements.test' instead. The '*_test' target will be removed in the next major release.
INFO: Analyzed 9 targets (152 packages loaded, 8037 targets configured).
INFO: Found 9 targets...
INFO: Elapsed time: 20.955s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
INFO: Vendoring dependencies for targets...
INFO: All external dependencies for the requested targets vendored successfully.
WARNING: /usr/local/google/home/amontanez/development/projects/bazel/rules_python/examples/pip_parse/BUILD.bazel:57:25: target '//:requirements_test' is deprecated: Use 'requirements.test' instead. The '*_test' target will be removed in the next major release.
ERROR: /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/repo_utils.bzl:83:16: Traceback (most recent call last):
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/extension.bzl", line 625, column 25, in _pip_impl
		mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/extension.bzl", line 501, column 36, in parse_modules
		out = _create_whl_repos(
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/extension.bzl", line 163, column 50, in _create_whl_repos
		requirements_by_platform = parse_requirements(
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/parse_requirements.bzl", line 162, column 51, in parse_requirements
		env_marker_target_platforms = evaluate_markers(ctx, reqs_with_env_markers)
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/extension.bzl", line 194, column 77, in lambda
		evaluate_markers = lambda module_ctx, requirements: evaluate_markers(
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/evaluate_markers.bzl", line 78, column 36, in evaluate_markers_py
		pypi_repo_utils.execute_checked(
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/pypi/pypi_repo_utils.bzl", line 140, column 38, in _execute_checked
		return repo_utils.execute_checked(
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/repo_utils.bzl", line 228, column 29, in _execute_checked
		return _execute_internal(fail_on_error = True, *args, **kwargs)
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/repo_utils.bzl", line 157, column 27, in _execute_internal
		return logger.fail((
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/repo_utils.bzl", line 93, column 39, in lambda
		fail = lambda message_cb: _log(-1, "FAIL", message_cb, fail),
	File "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+/python/private/repo_utils.bzl", line 83, column 16, in _log
		printer("\nrules_python:{} {}:".format(
Error in fail:
rules_python:pypi:create_whl_repos FAIL: repo.execute: ResolveRequirementEnvMarkers(/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/modextwd/rules_python++pip/requirements_with_markers.in.json): end: failure:
  command: /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python++python+python_3_9_13_host/python -B -m python.private.pypi.requirements_parser.resolve_target_platforms /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/modextwd/rules_python++pip/requirements_with_markers.in.json /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/modextwd/rules_python++pip/requirements_with_markers.out.json
  return code: 1
  working dir: <default: /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/modextwd/rules_python++pip>
  timeout: <default timeout>
  environment:
PYTHONPATH="/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python+:/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python++internal_deps+pypi__packaging"
<stdout empty>
===== stderr start =====
Could not import runpy module
ModuleNotFoundError: No module named 'runpy'
===== stderr end =====
Target //:requirements up-to-date (nothing to build)
ERROR: Analysis of target '//:_yamllint_gen' failed; build aborted: error evaluating module extension @@rules_python+//python/extensions:pip.bzl%pip
INFO: Elapsed time: 0.569s, Critical Path: 0.03s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching ...zl%pip; Running ResolveRequirementEnvMarkers(/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/modextwd/rules_python++pip/requirements_wi\
th_markers.in.json)

🌍 Your Environment

Operating System:

  
  Linux
  macOS
  

Output of bazel version:

  
  8.1.0
  

Rules_python version:

  
  0.40.0 -->  HEAD
  

Anything else relevant?

[ssm3375]

@armandomontanez I also encountered the same exact error using bazel 8.01.1 (I was using this for Python 3.10 though). Is there any way I can get this fixed for my version of Python? Thanks!

[rickeylev]

For my own edification, can you post a directory listing of the directory:

/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/a290cf3762b3f28daa422427a879b8fd/external/rules_python++python+python_3_9_13_host/

Using e.g. ls -l -h -a -F. I'm interested in seeing which files/dirs are real vs symlinks and where they point.

The basic logic Python uses to locate it's home directory is "if python.exe is a symlink, use the directory it points to. Otherwise, use the directory of python.exe". There's a bit more to it (see https://github.com/python/cpython/blob/main/Modules/getpath.py ) but that's the main part.

So my theory is vendoring does something with, idk, symlinks vs copying? And Python gets confused. (Recall that, under the hood, the python_x_y_host repos are calling rctx.symlink() from the host to the backend repo)

[armandomontanez]

Linux

$ ls -l -h -a -F /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/4c6b8bc5f5001b48ea2cf108a2d16db7/external/rules_python~~python~python_3_9_host/
total 52K
drwxr-x---   2 amontanez primarygroup 4.0K Jun 20 09:09 ./
drwxr-x--- 111 amontanez primarygroup  20K Jun 20 09:09 ../
lrwxrwxrwx   1 amontanez primarygroup   78 Jun 20 09:09 bin -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/bin/
-rwxr-x--x   1 amontanez primarygroup  115 Jun 20 09:08 BUILD.bazel*
lrwxrwxrwx   1 amontanez primarygroup   82 Jun 20 09:09 include -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/include/
lrwxrwxrwx   1 amontanez primarygroup   78 Jun 20 09:09 lib -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/lib/
lrwxrwxrwx   1 amontanez primarygroup   81 Jun 20 09:09 python -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/python*
-rw-r-----   1 amontanez primarygroup    0 Jun 20 09:08 REPO.bazel
lrwxrwxrwx   1 amontanez primarygroup   80 Jun 20 09:09 share -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/share/
lrwxrwxrwx   1 amontanez primarygroup   97 Jun 20 09:09 STANDALONE_INTERPRETER -> ../bazel-external/rules_python~~python~python_3_9_x86_64-unknown-linux-gnu/STANDALONE_INTERPRETER*
-rw-r-----   1 amontanez primarygroup    0 Jun 20 09:08 WORKSPACE

macOS

$ ls -l -h -a -F /private/var/tmp/_bazel_amontanez/39deb24577b3c19a16a92f91582192be/external/rules_python~~python~python_3_9_host/
total 8
drwxr-xr-x   11 amontanez  primarygroup   352B Jun 20 09:04 ./
drwxr-xr-x  221 amontanez  primarygroup   6.9K Jun 20 09:04 ../
lrwxr-xr-x    1 amontanez  primarygroup    73B Jun 20 09:04 bin@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/bin
-rwxr-xr-x    1 amontanez  wheel          115B Jun 20 09:03 BUILD.bazel*
lrwxr-xr-x    1 amontanez  primarygroup    77B Jun 20 09:04 include@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/include
lrwxr-xr-x    1 amontanez  primarygroup    73B Jun 20 09:04 lib@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/lib
lrwxr-xr-x    1 amontanez  primarygroup    76B Jun 20 09:04 python@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/python
-rw-r--r--    1 amontanez  wheel            0B Jun 20 09:03 REPO.bazel
lrwxr-xr-x    1 amontanez  primarygroup    75B Jun 20 09:04 share@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/share
lrwxr-xr-x    1 amontanez  primarygroup    92B Jun 20 09:04 STANDALONE_INTERPRETER@ -> ../bazel-external/rules_python~~python~python_3_9_x86_64-apple-darwin/STANDALONE_INTERPRETER
-rw-r--r--    1 amontanez  wheel            0B Jun 20 09:03 WORKSPACE

[armandomontanez]

@armandomontanez I also encountered the same exact error using bazel 8.01.1 (I was using this for Python 3.10 though). Is there any way I can get this fixed for my version of Python? Thanks!

The issue/fix isn't specific to a single Python version. Should work for you. :)