pip.parse attempts to download dependencies for unsupported platforms

[keith]

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

Unclear

Description

If you generate platform specific requirements.txt files, and then provide them to pip.parse with requirements_by_platform, and also set experimental_target_platforms, pip.parse still attempts to download dependencies that are platform specific.

🔬 Minimal Reproduction

On this branch #2633 run bazel query 'deps(...)' on a Linux machine

🔥 Exception or Error

ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++pip+pip//pywin32': BUILD file not found in directory 'pywin32' of external repository @@rules_python++pip+pip. Add a BUILD file to a directory to mark it as a package.
Loading: 7 packages loaded
    Fetching repository @@rules_python++pip+pip_312_certifi_sdist_3d5da692; Running IsStandaloneInterpreter
    Fetching repository @@rules_python++pip+pip_312_charset_normalizer_cp312_cp312_win_amd64_6ff8a4a6; Running IsStandaloneInterpreter
    Fetching repository @@rules_python++pip+pip_312_blinker_sdist_b4ce2265; Running IsStandaloneInterpreter
    Fetching repository @@rules_python++pip+pip_312_brotli_sdist_81de08ac; starting
    Fetching repository @@rules_python++pip+pip_312_configargparse_sdist_e7067471; starting
    Fetching repository @@rules_python++pip+pip_312_click_py3_none_any_63c132bb; starting
    Fetching repository @@rules_python++pip+pip_312_brotli_cp312_cp312_macosx_10_13_universal2_32d95b80; starting
    Fetching repository @@rules_python++pip+pip_312_click_sdist_ed53c9d8; starting ... (47 fetches)

In this case it attempts to find a windows specific dependency, even though windows isn't supported, and the current host isn't windows.

🌍 Your Environment

Operating System:

linux x86_64

Output of bazel version:

8.1.0

Rules_python version:

branch above

Anything else relevant?

I think there was another issue about this in the past but I can't find it.

[keith]

if you generate a universal lockfile to try and solve this you end up hitting #2622

[keith]

I think there's another subtle version of this where I've noticed it downloading the wheels for unsupported platforms too:

    Fetching https://download.pytorch.org/whl/cu121/torch-2.2.2%2Bcu121-cp312-cp312-win_amd64.whl; 1.9 GiB (84.1%) 7s

which like in this example can potentially be quite large

[aignas]

Regarding: https://github.com/bazelbuild/rules_python/pull/2633/files#r1976390864

Right now the way to restrict the requirements_lock platforms would be to use the following snippet:

requirements_by_platform = {
    "//my:requirements.txt": "osx_aarch64,linux_aarch64,linux_x86_64",
}

As opposed to experimental_target_platforms, which is only used for whl METADATA interpreting. I think when I am going to stabilize the APIs, I will borrow the general builder pattern from #2578 to avoid these sort of ambiguities and proliferation of dicts.

[keith]

Ah nice thanks! I didn't think to try that combo! I verified that solves the first repro, but I get a similar error with a universal requirements file with that change, repro here #2643

===== stderr start =====
ERROR: Could not find a version that satisfies the requirement pywin32==308 (from versions: none)
ERROR: No matching distribution found for pywin32==308
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/private/var/tmp/_bazel_ksmiley/f68ec50975d34d4f4e31b12c12e2d1fc/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/private/var/tmp/_bazel_ksmiley/f68ec50975d34d4f4e31b12c12e2d1fc/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_ksmiley/f68ec50975d34d4f4e31b12c12e2d1fc/external/rules_python++python+python_3_12_aarch64-apple-darwin/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_ksmiley/f68ec50975d34d4f4e31b12c12e2d1fc/external/rules_python++python+python_3_12_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index-url', 'https://pypi.org/simple', '-r', '/var/folders/68/rn88yggx76bdgkj66fnxc1hc0000gn/T/tmpuc8z4ueu']' returned non-zero exit status 1.
===== stderr end =====

but maybe we just roll that into #2622 ? the difference in my mind is the requirements markers are much less complicated in this case vs the torch case, since pywin should never be pulled

[aignas]

Let's combine this with #2622