Failure when trying to run pypi2nix on pantsbuild.pants

[copumpkin]

Description

Wow, I like your fancy auto-reporting machinery! Anyway, just trying out pypi2nix and got this immediately when giving it a go.

Traceback

% pypi2nix --version
1.8.0
% pypi2nix -V 3.5 -e pantsbuild.pants==1.2.1
Collecting pantsbuild.pants==1.2.1 (from -r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading pantsbuild.pants-1.2.1.tar.gz (895kB)

Collecting twitter.common.collections<0.4,>=0.3.1 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading twitter.common.collections-0.3.9.tar.gz

Collecting ansicolors==1.0.2 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading ansicolors-1.0.2.tar.gz

Collecting setproctitle==1.1.10 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading setproctitle-1.1.10.tar.gz

Collecting setuptools==5.4.1 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading setuptools-5.4.1.zip (832kB)

Collecting six<2,>=1.9.0 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading six-1.10.0.tar.gz

Collecting packaging==16.7 (from pantsbuild.pants==1.2.1->-r /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/editable.txt (line 1))

  Downloading packaging-16.7.tar.gz (44kB)

    Complete output from command python setup.py egg_info:

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/cache/pip/packaging/setup.py", line 72, in <module>

        "packaging",

      File "/nix/store/3jlvnszfs49k9b6i4qww1ax9q7yyygqc-python3-3.5.2/lib/python3.5/distutils/core.py", line 108, in setup

        _setup_distribution = dist = klass(attrs)

      File "/nix/store/rh7n7mvcgw6659l587rr5jhvpmir3mcd-pypi2nix-bootstrap/extra/setuptools/dist.py", line 320, in __init__

        _Distribution.__init__(self, attrs)

      File "/nix/store/3jlvnszfs49k9b6i4qww1ax9q7yyygqc-python3-3.5.2/lib/python3.5/distutils/dist.py", line 281, in __init__

        self.finalize_options()

      File "/nix/store/rh7n7mvcgw6659l587rr5jhvpmir3mcd-pypi2nix-bootstrap/extra/setuptools/dist.py", line 386, in finalize_options

        ep.require(installer=self.fetch_build_egg)

      File "/nix/store/rh7n7mvcgw6659l587rr5jhvpmir3mcd-pypi2nix-bootstrap/extra/pkg_resources/__init__.py", line 2318, in require

        items = working_set.resolve(reqs, env, installer, extras=self.extras)

      File "/nix/store/rh7n7mvcgw6659l587rr5jhvpmir3mcd-pypi2nix-bootstrap/extra/pkg_resources/__init__.py", line 859, in resolve

        raise VersionConflict(dist, req).with_context(dependent_req)

    pkg_resources.VersionConflict: (packaging 16.7 (/private/var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/cache/pip/packaging), Requirement.parse('packaging>=16.8'))

    

    ----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /var/folders/f2/18n30h057611nz0cy1ng9bhc0000gn/T/pypi2nix/cache/pip/packaging/

[garbas]

@copumpkin thank yo for using this reporting tool :) I makes my job (trying to reproduce) so much easier.

I am not sure who to "blame" for this bug, probably pip doesn't handle this good. Anyway, setuptools in recent releases gained few dependencies, one of which is packaging, which I update once a release (not sure this is best, but was working ok until now). Looks like you need to tell pip explicitly to update packaging package.

Here is the command that worked for me:

% pypi2nix -v -V 3.5 -e packaging -e "pex=1.2.2" -e pantsbuild.pants==1.2.1 -E libffi

You can see I also used -E option since one of the packages depends on libffi. I also had to force pip to use the latest version of pex otherwise we have conflicts with correct setuptools version. I'm not sure if forcing newer pex version will break something.

You will also need to patch pantsbuild.pants's setup.py a bit. This is what worked for me requirements_override.nix:

{ pkgs, python }:

self: super: {

  "pantsbuild.pants" = python.overrideDerivation super."pantsbuild.pants" (old: {
    patchPhase = ''
      sed -i \
        -e "s|pex==1.1.13|pex|" \
        -e "s|packaging==16.7|packaging|" \
        -e "s|setuptools==5.4.1|setuptools|" \
        setup.py
    '';
  });

}

Then building all packages works.

% nix-build requirements.nix -A interpreter
% ./result/bin/pants --help
No buildroot detected. Pants detects ...
% ./result/bin/pex --help
Usage: pex [-o OUTPUT.PEX] [options] [-- arg1 arg2 ...] ...

Hope this helps you, I know that it help me understand better how to improve pypi2nix that there would be less of this magic need. Keep the issues coming :)

[garbas]

i'm closing this issue since a solution is provided. please open if this i'm wrong.