[BUG] Setuptools exception in distutils mode since version 69.x

[Shixiaowei02]

setuptools version

setuptools == 69.x

Python version

Python 3.10

OS

Ubuntu 22.04

Additional environment information

As described in the issues mpi4py/mpi4py#431 and NVIDIA/TensorRT-LLM#447

Description

The installation of mpi4py suddenly unavailable since setuptools updated to version 69. (It works fine in older versions <= 68.x)

Expected behavior

Installation works as usual.

How to Reproduce

$ docker run -it --rm nvcr.io/nvidia/pytorch:23.10-py3
# apt update; apt install -y libopenmpi-dev
# pip install mpi4py

Output

pip install /tmp/mpi4py-3.1.5 --no-cache --verbose
Using pip 23.2.1 from /home/workspace/.local/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing ./mpi4py-3.1.5
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://pypi.ngc.nvidia.com
  Collecting setuptools<70.0.0,>=40.9.0
    Obtaining dependency information for setuptools<70.0.0,>=40.9.0 from https://files.pythonhosted.org/packages/f9/59/701df637517d6af0434cbb580bfc35a9c536aa7f47e0c2e222f1ef83547c/setuptools-69.0.1-py3-none-any.whl.metadata
    Downloading setuptools-69.0.1-py3-none-any.whl.metadata (6.3 kB)
  Collecting wheel
    Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/fa/7f/4c07234086edbce4a0a446209dc0cb08a19bb206a3ea53b2f56a403f983b/wheel-0.41.3-py3-none-any.whl.metadata
    Downloading wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB)
  Downloading setuptools-69.0.1-py3-none-any.whl (819 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.4/819.4 kB 13.7 MB/s eta 0:00:00
  Downloading wheel-0.41.3-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 25.3 MB/s eta 0:00:00
  Installing collected packages: wheel, setuptools
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  tensorrt-llm 0.5.0 requires mpi4py, which is not installed.
  tensorrt-llm 0.5.0 requires transformers==4.33.1, but you have transformers 4.35.2 which is incompatible.
  Successfully installed setuptools-69.0.1 wheel-0.41.3

  [notice] A new release of pip is available: 23.2.1 -> 23.3.1
  [notice] To update, run: python -m pip install --upgrade pip
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://pypi.ngc.nvidia.com
  Collecting Cython<3.0.0,>=0.27
    Obtaining dependency information for Cython<3.0.0,>=0.27 from https://files.pythonhosted.org/packages/f8/26/ca0f1bb049b83c25cafa39f3fa5287c826a6ab36e665c906209e07f4deac/Cython-0.29.36-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata
    Downloading Cython-0.29.36-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (3.1 kB)
  Downloading Cython-0.29.36-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 6.8 MB/s eta 0:00:00
  Installing collected packages: Cython
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  tensorrt-llm 0.5.0 requires mpi4py, which is not installed.
  tensorrt-llm 0.5.0 requires transformers==4.33.1, but you have transformers 4.35.2 which is incompatible.
  Successfully installed Cython-0.29.36

  [notice] A new release of pip is available: 23.2.1 -> 23.3.1
  [notice] To update, run: python -m pip install --upgrade pip
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info
  writing /tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/SOURCES.txt'
  dependency src/mpi4py.MPI.c won't be automatically included in the manifest: the path doesn't exist
  reading manifest file '/tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE.rst'
  writing manifest file '/tmp/pip-modern-metadata-6ybdk43b/mpi4py.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-6ybdk43b/mpi4py-3.1.5.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
  Running command Building wheel for mpi4py (pyproject.toml)
  running bdist_wheel
  running build
  running build_src
  using Cython version 0.29.36
  cythonizing 'mpi4py/MPI.pyx' -> 'mpi4py.MPI.c'
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.10
  creating build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/run.py -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/bench.py -> build/lib.linux-x86_64-3.10/mpi4py
  creating build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-3.10/mpi4py/futures
  creating build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/py.typed -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-3.10/mpi4py
  copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-3.10/mpi4py
  creating build/lib.linux-x86_64-3.10/mpi4py/include
  creating build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-3.10/mpi4py/include/mpi4py
  copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-3.10/mpi4py/futures
  copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-3.10/mpi4py/util
  copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-3.10/mpi4py/util
  running build_clib
  MPI configuration: [mpi] from 'mpi.cfg'
  MPI C compiler:    /usr/local/mpi/bin/mpicc
  MPI C++ compiler:  /usr/local/mpi/bin/mpicxx
  MPI F compiler:    /usr/local/mpi/bin/mpifort
  MPI F90 compiler:  /usr/local/mpi/bin/mpif90
  MPI F77 compiler:  /usr/local/mpi/bin/mpif77
  checking for library 'lmpe' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -llmpe -o _configtest
  /usr/bin/ld: cannot find -llmpe: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  building 'mpe' dylib library
  creating build/temp.linux-x86_64-3.10
  creating build/temp.linux-x86_64-3.10/src
  creating build/temp.linux-x86_64-3.10/src/lib-pmpi
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-3.10/src/lib-pmpi/mpe.o
  creating build/lib.linux-x86_64-3.10/mpi4py/lib-pmpi
  /usr/local/mpi/bin/mpicc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,--no-as-needed build/temp.linux-x86_64-3.10/src/lib-pmpi/mpe.o -o build/lib.linux-x86_64-3.10/mpi4py/lib-pmpi/libmpe.so
  checking for library 'vt-mpi' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt-mpi -o _configtest
  /usr/bin/ld: cannot find -lvt-mpi: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.mpi' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt.mpi -o _configtest
  /usr/bin/ld: cannot find -lvt.mpi: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  building 'vt' dylib library
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-3.10/src/lib-pmpi/vt.o
  /usr/local/mpi/bin/mpicc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,--no-as-needed build/temp.linux-x86_64-3.10/src/lib-pmpi/vt.o -o build/lib.linux-x86_64-3.10/mpi4py/lib-pmpi/libvt.so
  checking for library 'vt-mpi' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt-mpi -o _configtest
  /usr/bin/ld: cannot find -lvt-mpi: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.mpi' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt.mpi -o _configtest
  /usr/bin/ld: cannot find -lvt.mpi: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  building 'vt-mpi' dylib library
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-3.10/src/lib-pmpi/vt-mpi.o
  /usr/local/mpi/bin/mpicc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,--no-as-needed build/temp.linux-x86_64-3.10/src/lib-pmpi/vt-mpi.o -o build/lib.linux-x86_64-3.10/mpi4py/lib-pmpi/libvt-mpi.so
  checking for library 'vt-hyb' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt-hyb -o _configtest
  /usr/bin/ld: cannot find -lvt-hyb: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.ompi' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -lvt.ompi -o _configtest
  /usr/bin/ld: cannot find -lvt.ompi: No such file or directory
  collect2: error: ld returned 1 exit status
  failure.
  removing: _configtest.c _configtest.o
  building 'vt-hyb' dylib library
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-3.10/src/lib-pmpi/vt-hyb.o
  /usr/local/mpi/bin/mpicc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,--no-as-needed build/temp.linux-x86_64-3.10/src/lib-pmpi/vt-hyb.o -o build/lib.linux-x86_64-3.10/mpi4py/lib-pmpi/libvt-hyb.so
  running build_ext
  MPI configuration: [mpi] from 'mpi.cfg'
  MPI C compiler:    /usr/local/mpi/bin/mpicc
  MPI C++ compiler:  /usr/local/mpi/bin/mpicxx
  MPI F compiler:    /usr/local/mpi/bin/mpifort
  MPI F90 compiler:  /usr/local/mpi/bin/mpif90
  MPI F77 compiler:  /usr/local/mpi/bin/mpif77
  checking for dlopen() availability ...
  checking for header 'dlfcn.h' ...
  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  success!
  removing: _configtest.c _configtest.o
  success!
  checking for library 'dl' ...
  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  x86_64-linux-gnu-gcc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'dlopen' ...
  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  x86_64-linux-gnu-gcc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  building 'mpi4py.dl' extension
  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/usr/include/python3.10 -c src/dynload.c -o build/temp.linux-x86_64-3.10/src/dynload.o
  x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.10/src/dynload.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o build/lib.linux-x86_64-3.10/mpi4py/dl.cpython-310-x86_64-linux-gnu.so
  checking for MPI compile and link ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  success!
  removing: _configtest.c _configtest.o
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for missing MPI functions/symbols ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  success!
  removing: _configtest.c _configtest.o
  checking for function 'MPI_Type_create_f90_integer' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'MPI_Type_create_f90_real' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'MPI_Type_create_f90_complex' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'MPI_Status_c2f' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'MPI_Status_f2c' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for symbol 'MPI_LB' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for symbol 'MPI_UB' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for dlopen() availability ...
  checking for header 'dlfcn.h' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  success!
  removing: _configtest.c _configtest.o
  success!
  checking for library 'dl' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'dlopen' ...
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
  /usr/local/mpi/bin/mpicc _configtest.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  building 'mpi4py.MPI' extension
  /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/usr/include/python3.10 -c src/MPI.c -o build/temp.linux-x86_64-3.10/src/MPI.o
  /usr/local/mpi/bin/mpicc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.10/src/MPI.o -Lbuild/temp.linux-x86_64-3.10 -ldl -o build/lib.linux-x86_64-3.10/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so
  writing build/lib.linux-x86_64-3.10/mpi4py/mpi.cfg
  Traceback (most recent call last):
    File "/home/workspace/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/workspace/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/workspace/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 644, in <module>
    File "<string>", line 641, in main
    File "<string>", line 492, in run_setup
    File "/tmp/mpi4py-3.1.5/conf/mpidistutils.py", line 541, in setup
      return fcn_setup(**attrs)
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/wheel/bdist_wheel.py", line 371, in run
      install = self.reinitialize_command("install", reinit_subcommands=True)
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 216, in reinitialize_command
      cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
    File "/usr/lib/python3.10/distutils/cmd.py", line 305, in reinitialize_command
      return self.distribution.reinitialize_command(command,
    File "/usr/lib/python3.10/distutils/dist.py", line 938, in reinitialize_command
      command = self.get_command_obj(command_name)
    File "/usr/lib/python3.10/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 174, in __init__
      super().__init__(dist)
    File "/usr/lib/python3.10/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/tmp/pip-build-env-3u3m7v9k/overlay/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 50, in initialize_options
      orig.install.initialize_options(self)
    File "/usr/lib/python3.10/_distutils_system_mod.py", line 33, in initialize_options
      super().initialize_options()
  TypeError: super(type, obj): obj must be an instance or subtype of type
  error: subprocess-exited-with-error
  
  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python /home/workspace/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpey924q3u
  cwd: /tmp/mpi4py-3.1.5
  Building wheel for mpi4py (pyproject.toml) ... error
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects

[abravalheri]

This line looks a bit suspicious: /usr/lib/python3.10/_distutils_system_mod.py.

It seems that the incomparability is associated with a customisation implemented by the OS-provided package in the docker image.

@Shixiaowei02, what happens if you use SETUPTOOLS_USE_DISTUTILS=local when building/installing the package?

I am also assuming that the base image on the container is a Ubuntu 22.04. Can you please confirm that?

@jaraco, do you have any advice on that?

[davidxhive]

Getting the same issue with Nvidia's Pytorch 23.07 Docker Image . Doing SETUPTOOLS_USE_DISTUTILS=local when building/installing the package did not resolve the issue.

[jaraco]

Sorry for the long delay.

That error message indicates that the super() call is somehow not associated with the expected subclass. Thanks for providing the reproducer in the docker image. I'll see if I can reproduce it.

Unfortunately, at this time, the repro instructions no longer work:

root@c2d9b2c42b74:/workspace# pip install mpi4py
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting mpi4py
  Downloading mpi4py-3.1.5.tar.gz (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 18.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [158 lines of output]
      running bdist_wheel
 ...
      /usr/local/mpi/bin/mpicc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.10 -c _configtest.c -o _configtest.o
      _configtest.c:2:10: fatal error: mpi.h: No such file or directory
          2 | #include <mpi.h>
            |          ^~~~~~~
      compilation terminated.
      failure.
      removing: _configtest.c _configtest.o
      error: Cannot compile MPI programs. Check your configuration!!!
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects

I tried to find _distutils_system_mod.py online, but was unsuccessful. Here's a snippet of its contents:

import distutils.sysconfig
import distutils.command.install as orig_install
import distutils.command.install_egg_info as orig_install_egg_info
from distutils.command.install_egg_info import (
    to_filename,
    safe_name,
    safe_version,
    )
from distutils.errors import DistutilsOptionError


class install(orig_install.install):
    user_options = list(orig_install.install.user_options) + [
        ('install-layout=', None,
         "installation layout to choose (known values: deb, unix)"),
    ]

    def initialize_options(self):
        super().initialize_options()
        self.prefix_option = None
        self.install_layout = None

I suspect what's happening is that somehow that hook is getting called in a different order than before.

Can you provide refreshed repro instructions if the issue is still present?

[jaraco]

The changelog for v69 is here. Sadly, the mention of distutils changes doesn't link to #4103 where those were merged. The distutils changes don't seem to touch the install command at all, which makes me think the cause might be something else.

[lucy9527]

This line looks a bit suspicious: /usr/lib/python3.10/_distutils_system_mod.py.

It seems that the incomparability is associated with a customisation implemented by the OS-provided package in the docker image.

@Shixiaowei02, what happens if you use SETUPTOOLS_USE_DISTUTILS=local when building/installing the package?

I am also assuming that the base image on the container is a Ubuntu 22.04. Can you please confirm that?

@jaraco, do you have any advice on that?

I meet same problem, when set "SETUPTOOLS_USE_DISTUTILS=local", Then the mpi4py installed will be passed.

[jaraco]

when set "SETUPTOOLS_USE_DISTUTILS=local", Then the mpi4py installed will be passed.

Note that while this might serve as a bandaid, it won't work on Python 3.12 and won't work when we deprecate that hook, so use it with caution.

[DeclK]

This would work if you run the container and use the command line, but if you write a docker file and build the mpi4py inside the docker, it will give error when import torch

ImportError: /opt/hpcx/ucx/lib/libucs.so.0: undefined symbol: ucm_set_global_opts

You need to add the ucx lib path to the LD_LIBRARY_PATH
see thiss discussion for more details.

[jaraco]

In #4130, I spent some time root-causing this issue. It seem that only under certain circumstances, when setuptools._distutils is imported and even though SETUPTOOLS_USE_DISTUTILS=stdlib is configured, the _distutils_system_mod gets applied, leading to distutils.command.install getting patched after it's been subclassed by setuptools.command.install.

Long term, what needs to happen here is the _distutils_hack needs to go away, removing any possibility of SETUPTOOLS_USE_DISTUTILS=stdlib, and causing any _distutils_system_mod patching to happen early and unconditionally.

The short term fix, however, as suggested by @abravalheri is to stop using explicit references to the class in orig/distutils.command.install and instead use super() so that the super class gets called, whatever it is. This change will also have the benefit of avoiding any patched version from _distutils_system_mod if stdlib distutils is in use.