From 3518185b813b405a1a4d654a54bd2797092666bf Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Fri, 1 Dec 2023 11:08:57 +0100 Subject: [PATCH 01/13] Add discussion "Distribution package vs. import package" Resolves #1425 --- ...distribution-package-vs-import-package.rst | 90 +++++++++++++++++++ source/discussions/index.rst | 1 + source/glossary.rst | 6 +- .../guides/packaging-namespace-packages.rst | 2 + 4 files changed, 97 insertions(+), 2 deletions(-) create mode 100644 source/discussions/distribution-package-vs-import-package.rst diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst new file mode 100644 index 000000000..9ec708624 --- /dev/null +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -0,0 +1,90 @@ +.. _distribution-package-vs-import-package: + +======================================= +Distribution package vs. import package +======================================= + +A number of different concepts are commonly referred to by the word +"package". This page clarifies the differences between two distinct but +related meanings in Python packaging, "distribution package" and "import +package". + +What's a distribution package? +============================== + +A distribution package is a piece of software that you can install. +Most of the time, this is synonymous with "project". When you type ``pip +install pkg``, or when you write ``dependencies = ["pkg"]`` in your +``pyproject.toml``, ``pkg`` is the name of a distribution package. When +you search or browse PyPI_, the most widely known centralized source for +installing Python software, what you see is a list of distribution +packages. Alternatively, the term "distribution package" can be used to +refer to a specific file that contains a certain version of a project. + +Note that in the Linux world, "distribution package" refers to a package +provided by the system package manager, which is a different meaning. + + +What's an import package? +========================= + +An import package is a Python module. Thus, when you write ``import +pkg`` or ``from pkg import func`` in your Python code, ``pkg`` is the +name of an import package. More precisely, import packages are special +Python modules that can contain submodules. For example, the ``numpy`` +package contains modules like ``numpy.linalg`` and +``numpy.fft``. Usually, an import package is a directory on the file +system, containing modules as ``.py`` files and subpackages as +subdirectories. + +You can use an import package as soon as you have installed a distribution +package that provides it. + + +What are the links between distribution packages and import packages? +===================================================================== + +By convention, a distribution package usually provides one single import +package (or non-package module), with a matching name. For example, +``pip install numpy`` lets you ``import numpy``. + +However, this is only a convention. PyPI and other package indices do +not enforce any relationship between the name of a distribution package +and the import packages it provides. + +A distribution package could provide an import package with a different +name. An example of this is the popular Pillow_ library for image +processing. Its distribution package name is ``Pillow``, but it provides +the import package ``PIL``. This is for historical reasons: Pillow +started as a fork of the PIL library, thus it kept the import name +``PIL`` so that existing PIL users could switch to Pillow with little +effort. More generally, a fork of an existing library is a common reason +for differing names between the distribution package and the import +package. + +On a given package index (like PyPI), distribution package names must be +unique. On the other hand, import packages have no such requirement. +Import packages with the same name can be provided by several +distribution packages. Again, forks are a common reason for this. + +Conversely, a distribution package can provide several import packages, +although this is less common. + + +How do distribution package names and import package names compare? +=================================================================== + +Import packages should have valid Python identifiers as their name. In +particular, they use underscores ``_`` as word separator and they are +case-sensitive. + +On the other hand, distribution packages can use hyphens ``-`` or +underscores ``.``. They can also contain dots ``.``, which is sometimes +used for packaging a subpackage of a :ref:`namespace package +`. For most purposes, they are insensitive +to case and to ``-`` vs. ``_`` differences, e.g., ``pip install +Awesome_Package`` is the same as ``pip install awesome-package``. + + +.. _PyPI: https://pypi.org +.. _Pillow: https://pypi.org/project/Pillow diff --git a/source/discussions/index.rst b/source/discussions/index.rst index e5411ece3..b378ed810 100644 --- a/source/discussions/index.rst +++ b/source/discussions/index.rst @@ -12,5 +12,6 @@ specific topic. If you're just trying to get stuff done, see pip-vs-easy-install install-requires-vs-requirements wheel-vs-egg + distribution-package-vs-import-package src-layout-vs-flat-layout setup-py-deprecated diff --git a/source/glossary.rst b/source/glossary.rst index f9f2abc29..9bfcbbd3f 100644 --- a/source/glossary.rst +++ b/source/glossary.rst @@ -67,7 +67,8 @@ Glossary :term:`Import Package` (which is also commonly called a "package") or another kind of distribution (e.g. a Linux distribution or the Python language distribution), which are often referred to with the single term - "distribution". + "distribution". See :ref:`distribution-package-vs-import-package` + for a breakdown of the differences. Egg @@ -103,7 +104,8 @@ Glossary An import package is more commonly referred to with the single word "package", but this guide will use the expanded term when more clarity is needed to prevent confusion with a :term:`Distribution Package` which - is also commonly called a "package". + is also commonly called a "package". See :ref:`distribution-package-vs-import-package` + for a breakdown of the differences. Module diff --git a/source/guides/packaging-namespace-packages.rst b/source/guides/packaging-namespace-packages.rst index 548020239..3d929d527 100644 --- a/source/guides/packaging-namespace-packages.rst +++ b/source/guides/packaging-namespace-packages.rst @@ -1,3 +1,5 @@ +.. _packaging-namespace-packages: + ============================ Packaging namespace packages ============================ From 176f95637e0a0a1d4574da25e75fe135d95da9e7 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Fri, 1 Dec 2023 16:22:02 +0100 Subject: [PATCH 02/13] Update source/discussions/distribution-package-vs-import-package.rst Co-authored-by: sinoroc <5529267+sinoroc@users.noreply.github.com> --- source/discussions/distribution-package-vs-import-package.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 9ec708624..c54bcb523 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -79,7 +79,7 @@ particular, they use underscores ``_`` as word separator and they are case-sensitive. On the other hand, distribution packages can use hyphens ``-`` or -underscores ``.``. They can also contain dots ``.``, which is sometimes +underscores ``_``. They can also contain dots ``.``, which is sometimes used for packaging a subpackage of a :ref:`namespace package `. For most purposes, they are insensitive to case and to ``-`` vs. ``_`` differences, e.g., ``pip install From 790e7cf6674c9761d204a2930673c0a1418efb8b Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Fri, 1 Dec 2023 16:26:51 +0100 Subject: [PATCH 03/13] Update source/discussions/distribution-package-vs-import-package.rst --- source/discussions/distribution-package-vs-import-package.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index c54bcb523..129eec926 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -83,7 +83,9 @@ underscores ``_``. They can also contain dots ``.``, which is sometimes used for packaging a subpackage of a :ref:`namespace package `. For most purposes, they are insensitive to case and to ``-`` vs. ``_`` differences, e.g., ``pip install -Awesome_Package`` is the same as ``pip install awesome-package``. +Awesome_Package`` is the same as ``pip install awesome-package`` (the precise rules +are given in the :ref:`name normalization +specification `). .. _PyPI: https://pypi.org From fe9378a1935acf57149a7dd1d6c9137192939a8f Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sun, 3 Dec 2023 17:24:59 +0100 Subject: [PATCH 04/13] Mention attrs as a distribution package which provides several import packages --- .../discussions/distribution-package-vs-import-package.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 129eec926..09f61668a 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -68,7 +68,9 @@ Import packages with the same name can be provided by several distribution packages. Again, forks are a common reason for this. Conversely, a distribution package can provide several import packages, -although this is less common. +although this is less common. An example is the attrs_ distribution +package, which provides both an ``attrs`` import package with a newer +API, and an ``attr`` import package with an older but supported API. How do distribution package names and import package names compare? @@ -90,3 +92,4 @@ specification `). .. _PyPI: https://pypi.org .. _Pillow: https://pypi.org/project/Pillow +.. _attrs: https://pypi.org/project/attrs From 92380eca023d5fe22f5404b022b28eb3eb224713 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sun, 3 Dec 2023 17:25:20 +0100 Subject: [PATCH 05/13] Link to Python docs for exact rules on valid identifiers --- .../discussions/distribution-package-vs-import-package.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 09f61668a..f10dced75 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -76,9 +76,10 @@ API, and an ``attr`` import package with an older but supported API. How do distribution package names and import package names compare? =================================================================== -Import packages should have valid Python identifiers as their name. In -particular, they use underscores ``_`` as word separator and they are -case-sensitive. +Import packages should have valid Python identifiers as their name (the +:ref:`exact rules ` are found in the Python +documentation). In particular, they use underscores ``_`` as word +separator and they are case-sensitive. On the other hand, distribution packages can use hyphens ``-`` or underscores ``_``. They can also contain dots ``.``, which is sometimes From a7592e8aa6b4e619a80352bfbb19ba2334f74773 Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sun, 3 Dec 2023 17:37:06 +0100 Subject: [PATCH 06/13] Rewrap --- .../discussions/distribution-package-vs-import-package.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index f10dced75..967328620 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -86,9 +86,9 @@ underscores ``_``. They can also contain dots ``.``, which is sometimes used for packaging a subpackage of a :ref:`namespace package `. For most purposes, they are insensitive to case and to ``-`` vs. ``_`` differences, e.g., ``pip install -Awesome_Package`` is the same as ``pip install awesome-package`` (the precise rules -are given in the :ref:`name normalization -specification `). +Awesome_Package`` is the same as ``pip install awesome-package`` (the +precise rules are given in the :ref:`name normalization specification +`). .. _PyPI: https://pypi.org From 38675a15eeb7d1b03a1f1d16617d7ccef164179c Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Thu, 14 Dec 2023 22:19:00 +0100 Subject: [PATCH 07/13] Address review comments --- .../distribution-package-vs-import-package.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 967328620..36f11d963 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -44,13 +44,15 @@ package that provides it. What are the links between distribution packages and import packages? ===================================================================== -By convention, a distribution package usually provides one single import +Most of the time, a distribution package provides one single import package (or non-package module), with a matching name. For example, ``pip install numpy`` lets you ``import numpy``. -However, this is only a convention. PyPI and other package indices do -not enforce any relationship between the name of a distribution package -and the import packages it provides. +However, this is only a convention. PyPI and other package indices *do not +enforce any relationship* between the name of a distribution package and the +import packages it provides. (A consequence of this is that you cannot blindly +install the PyPI package ``foo`` if you see ``import foo``; this may install an +unintended, and potentially even malicious package.) A distribution package could provide an import package with a different name. An example of this is the popular Pillow_ library for image From d4164d2d1b8b2f04f2c5961af5e388ef42974c2b Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 08:26:54 +0100 Subject: [PATCH 08/13] The PyPI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) --- source/discussions/distribution-package-vs-import-package.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 36f11d963..005f65d14 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -16,7 +16,7 @@ A distribution package is a piece of software that you can install. Most of the time, this is synonymous with "project". When you type ``pip install pkg``, or when you write ``dependencies = ["pkg"]`` in your ``pyproject.toml``, ``pkg`` is the name of a distribution package. When -you search or browse PyPI_, the most widely known centralized source for +you search or browse the PyPI_, the most widely known centralized source for installing Python software, what you see is a list of distribution packages. Alternatively, the term "distribution package" can be used to refer to a specific file that contains a certain version of a project. From 62273fef03a2a6e678f5bd5af3478914233191d7 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 08:28:42 +0100 Subject: [PATCH 09/13] Python software -> Python libraries and tools --- source/discussions/distribution-package-vs-import-package.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 005f65d14..8d847cca7 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -17,7 +17,7 @@ Most of the time, this is synonymous with "project". When you type ``pip install pkg``, or when you write ``dependencies = ["pkg"]`` in your ``pyproject.toml``, ``pkg`` is the name of a distribution package. When you search or browse the PyPI_, the most widely known centralized source for -installing Python software, what you see is a list of distribution +installing Python libraries and tools, what you see is a list of distribution packages. Alternatively, the term "distribution package" can be used to refer to a specific file that contains a certain version of a project. From bbbab1a4bee5c1332ce9326015a2108429a7f8ac Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 08:47:11 +0100 Subject: [PATCH 10/13] Clarify reference to Linux distro packages --- .../discussions/distribution-package-vs-import-package.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 8d847cca7..b00e3bde2 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -21,8 +21,10 @@ installing Python libraries and tools, what you see is a list of distribution packages. Alternatively, the term "distribution package" can be used to refer to a specific file that contains a certain version of a project. -Note that in the Linux world, "distribution package" refers to a package -provided by the system package manager, which is a different meaning. +Note that in the Linux world, a "distribution package", +most commonly abbreviated as "distro package" or just "package", +is something provided by the system package manager of the `Linux distribution `_, +which is a different meaning. What's an import package? @@ -93,6 +95,7 @@ precise rules are given in the :ref:`name normalization specification `). +.. _distro: https://en.wikipedia.org/wiki/Linux_distribution .. _PyPI: https://pypi.org .. _Pillow: https://pypi.org/project/Pillow .. _attrs: https://pypi.org/project/attrs From 0335a1fbca21752135619e689ecc580dc78737d0 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 08:55:29 +0100 Subject: [PATCH 11/13] Add importlib pointer --- .../distribution-package-vs-import-package.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index b00e3bde2..663225bcc 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -82,7 +82,7 @@ How do distribution package names and import package names compare? Import packages should have valid Python identifiers as their name (the :ref:`exact rules ` are found in the Python -documentation). In particular, they use underscores ``_`` as word +documentation) #[non-identifier-mod-name]_. In particular, they use underscores ``_`` as word separator and they are case-sensitive. On the other hand, distribution packages can use hyphens ``-`` or @@ -95,6 +95,16 @@ precise rules are given in the :ref:`name normalization specification `). + +- +--------------------------- + +.. [#non-identifier-mod-name] Although is technically possible + to import packages/modules that do not have a valid Python identifier as + their name, using :doc:`importlib `, + this is vanishingly rare and strongly discouraged. + + .. _distro: https://en.wikipedia.org/wiki/Linux_distribution .. _PyPI: https://pypi.org .. _Pillow: https://pypi.org/project/Pillow From 3b42e3f98a9737a2e26ddd78f0a4569539bc6a45 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 09:02:50 +0100 Subject: [PATCH 12/13] Syntax fixes --- source/discussions/distribution-package-vs-import-package.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index 663225bcc..d4a754310 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -82,7 +82,7 @@ How do distribution package names and import package names compare? Import packages should have valid Python identifiers as their name (the :ref:`exact rules ` are found in the Python -documentation) #[non-identifier-mod-name]_. In particular, they use underscores ``_`` as word +documentation) [#non-identifier-mod-name]_. In particular, they use underscores ``_`` as word separator and they are case-sensitive. On the other hand, distribution packages can use hyphens ``-`` or @@ -96,7 +96,6 @@ precise rules are given in the :ref:`name normalization specification -- --------------------------- .. [#non-identifier-mod-name] Although is technically possible From 9b3655c495b0ca8f8634b48d5b558b8625bd8999 Mon Sep 17 00:00:00 2001 From: Jean Abou-Samra Date: Sat, 16 Dec 2023 09:08:02 +0100 Subject: [PATCH 13/13] Missing word --- source/discussions/distribution-package-vs-import-package.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/discussions/distribution-package-vs-import-package.rst b/source/discussions/distribution-package-vs-import-package.rst index d4a754310..65e7019c6 100644 --- a/source/discussions/distribution-package-vs-import-package.rst +++ b/source/discussions/distribution-package-vs-import-package.rst @@ -98,7 +98,7 @@ precise rules are given in the :ref:`name normalization specification --------------------------- -.. [#non-identifier-mod-name] Although is technically possible +.. [#non-identifier-mod-name] Although it is technically possible to import packages/modules that do not have a valid Python identifier as their name, using :doc:`importlib `, this is vanishingly rare and strongly discouraged.