Skip to content

Conversation

@mahmoud
Copy link
Contributor

@mahmoud mahmoud commented May 31, 2018

Following up on @theacodes and my discussion at the PyCon 2018 packaging sprint, this PR brings in a new Overview document, meant to be linked from the packaging.python.org landing page. Also per Thea's instruction, I'm submitting the PR much closer to an "outline" state than a finished document.

This PR also addresses #310, providing much more direct coverage of packaging applications written in Python. This initial PR took a bit longer than expected because there are some intentional changes in direction from the original inspiration:

  1. More direct approach. Not assuming the audience has packaged Python before.
  2. Coverage of web, mobile, PaaS, and other frameworks.
  3. Brief explanation of tools -- (often smaller) applications with a developer audience
  4. Shorter sections so as to function more like a portal

TODOs for this document before the draft is final:

  • Flesh out and edit down sections for an even reading experience.
  • Documentation links for each technology.
  • Open-source project example links (the idea for which gave me a great idea for a survey we could run...)
  • Possibly import illustrations, if folks are interested

Some features I was planning for this doc may best be broken out into dedicated articles for each approach, e.g., "Packaging Python Applications with Snappy", just to keep it maximally browsable. Specifically I think a "Packaging Python Applications with Freezers" would be a great place for a table comparing the myriad options.

Anyways, hope you like the new direction, looking forward to continuing work on this!

@mahmoud mahmoud changed the title Add outline / rough draft of new overview page Add outline / rough draft of new Overview page May 31, 2018
Copy link
Member

@ncoghlan ncoghlan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the direction you're going with this - assorted more detailed comments in line :)

^^^^^^^^^^^^^^^^^^^^^^^^^^^

If your code consists of multiple Python files, it's usually organized
into a *package*. A package is any directory containing Python files,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have both import package and distribution package defined in the glossary, so I'd suggest adjusting this section a bit to cover the ambiguity of the "package" term, and linking to those terms the first time they're used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sure! Definitely want lots of internal links, that's the reason for the season :)

Service platforms
*****************

If you're developing a "Platform-as-a-Service" or "PaaS" like Heroku
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing "for" after "developing"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, good catch. Developing a new PaaS would at the very least be a separate document.

* Heroku
* Google App Engine
* PythonAnywhere
* OpenShift
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zappa for AWS Lambda would also be worth mentioning here.

libraries and local services. Images can be built to include
everything.
Depending on a framework
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"deployment system" might be a better phrase here (I initially thought you meant web framework)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was kind of looking for a term that could encapsulate the conventions of Heroku/PaaS as well as stuff like Kivy/Beeware, where as long as you stay within the lines, stuff should just work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I could include a clarifications that not all frameworks include deployment systems, so evaluate carefully, especially if the option isn't on the list.

If you're developing software that will be deployed to machines you
own, users' personal computers, or any other arrangement, read on.

Web and mobile platforms
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While technically accurate, I think mentioning "Web and ..." in the heading here is more confusing than helpful (since the previous section also mostly targets the web, just for backend network services rather than frontend single-page applications).

Alternatively, expanding it to Web browsers and mobile platforms would likely also work, as that makes it clearer that it isn't referring to backend web services the way the previous section does.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, web clients/browsers and mobile apps would definitely be the better explicit choice. I'll make that change.

************************

Python's advances are leading it into new spaces. These days you can
write a mobile app or web application frontend in Python. While the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Along a similar vein to tinkering with the section title, I think sticking a "single page" qualifier before "web application" will help distinguish the scope of this from more traditional server-side template rendering in a web framework.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, yeah, I'll try that out!

and Mac OS version for many years now.

* PEX
* zipapp (doesn't include library dependencies, requires Python 3.5+)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/include/help manage/ (if you use pip install --target to add zip-compatible pure Python deps, zipapp will happily add them to the archive)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the bullets in these sections are really anemic. I'll be sure to include more discussion of the pros and cons of each option. Even then, I might summarize by saying "minimal support" or something, because that is a really small subset of the libraries out there :)

installed, like Anaconda. Increasingly common in academic, analytical,
and other data-oriented environments. Also used in production services.

* conda/Anaconda
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible alternative structure here: rather than saying "new Python ecosystem", perhaps say "language independent packaging ecosystem", and mention:

  • conda/Anaconda (cross-platform)
  • homebrew (Mac OS X only)
  • rpm (selected Linux distros only)
  • deb (selected Linux distros only)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Yeah, I suppose there are Python applications out there that depend on RPM-packaged/managed libraries (python-*.i386.rpm, etc.). I'm not sure if I'd recommend it. I probably would want to give conda its own heading regardless, sort of like omnibus, below, because it does represent a pretty unique solution.


Embed your code on an Adafruit or a Micropython, or some other
hardware, and just ship it to the datacenter, or your users' homes,
and call it good.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, I resemble this remark :)

(DC fast chargers are a teensy bit bigger than a micro:bit, though)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to find more projects like that to link to from here. I'm pretty confident we can find open-source exemplars of applications for all the solutions above, but do keep an eye out for good hardware applications.

@ncoghlan
Copy link
Member

ncoghlan commented Jun 1, 2018

If you have images you want to include, I think that could definitely be a good idea - I know a lot of folks found the layered diagrams useful for explain IaaS vs PaaS vs SaaS, and much of that applies here as well (at least on the application packaging side).

@mahmoud
Copy link
Contributor Author

mahmoud commented Jun 1, 2018

@ncoghlan Glad you like it, and thanks for the initial review! As far as pictures, I think you're right. A diagram is more helpful than the flavor illustrations of the original blog post. I'll probably make a version of this:

image

I'll ping back when I integrate these changes and make some progress on the TODOs.

@ncoghlan
Copy link
Member

ncoghlan commented Jun 2, 2018

If you want more examples of "Anaconda-like" situations, you might consider situations like ESRi's ArcGIS Pro, Enthought Canopy, VFX Platform, etc.

VFX Platform in particular is an interesting example, since it's a platform spec that agrees on common versions of various dependencies rather than a particular implementation: http://www.vfxplatform.com/

@mahmoud
Copy link
Contributor Author

mahmoud commented Jun 3, 2018

Just pushed a commit that addresses most of the points above. Still need to address my initial TODOs.

As far as Canopy and VFX Platform, I think the main thing I didn't find for those was an actual guide to making your own package for the platform. VFX is great, and kinda reminds me of manylinux, but without a reference/repo with the docker image for building and auditing.

Canopy is definitely like Anaconda in that it's a distribution, but all the docs I found were about using it as a consumer, not as a distributor of an original application.

Anaconda has multiple tutorials for building packages (applicable for both library and applications), which can, like OS package managers, depend on Postgres and other non-Python dependencies. Unlike OS package managers, it's OS agnostic and has a tutorial for smoothly transitioning from a Python-native ecosystem (PyPI), to the bigger pond of cross-language.

Like, I'm tempted to include it in a more comprehensive OS-native distribution section, but conda really is a pretty unique case, especially taking into account its particular mutual affinity with Python community.

Copy link
Member

@pradyunsg pradyunsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments inline.

Packaging libraries and tools
-----------------------------

You may have heard about PyPI and ``setup.py`` and wheel files. These
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: two "and", first can become a comma.

Packaging Python
================

Python is a general-purpose programming lanaguage, meaning you can use
Copy link
Member

@pradyunsg pradyunsg Jun 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: language

check out the following frameworks, and refer to their packaging
guides:

* Kivy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we link to these project's pages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely! I actually have quite a few projects I still haven't mentioned, but when I get a spare hour or two, I plan on going through and hyperlinking everything :)

@ncoghlan
Copy link
Member

ncoghlan commented Jun 4, 2018

Fair points regarding conda vs the other alternatives I mentioned. While it's HPC focused rather than general user environment focused, I'd say https://spack.io/ is the closest thing I'm aware to a comparable system. (They submitted an entry for the "Other projects" page a while back: https://packaging.python.org/key_projects/#spack )

Copy link
Member

@theacodes theacodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some top-level comments. This is looking amazing! Let me know once you're ready for final editorial review and we'll get it merged. :)

@@ -0,0 +1,256 @@
================
Packaging Python
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you make the title something like "An overview of Python packaging"? It makes it better for linking and SEO.

it for many things. You can build robots or server software or a game
for your friends to play.

For this reason, the first step in every Python project must be to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this paragraph should merge with the one above (just delete the newlines).

It might seem strange to think about packaging before writing code,
but this process does wonders for avoiding headaches later on.

* Who are your software's users? Are they other developers doing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduce this list with something like:

"Some of the questions you may need to ask to determine your packaging strategy are:"

multiple files?). It's easier to get incomplete transfers, and harder
to guarantee code integrity at the destination.

So long as your code contains nothing but pure Python code, and you
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this is a bit weird. Even packages with native components have source distributions, and conversely, pure-Python packages can still have built distributions (wheels).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. I've added a graf to the (very underdeveloped) binary distribution section about how binary is best served up with source available, too. Still, because this is an overview, I'm going to leave some of the complexity (e.g., pure-Python wheels) offloaded to some of the other pages. I think next week sometime I'll be able to trawl the full guide for all of the crossrefs :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:

  1. If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's --only-binary option, whereas even pure Python sdists will fail to pass that check.
  2. However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.

So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:

  1. Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
  2. Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover

This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)

Python binary distribution
^^^^^^^^^^^^^^^^^^^^^^^^^^

Python's real power comes from its ability to integrate with the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Python's real power" can read as dismissive. I'd recommend "A great feature of Python is its ability to..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally. This section is very underdeveloped, but I've adjusted this in my most recent commit..

@pradyunsg
Copy link
Member

This PR has been idle for a month. Is there anything I can do to help get this moving again; by contributing content or otherwise?

@mahmoud
Copy link
Contributor Author

mahmoud commented Jul 15, 2018

Sorry @pradyunsg I promise I haven't forgotten, just running a bit late with some life stuff. Was going to work on it this weekend, but wedding planning flared up again. Thanks for the check-in!

@pradyunsg
Copy link
Member

No issues @mahmoud. :)

Thanks for making the time to work on this again.

@mahmoud
Copy link
Contributor Author

mahmoud commented Jul 26, 2018

I daresay this PR is finally done! I only imported diagrams, not illustrations, and there are a couple ideas for sections commented, but otherwise the overview reads and looks great on my local. Added a link to the home page, and I think this is ready to merge. Let me know if there are any requested changes. Otherwise, very much looking forward to seeing this go live!

@theacodes
Copy link
Member

theacodes commented Jul 26, 2018 via email

* `constructor <https://github.com/conda/constructor>`_ - For command-line installers
* `py2exe <http://www.py2exe.org/>`_ - Windows only
* `py2app <https://py2app.readthedocs.io/en/latest/>`_ - Mac only
* `bbFreeze <https://pypi.python.org/pypi/bbfreeze>`_ - Windows, Linux, Python 2 only
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial thing: should be https://pypi.org/project/bbfreeze/ (though this URL will redirect correctly)😁

P.S. Very excited about having this guide!

@mahmoud
Copy link
Contributor Author

mahmoud commented Jul 27, 2018

@gtback, dangit, I got all of them but one! Good catch.

Fixed, with some additional improvements as well. No TODOs left in the doc :)


This Python packaging superpower makes it possible for Pillow to be
a drop-in replacement for PIL, just by changing your project's
`q`install_requires`` or ``requirements.txt``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There sees to be a typo here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks :)

@theacodes
Copy link
Member

Preview at d366f79 staged here: http://temp.theadora.io/pypug/overview.html if anyone wants to look at it.

Copy link
Member

@theacodes theacodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mostly looks great except for one comment. Thank you so much for doing this!

looking for ways to package Python for a non-technical audience and/or
a production setting, skip ahead to :ref:`packaging-applications`.

Python modules
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section reads a bit bare and doesn't quite mesh with the context. What are we trying to convey to the user here? What action do we expect them to take with this information?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this section is under two pressures:

  1. Python developers from their first pip install are steeped in setup.py and "for-other-developers" packaging. I vividly remember the times I really struggled to make the mental leap from "I should build an egg" to "I should build an RPM" for my production application. It wasn't until much later that I realized the deployment model and even the audience were completely different. So this section is trying to place these technologies in the decision tree, as early and separate as possible from the applications. Without it, the immediate question becomes "well, what about wheels/sdists/pip/eggs/etc."?
  2. The section is on the bare side (even shorter than the sections in the initial post) because while I want to acknowledge the technologies, the rest of the site is a much better guide. I wanted it to be more content-light and link-heavy.

If you think it could stand do be expanded a bit more without overshadowing more extensive writings on the rest of the guide, I'd be happy to do what I can. Alternatively, as someone who knows the guide much better than me, I would really welcome more links to things that need linking!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm thinking we can either (1) remove it, or (2) expand on it just a little:

In the most simple of cases such as a single Python file that only depends on the Python standard library, you can simply copy and use the script however you'd like. This is great for sharing simple scripts between people who both have compatible Python versions (such as via email, StackOverflow, or GitHub gists). There are even some entire Python libraries that offer this as an option, such as Bottle. However, this pattern doesn't hold well for projects that consist of multiple files, need specific additional Python or OS libraries, or need a specific version of Python.

Copy link
Member

@ncoghlan ncoghlan Jul 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely! Added, with a link to bottle, and boltons, only because I specifically talk about this in the docs.

Copy link
Member

@ncoghlan ncoghlan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you :)

I think there's really only one substantive point which we still need to resolve, which is the information on distributing pure Python packages to other developers: I think we want to actively promote the use of universal wheel archives for pure Python projects, but even mentioning that as a potential question worth asking yourself will require a bit of restructuring of that entire section.

Beyond that, my comments are mainly copy-editing nits and a few minor technical corrections :)

* Who are your software's users? Will your software be installed by
other developers doing software development, operations people in a
datacenter, or a less software-savvy group?
* Is your software intended to run on servers, desktops, or embedded
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slight tweak to specifically mention mobile devices:

  • Is your software intended to run on servers, desktops, mobile clients (phones, tablets, etc), or embedded ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure!

looking for ways to package Python for a non-technical audience and/or
a production setting, skip ahead to :ref:`packaging-applications`.

Python modules
Copy link
Member

@ncoghlan ncoghlan Jul 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.

If your code consists of multiple Python files, it's usually organized
into a directory structure. Any directory containing Python files,
provided one of those files is named ``__init__.py``, comprises an
:term:`import package`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the introduction of namespace packages, this qualifier isn't necessary - __init__.py just marks the distinction between a self-contained packaged and a native namespace package, rather than being a pre-requisite for making an import package at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the qualifier, but it's kind of tricky. Maybe I'm getting philosophical here, but a directory without an expression of intent like __init__.py doesn't necessarily become an import package until someone imports it. And that was more a choice of the importer than the publisher.

I think I'll always prefer to the __init__.py for explicit package creation to avoid this sort of quandary. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I default to __init__.py as well, and there are still plenty of tools that only handle explicit import packages by default.

Perhaps we could say something like:

Any directory containing Python files can optionally be accessed at runtime as an :term:import package, rather than as a collection of independent scripts (and if one of the files in the directory is named __init__.py, it makes it explicit that the submodules are intended to be accessed via the package rather than directly).

multiple files?). It's easier to get incomplete transfers, and harder
to guarantee code integrity at the destination.

So long as your code contains nothing but pure Python code, and you
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:

  1. If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's --only-binary option, whereas even pure Python sdists will fail to pass that check.
  2. However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.

So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:

  1. Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
  2. Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover

This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)

a drop-in replacement for PIL, just by changing your project's
``install_requires`` or ``requirements.txt``.

Python binary distributions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continuing the above discussion, I'm wondering if it may make sense to add a third subsection after this one: "Publishing binary distributions for pure Python projects".

Then the paragraph above about specific example external dependencies can move down into the second subsection (since we'll want folks to keep reading regardless of whether they have external dependencies or not), and the final subsection can adopt the perspective of "Even if you're publishing a pure Python sdist, you may also want to publish a wheel archive anyway, for these reasons..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I also want to recommend universal wheels for pure-Python packages. That said, in trying to "ramp" users based on their needs, I also don't want to overwhelm them. I've put a couple sentences in to tie the two together as best practice, but more technical discussions should probably be on the linked article (tutorials/packaging-projects.rst). Maybe an aside to explicitly recommend doing both could work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a tricky balance. I think an aside probably isn't strong enough for this point, but I also don't think we really want to include in either of the existing subsections - hence the idea of make it a very short subsection in its own right.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I guess my main concern there is that right now the document is organized by "class of distributable artifact, and what it is capable of distributing". If I'm understanding correctly, this would add a section about wheels, which may be a confusing redundancy.

Another point is that sdist + wheel is absolutely a best practice on PyPI, but not necessarily so in other packaging workflows. I've used these formats "off PyPI" and always sdist only or wheel only. I'm not sure having both would have done much in those other packaging approaches. So in the interest of maintaining the introductory tone, I think it helps to introduce the formats and their intended purposes in this overview, then the best practices in the practical docs. (If someone manages to figure out how to push to PyPI from the overview alone, I'll be very surprised!)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see "class of distributable artifact" as the main organising principle in the current overview - I see it as being primarily organised by target audience, and then working back from that to the appropriate artifacts to use.

And when the target audience is "other Python developers", the appropriate artifacts to publish are both an sdist and a wheel archive.

The only time it makes sense to choose one or the other exclusively is when the target audience is "next stage in a specific automated build pipeline", and I don't think that's a scenario the overview really needs to emphasise.

At most, that use case feels like a potential side note along the lines of "Default to publishing both sdist and wheel archives, unless you're creating artifacts for a very specific use case where you know the recipient only needs one or the other".

Copy link
Contributor Author

@mahmoud mahmoud Aug 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I liked that last line, incorporated it in one of the recent commits. :)

I have to say, when I wrote it, my target audience was squarely "intermediate developers", i.e., devs experienced enough to get something to a shippable state who are also smart enough to search out some prior art. I'm willing to bet most of them are already familiar with several formats at this point, just from package managers, app stores, pip, and so forth. So technologies became the organizing principle because it's about creating a taxonomy to avoid getting lost in a sea of apples and oranges.

I'd love to think that some folks might consult the guide before writing any code destined for production, but I'll still put my money on this not being their first rodeo. In fact, many are probably looking for ways to create a rational build pipeline, such as a wheel into a container, which is why I think this arrangement works better than a "wizard" approach of segmenting by application/developer type.

Anyways, I wish I had more weekends like this to go on, but I'm gettin married in under a month, so I think this is all I can muster for now. I hope it can do some good for now.


.. _depending-on-a-new-python:

Depending on a new Python ecosystem
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to adjust the title here, as "new" is ambiguous: it could refer to "new-to-the-target-environment" or "new-to-the-world-at-large" (and neither is entirely accurate for the scope of the section anyway).

My suggestion would be "Depending on another software distribution ecosystem". That way if we later decide to add other options like Nix or homebrew in here, they can just be an extra paragraph at the end of this section, rather than needing their own section.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(working from bottom to top has its disadvantages, I see what you mean from down below)

would require only electricity.

Whereas the virtual machines described above are primarily reserved
for the tech-savvy, hardware appliances used by the most advanced data
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: missing "are" before "used"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

^^^^^^^^^^^^^^^^^^^^^^^^^

As mentioned in :ref:`depending-on-a-new-python` above, some operating
have package managers of their own. If you're very sure of the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: missing "systems" in "operating systems"

This cross-ref will also need to change if that section title changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch! wrt the crossref, I think I made an explicit heading anchor ref thing in this case.

Debian, Ubuntu, etc.) or `RPM
<https://en.wikipedia.org/wiki/RPM_Package_Manager>`_ (for Red Hat,
Fedora, etc.), and use that built-in package manager to take care of
installation, and even deployment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to directly mention FPM here, since that can take a fair bit of pain out of converting Python projects into Linux system packages: https://fpm.readthedocs.io/en/latest/source/virtualenv.html

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

ride. This impression is mostly a byproduct of Python's
versatility. Once you understand the natural boundaries between each
packaging solution, you begin to realize that the varied landscape is
a small price Python programmers pay for using the most balanced,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd tweak the praise of the language in this last sentence to be something unequivocally true. Something like:

... for using one of the most balanced, flexible, and broadly applicable languages available.

(The "one of" does the equivocation work there: "most" is debatable, "one of the most" is not)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, I'd take that debate anyday! But sure, not trying to make any enemies of anyone who read to the bottom of the page :)

@mahmoud
Copy link
Contributor Author

mahmoud commented Jul 29, 2018

OK, made some changes which I think mostly address the concerns raised.

@ncoghlan, regarding your big point, I'm OK expanding the packaging-libraries section somewhat, I just want to avoid having a big technical bump in the middle covering a topic that's really well-covered by links to the rest of the guide. Take a look at the sentences added in 8d3aae2, and if those aren't enough, would you mind writing a short aside (something I can put in a .. note:) as mentioned in the comment above?


.. _depending-on-a-new-ecosystem:

Depending on a new software distribution ecosystem
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"new" is still a bit ambiguous here ("new" in general? "new" to you? "new" to your users?).

Perhaps "a new" -> "another"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, meant to fix that one before, fixed now!

@mahmoud
Copy link
Contributor Author

mahmoud commented Aug 11, 2018

Just finished adding a few more tweaks based on feedback, and I think in many ways this document is actually a lot more comprehensive and broadly-targeted than the original post, in all the best ways. I think it's safe to say at this point I think it is/I am done :)

For the moment, I've got a (my) wedding to attend to, so I don't think I'll have much more time to work on it. I think even for any of its remaining, arguable flaws, having the overview available on the primary site would help a lot more than it would hurt, so I would encourage publishing it in its current state and tweaking it further in the form of future PRs. Thanks again for all the feedback, everyone!

Copy link
Member

@pradyunsg pradyunsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. I agree that this should be good enough to publish as is; we can make any additional changes needed in the near future.

Thanks for working on this @mahmoud! ^.^

@theacodes
Copy link
Member

LGTM, too! @ncoghlan if you have any further concerns please let me know (or file bugs) and I'll be happy to fix them in follow-up PRs.

@mahmoud this is monumental work, thank you so much!

@theacodes theacodes dismissed ncoghlan’s stale review August 12, 2018 23:02

We can address any remaining issues in follow-ups.

@theacodes theacodes merged commit 4b5e860 into pypa:master Aug 12, 2018
@ncoghlan
Copy link
Member

8d3aae2 and 88a1bc9 cover my wheel vs sdist concerns without requiring major changes to that section, so +1 for that approach :)

Thanks again @mahmoud for this incredibly useful addition!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants