-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add outline / rough draft of new Overview page #519
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like the direction you're going with this - assorted more detailed comments in line :)
source/overview.rst
Outdated
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| If your code consists of multiple Python files, it's usually organized | ||
| into a *package*. A package is any directory containing Python files, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have both import package and distribution package defined in the glossary, so I'd suggest adjusting this section a bit to cover the ambiguity of the "package" term, and linking to those terms the first time they're used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For sure! Definitely want lots of internal links, that's the reason for the season :)
source/overview.rst
Outdated
| Service platforms | ||
| ***************** | ||
|
|
||
| If you're developing a "Platform-as-a-Service" or "PaaS" like Heroku |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing "for" after "developing"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haha, good catch. Developing a new PaaS would at the very least be a separate document.
source/overview.rst
Outdated
| * Heroku | ||
| * Google App Engine | ||
| * PythonAnywhere | ||
| * OpenShift |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Zappa for AWS Lambda would also be worth mentioning here.
| libraries and local services. Images can be built to include | ||
| everything. | ||
| Depending on a framework |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"deployment system" might be a better phrase here (I initially thought you meant web framework)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was kind of looking for a term that could encapsulate the conventions of Heroku/PaaS as well as stuff like Kivy/Beeware, where as long as you stay within the lines, stuff should just work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess I could include a clarifications that not all frameworks include deployment systems, so evaluate carefully, especially if the option isn't on the list.
source/overview.rst
Outdated
| If you're developing software that will be deployed to machines you | ||
| own, users' personal computers, or any other arrangement, read on. | ||
|
|
||
| Web and mobile platforms |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While technically accurate, I think mentioning "Web and ..." in the heading here is more confusing than helpful (since the previous section also mostly targets the web, just for backend network services rather than frontend single-page applications).
Alternatively, expanding it to Web browsers and mobile platforms would likely also work, as that makes it clearer that it isn't referring to backend web services the way the previous section does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, web clients/browsers and mobile apps would definitely be the better explicit choice. I'll make that change.
source/overview.rst
Outdated
| ************************ | ||
|
|
||
| Python's advances are leading it into new spaces. These days you can | ||
| write a mobile app or web application frontend in Python. While the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Along a similar vein to tinkering with the section title, I think sticking a "single page" qualifier before "web application" will help distinguish the scope of this from more traditional server-side template rendering in a web framework.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, yeah, I'll try that out!
source/overview.rst
Outdated
| and Mac OS version for many years now. | ||
|
|
||
| * PEX | ||
| * zipapp (doesn't include library dependencies, requires Python 3.5+) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/include/help manage/ (if you use pip install --target to add zip-compatible pure Python deps, zipapp will happily add them to the archive)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the bullets in these sections are really anemic. I'll be sure to include more discussion of the pros and cons of each option. Even then, I might summarize by saying "minimal support" or something, because that is a really small subset of the libraries out there :)
source/overview.rst
Outdated
| installed, like Anaconda. Increasingly common in academic, analytical, | ||
| and other data-oriented environments. Also used in production services. | ||
|
|
||
| * conda/Anaconda |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possible alternative structure here: rather than saying "new Python ecosystem", perhaps say "language independent packaging ecosystem", and mention:
- conda/Anaconda (cross-platform)
- homebrew (Mac OS X only)
- rpm (selected Linux distros only)
- deb (selected Linux distros only)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Yeah, I suppose there are Python applications out there that depend on RPM-packaged/managed libraries (python-*.i386.rpm, etc.). I'm not sure if I'd recommend it. I probably would want to give conda its own heading regardless, sort of like omnibus, below, because it does represent a pretty unique solution.
source/overview.rst
Outdated
|
|
||
| Embed your code on an Adafruit or a Micropython, or some other | ||
| hardware, and just ship it to the datacenter, or your users' homes, | ||
| and call it good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey, I resemble this remark :)
(DC fast chargers are a teensy bit bigger than a micro:bit, though)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to find more projects like that to link to from here. I'm pretty confident we can find open-source exemplars of applications for all the solutions above, but do keep an eye out for good hardware applications.
|
If you have images you want to include, I think that could definitely be a good idea - I know a lot of folks found the layered diagrams useful for explain IaaS vs PaaS vs SaaS, and much of that applies here as well (at least on the application packaging side). |
|
@ncoghlan Glad you like it, and thanks for the initial review! As far as pictures, I think you're right. A diagram is more helpful than the flavor illustrations of the original blog post. I'll probably make a version of this: I'll ping back when I integrate these changes and make some progress on the TODOs. |
|
If you want more examples of "Anaconda-like" situations, you might consider situations like ESRi's ArcGIS Pro, Enthought Canopy, VFX Platform, etc. VFX Platform in particular is an interesting example, since it's a platform spec that agrees on common versions of various dependencies rather than a particular implementation: http://www.vfxplatform.com/ |
|
Just pushed a commit that addresses most of the points above. Still need to address my initial TODOs. As far as Canopy and VFX Platform, I think the main thing I didn't find for those was an actual guide to making your own package for the platform. VFX is great, and kinda reminds me of manylinux, but without a reference/repo with the docker image for building and auditing. Canopy is definitely like Anaconda in that it's a distribution, but all the docs I found were about using it as a consumer, not as a distributor of an original application. Anaconda has multiple tutorials for building packages (applicable for both library and applications), which can, like OS package managers, depend on Postgres and other non-Python dependencies. Unlike OS package managers, it's OS agnostic and has a tutorial for smoothly transitioning from a Python-native ecosystem (PyPI), to the bigger pond of cross-language. Like, I'm tempted to include it in a more comprehensive OS-native distribution section, but conda really is a pretty unique case, especially taking into account its particular mutual affinity with Python community. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor comments inline.
source/overview.rst
Outdated
| Packaging libraries and tools | ||
| ----------------------------- | ||
|
|
||
| You may have heard about PyPI and ``setup.py`` and wheel files. These |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: two "and", first can become a comma.
source/overview.rst
Outdated
| Packaging Python | ||
| ================ | ||
|
|
||
| Python is a general-purpose programming lanaguage, meaning you can use |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: language
source/overview.rst
Outdated
| check out the following frameworks, and refer to their packaging | ||
| guides: | ||
|
|
||
| * Kivy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we link to these project's pages?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely! I actually have quite a few projects I still haven't mentioned, but when I get a spare hour or two, I plan on going through and hyperlinking everything :)
|
Fair points regarding |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some top-level comments. This is looking amazing! Let me know once you're ready for final editorial review and we'll get it merged. :)
source/overview.rst
Outdated
| @@ -0,0 +1,256 @@ | |||
| ================ | |||
| Packaging Python | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you make the title something like "An overview of Python packaging"? It makes it better for linking and SEO.
source/overview.rst
Outdated
| it for many things. You can build robots or server software or a game | ||
| for your friends to play. | ||
|
|
||
| For this reason, the first step in every Python project must be to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: this paragraph should merge with the one above (just delete the newlines).
source/overview.rst
Outdated
| It might seem strange to think about packaging before writing code, | ||
| but this process does wonders for avoiding headaches later on. | ||
|
|
||
| * Who are your software's users? Are they other developers doing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Introduce this list with something like:
"Some of the questions you may need to ask to determine your packaging strategy are:"
| multiple files?). It's easier to get incomplete transfers, and harder | ||
| to guarantee code integrity at the destination. | ||
|
|
||
| So long as your code contains nothing but pure Python code, and you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, this is a bit weird. Even packages with native components have source distributions, and conversely, pure-Python packages can still have built distributions (wheels).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Absolutely. I've added a graf to the (very underdeveloped) binary distribution section about how binary is best served up with source available, too. Still, because this is an overview, I'm going to leave some of the complexity (e.g., pure-Python wheels) offloaded to some of the other pages. I think next week sometime I'll be able to trawl the full guide for all of the crossrefs :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:
- If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's
--only-binaryoption, whereas even pure Python sdists will fail to pass that check. - However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.
So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:
- Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
- Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover
This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)
source/overview.rst
Outdated
| Python binary distribution | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| Python's real power comes from its ability to integrate with the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Python's real power" can read as dismissive. I'd recommend "A great feature of Python is its ability to..."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally. This section is very underdeveloped, but I've adjusted this in my most recent commit..
…section, all per @theacodes comments
|
This PR has been idle for a month. Is there anything I can do to help get this moving again; by contributing content or otherwise? |
|
Sorry @pradyunsg I promise I haven't forgotten, just running a bit late with some life stuff. Was going to work on it this weekend, but wedding planning flared up again. Thanks for the check-in! |
|
No issues @mahmoud. :) Thanks for making the time to work on this again. |
…the only thing left)
…linked summary of wheels
|
I daresay this PR is finally done! I only imported diagrams, not illustrations, and there are a couple ideas for sections commented, but otherwise the overview reads and looks great on my local. Added a link to the home page, and I think this is ready to merge. Let me know if there are any requested changes. Otherwise, very much looking forward to seeing this go live! |
|
Excellent! I will try to review this tomorrow. Thank you so much!
…On Thu, Jul 26, 2018 at 10:05 AM Mahmoud Hashemi ***@***.***> wrote:
I daresay this PR is finally done! I only imported diagrams, not
illustrations, and there are a couple ideas for sections commented, but
otherwise the overview reads and looks great on my local. Added a link to
the home page, and I think this is ready to merge. Let me know if there are
any requested changes. Otherwise, very much looking forward to seeing this
go live!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#519 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPUc0blM2lsUD0JJ01EkI06XI-txW0Fks5uKfbNgaJpZM4UVY8Q>
.
|
source/overview.rst
Outdated
| * `constructor <https://github.com/conda/constructor>`_ - For command-line installers | ||
| * `py2exe <http://www.py2exe.org/>`_ - Windows only | ||
| * `py2app <https://py2app.readthedocs.io/en/latest/>`_ - Mac only | ||
| * `bbFreeze <https://pypi.python.org/pypi/bbfreeze>`_ - Windows, Linux, Python 2 only |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Trivial thing: should be https://pypi.org/project/bbfreeze/ (though this URL will redirect correctly)😁
P.S. Very excited about having this guide!
|
@gtback, dangit, I got all of them but one! Good catch. Fixed, with some additional improvements as well. No TODOs left in the doc :) |
source/overview.rst
Outdated
|
|
||
| This Python packaging superpower makes it possible for Pillow to be | ||
| a drop-in replacement for PIL, just by changing your project's | ||
| `q`install_requires`` or ``requirements.txt``. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There sees to be a typo here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed, thanks :)
|
Preview at d366f79 staged here: http://temp.theadora.io/pypug/overview.html if anyone wants to look at it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mostly looks great except for one comment. Thank you so much for doing this!
| looking for ways to package Python for a non-technical audience and/or | ||
| a production setting, skip ahead to :ref:`packaging-applications`. | ||
|
|
||
| Python modules |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This section reads a bit bare and doesn't quite mesh with the context. What are we trying to convey to the user here? What action do we expect them to take with this information?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this section is under two pressures:
- Python developers from their first
pip installare steeped insetup.pyand "for-other-developers" packaging. I vividly remember the times I really struggled to make the mental leap from "I should build an egg" to "I should build an RPM" for my production application. It wasn't until much later that I realized the deployment model and even the audience were completely different. So this section is trying to place these technologies in the decision tree, as early and separate as possible from the applications. Without it, the immediate question becomes "well, what about wheels/sdists/pip/eggs/etc."? - The section is on the bare side (even shorter than the sections in the initial post) because while I want to acknowledge the technologies, the rest of the site is a much better guide. I wanted it to be more content-light and link-heavy.
If you think it could stand do be expanded a bit more without overshadowing more extensive writings on the rest of the guide, I'd be happy to do what I can. Alternatively, as someone who knows the guide much better than me, I would really welcome more links to things that need linking!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'm thinking we can either (1) remove it, or (2) expand on it just a little:
In the most simple of cases such as a single Python file that only depends on the Python standard library, you can simply copy and use the script however you'd like. This is great for sharing simple scripts between people who both have compatible Python versions (such as via email, StackOverflow, or GitHub gists). There are even some entire Python libraries that offer this as an option, such as Bottle. However, this pattern doesn't hold well for projects that consist of multiple files, need specific additional Python or OS libraries, or need a specific version of Python.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Absolutely! Added, with a link to bottle, and boltons, only because I specifically talk about this in the docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great, thank you :)
I think there's really only one substantive point which we still need to resolve, which is the information on distributing pure Python packages to other developers: I think we want to actively promote the use of universal wheel archives for pure Python projects, but even mentioning that as a potential question worth asking yourself will require a bit of restructuring of that entire section.
Beyond that, my comments are mainly copy-editing nits and a few minor technical corrections :)
source/overview.rst
Outdated
| * Who are your software's users? Will your software be installed by | ||
| other developers doing software development, operations people in a | ||
| datacenter, or a less software-savvy group? | ||
| * Is your software intended to run on servers, desktops, or embedded |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Slight tweak to specifically mention mobile devices:
- Is your software intended to run on servers, desktops, mobile clients (phones, tablets, etc), or embedded ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure!
| looking for ways to package Python for a non-technical audience and/or | ||
| a production setting, skip ahead to :ref:`packaging-applications`. | ||
|
|
||
| Python modules |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like @theacodes's suggested additional paragraph here - it does a bit of extra scene-setting to make it clear that even within the "fellow Python users" audience there's still quite a bit of potential variety in needs and expectations.
source/overview.rst
Outdated
| If your code consists of multiple Python files, it's usually organized | ||
| into a directory structure. Any directory containing Python files, | ||
| provided one of those files is named ``__init__.py``, comprises an | ||
| :term:`import package`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the introduction of namespace packages, this qualifier isn't necessary - __init__.py just marks the distinction between a self-contained packaged and a native namespace package, rather than being a pre-requisite for making an import package at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've removed the qualifier, but it's kind of tricky. Maybe I'm getting philosophical here, but a directory without an expression of intent like __init__.py doesn't necessarily become an import package until someone imports it. And that was more a choice of the importer than the publisher.
I think I'll always prefer to the __init__.py for explicit package creation to avoid this sort of quandary. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I default to __init__.py as well, and there are still plenty of tools that only handle explicit import packages by default.
Perhaps we could say something like:
Any directory containing Python files can optionally be accessed at runtime as an :term:
import package, rather than as a collection of independent scripts (and if one of the files in the directory is named__init__.py, it makes it explicit that the submodules are intended to be accessed via the package rather than directly).
| multiple files?). It's easier to get incomplete transfers, and harder | ||
| to guarantee code integrity at the destination. | ||
|
|
||
| So long as your code contains nothing but pure Python code, and you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this section is currently fairly confusing as well, since it ends up conflating two different aspects:
- If you're publishing a pure Python package, you may not technically need a wheel file, since it's unlikely anybody will have issues installing your file. However, it's still a good idea to publish one, since wheels have a secondary purpose: they provide much richer static metadata than sdists do. Publishing a universal wheel also makes your project compatible with pip's
--only-binaryoption, whereas even pure Python sdists will fail to pass that check. - However, if you're publishing a package with native extensions, then the reason for publishing an sdist in addition to any wheel files you publish is to give folks the option of building it for themselves.
So rather than saying "Pure Python projects don't need to publish wheel archives", I think we want to instead have this section convey the perspective:
- Even pure Python projects should publish a universal wheel archive, as it's the presence of that wheel archive that makes it clear that they're either a pure Python project, or else the built archive contains only cross-platform binaries accessed through platform-independent mechanisms.
- Even projects that provided pre-compiled binary archives should still publish platform independent source archives to handle the cases that the pre-compiled archives don't cover
This change does mean the image from the Packaging Gradient article won't be usable as is, but it was also the image that I found confusing (which then allowed me to notice the issue in the text), so that may not be a bad thing :)
| a drop-in replacement for PIL, just by changing your project's | ||
| ``install_requires`` or ``requirements.txt``. | ||
|
|
||
| Python binary distributions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Continuing the above discussion, I'm wondering if it may make sense to add a third subsection after this one: "Publishing binary distributions for pure Python projects".
Then the paragraph above about specific example external dependencies can move down into the second subsection (since we'll want folks to keep reading regardless of whether they have external dependencies or not), and the final subsection can adopt the perspective of "Even if you're publishing a pure Python sdist, you may also want to publish a wheel archive anyway, for these reasons..."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I also want to recommend universal wheels for pure-Python packages. That said, in trying to "ramp" users based on their needs, I also don't want to overwhelm them. I've put a couple sentences in to tie the two together as best practice, but more technical discussions should probably be on the linked article (tutorials/packaging-projects.rst). Maybe an aside to explicitly recommend doing both could work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's a tricky balance. I think an aside probably isn't strong enough for this point, but I also don't think we really want to include in either of the existing subsections - hence the idea of make it a very short subsection in its own right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I guess my main concern there is that right now the document is organized by "class of distributable artifact, and what it is capable of distributing". If I'm understanding correctly, this would add a section about wheels, which may be a confusing redundancy.
Another point is that sdist + wheel is absolutely a best practice on PyPI, but not necessarily so in other packaging workflows. I've used these formats "off PyPI" and always sdist only or wheel only. I'm not sure having both would have done much in those other packaging approaches. So in the interest of maintaining the introductory tone, I think it helps to introduce the formats and their intended purposes in this overview, then the best practices in the practical docs. (If someone manages to figure out how to push to PyPI from the overview alone, I'll be very surprised!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see "class of distributable artifact" as the main organising principle in the current overview - I see it as being primarily organised by target audience, and then working back from that to the appropriate artifacts to use.
And when the target audience is "other Python developers", the appropriate artifacts to publish are both an sdist and a wheel archive.
The only time it makes sense to choose one or the other exclusively is when the target audience is "next stage in a specific automated build pipeline", and I don't think that's a scenario the overview really needs to emphasise.
At most, that use case feels like a potential side note along the lines of "Default to publishing both sdist and wheel archives, unless you're creating artifacts for a very specific use case where you know the recipient only needs one or the other".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I liked that last line, incorporated it in one of the recent commits. :)
I have to say, when I wrote it, my target audience was squarely "intermediate developers", i.e., devs experienced enough to get something to a shippable state who are also smart enough to search out some prior art. I'm willing to bet most of them are already familiar with several formats at this point, just from package managers, app stores, pip, and so forth. So technologies became the organizing principle because it's about creating a taxonomy to avoid getting lost in a sea of apples and oranges.
I'd love to think that some folks might consult the guide before writing any code destined for production, but I'll still put my money on this not being their first rodeo. In fact, many are probably looking for ways to create a rational build pipeline, such as a wheel into a container, which is why I think this arrangement works better than a "wizard" approach of segmenting by application/developer type.
Anyways, I wish I had more weekends like this to go on, but I'm gettin married in under a month, so I think this is all I can muster for now. I hope it can do some good for now.
source/overview.rst
Outdated
|
|
||
| .. _depending-on-a-new-python: | ||
|
|
||
| Depending on a new Python ecosystem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to adjust the title here, as "new" is ambiguous: it could refer to "new-to-the-target-environment" or "new-to-the-world-at-large" (and neither is entirely accurate for the scope of the section anyway).
My suggestion would be "Depending on another software distribution ecosystem". That way if we later decide to add other options like Nix or homebrew in here, they can just be an extra paragraph at the end of this section, rather than needing their own section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(working from bottom to top has its disadvantages, I see what you mean from down below)
source/overview.rst
Outdated
| would require only electricity. | ||
|
|
||
| Whereas the virtual machines described above are primarily reserved | ||
| for the tech-savvy, hardware appliances used by the most advanced data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo: missing "are" before "used"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed!
source/overview.rst
Outdated
| ^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| As mentioned in :ref:`depending-on-a-new-python` above, some operating | ||
| have package managers of their own. If you're very sure of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo: missing "systems" in "operating systems"
This cross-ref will also need to change if that section title changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch! wrt the crossref, I think I made an explicit heading anchor ref thing in this case.
source/overview.rst
Outdated
| Debian, Ubuntu, etc.) or `RPM | ||
| <https://en.wikipedia.org/wiki/RPM_Package_Manager>`_ (for Red Hat, | ||
| Fedora, etc.), and use that built-in package manager to take care of | ||
| installation, and even deployment. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to directly mention FPM here, since that can take a fair bit of pain out of converting Python projects into Linux system packages: https://fpm.readthedocs.io/en/latest/source/virtualenv.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added
source/overview.rst
Outdated
| ride. This impression is mostly a byproduct of Python's | ||
| versatility. Once you understand the natural boundaries between each | ||
| packaging solution, you begin to realize that the varied landscape is | ||
| a small price Python programmers pay for using the most balanced, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd tweak the praise of the language in this last sentence to be something unequivocally true. Something like:
... for using one of the most balanced, flexible, and broadly applicable languages available.
(The "one of" does the equivocation work there: "most" is debatable, "one of the most" is not)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Heh, I'd take that debate anyday! But sure, not trying to make any enemies of anyone who read to the bottom of the page :)
…ansion of the modules section.
|
OK, made some changes which I think mostly address the concerns raised. @ncoghlan, regarding your big point, I'm OK expanding the packaging-libraries section somewhat, I just want to avoid having a big technical bump in the middle covering a topic that's really well-covered by links to the rest of the guide. Take a look at the sentences added in 8d3aae2, and if those aren't enough, would you mind writing a short aside (something I can put in a |
source/overview.rst
Outdated
|
|
||
| .. _depending-on-a-new-ecosystem: | ||
|
|
||
| Depending on a new software distribution ecosystem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"new" is still a bit ambiguous here ("new" in general? "new" to you? "new" to your users?).
Perhaps "a new" -> "another"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, meant to fix that one before, fixed now!
|
Just finished adding a few more tweaks based on feedback, and I think in many ways this document is actually a lot more comprehensive and broadly-targeted than the original post, in all the best ways. I think it's safe to say at this point I think it is/I am done :) For the moment, I've got a (my) wedding to attend to, so I don't think I'll have much more time to work on it. I think even for any of its remaining, arguable flaws, having the overview available on the primary site would help a lot more than it would hurt, so I would encourage publishing it in its current state and tweaking it further in the form of future PRs. Thanks again for all the feedback, everyone! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. I agree that this should be good enough to publish as is; we can make any additional changes needed in the near future.
Thanks for working on this @mahmoud! ^.^
We can address any remaining issues in follow-ups.

Following up on @theacodes and my discussion at the PyCon 2018 packaging sprint, this PR brings in a new Overview document, meant to be linked from the packaging.python.org landing page. Also per Thea's instruction, I'm submitting the PR much closer to an "outline" state than a finished document.
This PR also addresses #310, providing much more direct coverage of packaging applications written in Python. This initial PR took a bit longer than expected because there are some intentional changes in direction from the original inspiration:
TODOs for this document before the draft is final:
Some features I was planning for this doc may best be broken out into dedicated articles for each approach, e.g., "Packaging Python Applications with Snappy", just to keep it maximally browsable. Specifically I think a "Packaging Python Applications with Freezers" would be a great place for a table comparing the myriad options.
Anyways, hope you like the new direction, looking forward to continuing work on this!