5 tips for packaging your Python projects
by Tarek Ziadé
Next week I am keynoting at Pycon Japan, and one thing I will talk about is packaging of course. And in particular: what advice can I give my audience on how to package Python projects today ?
This is a hard task, because we are in some kind of transitional state.
Anyways, I wrote down a list of advices and removed everything that was dependent on the tools we did not release yet — that’s another part in my keynote.
Here’s a list. Most of them are not controversial. If you see something missing or want to rant about one, please comment.
Tip # 1 — Use a PEP 386 compatible scheme for your versions
Having several version scheme in our eco-system is pure madness. It breaks interoperability, and makes it impossible to write tools that handle versions properly. By using a PEP 386-friendly scheme now, you are making your project future-proof !
PyPI already rejects any Metadata 1.2 project that does not comply to this policy. You probably don’t know this because no tools produces Metadata 1.2 packages yet. But that’s going to be the default in Python 3.3 and distutils2.
So long “devdevdev123” and “3765-2011-test” versions !
Tip #2 — try to make setup.py as dumb and simple as possible
setup.py is not your personal build system. I have seen crazy things in some projects. Remember that setup.py is used by installers for a lot of different tasks. Like getting the metadata fields of the project.
Here’s a simple test: make sure “python setup.py –name” (double dash) eturns the name field without any external dependency, and without calling any function or method.
Remember that setup.py is going away in Python 3.3 and distutils2, replaced by simple options in setup.cfg. Don’t be scared, you will still able to do complex tasks.
My advice: don’t do anything else that feeding setup() with options in there. Put all your build things in another place, and if they need to be called by setup.py, make sure they are called only when needed.
Tip #3 — Do not make any assumption about which installer will be used
Make sure your setup.py can be run by a vanilla Python (==distutils). Even if you use setuptools or distribute, in most case you can manage to have it working in both tools. You can always tell the user to do extra steps manually if he needs to.
Forcing the installation of an installer, by using the ez_setup script for instance, without asking, is a bit rude to the end-user. It’s basically forcing the end user to use a new installer. If you do this in your setup.py, ask first !
Or simply tell the user “This project only works with the XXX installer — install it if you want. Aborting.”
Tip #4 — Do not release unstable releases at pypi
Our installers are not —yet— smart enough to prefer stable releases when they are asked to get a project at PyPI. That’s how PyPI is built: every project has a directory with all releases and it’s up to the installer to decide which one is the “latest”. The only tool out there that’s smart about it is zc.buildout.
So when you push an alpha release or a rc release at PyPI, it’s going to land in people environments unless they have mature processes to update their stuff — or simply because they make the assumption that PyPI is where stable release go. So do not make assumptions about how your users are updating your project.
Prefer another explicit channel for your beta testers. All installers know how to install from any url or directory.
Tip #5 — Be cautious about your data files
Distutils or Distribute or Python itself have no way to explicitly make a difference between a doc file or a media file or a configuration file. They are all data files. Worse, since they are no universal place for data files on the various OSes, people tend to treat their data files like Python modules so they are able to find them back on the target system without trouble.
Yeah that’s broken, and we’ve fixed it in 3.3. But until then, that’s unfortunately the most protable way to do this. So what you can do is document clearly how you handle your data files and create a single function or module that reads them. That’ll help the downstream maintainers to handle your project.
I’m not really sure what’s the best way to follow #2. I have my version in my module, so I import it in setup.py to set `version=mod.__version__`.
It’s probably bad for projects with dependencies, since there is possibility that you need to install dependency before setup.py has ability to installed, but… what should I do? I have zero desire writing version in two different files.
How do you recommend reconciling #2 with useful features that live in setuptools (or distribute) such as “python setup.py test” or even just the convenience of not needing a MANIFEST.in? Seems like a bit of a step backwards from 2005 or whenever it was when setuptools became commonplace.
@Alexander — your __version__ can probably be isolated in a package __init__ module, thus avoiding any dependency.
@Bob — It’s quite simple to run all tests on any project using Nose or unittest2 discovery script, and in fact that does not really impact setup.py, In other words, as long as you keep your tests written in a standard fashion they can be discovered and run by any test runner out there, be it setuptools’ test command or another runner. The only thing we don’t deal with is test dependencies. This topic is a bit fuzzy still.
Btw, “test” has been added in Python 3.3 and distutils2, and will be easy to configure to pick your preferred runner. We want people to do “pysetup -c test” on any project — with no setup.py involved.
For the MANIFEST.in I am not sure what you are using, but I don’t think Setuptools options are any better.
For instance, making setup.py rely on the VCS information (.svn or .hg) to collect what gets into the release is quite hackish — you can invoke hg manifest > MANIFEST to do this and not make your setup.py anymore complex. — e.g. it’s a pre-built step
We have improved the story to do this in 3.3 : in setup.cfg you can define what gets into the release, in a fine-grained way.
Now if you setup.py has a loop to collect file, I guess that still makes it dumb enough 🙂
All in all, setup.py has became this place where we do too many things: the build steps, how we install the project, what it contains etc..
So for all the things you have mentioned I think there are ways to avoid building a super-setup.py that is a build system, an installer and a metadata holder. I’d be happy to build a FAQ if you think that can be helpfull.
Moving the metadata to setup.cfg, “killing” setup.py, and let people use pre- and post- command hooks is what we intend to provide in 3.3. setup.cfg will even be published at PyPI making it possible for installer to introspect dependencies etc (look at 345)
I should say I started with one question in mind and finished with completely different one. 🙂 If we have setup.cfg instead of setup.py, how do I import my module and put it’s version as version in setup.py? 🙂 Or if it’s in some file on filesystem, right now I can just do `file(‘version’).read()`.
Though looking at your comment – probably through pre- hook everything’s possible. 🙂
Look at : http://www.python.org/dev/peps/pep-0396/#distutils2
Thanks Tarek, great suggestions. I know that specifying requirements is not standard yet but IMHO the best thing to do today is to make a requirements.txt file that lists each module per line and maybe has hash comments (like Python code). This can be fed into pip and without pip it gives users a readable text file to easily find and install the requirements. Maybe you have another recommendation for something that will transition better into distutils2.
Yeah that can’t hurt. Distutils2 lists them in the setup.cfg file
There is a small typo in Tip #2:
python setup.py -name
should be:
python setup.py –name
(double-dash)
fixing, thx
mmm it’s a double-dash, the font seems to merge it though
It’s nice that “setup.py test” is in distutils2 / Python 3.3… but we shouldn’t use distutils2, right?
With regard to the VCS stuff, sure, setuptools doesn’t quite do the right thing… but it’s convenient. Why doesn’t it generate a MANIFEST for sdist tarballs? I don’t know. You’re probably right that we should just a generate MANIFEST, but it’s annoying to have to do this manually. There’s no “git manifest” and most people are using git these days (including myself).
I agree that we do too many things with setup.py, it was never a great idea to do it this way, but it doesn’t seem like there’s a sensible alternative for open source packages that support Python 2.x.
Is there some way to use setup.cfg and setup.py, where the setup.py is simply a stub that works with the data from setup.cfg? This seems like it could be useful for transition.
> It’s nice that “setup.py test” is in distutils2 / Python > 3.3… but we shouldn’t use distutils2, right?
> With regard to the VCS stuff, sure, setuptools
> doesn’t quite do the right thing… but it’s
> convenient. Why doesn’t it generate a MANIFEST > for sdist tarballs? I don’t know.
But setuptools installer will correctly install a distutils project that has the MANIFEST
> You’re probably right that we should just a
> generate MANIFEST, but it’s annoying to have
> to do this manually. There’s no “git manifest”
> and most people are using git these days (including myself).
> I agree that we do too many things with setup.py,
> it was never a great idea to do it this way, but
> it doesn’t seem like there’s a sensible
> alternative for open source packages that support Python 2.x.
I think some features like this one can live without a dependency on setup.py files. It should be possible to write a script that does:
1/ collect the VCS-ed files to build the manifest
2/ run python setup.py sdist
and work for all projects.
> Is there some way to use setup.cfg and setup.py,
> where the setup.py is simply a stub that works with
> the data from setup.cfg?
> This seems like it could be useful for transition.
Yeah we provide this: http://hg.python.org/cpython/file/b09d07d1f696/Lib/packaging/util.py#l1122
A wizard generates a standalone setup.py that is a wrapper around setup.cfg
It’s going to be available when distutils2 is out and can be used (it’s documented)
That’s helpful to provide d2 support and keep d1, setuptools support without duplicating metadata
[…] a hard task, because we are in some kind of transitional state. Anyways, I wrote down a list of… Read more… Categories: Python Share | Related […]
This script is awfully convenient, it can automatically install man pages, translations, ui files, and GNOME help pages.
https://launchpad.net/python-distutils-extra
I think:
s/This script is awfully convenient/ those commands are awfully convenient/
This is not going away ! You’ll still be able to run commands, create new ones. It’s also going to be easier to manage the set of commands (ala Mercurial plugins)
PEP 386 versioning is pretty close in line with ‘Semantic Versioning’ at http://semver.org/. I encourage everyone to follow that – not only for keeping uniform release numbers, but also ensuring that they have meaning.
I’m still sometimes a bit confused concerning how to do certain things. Even basic things are a bit confusing like how do you make a 1 file pypi module without putting everything in __init__.py? Is there an exemplary, up-to-date best practices, package out there?
the minimal project is one setup.py with one python module. The Python module is declared in the py_modules option.
As for the best practice guide, we’ve started to write content in 3.3 doc. Also look at http://guide.python-distribute.org/
Tarek, are we able to take advantage of the (great!) work you are doing in the “packaging” project for python 3.3 now? I’d like to start using setup.cfg and pysetup right now, but I am stuck on Python 2.7.2 because of dependencies on other packages that are not yet 3.0 compliant.
I’ve read up a bit on hitchhikers guide to packaging, etc. and know that you are doing a lot of backports to distutils2, but when I install distutils2 (via pip), I don’t see pysetup installed into Python-2.7.2/bin. I also can’t seem to find it in distutils2 site-packages section.
Am I missing something obvious, or is it just not available for 2.7.2 yet?
I’d really like to get involved here and the first step is playing around with the alpha release of distutils2.
Thanks so much for all your hard work, I think this is really important stuff for moving the language forward!
–Dennis