Do not upload dev releases at PyPI
by Tarek Ziadé
The discussion we had on the SQLAlchemy mailing list, triggered by the release of 0.7.1b, made me realize that we need a heads-up on this problem. EDIT: a similar problem occurred for psycopg2 today.
Until our packaging ecosystem knows how to handle properly development releases, the best practice for mature projects is to avoid publishing anything that is not a final release at PyPI (or any download link that points to a development release)
By development releases, as opposed to final releases, I include:
- any release that is a snapshot of the trunk or tip of the project
- any alpha, beta, or release candidate
And by mature project, I mean any project that already published one stable release at PyPI.
The reason is that our current set of packaging tools do not know how to make a difference between a final release and a development release.
Setuptools’ easy_install script will scan the simple index page, order the releases number via its version sorting algorithm, then take the “latest” version.
So calling:
$ easy_install Foo
will install the latest uploaded release for the Foo project, even if it’s a development version. If you use Setuptools’ install_requires option in your setup.py, the same thing will happen.
To prevent it, you can tell the tool which version you want, and even provide a complex condition, like:
$ easy_install SQLAlchemy >=0.5, <0.7
But there’s no way to tell it to get the latest final.
zc.buildout is the only tool in my knowledge that prevents this, with the prefer-final option.
How Distutils2 and PEP 386 fixes the problem
But uploading release candidates is a great way to get feedback from the community, and it’s currently frustrating not to be able to push development versions at PyPI. Because depending on the installers our users will use, they might be unable to control if they want to opt in using non final releases.
And, well, what’s a final release ? what’s a beta ? Are we sure all tools agree on the versionning schemes ?
How can I make sure my beta version will be recognized as a beta release by all tools out there ?
PEP 386 solves this by defining a version scheme.
And guess what:
- The new Metadata 1.2 – PEP 345, implemented in Distutils2, recognizes only PEP 386 versions
- PyPI will reject any project that uploads metadata 1.2 with a non-PEP 386 version. And this is already activated at PyPI.
- The Distutils2 installer recognizes development releases and let you decide which one to pick.
Granted, it’s going to take a while before all installers use the new standards, and all projects out there make the jump to Metadata 1.2, but at least we know the problem and implemented the solution..
And Pycon should be an important milestone since the first beta of Distutils2 will be released at… PyPI and will be usable by any project.
Fetchez le Python 
zc.buildout provides a prefer-final setting that helps with this (to default to true in some future release), but I’ve never seen development releases ever prove anything but painful.
I agree this is something that release managers should be cognizant of; there’s just no good reason to publish non-final releases on PyPI.
Yeah I was mentioning prefer-final, I think it’s great to have it. I am wondering how hard it would be to push it into Distribute as well.
Now, I can see one good reason to push non-final releases to PyPI: that’s an easy media to publish at.
You can upload it there and tell people to “pip install” or “easy_install” that particular version, which is quite convenient imho.
Uploading any type of release to PyPi is just convenient to make it available for people to test and try out. And what one project might call a 0.9.15 release, another might call 1.0b10 and yet another call it 3.2. In the end each developer using a project needs to pick one exact version of the project, download and test it and then stick to it.
If you allow any of your installers to pick random or latest versions for you, you shouldn’t expect to get anything stable or working. PyPi is the “unstable” or “world” equivalent of packaging indexes. If you want a stable set of software that is tested together, you need to built something on top of it. In the Zope/Plone communities we do this by having “known good sets” – which serve a similar purpose as Linux distribution releases. PyPi isn’t the right place or tool to maintain the kind of “works with” or compatibility information – or at least not yet.
> If you allow any of your installers to
> pick random or latest versions for you,
> you shouldn’t expect to get anything stable or working.
That’s unfortunately how things are today. Most software documentation that proposes to use easy_install to install an app will say:
$ easy_install -U Foo
Implicitly meaning “Pick the latest stable version”
That’s the end-user Pov.
I think Plone and equivalent beasts are a bit different. What you are releasing is like a “software appliance”, a meta-package, where you define precisely all your dependencies e.g. your known good state. And your customer is a developer, not an end-user.
An end-user will not choose the right versions for the libs Plone uses. It will just ask for the latest Plone and you have done the pinning work before he gets it.
So if you’d offer “easy_install -U Plone” you would have the same problem. You’d need the latest Plone version at PyPI to be a stable.
So, for any end-user project, or for libraries like SQLAlchemy, I don’t think it bring any value with our current ecosystem to release non final releases to PyPI. Just problems.
> $ easy_install -U Foo
> Implicitly meaning “Pick the latest stable version”
I don’t know why you get that implication. Rather, the implication is “Upgrade/Install to the latest version at PyPI”.
Leaving beta releases off of pypi isn’t enough, you also need to ensure that your pypi page doesn’t link to a page from which the beta release is available. Remember that easy_install follows links…
Right. and it just happened to psycopg2: http://mail.python.org/pipermail/distutils-sig/2011-February/017355.html
It’s great that distribute is fixing this, but wouldn’t it be plausible to fix for all versions of Python packaging tools by making a prerelease.pypi.org staging area, which could be explicitly selected / enabled?
Sorry, that distutils2 is fixing this. It’s hard to keep track of the latest and greatest packaging whatever, since the name changes every week ;-).
Oh… that’s a very good idea ! I’ll bring this up at Catalog SIG
easy_install SQLAlchemy >=0.5, <0.7 does not even help here since 0.7b1 is considered to be <0.7.
People need to learn how to use their tools better (or use a better tool and there is of course lots of room for making the tools easier). Presuming that there are some incompatibilities between SQLAlchemy 0.6.* and 0.7.*, holding back on not releasing to PyPI until 0.7 final is just going to delay the point until people complain that easy_install is giving them the “wrong version that doesn’t work with their code”.
I have the same problem with releasing 3rd party Plone packages, where people want to just get the “correct” version and if the package is geared for Plone 4 and they are on Plone 3, they complain that it’s “broken”. These user’s are even already using Buildout, which allows them to easily specify the version they need, but people tend to want things to “just work” and when it doesn’t they’ve heard “python packaging is a mess” and they blame that.
However, being able to tell your tool to “install alpha quality”, “install beta quality” or “install final” based on PEP 386 version numbering will be a beautiful thing though!
I’ve hacked together a pipi mirror/proxy that includes only final releases: http://pypi-stable.maluke.com/
For more info see https://groups.google.com/d/msg/paste-users/1UAuadQjKng/qxVuF6lNU6QJ
So, what happens when you need to target a development version in the requirements list in setup.py if you can’t publish development versions on pypi? I found that you can use something like the following:
install_requires = [
‘AuthBWC ==dev, >0.1.9’,
],
More details here:
http://rsyring.blogspot.com/2012/01/targeting-development-versions-of.html