The strange world of packaging – forking setuptools
by Tarek Ziadé
Again, like a year ago, people had enough of the fact that the setuptools project is not maintained since 9 months.
Phillip Eby explained that he doesn’t have time to do it unless someone would pay him for that. But in the meantime, he doesn’t bless anyone to do it. Well, he has blessed some people to do it (Ian Bicking and Jim Fulton), but unfortunately these people are not willing to do it because they have a lot of other projects going on. Other people that could maintain it, including me, fail in his “unqualified people” category 🙂
So we are all locked in a strange situation where tons of patches are ready to be commited in the setuptools tracker but are not making it. Several non-public forks have started to appear around of course.
So again, I decided with some other people to create a fork called “Distribute”. It’s a real fork located here : http://bitbucket.org/tarek/distribute.
By real I mean that this fork was not created with the purpose of forcing Phillip to do a release like we did last year for the 0.6c9 release, but with the intention to free us from that strange situation where we all depend on his wills and (lack of) time.
The plan is to release a first version next week, that corresponds to the setuptools 0.6 branch, with some patches applied.
Next, we are planning to start a 0.7 version where the code will be splitted in several distributions:
- a distribution for pkg_resources
- a distribution for the setuptools package itself
- a distribution for easy_install
A little bit of bikeshedding is going on to pick a name for that fork, and we ended up running a poll. (vote!)
Now, right after I have announced this plan on Distutils-SIG, Phillip reacted by annoucing a similar plan, e.g. splitting the setuptools project in several distributions. But since he previously said that he didn’t have the time to do it, I doubt that it’ll work out unless he’s opening its development to a wider range of developers and maintainers.
That’s the strange world of packaging…
Yeah, I don’t know what it is with packaging that makes people so fantastically angry and stubborn… Very strange.
Good luck! Breaking the package into three is a very good idea — the concept of building and publishing a package is a very different one than the mechanics of installation, and I am glad to hear that those building blocks are now going to be split out separately.
As you know, I am in over in the make-things-much-simpler minority who would do crazy things like enforce one-package-per-project rules on all new submissions to PyPI (while understanding that old “packages” that actually contain several Python packages need to be grandfathered in), which would begin to make PyPI an actual “package” index (which it’s not today), and go for the very simplest versioning scheme we can conceive of. So, of course, I won’t get everything that I want. 🙂 But this community move that you are coordinating will put us in a great position to wring from the current situation what improvements we can, while making much clearer building blocks for the future. Bravo!
Good luck Tarek and thanks for all your efforts in this area.
PJE’s actions (or non-actions) amount to a refusal to fix setuptools for the latest subversion format change. Despite patches being available for this embarrassing issue.
That alone gives you the right to fork.
I’m looking forward to a brighter setuptools/distribute situation when I get back from my holiday 🙂
Thanks Tarek! KEEP UP THE GREAT WORK you and all the other guys that together decided to FINALLY change this embarrassing situation! For what is worth you have all my support…
WOW, guys I’m reading some of the (many) emails I’ve not read during the weekend after you’ve announced the fork, just one thing: DON’T LET ALL THE STOP ENERGY PJE IS THROWING AT YOU STOP WHAT YOU ARE DOING!
Damn, I’m astonished by his behavior, it’s really embarrassing.
Thanks again!
Hmmm, I find this all rather amusing. 🙂
This is because the WSGI specification is in some respects also affected by the PJE limbo force.
Specifically, although there are various parties who would like to see at least amendments made to the existing WSGI 1.0 specification to bring it in line with actual practice, the Python WEB-SIG as a group never seems to get together and do anything about it. I see this as being in part due to PJE being the original author and no one wanting to go and make the changes without PJE’s blessing, which has never been given. It also seems that when it looks like something might happen, he steps in and says just enough to cause any action to be forestalled. Thus it is stuck in limbo and nothing moves forward. The problem is that since WSGI is a specification, and so much relies on it, is somewhat harder to fork it than an implementation.
Anyway, this can’t go on indefinitely because the WSGI specification doesn’t really cover Python 3.0. Frankly, I think it will probably need to come to a point where Guido himself steps in and says ‘enough is enough’.
how does forking something as broken as setuptools do more good than going and fixing/enhancing more nice things like pip and virtualenv
i have never seen anything pje did that didn’t deserve the term horrible
i dont see how people can keep up with seeing him as the “owner that should give his ok” and not be completely ashamed
Good luck on the fork; I’m very happy about this move to make setuptools a community-run project.
@RonnyPfannschmidt: pip/virtualenv are based on setuptools so you need to fix/evolve it TOO to move things forward…
@Graham: yep, you’re 110% right, it’s the same embarrassing situation. While in ruby land they created Rack taking inspiration from WSGI (but done right, without start_response) and it gained BIG traction on every framework and with a myriad of middlewares, in python we’re stuck with WSGI 1.0 (where writing a *good behaving middleware* without breaking things is really hard http://svn.repoze.org/repoze.retry/trunk/CHANGES.txt), now Guido stepping up would be wonderful, but let me say that IMHO you’re a big expert (if not the biggest) on WSGI things (its strengths and shortcomings) so you AT LEAST FOR ME could do it too! 😉
@michele uh, not really, pip only uses the version string parsing, setuptools only installs it as default packages installer since its what people are used
its very orthogonal and replacable in those places
@RonnyPfannschmidt: wrong 😉
I suggest your read this explanation by Ian on the subject:
http://blog.ianbicking.org/2008/12/14/a-few-corrections-to-on-packaging/
On pip he says:
– This is an alternative to easy_install
– It uses Setuptools to generate this metadata from a setup.py file, and uses pkg_resources to parse this metadata. It then installs packages with the setuptools monkeypatches applied
– Pip installs eggs!
– Pip is not a repudiation of Setuptools or the basic mechanisms that easy_install uses
So, it is NOT orthogonal.
@Lennart : I think this is due to the fact that the topic is very wide. It’s easy to get frustrated because we don’t understand each other, since we all may have different experiences in the domain.
@Brandon: Thanks for cheering ! Yes I think one of the mistake of setuptools was that it had too much features in one single distribution. If they’ve had been separated from the beginning, things would have been better. Now with starting at Distribute 0.7, we will have at least three separated packages.
I am not found of the one-package-per-project rule though, because it makes it harder to reuse and share packages among projects, dependency-wise. I think the best approach to group you code for your application is somewhere between one single package ala Django, and the “craziness” we can see in the Zope/Plone world, where you can have like 200+ packages in one running application, ala-Perl. (No criticism here, I konda like that 🙂 )
@Michael, michele, Martijn : thanks for cheering guys !
@Graham, Reinout, RonnyPfannschmidt: Notice that I have a lot of respect with what PJE achieved in the packaging world. He solved a lot of problems people had with packaging and brought Python to a new dimension in that area. And the fact that people are angry because the project is not maintained proves how
that it is used a *lot* out there. I am just doing what I feel (and others too) has to be done to make packaging move forward again.
Imho a community-driven project is just what we miss on the top of distutils to make the innovation in that area go way faster. I’ll also write a blog asap about what I think “community-driven” will mean in Distribute.
@Tarek: for all of my projects i stopped to use setuptools as it broke some of them in various aspects
for me setuptools mostly meant new dimensions of pain, cause it’s certainly badly broken in various cases
if you manage to extract the few good things in a reasonable way and give people a reasonable entrypoint system,
bags of bags of kudos to you
FYI, I did not propose a similar plan. I proposed doing something that I thought would help you implement YOUR plan.
Thank you! This is just what we needed and it seem like a new logical step for your pypi work. As others have said the core ideas of setuptools are (mostly great) and I totally agree the project has grown beyond the “single author software” specially because of the fact that no one is a master of everything and packages are exactly that. They are too wide.
[…] Forking setuptools by Tarek Ziade […]