Distribute, end of the fork — or the start of a new hope
by Tarek Ziadé
(This post is a work in progress, as things still evolve)
In a way, I am glad I have made a fork, and I am glad this is going to be the shortest fork ever. A lot of people reacted on my proposal and I could get a very clear picture of what is wrong in the distutils/setuptools world, and how I could help on this.
This is how I interpret it:
Guido explained that the mistake made was not to integrate setuptools in the core from the very begining. The current distutils code did not evolve in the meantime and Phillip Eby worked on setuptools to make it evolve. setuptools now reaches a point where it needs other contributors to fit all the needs and solve more problems. It had one contributor in the past, (Jim Fulton) but it needs more. It also became a mandatory package for many folks in the Python world because Phillip did a great work on it : it solves most problems.
From what I could see, all the people that gets frustrated at some point with setuptools, either don’t use it, either try to see how it could be changed. But when they try that, they take the same paths others have taken before because there is a lack of info and documentation on setuptools current status and future. And these paths are quite long for the brain before you are able to provide a well-thaught idea or patch.
I have tried to help myself on setuptools, then I have started to blame Phillip saying that he did not have enough time to make setuptools evolve. But that was the wrong approach, because the only problem really, is a lack of visibility in setuptools development. And this is something that can be fixed quite easily I believe. The bug tracker added some months ago helped a lot. A roadmap and one or two page around it and that should be it.
Guido also stated that it was merely impossible to work on the next generation of distutils outside Python core. But in the meantime this means that we need some people from the core to help us in there. And they are really busy (I can understand that) on other matters in Python code. We can write patches for some fixes for sure, and there are hundreds of them to do in distutils.
But to boost it we would need a core developer that gives some love to that package. I mean, for example I still have a patch waiting for reviewing, wich only adds some unit tests that are lacking. There is no code there, just unit tests, and it is pending since 6 months… Guido said that I could try to become a core developer, that this was not impossible, if I started to contribute patches often to other parts of Python as well. But that’s quite a challenge.
What frustrated me is that some people like Jim Fulton said they were willing to work on a new distutils from scratch, and in the meantime I understand what Guido says. he pointed us to Joel entry on rewriting from scratch.
My Plans
I don’t have the brain to drive a fork of setuptools at this time. I knew it from the beginning, I just wanted to make people react. I think that setuptools can be the laboratory where we can experiment things, and distutils the place where we can apply them at some point, with Phillip help because he has a pretty big overview of all that. It is going to take years, but well, we are young.
To help on this I will:
- try to gather people at the PloneConf to talk about it, maybe even sprint
- try to document setuptools and distutils at my level, and write a roadmap for people to know what is going on. Either through a Sphinx site either on Python.org wiki.
- work on patches for distutils, and try to point things in setuptools that might be brought into distutils
- try to see if I have the brain to understand and work on other parts of Python.
Fetchez le Python 
[…] Posted in plone, python, zope tagged plone, setuptools at 1:10 pm by Tarek Ziadé EDIT: Read this new entry […]
Any chance we could get setuptools moved somewhere that we could have ReviewBoard (or something similar) for code review? I’ve found recently that good code review tools can go a long way towards keeping a project active.
@Alex: This apps looks nice ! otho, the current roundup here http://bugs.python.org/setuptools/ is still being used. But maybe we could suggest ReviewBoard if it speeds up the reviewing work.
I will ask PJE what he thinks about moving to a dvcs-based system too
I honestly don’t think the problem is in the lack of reviews and co.
Sure, it may help a bit to have DVCS, reviewboard and whatnot, but the real problem is that core issues with setuptools are distutils issues, and nobody wants to touch that code, because distutils is a POS. It cannot be refactored (honestly, it would already have been otherwise): any function change can breaks something. The reference to ‘never rewrite from scratch’ is not really applicable here IMHO: first, distutils code is small (10000loc), but horrible. There is no defined API: only the setup.py is somewhat defined, but internally, everything is spaghetti code and extremely fragile. So the argument that it would take less time to refactor than rewrite from scratch is not valid.
I think Guido suggestion to move things gradually from distutils is a something to think about: for example, adding a new distutils command which does nothing by itself but gathering options from distutils, and passing options to a totally separate module. That’s what I have done with numscons, and after having thought a bit about it, I believe this could well be applicable in general, if it is sanctified in python core quickly. It could first be something really simple, but extensible and well defined.
@david: I tend to agree. And the numscons approach seem wise.
In the meantime, some people didn’t even know where was setuptools tracker. On the Mailing list, people reported the same bug 5 or 6 times (the import error due to svn 1.5). I have asked pje to add it in the pypi page, what he did on 0.6c9. That is a first step, so anything that can make the development process more visible is a good thing.
As an experiment, you could embark on a refactoring of Distutils without worrying about what breaks, and then see what actually breaks. Yes, it’s certainly true that any change might break some users, but it’s hard to tell if the number of such users is significant.
Also, it would be interesting to hear what people would do if they started with a blank sheet of paper. Forget about existing Distutils; forget about compatibility. If you (the non-specific ‘you’) were trying to solve package installation for the first time today, what would you do?
@amk: Yes, I think some people are starting to claim that in the long thread at distutils.
For the other point I think I might try to organize some kind of BoF ar the Plone Conf, trying to think about a system from scratch. It will help at least, to list what could be done from some people point of view.
Hi Tarek!
I’ve been following your blog postings about distutils and setuptools for some time now. I understand your frustration but I hope you also understand that Python 3.0 is taking up most of our resources. We are working on two releases at the same time and we are spread thin on developer time.
Also your patches came too late to include them in the upcoming releases. A year ago you’d have the change to make a difference. Python is more conservative about changes than e.g. Plone. (Just ask Limi about the Plone 1.1 debacle *g*)
I suggest you:
* get together with everybody who is working on, complaining about and/or writing a substitute for setuptools and distutils. Find common goals! Collect real world use cases!
* Write a PEP about how distutils should be enhanced.
* Write another PEP about setuptools integration into distutils.
* Write unit tests for distutils to make sure you won’t break it. Distutils has a low test coverage.
* Implement your PEPs
I’m sure you can get commit privileges for the core if you can provide us with a good PEP. I’m going to champion you.
Hi Christian, thanks a lot for your support !
It’s funny: while you were writing this comment, I was just working on a first PEP to improve some parts of distutils.
I’ll follow your advice and try to come up with some detailed PEPs and to get people together to find common goals.
hehe!
Before you start with your PEPs please start a discussion on the Python ideas mailing list. Ask your fellow developers what they like about distutils and setuptools, what they dislike and which features they are missing.
The task is so large and complex that you are going to need all help you can get. You may even have to write a meta PEP.
@Christian: Yes I am going a bit too fast to write down some PEPs. I am just afraid that nothing comes out of the discussion that has been going on on distutils-SIG in the thread I have initiated.
PJE and Ian Bicking have made some relevants points in that thread though and maybe people will find some consensus naturally and we will be able to come up with some PEPs there.
[…] The distribute project didn’t last long, as it’s already been declared officially dead. […]