Help us ironing Packaging
by Tarek Ziadé
packaging has landed in the standard library, but the road to Python 3.3 is still filled with a lot of work. We’ve pushed the Documentation yesterday in the tip, and it now appears here: http://docs.python.org/dev/packaging/
There are a lot of stuff you can do to help us improving packaging. If you wish to help out, read up.
1. Install a Python 3 development environment
The first step is to install a Python development environment
There’s a full dev guide here: http://docs.python.org/devguide/ but it basically boils down to run make on the tip:
$ hg clone https://hg.python.org/cpython (very long) $ cd cpython $ ./configure && make
Once this is done, you’ll have a Python interpreter you can run:
$ ./python
Python 3.3a0 (default:94066c3e2236+, May 31 2011, 08:29:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print('Python 3, yay !')
Python 3, yay !
2. Try out the pysetup script, as an end-user
This script is a global script people will be able to use to check what’s installed on a Python installation, to install things, remove them, etc. The script has still a lot of rough edges, which is a shame since it’s just the tip of a feature-rich system.
It’s located in Tools/script/pysetup3 in a dev environment, and here’s a demonstration of how to install the lastest Mako release, check that it’s installed, look at some of its metadata, then remove it:
$ sudo ./python Tools/scripts/pysetup3 install Mako Checking the installation location... Getting information about 'Mako'... Installing 'mako' 0.4.1... [lots of output] $ ./python Tools/scripts/pysetup3 list SQLAlchemy 0.7.0 at /usr/local/lib/python3.3/site-packages/SQLAlchemy-0.7.0-py3.3.dist-info distribute 0.6.17 at /usr/local/lib/python3.3/site-packages/distribute-0.6.17-py3.3.dist-info Mako 0.4.1 at /usr/local/lib/python3.3/site-packages/Mako-0.4.1-py3.3.dist-info Found 3 projects installed. $ ./python Tools/scripts/pysetup3 metadata Mako -f Version Version: 0.4.1 $ ./python Tools/scripts/pysetup3 metadata Mako -f Author Author: Mike Bayer $ sudo ./python Tools/scripts/pysetup3 remove Mako Removing 'Mako': /usr/local/lib/python3.3/site-packages/mako/parsetree.py [lots of lines] /usr/local/lib/python3.3/site-packages/Mako-0.4.1-py3.3.dist-info/RECORD Proceed (y/n)? y Success: removed 52 files and 2 dirs
So go ahead, play with this script, discover its features and:
- tell us what feels wrong
- tell us what kind of features you wish you had in this script
- found a bug, have a patch, tell us !
3. Make your project packaging-ready, as a developer
The sweet thing is that adding packaging support in your project is risk-free because it’s just adding a few sections in your setup.cfg file. setup.py can stick around, and older installers will still pick it up.
So, here is how you can do: You can tollow the tutorial if you want to do something from scratch: http://docs.python.org/dev/packaging/tutorial.html
Or you can use the magic create command in your Project root directory, to create a setup,cfg out of your setup.py file !
$ sudo ./python Tools/scripts/pysetup3 create A legacy setup.py has been found. Would you like to convert it to a setup.cfg? (y/n) [y]: y Wrote "setup.cfg".
If you’re starting your project from scratch You can also generate a setup.py that will extract the options out of setup.cfg. Very handy to provide backward compatibility and avoid maintaining two files !
$ sudo ./python Tools/scripts/pysetup3 generate-setup The setup.py was generated
So go ahead, learn how setup,cfg works, reads its specs at http://docs.python.org/dev/packaging/setupcfg.html and:
- tell us what feels wrong
- tell us what kind of features you wish you had in this file
- found a bug, have a patch, tell us !
- tell us if you were unable to convert your project
Once the project is packaging ready, you can even register and upload a new version of it at PyPI and check that pysetup knows how to install it
4. Give use some feedback
- You can add new bugs/feature requests at http://bugs.python.org/ under the Distutils2 component, that will be really really helpful.
- You can tell us what’s weird with our documentation, what misses, etc. That goes to the Documentation+Distutils2 components
Fetchez le Python 
Does pysetup do signature checks when the PyPI packages have been signed with GPG?
As the dynamic install story gets better, the same security issues that yum and apt have to deal with are going to arise for pysetup as well.
Not yet, but this should be fairly simple since we’re already looking for checking the mirrors authenticity. Could you please add an issue w/ alexis in copy
Thanks
Hi,
it seems much simpler to package something with pysetup.
sudo feels wrong. Having to install python from source to use it feels wrong. The output of -f on two lines feels wrong. The tutorial seems very short (probably a good thing), it should tell people there is a lot more detail in the packaging documentation and link to it.
Maybe there are some details missing from the documentation still. I could not see how to specify a C extension module or if that is supported at all. Perhaps there also needs to be a section on packaging for older versions of python. I could not find links to example packages to look at.
That is all I could think of.
> sudo feels wrong
This is mandatory when you install something system-wide
> Having to install python from source to use it feels wrong
Python 3.3 is not out yet, that’s why I am asking people to install the dev version. Once it’s out, this won’t be the case anymore
> The tutorial seems very short (probably a good thing), it should tell people there is a lot more detail in the packaging documentation and link to it.
Good idea, I’ll see what we can do
> Maybe there are some details missing from the documentation still. I could not see how to specify a C extension module or if that is supported at all. Perhaps there also needs to be a section on packaging for older versions of python. I could not find links to example packages to look at.
Right, these are missing. Would you mind adding an issue to ask for it.
Thanks a lot for the feedback
>> sudo feels wrong
> This is mandatory when you install something system-wide
Installing system-wide is wrong 🙂 Let’s recommend using –prefix $TMP or –user for testing.
> > sudo feels wrong
> This is mandatory when you install something system-wide
That I can understand, but sudo for creating a setup.cfg/setup.py in the current working directory??? Or was that just an accidental copy and paste from earlier examples?
I had to use `hg clone http://hg.python.org/cpython` instead of https. It was giving me a “network is unreachable” error. Just an FYI, if anyone else hits that problem
same here
a weird, thanks for the heads up, I’ll change the blog entry
Hi. There is one thing annoying me.
With the current python enviroments, I can install packages as a normal user in my user space, and use os.path to use them.
There is any way to do the same thing with pysetup?
I suppose the answer is yes and I am missing something in the docs.
The options do exist when installing a single package wth “pysetup run install_dist”, but not with “pysetup install”. We need to expose them. would you mind adding a bug for this, with alexis in cc ?
Thanks a lot
I noticed that in the documentation there isn’t any mention of namespace packages. Is this intentional?
There’s a PEP for this feature but that was not implemented yet. It will be a separated stdlib feature, hipefully for python 3.3
I have been quite thoroughly burned by pkgutil’s and setuptools’ implementation of namespace packages in the past, because they are so inconsistently supported.
You simply do not know what is going to happen when you try to import a package in a namespace, unless you had control over the installation of *every* package in that namespace, and even then external tools like PyDev and PyInstaller might not know what’s going on.
PEP 382 sounds, to me, like a very naive solution. It doesn’t deal with many of the installation issues and inconsistencies they have created and will create, and it just flat out says that third-party tools will have to adapt. They probably won’t.
Tarek, your work on packaging has been excellent and I don’t want you to ever have to mess it up with namespace packages. I don’t see it as a flaw at all that packaging doesn’t support namespace packages, when realistically the Python language doesn’t either.
I’d encourage Luca to rename his/her packages to replace the period after the namespace with an underscore. It’s not as pretty-looking, but at least it’s universally understandable.
Hi. Congratulation on merging all this work.
I’m interested in the fate of the old distutils. I looked through the mailing lists, but couldn’t find any statement on this topic. Will it remain in the stdlib in a frozen state or is there a plan to remove it at some point?
This command will not pull 2.x changesets:
$ hg clone http://hg.python.org/cpython#default
Rob: (don’t know why the blog doesn’t give a reply button to your message)
> You simply do not know what is going to happen when
> you try to import a package in a namespace
Import semantics are a bit of a mess, but there are PEPs and docs out there, and the work in importlib aims to make the implementation more robust.
> PEP 382 sounds, to me, like a very naive solution. It
> doesn’t deal with many of the installation issues and
> inconsistencies they have created and will create,
Please share your comments on the python-dev mailing list.
> I don’t see it as a flaw at all that packaging
> doesn’t support namespace packages, when
> realistically the Python language doesn’t either.
Well, PEP 383 is about adding namespace packages to Python, so packaging will have to get support for them.
382*
A brave soul could investigate why test_packaging is still leaking with -R ::.
I’ll look into it, thanks to http://blog.ccpgames.com/kristjan/2010/12/08/finding-c-reference-leaks-using-the-gc-module/ 🙂
[…] the Original article Fatal error: Call to undefined function related_posts() in […]
Cool!
But you’ll get more real-world feedback from the Python 2.x world, won’t you? The plan is to help people transition towards this, right?
Is it possible to supply external .cfg files for existing Python packages? That should make it possible to install a lot of existing packages without having to go and modify them right away. That would make a large-scale real world test in Python 2.x land possible.
It should also be possible to share such external setup.cfg files on pypi as well. And it should be easy for the original package author to find them so they can include them into their package.
And if you want to make overrides possible, it should be possible to specify which one you prefer: the one from the package itself or an overridden version.
Does the new packaging stuff have a good API so that tools such as buildout can be integrated with them? I know that the integration with setuptools (especially the easy_install part) required quite a bit of hacking.
> But you’ll get more real-world feedback from the
> Python 2.x world, won’t you? The plan is to help
> people transition towards this, right?
Yes. We will backport packaging to Python 2.x, under the name distutils, so that it can be used by all projects out there. I’m also working on a migration howto.
> Is it possible to supply external .cfg files for
> existing Python packages? That should make it
> possible to install a lot of existing packages
> without having to go and modify them right away.
Yes, it’s a goal.
> It should also be possible to share such external
> setup.cfg files on pypi as well.
That too!
> And it should be easy for the original package author
> to find them so they can include them into their
> package.
Good idea.
> Does the new packaging stuff have a good API so that
> tools such as buildout can be integrated with them?
Yes, one of the goals of packaging is to offer building blocks for other packaging tools, independently of the built-in pysetup installer and the command machinery. See the reference documentation at http://docs.python.org/dev/library/packaging
needs update:
$ sudo ./python Tools/scripts/pysetup3 install Mako
installing third-party projects from an uninstalled Python is not supported