Thanks to the sprinters in Washington D.C., (Andrew who worked on merging it, Alex, etc.) and people that helped in the discussion (Fred, Martin), my patch to Python for multiple servers handling in .pypirc file has been integrated in 2.6 trunk.

This patch will allow using distutils register and upload commands with several servers, using the -r option and storing for each one of them the appropriate username/password in the configuration file. See http://wiki.python.org/moin/EnhancedPyPI for more details.

The collective.dist package provides the same feature for Python 2.4 and 2.5, through new commands: mregister and mupload.

The new format is also pretty convenient to store several profiles per servers. For instance, if you have several accounts on PyPI, one for your company (acme) and one for your own usage, you can define them like this:

[distutils]
index-servers =
    pypi
    acme

[pypi]
username:user
password:password

[acme]
username:acme_user
password:password

Then use them depending on the package you work in:

$ python setup.py register -r acme
$ python setup.py register    # default, which is pypi

When plone.org will go Plone 3 (this is happening soon), the pypirc file can be extended with this new target:

[distutils]
index-servers =
    pypi
    acme
    plone

[pypi]
username:user
password:password
[acme]
username:acme_user
password:password
[plone]
repository:http://plone.org/products
username:plone_user
password:password

From there, setuptools aliases can be used in each package to simplify things:

$ python setup.py alias plone_org register -r plone sdist bdist_egg upload -r plone

Which will allow you to upload the package to the website in one simple command:

$ python setup.py plone_org

The patch aslo corrects a few minor bugs in distutils, such as:

• .pypirc was not found on Windows
• pydistutils.cfg was not found on Windows

It is interesting to look at the usage statistics at PyPI: http://pypi.python.org/webstats/usage_200805.html#DAYSTATS

Besides the front page, the most visited pages are in order (same thing in April):

• zc.buildout
• setuptools
• zc.recipe.egg
• plone.recipe.zope2instance
• plone.recipe.distros
• python-openid
• elementtree
• plone.recipe.zope2install

Then the packages in the top 60 are mainly zope eggs. This corresponds to the sequence done by a Plone or Zope being build in a buildout.

If you take a closer look at http://pypi.python.org/webstats/url_200804.html, the  next top hits  corresponds to zope.* then plone.* packages.

Pycon FR will take place in Paris, may 17/18, at the Cité des sciences et de la Villette.

A very rich program will be presented:

• Learn Python first ! (Eric LEBIGOT)
• Graphviz made easier with GvGen (Sebastien Tricaud)
• Genes research with Python (Andre Espaze)
• CouchDB (Benoit Chesneau)
• PyPy explained (Victor Stinner)
• Quality Assurance (Julien Jehannet)
• Zope 3 overview (Christophe Combelles)
• Scapy in action (Renaud Lifchitz)
• E-sonoclaste, multimedia annotations (Vincent Rioux)
• Why Django ? (David Larlet)
• PLUIE project (Michel Claveau)
• WSGI in practice using Paste (Gael Pasgrimaud)
• Distributed Version Control with Mercurial (Michael SCHERER)
• Create and deploy Python apps with zc.buildout (Tarek Ziadé)
• Django everyday: quality and performance (David Larlet)
• Python 3000 (Victor Stinner)

There will be lightning talks as well, and the French speaking user group (Afpy) will have its annual meeting on sunday morning. A second room will be available for sprints and BoF sessions as well. All the talks will be in French this year

If you are around Paris, please join use ! It free !

Otherwise, watch the talks in the live stream: http://fr.pycon.org/stream-live

Thanks to our partners, for making this happen:

• The Python Software Foundation
• Logilab
• Nerim
• Ingeniweb
• Resolver Systems
• Pilot Systems
• WingWare
• Makina Corpus
• Toonux
• ActiveState
• Gymglish
• Eyrolles
• Cité des Sciences et de la Vilette

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:

• make distributions of packages (tarball, eggs, ..)
• build the application with buildout, then release a source distribution
• and provide a way for other developers to build the application on their own, by:
  • publishing the buildout configuration files
  • pushing the packages the buildout uses into to a server that is reachable by developers
  • make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the –repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn’t happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei’s work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here…

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :

• making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch.
• finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org.
• extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that’s it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: “Why changing that ? we have only one package server, which is the cheeseshop”.

It can also sound to some of them like I am (together with the team of 20 developers we have ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in… a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.

The Pimp my Buildbot project that was started here at Ingeniweb some time ago, to be able to set up a buildbot in a matter of minutes with zc.buildout, was continued during the sprint, and the guys did a great job on it.

It will be used here in customer projects with a Paster that adds buildbot support when a project starts, because it is a waste of time for the developers to set everything everytime.

Jean-Francois Roche, Kai Lautaportti and Gael Pasgrimaud added extensive configuration options (mail, scheduling), and made the SVN Poller works. This feature allows for instance to make the buildbot watch a SVN repository without having to add a hook in the server (post-commit hook for instance), when you don’t own it (SourceForge, collective, etc)

The tool is released in the collective, and available at the Cheeseshop in one single package !

If you want to set a buildbot

• provide for each one of your project a buildout that has a test script
• make sure the test script returns exit code (–with-exit-status with zope.testing)
• create a buildout cfg file using collective.buildbot
• run buildout, that’s it !
• run the master, slaves scripts, and go to the /waterfall page

Just try out our own buildbot by running this sequence:

$ cd /tmp/
$ mkdir my_bot
$ cd my_bot/
$ svn co https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/buildbot/trunk .
$ python bootstrap.py
$ bin/buildout
$ bin/master start
$ bin/linux_debian start   (that's our slave)

You should have a buildbot running then at http://localhost:9000/waterfall

The tool, without the polling stuff, also works with Mercurial and Bzr, but probably needs more tests with these repositories. We also need to make sure the slaves works fine under Windows, and add a nice front page to the buildbot.

If you use it let us know !

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,

We are working here at the Paris Sprint, and many things are going on. I wanted to do a quick wrapup before a longer and final wrapup.

Here’s all the tasks going on (I might miss some):

• Youenn and Cartsen are working on plone.recipe.apache, that will let you build and/or setup Apache in a buildout
• Michel presented a very interested talk yesterday on how insecure Plone can be (Man in the Middle attacks, etc), on the various existing solutions, and works on things to enhance it
• Maik leads a Funittest group, teaching the tool to people, and enhancing i
• Olivier and Christophe are working on Grok to learn it. Christophe worked yesterday on filling content to the new zope.org web site
• Matthew, Alex and I are working on PloneSoftwareCenter and plone.org, so the website can be ready to move to an egg-enabled Plone 3. wOOt !
• Gael, Jean-Francois (and Kai yesterday) are working on “Pimp my Buildbot“: configure and launch a complete buildbot system in a matter of minutes, using a simple buildout cfg file and running a simple command. I think this could be used for Plone repository and the collective as well ! The tool will be move to the collective at the end of the sprint.
• Gilles, Cyrille, and others (sorry, can’t keep up with all names), are working on iw.fss (FileSystemStorage for plone 3)
• Mustapha and Anton are making sweet things in ZopeSkel
• Anton worked also on finding the best way to do aysnhronous tasks in Plone
  
• There’s a big PloneGov team that is working on making the tool friendly with various government systems.
• Jean-Nicolas leads the work on kss/eventPush, so we all get a upload widget with live feedback (no flash inside ;))
• Godefroid and Lennart are Grok-ing as well;
• Christophe is preparing collective.kss.fly_guy, a tool that provides drag and drop in folder_contents
• Harito, Christian and Wim are making a sweet new theme for Plone: “Notre Dame“. The theme is kept quite generic, with all the plone tools/portlets designed in blue tones, kept in mind to be easy adjustable and changed to other color schemes. It will be a table-less egg for Plone 3.

More blogging tomorrow, I need to get back to my task

Alex is trying to raise money for his travel expenses, to go to the Plone Paris Sprint.

Alex will work on making Plone.org go Plone 3, so this is important !

Help him here : http://www.aclark.net/Members/aclark/help-me-upgrade-plone.org

edit: I have changed the blog title, which was not correct in English, thanks Maik

I wanted to react on Martinj’s blog to make things clear about Pycon FR

Background

First of all here’s a little background about Pycon FR.

Afpy (Association Francophone Python) is a french Python user group created four years ago. The group counts around 100 members and is the only one of this size in french-speaking countries as far as we know. It has members from France, Belgium, Canada and North Africa countries.

Many meetings are organized throughout the year, mainly in Paris, and sometimes in Charleroi in Belgium.

An important event took place at the “Cité des sciences” in Paris last summer, called “Journées Python” We had around 80 attendees and a rich program was given during two days. Topics where presented by french speaking people from France and Belgium, and we had talks about AI, Django, Zope, TuxDroid, etc

Some videos are available on the web, you can probably reach them from here: http://video.google.fr/videosearch?q=journees.afpy.org

The conference was 100% free for attendees. We also decided to chose carefully the dates so the conference would not compete with other events like Europython.

This event was a *true* success.

This event is now called Pycon FR, to be in the same line of what happens in some other countries. This makes a lot of sense indeed, to have national Python events, with standardized names.

Attendees

Pycon FR attendees are mainly people that speaks english but for some of them not enough to easily follow English talks. Most of those people would not make it to Europython because of this language barrier, and a national conference held in Paris is something that was quite missing at this time, for them.

So PyCon FR is just a sign of the growing interest about Python, and should be seen as a 100% positive and constructive thing, believe me.

Together with Europython and Pycon US, the “local” PyCons are in my opinion, a very good thing to spread the good word. We will, like last year, promote Europython there. Maybe we could even do a survey this year to know exactly who goes to Europython, etc.

Pycon FR expansion plans

When I said in my last blog entry that we were working on making it less french-centric, and having international attendees, I should have explained it better to make sure no one thinks Pycon FR might be a bad thing for Europython.

• If we do invite international speakers, it will be done only if we are able to provide a live translation system.
• The event date will always be picked carefully, as usual, so it does not compete with Europython.

Europython and local PyCons expansion

Here’s my opinion about what Europython strategy could be: since it is an international english-speaking event that gathers a lot of people. It could be the seed to set up a local Pycon event in places where it does not exist yet, using the momentum of the event.

This could be done by working with the local user group this way, by organizing maybe two years in a row Europython there. So, maybe we could see “Pycon Lithunia” next year ? Then move EuroPython to Amsterdam and see “Pycon The Netherlands” in two years ?

Europython exists to promote the language at an international level, so that could be a good way to spread imho.

But please, pretty please, don’t see local Pycons as Europython competitors, this is not true, this happens just because Python is growing.

Everyone should be happy about that I think. I disagree with the fact that each conference might fight to have the “best international speakers” for a very simple reason: the number of speakers is growing too, and I don’t think either Pycon either EuroPython will ever miss of interesting talks.

The next smart move I think is to organize and synchronize all these events, and that is where the Python Software Foundation has a role to play in my opinion. They have started anyway, and I am really confident about what will happen next.

PyCon FR is coming up ! We published yesterday the final program, and I must say that this second annual meeting of Pythoneers in Paris, looks very good.

Extracts:

• Why you should learn Python
• GraphViz with GvGen
• Gene search
• CouchDB
• PyPy
• Quality Assurance
• Scapy
• Zope 3
• zc.buildout
• Django
• WSGI
• etc…

This is going to be two intense days in Paris, so if you are around, please join us, it’s free.

Paris - Cité des Sciences et de la Vilette - 16/17 May .

The talks will be in french, but the organization team works hard to have english talks as well next year. Anyway, you don’t need to speak french to join us and have a good time