This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,

We are working here at the Paris Sprint, and many things are going on. I wanted to do a quick wrapup before a longer and final wrapup.

Here’s all the tasks going on (I might miss some):

• Youenn and Cartsen are working on plone.recipe.apache, that will let you build and/or setup Apache in a buildout
• Michel presented a very interested talk yesterday on how insecure Plone can be (Man in the Middle attacks, etc), on the various existing solutions, and works on things to enhance it
• Maik leads a Funittest group, teaching the tool to people, and enhancing i
• Olivier and Christophe are working on Grok to learn it. Christophe worked yesterday on filling content to the new zope.org web site
• Matthew, Alex and I are working on PloneSoftwareCenter and plone.org, so the website can be ready to move to an egg-enabled Plone 3. wOOt !
• Gael, Jean-Francois (and Kai yesterday) are working on “Pimp my Buildbot“: configure and launch a complete buildbot system in a matter of minutes, using a simple buildout cfg file and running a simple command. I think this could be used for Plone repository and the collective as well ! The tool will be move to the collective at the end of the sprint.
• Gilles, Cyrille, and others (sorry, can’t keep up with all names), are working on iw.fss (FileSystemStorage for plone 3)
• Mustapha and Anton are making sweet things in ZopeSkel
• Anton worked also on finding the best way to do aysnhronous tasks in Plone
  
• There’s a big PloneGov team that is working on making the tool friendly with various government systems.
• Jean-Nicolas leads the work on kss/eventPush, so we all get a upload widget with live feedback (no flash inside ;))
• Godefroid and Lennart are Grok-ing as well;
• Christophe is preparing collective.kss.fly_guy, a tool that provides drag and drop in folder_contents
• Harito, Christian and Wim are making a sweet new theme for Plone: “Notre Dame“. The theme is kept quite generic, with all the plone tools/portlets designed in blue tones, kept in mind to be easy adjustable and changed to other color schemes. It will be a table-less egg for Plone 3.

More blogging tomorrow, I need to get back to my task

Alex is trying to raise money for his travel expenses, to go to the Plone Paris Sprint.

Alex will work on making Plone.org go Plone 3, so this is important !

Help him here : http://www.aclark.net/Members/aclark/help-me-upgrade-plone.org

edit: I have changed the blog title, which was not correct in English, thanks Maik

I wanted to react on Martinj’s blog to make things clear about Pycon FR

Background

First of all here’s a little background about Pycon FR.

Afpy (Association Francophone Python) is a french Python user group created four years ago. The group counts around 100 members and is the only one of this size in french-speaking countries as far as we know. It has members from France, Belgium, Canada and North Africa countries.

Many meetings are organized throughout the year, mainly in Paris, and sometimes in Charleroi in Belgium.

An important event took place at the “Cité des sciences” in Paris last summer, called “Journées Python” We had around 80 attendees and a rich program was given during two days. Topics where presented by french speaking people from France and Belgium, and we had talks about AI, Django, Zope, TuxDroid, etc

Some videos are available on the web, you can probably reach them from here: http://video.google.fr/videosearch?q=journees.afpy.org

The conference was 100% free for attendees. We also decided to chose carefully the dates so the conference would not compete with other events like Europython.

This event was a *true* success.

This event is now called Pycon FR, to be in the same line of what happens in some other countries. This makes a lot of sense indeed, to have national Python events, with standardized names.

Attendees

Pycon FR attendees are mainly people that speaks english but for some of them not enough to easily follow English talks. Most of those people would not make it to Europython because of this language barrier, and a national conference held in Paris is something that was quite missing at this time, for them.

So PyCon FR is just a sign of the growing interest about Python, and should be seen as a 100% positive and constructive thing, believe me.

Together with Europython and Pycon US, the “local” PyCons are in my opinion, a very good thing to spread the good word. We will, like last year, promote Europython there. Maybe we could even do a survey this year to know exactly who goes to Europython, etc.

Pycon FR expansion plans

When I said in my last blog entry that we were working on making it less french-centric, and having international attendees, I should have explained it better to make sure no one thinks Pycon FR might be a bad thing for Europython.

• If we do invite international speakers, it will be done only if we are able to provide a live translation system.
• The event date will always be picked carefully, as usual, so it does not compete with Europython.

Europython and local PyCons expansion

Here’s my opinion about what Europython strategy could be: since it is an international english-speaking event that gathers a lot of people. It could be the seed to set up a local Pycon event in places where it does not exist yet, using the momentum of the event.

This could be done by working with the local user group this way, by organizing maybe two years in a row Europython there. So, maybe we could see “Pycon Lithunia” next year ? Then move EuroPython to Amsterdam and see “Pycon The Netherlands” in two years ?

Europython exists to promote the language at an international level, so that could be a good way to spread imho.

But please, pretty please, don’t see local Pycons as Europython competitors, this is not true, this happens just because Python is growing.

Everyone should be happy about that I think. I disagree with the fact that each conference might fight to have the “best international speakers” for a very simple reason: the number of speakers is growing too, and I don’t think either Pycon either EuroPython will ever miss of interesting talks.

The next smart move I think is to organize and synchronize all these events, and that is where the Python Software Foundation has a role to play in my opinion. They have started anyway, and I am really confident about what will happen next.

PyCon FR is coming up ! We published yesterday the final program, and I must say that this second annual meeting of Pythoneers in Paris, looks very good.

Extracts:

• Why you should learn Python
• GraphViz with GvGen
• Gene search
• CouchDB
• PyPy
• Quality Assurance
• Scapy
• Zope 3
• zc.buildout
• Django
• WSGI
• etc…

This is going to be two intense days in Paris, so if you are around, please join us, it’s free.

Paris - Cité des Sciences et de la Vilette - 16/17 May .

The talks will be in french, but the organization team works hard to have english talks as well next year. Anyway, you don’t need to speak french to join us and have a good time

When a site used by your buildout is not responding, you can stare at it for … ever

Add these two lines in your bin/buildout script:

import socket
socket.setdefaulttimeout(10)

With this, the buildout will go to the next link after ten seconds. This trick made my day

Ben Bangert was reacting about zc.buildout doctests, saying that they are hard to read from the PyPI page, and the examples hard to use and follow.

I agree with Ben as these doctests are very hard to read when you are not familiar with zc.buildout testing modules, which provides a set of API the doctests relies on.

But from a developer point of view, adding a feature to such a package is best done through doctests, using zc.buildout.testing goodies. And a developer that is familiar with this package, will find this doctest very useful.

zc.buildout in any case, is trying to structurize its PyPI front page, and push a maximum amount of doc for users, so … kudos !

I think the problem is more about specifying the target readership.

I would like to point another example that comes in zc.buildout: dependency-links.

The main doctest that appears at PyPI as a light, human-readable section: http://pypi.python.org/pypi/zc.buildout#dependency-links

And the very same section is continued in a specific doctest that does not appear on the main page: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/dependencylinks.txt?rev=81182&view=markup

So what happened here is that the developer specified two kind of readers:

• people that will reach the package through its PyPI page
• people that will go deeper in how the package works, through recipes or tutorials

From there I think there’s a simple guideline that could be applied to enhance the package documentation when adding a feature:

• resume the feature in the main page (long_description) with examples that does not rely on specific testing API. In other words it should be made of plain english and plain python when needed;
• leave doctest that relies on internal testing API as complementary documentation.
• define for each doctest its nature (recipe, tutorial, etc)

How could we help people doing such structuration ?

The distutils metadata could be a good place to do it, by adding an extra_doctests list for example, that would contain a list of doctests. From there, PyPI could display the long_description text as usual, and add a “more information section”.

Let’s take an example:

def _(f)
return open(f).read()

setup('my.package', ...
long_decription=_('README.txt'),
extra_description=
{'recipe': [_('create_this.txt'),
_('do_that.txt')],
‘tutorial’: [_('how_to_use.txt')
_('how_to_2.txt')]}

From there, PyPI could provide a Table of content, with a structurized documentation, and additional pages for the package, grouped by types (recipe, tutorial) etc. -> Maybe a Sphinx-powered PyPI ?

By the way, I have another post related, which tries to summarized good pratices in technical writing : http://tarekziade.wordpress.com/2007/02/23/technical-writing-the-seven-laws

When a project team use Test-Driven Development to build the code (everyone should), the next step is to set up automate builds, as explained in continous integration.

This is where Buildbot is great. It is easy and flexible to install, even more since Twisted has been eggified. A few easy_install steps are now sufficient to create a buildbot waterfall. Configuring a buildbot requires writing a few Python scripts though, and this has to be done everytime a project starts.

In my work, I need to be able to set buildbots in a matter of minutes, and they are always similar. They just need a buildmaster, a buildslave, and a few rules.

A first attempt to make things easier is to write a Python Paste script that generates default files. This is not very flexible though, as upgrades are still a bit tedious.

A more interesting solution is to provide zc.buildout recipes that take care of buildmaster and buildslave generation, through a very simple configuration file.

We have started this project, in order to be able to launch buildbot within a buildout.

The project has three parts:

• iw.buildbot: a thin layer on the top of Buildbot that allows to drive it with configuration files, instead of Python code. In other words, it makes buildbot configuration based on declarative configuration file and dynamic Python code, instead of declarative Python code.
• iw.recipe.buildmaster: a zc.buildout recipe that creates a buildbot instance together with configuration files.

• iw.recipe.buildslave: a zc.buildout recipe that creates a buildslave

The result is that creating a buildbot is done in a few lines in the buildout.cfg file:

[buildout]
parts =
  buildmaster
  linux_debian

[buildmaster]
recipe = iw.recipe.buildmaster
project-name = Ingeniweb Public buildbot
project-url = http://ingeniweb.com
port = 8999
wport = 9000
url = http://buildbot.ingeniweb.com
slaves =
  linux_debian    xxxxx
projects =
  iw.recipes 

[linux_debian]
recipe = iw.recipe.buildslave
host = localhost
port = ${buildmaster:port}
password = xxxx 

[iw.recipes]
slave-name=linux_debian
base-url=http://ingeniweb.svn.sourceforge.net
repository=/svnroot/ingeniweb/projects/iw.recipes/
branch=buildout
build-sequence =
    python bootstrap.py
    bin/buildout 

test-sequence =
    bin/test -v –exit-with-status

• buildmaster defines the project name and url, the buildbot web port, slave port and url, and a list of slaves and projects.
• each project has his own section, where the Subversion path is defined, as well as the build sequence and the test sequence.
• each slave is defined in its section

From there other buildouts can be created to group packages to be tested, and to define a script that can be used for tests. For Zope applications, that would be zopectl of course, as long as it is used with –exit-with-status.

This is the case for iw.recipes: it grabs all iw.recipes.* eggs to hook them to a testrunner.

The result can be see here: http://buildbot.ingeniweb.com

The next steps are to make sure everything works fine under Windows, and see how things goes in various projects, then make it work with all kinds of VCS and schedulers.

I will probably propose a sprint task on this in Paris sprint and see if the package meets interest.