This is a reminder if you wish to join us in Paris, April 25-27. (week-end)

There will be a Plone sprint, and we expect around 30 people so far.

All infos are located on the OpenPlans page here: http://www.openplans.org/projects/plone-3-paris-sprint/project-home

There’s also a Grok sprint a few days after Paris one : http://wiki.zope.org/grok/GrokkerdamSprint
so this is a good opportunity to do a Python european tour

A new recipe has been added to the iw.recipe.* family: iw.recipe.backup.

If you use zc.buildout to deploy Python applications in production, this one can be used to backup a buildout folder, before any upgrade is attempted, or for a daily backup. It creates an archive with a timestamp on its name, in a defined folder, with the whole folder content. Some directories can be excluded if needed.

This script is not doing fine-grained backups of the Data.fs like repozo.py does, but can be combined with it if this finesse is required. The recipe creates two scripts in the bin folder: backup and restore.

backup creates an archive and takes no parameters, whereas restore takes an archive name and decompress it into the buildout folder. The latter does a backup before it decompresses the archive for more safety.

Last, it will raise an error if the backup folder is located within the buildout folder ;), and logs all commands into a log file. This is a tiny recipe, but provides a simple way to backup your buildouts in a single call, no matter the platform you are in.

Other recipes we provide at this time are:

• iw.recipe.cmd: provides a way to execute shell calls, or inline Python calls. This is useful when you need to set a few things when a buildout is built, that does not worth creating a new recipe;
• iw.recipe.fetcher: similar to wget. We use it to build python-win32.zip archive, to get all installers on the web;
• iw.recipe.fss: will let you configure FileSystemStorage, so no extra step is required;
• iw.recipe.pound: compiles Pound load balancer, and creates its configuration file, so it can be run directly;
• iw.recipe.sendmail: set zope.sendmail as the default mail sender in a Zope 2-based application. This is useful if you wish to have a fast and robust mailhost service in your Plone sites;
• iw.recipe.squid: This recipe installs all parts needed to run a Squid server dedicated to serve a Zope application, staying friendly with Apache and setting things nicely for Plone;
• iw.recipe.subversion: recipe to checkout a svn location into a part. It is different from infrae.subversion because it creates a tarball out of the subversion checkout, and put it in the downloads folder. In other words, it works offline as well as long as the tarball was built once. It doesn’t have the nice feature infrae.subversion provides which is a code checker that will raise an error when buildout tries to remove a changed file. But as we use the develop feature of buildout to create our code, we didn’t need that feature. Subversion checkouts are useful to us when we want to use a bleeding-edge version of a third-party package that has not been released yet;
• iw.recipe.template: this one provides a way to write files based on Cheetah templates, like Python Paste does, and is useful for micro-needs, like iw.recipe.cmd.

If you use one of those recipes and whish to make a feature request or a bug report, I have created a new space on plone.org to have a bug tracker for all recipes: http://plone.org/products/iw-recipes/issues

Last year, I worked on a documentation builder based on doctest and reStructuredText format called PyCommunity. This tool is collecting doctests from Python packages and from special places on a source repository, basing its work on a set of project good practices I had presented and used in a Pycon 2007 tutorial (slides).

Basically, it uses the best practices described in Andreas Ruping book, called Agile Documentation, applied to Python projects using TDD and doctests. It can be called Document-Driven Development.

I have never found the time to finish the tool and I was looking forward to get back to it. As a matter of fact, George Brandl has released the tool that is used to generate Python documentation, called Sphinx, which does many things I still have in my TODO list, like pygments integration, and many other things.

See how Sphinx generates Python doc here: http://docs.python.org/dev

From there, it becomes dead simple to generate a website for a Python project, based on packages doctests and text files and on a specific doc/ structure.

Sphinx annoucement is really exciting, and it shouldn’t be too much pain to bundle it in a buildout recipe to manage a project documentation. Since it is based on templates and configuration files, a default structure can be generated to startup a project documentation together with a code base.

I have to try and see if Sphinx allows me to set everything up. In other words, replay my Pycon tutorial with it.

Cool, my proposal for OSCON 2008 has been accepted, and will be about Plone 3, zc.buildout and the work on creating private PyPI servers using PloneSoftwareCenter.

It will present how we work at Ingeniweb with both public and private packages, to deliver Plone applications to customers, using eggs and buildout.

It’s funny because since a few days, there are a lot of discussions around distutils, PyPI and setuptools, and about making things better at Python level on how to distribute packages and applications. So it seems to be a hot topic.

From my point of view, PyPI brought a lot in the past years in this area, and being able to deploy a pypi-compatible software center in a company helps a lot in using the same set of command line tool. (distutils/setuptools)

So, I am pretty happy nowadays with zc.buildout and setuptools (thanks to Mr Fulton, Eby and al), despite all the critical that has been made about setuptools in the last few days, and despite the fact that it is *so hard* to make a tiny little change make it to distutils trunk :’( …

Anyways, if you do Plone or Zope dev, and if you are interested about software delivery, I’d be glad to exchange about it, to see how you work and deliver Zope apps, to get other point of views before OSCON.

Here’s the abstract of my talk:

There is something that is a bit missing in a buildout environment to be able to drive it with no pain : a gobal script that allows starting, stopping or restarting every software that is bundled in it.

For instance, a buildout often has a Zeo server, a Zope instance and some other servers like Pound, Apache or Squid (just control scripts and sometimes a whole build. A global script can handle this by running a suite of commands for each action: start, stop, restart.)

Let’s take an example: we want to run a Zeo server, a Zope client, and a pound load balancer. These actions can be listed in a buildout section like this:

[buildout] …

[cluster]

recipe = plone.recipe.cluster
poundctl = ${buildout:bin-directory}/pound -f ${buildout:directory}/parts/pound/etc/pound.cfg -c ${buildout:directory}/var/pound.pid

start =
    ${buildout:bin-directory}/zeoserver start
    ${buildout:bin-directory}/instance start
    ${cluster:poundctl}

stop =
    ${buildout:bin-directory}/zeoserver stop
    ${buildout:bin-directory}/instance stop
    pid:${buildout:directory}/var/pound.pid

restart =
    ${buildout:bin-directory}/zeoserver restart
    ${buildout:bin-directory}/instance restart
    ${cluster:poundctl}

The zeoserver and instance scripts are daemons that just need to be called, and the pound script which is also running as a daemon, do not have any option to be stopped, but will let the user define a pid file. So having a way to kill the pid defined by this file is enough in this case.

The recipe can therefore build a control script that launches the set of commands for each action:

$ bin/cluster start
Starting Cluster...
$ bin/cluster stop
Stopping cluster...

Let’s take another example: a buildout that runs a simple script that is not daemonized. In other words, it needs to be run in the background and its pid saved, so the cluster recipe will know how to stop it:

[cluster]

recipe = plone.recipe.cluster
start =
    background:${buildout:directory}/bin/script
    ${buildout:bin-directory}/instance start

stop =
    ${buildout:bin-directory}/instance stop

restart =
    background:${buildout:directory}/bin/script
    ${buildout:bin-directory}/instance restart

In this case, the background: prefix will indicated the recipe to execute the command in the background, and to keep its PID in a file. When the stop command will be called, it will be stopped automatically.

I think that all cases are covered by this recipe, and I have a prototype working for Linux/Mac here:
http://svn.plone.org/svn/collective/buildout/plone.recipe.cluster/trunk . I took back and adapted some work done by Blue Dynamics (bda.daemon). System programming is a bit over my head so it needs some more work in the coming days, but it is working already.

Then, the next step will be to make it win32 compliant, using NT Services for this, and hopefully releasing a first 0.1.0 version.

If you feel that there’s some missing use cases, let me know ! (the full README is here)

iw.rotatezlogs is a very useful package made by Gilles Lenfant to rotate log file in Zope. This is quite useful to make sure they don’t get huge.

I have made the necessary changes in plone.recipe.zope2instance and plone.recipe.zope2zeoserver to make it available in a buildout environment. Since those changes are now released, here’s the way to add the log rotator in your buildout if you wish to use it:

[buildout]

…

[instance]

…

event-log-custom =
    %import iw.rotatezlogs
    <rotatelogfile>
        path ${buildout:folder}/var/log/event.log
        max-bytes 1MB
        backup-count 5
    </rotatelogfile>

access-log-custom =
    %import iw.rotatezlogs
    <rotatelogfile>
        path ${buildout:folder}/var/log/instance-Z2.log
        max-bytes 1MB
        backup-count 5
    </rotatelogfile>

eggs =
    …
    iw.rotatezlogs

[zeo]

…

zeo-log-custom =
    %import iw.rotatezlogs
    <rotatelogfile>
        path ${buildout:folder}/var/log/zeoserver.log
        max-bytes 1MB
        backup-count 5
    </rotatelogfile>

eggs =
    …
    iw.rotatezlogs

The package is added in the eggs section to make sure the recipes adds it in the Python path when the script (zope or zeo) is loaded.

Gilles told us that the rotator will be integrated in ZConfig itself sometimes, so it will be even simpler to use.