Annoucing collective.eggproxy, the smart PyPI mirror

I just wanted to announce the release of collective.eggproxy (http://pypi.python.org/pypi/collective.eggproxy/0.2.0)

collective.eggproxy is a smart mirror for PyPI, thaught and coded by my colleague Bertrand Mathieu.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally.

Want to give a try ? try it in two lines with easy_install:

easy_install collective.eggproxy
eggproxy_run

That’s it !

iw.eggproxy, a smart PyPI mirror

Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

The new .pypirc format in Python / distutils

Thanks to the sprinters in Washington D.C., (Andrew who worked on merging it, Alex, etc.) and people that helped in the discussion (Fred, Martin), my patch to Python for multiple servers handling in .pypirc file has been integrated in 2.6 trunk.

This patch will allow using distutils register and upload commands with several servers, using the -r option and storing for each one of them the appropriate username/password in the configuration file. See http://wiki.python.org/moin/EnhancedPyPI for more details.

The collective.dist package provides the same feature for Python 2.4 and 2.5, through new commands: mregister and mupload.

The new format is also pretty convenient to store several profiles per servers. For instance, if you have several accounts on PyPI, one for your company (acme) and one for your own usage, you can define them like this:

[distutils]
index-servers =
    pypi
    acme

[pypi]
username:user
password:password

[acme]
username:acme_user
password:password

Then use them depending on the package you work in:

$ python setup.py register -r acme
$ python setup.py register    # default, which is pypi

When plone.org will go Plone 3 (this is happening soon), the pypirc file can be extended with this new target:

[distutils]
index-servers =
    pypi
    acme
    plone

[pypi]
username:user
password:password
[acme]
username:acme_user
password:password
[plone]
repository:http://plone.org/products
username:plone_user
password:password

From there, setuptools aliases can be used in each package to simplify things:

$ python setup.py alias plone_org register -r plone sdist bdist_egg upload -r plone

Which will allow you to upload the package to the website in one simple command:

$ python setup.py plone_org

The patch aslo corrects a few minor bugs in distutils, such as:

• .pypirc was not found on Windows
• pydistutils.cfg was not found on Windows