03.20.08

how to run your own private PyPI (Cheeseshop) server

Posted in plone, python, zope at 11:26 am by Tarek Ziadé

PloneSoftwareCenter 1.5 is heavily developed and not yet released, but the current trunk code is working well to use it as a PyPI-like server. It can be really useful for companies that develop Python software and are looking for a way to centralize their eggs internally. That’s what we use now at Ingeniweb to work on customer projects.
This tutorial explains how to set a cuttting-edge PloneSoftwareCenter server, if you want to be an early-adopter of what will be the next version running on plone.org software center in a few months (but this code is to be used at your own risks of course ;))

Why a private PyPI ?

You have some python packages you want to treat the same way the Cheeseshop does. In other words you want to work with them with easy_install, zc.buildout, etc..

But these packages are private to your company…

The simplest way is to store your eggs on some network folder. But distutils and setuptools provide a nice set of commands to automatically build and upload eggs at PyPI or any server that implements PyPI apis.

How ? PloneSoftwareCenter !

The Plone community provides a nice tool to manage Python packages, and it is PyPI compatible. In other words it can act like Cheesehop to interact with your command line tools.

It also provides an extensive set of features to manage your releases, run your bug trackers, etc.

See the plone.org products section, it is powered by PSC.

4 steps to install

Thanks to zc.buildout, a Plone Software Center (PSC) is really easy to setup. There’s no binary distribution yet though, so you need to compile a few things.

Step 1 - Pre-requests

If you are under Windows, grab this archive : http://release.ingeniweb.com/third-party-dist/python2.4.4-win32.zip
decompress it, and run “install.bat”. It will install Python 2.4 together with a set of tools, and PATH updates.

If you are under Linux, make sure you have gcc, subversion and make installed. Then install easy_install and PIL,
like this:

$ wget http://peak.telecommunity.com/dist/ez_setup.py
$ python ez_setup.py
$ easy_install http://release.ingeniweb.com/third-party-dist/PILwoTk-1.1.6.4.tar.gz

Step 2 - installing PSC

  1. Make a directory on your system called softwarecenter

  2. Get into it and grab PSC code with the svn command line:

    $ svn co http://svn.plone.org/svn/collective/Products.PloneSoftwareCenter/buildout/trunk .

  3. run the buildout with this set of commands:

    $ python bootstrap.py
$ bin/buildout

It will take some time, to grab all elements needed to build PSC.

  1. run the server 
    $ bin/instance start

Step 3 - setting up PSC

Let’s create a Plone website with a PloneSoftwareCenter instance:

  1. Open a browser and go to http://localhost:8080/manage. The login/password is admin/admin.
  2. On the left part, there’s a dropbox, select “Plone Site” the hit the add button
  3. In the form, set the id to “plone” and hit enter.
  4. Go to http://localhost:8080/plone/prefs_install_products_form
  5. Check “PloneSoftwareCenter” on the left side and hit “Install”
  6. Go to http://localhost:8080/plone
  7. In the “Add new…” menu, Click on “software center”
  8. In the form, in the Title, put “Catalog”
  9. Check Use Classifiers to display Categories (with Topic :: *) under Classifiers
  10. Hit the Save button

Your Software Center is ready and available at http://localhost:8080/plone/catalog

Step 4 - setting up the client-side

Now let’s set the client-side so people can use your Software Center:

  1. install iw.dist:

    $ easy_install iw.dist

  2. create a file in your home directory, called .pypirc with this content:

    [distutils]
index-servers =
  pypi
  local

[pypi]
username:YOUR_PYPI_LOGIN
password:YOUR_PYPI_PASSWORD

[local]
repository:http://localhost:8080/plone/catalog/
username:admin
password:admin

Of course, the localhost value will differ if you are located on another machine..

iw.dist adds two new commands in distutils: mregister and mupload. These commands enhance register and upload to make distutils work with multiple servers. This should be merged hopefully in Python 2.6 very soon.

Let’s use it !

Now, you will have two new commands in distutils, called ‘mregister‘ and ‘mupload‘ that will let you use either your PSC either PyPI.

Let’s upload an egg into PSC:

$ python setup.py mregister sdist bdist_egg mupload -r local

Let’s upload an egg into PyPI:

$ python setup.py mregister sdist bdist_egg mupload -r pypi

if -r is omited, pypi is the default one.

If you want to use PSC in zc.buildout or easy_install, you can provide http://localhost:8080/plone/catalog/simple as a find-links or index value:

[buildout]  find-links = http://localhost:8080/plone/catalog/simple

Or:

$ easy_install -f http://localhost:8080/plone/catalog/simple my.egg

That’s it !

Possibly related posts: (automatically generated)

13 Comments »

  1. Andreas Jung said,

    March 20, 2008 at 1:38 pm

    And if you want it cool, slim and grok-y:
    use haufe.eggserver :-)

  2. Tarek Ziadé said,

    March 20, 2008 at 2:51 pm

    @Andreas: yeah I saw that new project, very cool too. You should use the standard distutils protocol though, imho, instead of creating a new one for your uploads. Btw, when getting it, I bumped into a pdb in the released code, just to let you know:

    $ more upload.py

    def upload(*args, **kw):
    import pdb; pdb.set_trace()

    (dunno if it hurts, but..)

  3. Andreas Jung said,

    March 21, 2008 at 7:58 am

    The pdb bug is resolved since days.
    Concerning the distutils protocol: this is by design. haufe.eggserver only deals with distribution files, not with metadata. The implementation should be easy and won’t see support for fancy metadata support. Keep it easy, keep it simple, keep it working :-)

  4. Tarek Ziadé said,

    March 21, 2008 at 8:06 am

    @Andreas: I think it could be nice if haufe.eggserver would support the regular distutils “upload” command (even more KISS), even if it doesn’t process the metadata.

    That would make it a nice ligth solution I would probably use when I don’t want to deploy a whole PSC. But that’s just my point of view ;)

  5. Marc said,

    April 23, 2008 at 8:51 am

    How do you create a .pypirc file on windows and where do you put it?

    cheers!

  6. Tarek Ziadé said,

    April 23, 2008 at 9:01 am

    @Marc: cut and paste the .pypirc content in a text file, and save it in your home folder. This folder is usually under c:\Documents and Settings\User.

    It corresponds to the HOMEDRIVE + HOMEPATH environment variables you can check in your system environement

  7. Marc said,

    April 23, 2008 at 9:07 am

    Thanks for that. Is there a filename though? Windows doesnt like .extension files and forces a filename. Something like config.pypirc? Or is there a way to add a .pypirc file with no filename?

    Cheers!

  8. Tarek Ziadé said,

    April 23, 2008 at 9:14 am

    @Marc, right this is a bug that is being corrected in distutils. Il add a fix in iw.dist so “pypirc” can be used as well. Until then you can rename it through a MS-DOS prompt.

  9. Marc said,

    April 23, 2008 at 9:29 am

    Many thanks.

  10. Marc said,

    May 1, 2008 at 9:10 am

    Hi, hopfully last thing…..

    All of the packages i have uploaded to my PSC upload fine, but when i get them down via buildout i get a ValueError ‘Expected Version spec in’,,’packagename’.

    Any suggestions?

    ta!

  11. Tarek Ziadé said,

    May 1, 2008 at 9:41 am

    @Marc, I need to dig on this, can you provide me the full traceback ? (by email tarek_at_ziade.org)

  12. Marc said,

    May 1, 2008 at 9:47 am

    will do….

  13. Plone Paris Sprint wrapup #3, new.plone.org, collective.dist released ! « Carpet Python said,

    May 6, 2008 at 12:50 pm

    [...] are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible [...]

Leave a Comment