Fetchez le Python

Technical blog on the Python programming language, in a pure Frenglish style. Main topics are Python and Mozilla – This blog does not engage my employer

Mozillians, win a free pass for Pycon US – take 2

I am extending the contest until Feb the 1st – Mozillians, win a free pass for Pycon US

The fear of CRUD

Cornice is growing steadily, and we are thinking about the different ways to use it for our needs. One use case that comes often when we build web services is the need to publish a SQL Database via HTTP.

For instance, in a project I am working on, we might expose a list of servers and some information about them, that are stored in a SQL DB . The goal is to allow some management scripts to interact with the DB, to set and retrieve information about the servers, like: “can I use server 12 as a node for application X ?”

Interacting with CURL or a similar tool is simpler and more portable than coding yet another SQL client for this, so the idea is to see how this kind of web service can be done is the minimum pain with Cornice.

What I am thinking about building is a small CRUD interface that glues Cornice and SQLAlchemy. The latter has a way to define a database schema explicitly via mappings meaning that it’s easy to write a generic layer that exposes the database to the web via Cornice definitions. The work consists of transforming POST & PUT requests that contains data to write to the DB into SQLAlchemy objects, and transforming select results asked via GET requests into the proper responses.

Nothing very new, there are tons of existing systems that implement CRUD on the top of ORMs or plain SQL libraries. The only reason to build yet another one is to use it in the context of our current toolset which is composed of Cornice, Pyramid & SQLAlchemy for most projects. The whole code will probably be less than 300 lines at the end anyways.

Oh my.

Turns out this idea is really freaking out some people around me. There’s a strong aversion of some coders against anything that looks a bit like Active Records — in the Rails Context. In other words anything that would completely automate the serialization & deserialization layer and make it hard to tweak some code.

Another criticism is that a CRUD system would not be able to scale in the context of a big database, like Firefox Sync, that uses numerous databases to shard data.

Turns out building a CRUD on tools like SQLAlchemy or Pyramid is not really going to ruin your scalability as long as:

  • you can tweak the serialization / deserialization
  • you can override any operation in the CRUD operations when needed
  • you don’t shoot yourself in the foot by using CRUD with some code or DB that is not meant to be used that way
  • you can use the power of the underlying tools without being blocked by the lib

For the latter, Ben Bangert was pointing me at SQLAlchemy horizontal feature, which is basically what I wrote from scratch last year to make the Sync server shard across databases… At this point I sense that Firefox Sync could have been built with a CRUD lib, and be as efficient as it is today, because when I look at the queries produced by the code and the one a CRUD lib would produce, we are one or two tweaks away.

Anyways, here’s a first attempt at such a library.

Defining the model

In SQLAlchemy, you can define the DB model using mappings, which are simple classes containing a description of the tables.

For example, if I have a class “users” with a field “id” and a field “name”, the mapping will look like this:

class Users(_Base):                                           
    __tablename__ = 'users'                                      
    id = Column(Integer, primary_key=True)                    
    name = Column(String(256), nullable=False)

What I started to do is write a meta class one can use in a class to publish the mapping via HTTP.

Here’s an example:

from cornicesqla import MetaDBView
from myapp import Users, DBSession

class UsersView(object):
    __metaclass__ = MetaDBView
    mapping = Users
    path = '/users/{id}'
    collection_path = '/users'
    session = DBSession

What we have here is the definition of a view for the Users mapping. The class defines an URI for the collection (collection_path) and for each user (path). The session attribute is an SQLAlchemy session object you usually define when you work with that tool.

That’s it.

The model gets published, and you can GET, PUT, POST and DELETE on /users and /users/someid.

The code of the prototype is here and you can find a working example in the tests here. It’s called cornice-sqla

Tweaking serialization & data validation

By default, cornice-sqla will serialize and deserialize using JSON but you can tweak these steps by providing a custom serializer, or deserializer (or both.)

Let’s say you want to use the Colander libary to validate and serialize the data. To do this, you just have to write your serializer method into the view class

class UsersValidation(colander.MappingSchema):
    name = colander.SchemaNode(colander.String())

class UsersView(object):
    __metaclass__ = MetaDBView

    mapping = Users
    path = '/users/{id}'
    collection_path = '/users'
    session = DBSession

    def serialize(self):
        """Unserialize the data from the request, to serialize it for the DB"""
            user = json.loads(self.request.body)
        except ValueError:
            request.errors.add('body', 'item', 'Bad Json data!')
            # let's quit

        schema = UsersValidation()
            deserialized = schema.deserialize(user)
        except Invalid, e:
            # the struct is invalid
            request.errors.add('body', 'item', e.message)

        return deserialized

Colander is used here to validate the incoming request and create a flat mapping we can push into the DB. Cornice’s error system is in usage here, as explained here.

You can tweak the data that gets back from the DB with unserialize(), and for the collection URI, use collection_serialize() and collection_unserialize().

Tweaking C, R, U or D

cornice-sqla is based on a fresh feature Gael added into Cornice lately: resources. A resource is a class where you can define get(), post(), delete() and put() methods for a given URI.

cornice-sqla views are based on resources, meaning that you can override anyone of those methods and do whatever you want if you don’t want the CRUD part.

What’s next

I need to make sure everything you can do in Cornice (acls various options etc) can still be done in cornice-sqla, and start to work with more complex DB schema that include relations etc.  I also need to add basic missing features like batching and some docs.

My hope at the end is that this small library will reduce considerably the code needed in some of our projects that interact with SQL.

Pyramid @ Python 3

If you have been following closely the latest work done by Chris on WebOb, you know that WebOb and eventually Pyramid became Python 3 compatible.

That makes Python 3 a very tempting target for a new web project.

Paste & PasteScript still need to be ported to Python 3 and the Pyramid team has chosen not to. They have created their own paster replacer instead, which can be used to initiate a Pyramid project or run the app using the .ini file.

I am wondering if it would not be simpler at this point to drop Paste and use this replacer for all Python 3 frameworks that are using the Paste script and templates features.

Besides all the features Pyramid and its libs turns out most of the libs you usually need to build a classical web app already support Python 3, like SQLALchemy and PyMysql for MySQL access, Pylibmc for Memcached;

Things I am still missing in Python 3:

  • gevent
  • gunicorn
  • python-ldap
  • Cornice — I will port it soon

If you want to give it a shot, get the latest Python 3.2 and grab more details at : https://github.com/Pylons/pyramid/wiki/Python-3-Porting

And if you miss one lib, add it here

Merry Christmas !

Tutorial – build your web services with Cornice

At this stage, I think we’ve added enough helpers in Cornice to get anyone started in building web services in Python.

As a reminder, Cornice provides helpers to build & document REST-ish Web Services with Pyramid, a Python web framework. The main benefits of Cornice are:

  • automatic handling of some HTTP errors – Ask yourself: is your app handling properly 405 or 406 errors?
  • automatic web service documentation via a Sphinx extension.
  • a simple way to validate and convert requests data, and return structured 400 responses.

This is a small tutorial, extracted from our documentation.

Let’s create a full working application with Cornice. We want to create a light messaging service.

You can find its whole source code at https://github.com/mozilla-services/cornice/blob/master/examples/messaging


  • users can register to the service
  • users can list all registered users
  • users can send messages
  • users can retrieve the latest messages
  • messages have three fields: sender, content, color (red or black)
  • adding a message is done through authentication


  • there’s a single channel for all messages.
  • if a user with the same name is already registered, he cannot register.
  • all messages and users are kept in memory.


The application provides two services:

  • users, at /users: where you can list all users or register a new one
  • messages, at /: where you can read the messages or add new ones

On the server, the data is kept in memory.

We’ll provide a single CLI client in Python, using Curses.

Setting up the development environment

To create this application, we’ll use Python 2.7. Make sure you have it on your system, then install virtualenv (see http://pypi.python.org/pypi/virtualenv.)

Create a new directory and a virtualenv in it:

$ mkdir messaging
$ cd messaging
$ virtualenv --no-site packages .

Once you have it, install Cornice in it with Pip:

$ bin/pip install Cornice

Cornice provides a Paster Template you can use to create a new application:

$ bin/paster create -t cornice messaging
Selected and implied templates:
cornice#cornice  A Cornice application

egg:      messaging
package:  messaging
project:  messaging
Enter appname (Application name) ['']: Messaging
Enter description (One-line description of the project) ['']: A simple messaging service.
Enter author (Author name) ['']: Tarek
Creating template cornice
Generating Application...
Running python2.7 setup.py egg_info

Once your application is generated, go there and call develop against it:

$ cd messaging
$ ../bin/python setup.py develop

The application can now be launched via Paster, it provides a default “Hello” service, you can check:

$ ../bin/paster serve messaging.ini
Starting server in PID 7618.
serving on view at

Once the application is running, visit in your browser or Curl and make sure you get:

{'Hello': 'World'}

Defining the services

Let’s open the file in messaging/views.py, it contains all the Services:

from cornice import Service

hello = Service(name='hello', path='/', description="Simplest app")

def get_info(request):
    """Returns Hello in JSON."""
    return {'Hello': 'World'}

Users managment

We’re going to get rid of the Hello service, and change this file in order to add our first service – the users managment

_USERS = {}

users = Service(name='users', path='/users', description="Users"0

def get_users(request):
    """Returns a list of all users."""
    return {'users': _USERS.keys()}

def create_user(request):
    """Adds a new user."""
    user = request.validated['user']
    _USERS[user['name']] = user['token']
    return {'token': '%s-%s' % (user['name'], user['token'])}

def del_user(request):
    """Removes the user."""
    user = request.validated['user']
    del _USERS[user['name']]
    return {'goodbye': user['name']}

What we have here is 3 methods on /users:

  • GET: simply return the list of users names – the keys of _USERS
  • PUT: adds a new user and returns a unique token
  • DELETE: removes the user.


  • PUT uses the unique validator to make sure that the user name is not already taken. That validator is also in charge of generating a unique token associated with the user.
  • GET users the valid_token to verify that a X-Messaging-Token header is provided in the request, with a valid token. That also identifies the user.
  • DELETE also identifies the user then removes it.

Validators are filling the request.validated mapping, the service can then use.

Here’s their code:

import os
import binascii
from webob import exc

def _create_token():
    return binascii.b2a_hex(os.urandom(20))

def valid_token(request):
    header = 'X-Messaging-Token'

    token = request.headers.get(header)
    if token is None:
        raise exc.HTTPUnauthorized()

    token = token.split('-')
    if len(token) != 2:
        raise exc.HTTPUnauthorized()

    user, token = token

    valid = user in _USERS and _USERS[user] == token
    if not valid:
        raise exc.HTTPUnauthorized()

    request.validated['user'] = user

def unique(request):
    name = request.body
    if name in _USERS:
        request.errors.add('url', 'name', 'This user exists!')
        user = {'name': name, 'token': _create_token()}
        request.validated['user'] = user

When the validator finds errors, it adds them to the request.errors mapping, and that will return a 400 with the errors.

Let’s try our application so far with CURL:

$ curl http://localhost:5000/users
{"status": "error", "errors": [{"location": "header",
                                "name": "X-Messaging-Token",
                                "description": "No token"}]}

$ curl -X PUT http://localhost:5000/users -d 'tarek'
{"token": "tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"}

$ curl http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'users': ['tarek']}

$ curl -X DELETE http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'Goodbye': 'tarek}

Messages managment

Now that we have users, let’s post and get messages. This is done via two very simple functions we’re adding in the views.py file:

messages = Service(name='messages', path='/', description="Messages")


def get_messages(request):
    """Returns the 5 latest messages"""
    return _MESSAGES[:5]

@messages.post(validator=(valid_token, valid_message))
def post_message(request):
    """Adds a message"""
    _MESSAGES.insert(0, request.validated['message'])
    return {'status': 'added'}

The first one simply returns the five first messages in a list, and the second one inserts a new message in the beginning of the list.

The POST uses two validators:

  • valid_token(): the function we used previously that makes sure the user is registered
  • valid_message(): a function that looks at the message provided in the POST body, and puts it in the validated dict.

Here’s the valid_message() function:

def valid_message(request):
        message = json.loads(request.body)
    except ValueError:
        request.errors.add('body', 'message', 'Not valid JSON')

    # make sure we have the fields we want
    if 'text' not in message:
        request.errors.add('body', 'text', 'Missing text')

    if 'color' in message and message['color'] not in ('red', 'black'):
        request.errors.add('body', 'color', 'only red and black supported')
    elif 'color' not in message:
        message['color'] = 'black'

    message['user'] = request.validated['user']
    request.validated['message'] = message

This function extracts the json body, then checks that it contains a text key at least. It adds a color or use the one that was provided, and reuse the user name provided by the previous validator with the token control.

Generating the documentation

Now that we have a nifty web application, let’s add some doc.

Go back to the root of your project and install Sphinx:

$ bin/pip install Sphinx

Then create a Sphinx structure with sphinx-quickstart:

$ mkdir docs
$ sphinx-quickstart
Welcome to the Sphinx 1.0.7 quickstart utility.


Enter the root path for documentation.
> Root path for the documentation [.]: docs
> Separate source and build directories (y/N) [n]: y
> Project name: Messaging
> Author name(s): Tarek
> Project version: 1.0
> Create Makefile? (Y/n) [y]:
> Create Windows command file? (Y/n) [y]:

Once the initial structure is created, we need to declare the Cornice extension, by editing the source/conf.py file. We want to change extensions = [] into:

import cornice
extensions = ['cornice.sphinxext']

The last step is to document your services by editing the source/index.rst file like this:

Welcome to Messaging's documentation!

.. services::
   :package: messaging

The services directive is told to look at the services in the messaging package. When the documentation is built, you will get a nice output of all the services we’ve described earlier.

The Client

A simple client to use against our service can do three things:

  1. let the user register a name
  2. poll for the latest messages
  3. let the user send a message !

Without going into great details, there’s a Python CLI against messaging that uses Curses.

See https://github.com/mozilla-services/cornice/blob/master/examples/messaging/messaging/client.py

Going deeper

If you want to dig deeper, here are a few links:

We’d love feedback & new contributors !