Mozillians, win a free pass for Pycon US – take 2
I am extending the contest until Feb the 1st – Mozillians, win a free pass for Pycon US
-
Subscribe
Subscribed
Already have a WordPress.com account? Log in now.
I am extending the contest until Feb the 1st – Mozillians, win a free pass for Pycon US
Cornice is growing steadily, and we are thinking about the different ways to use it for our needs. One use case that comes often when we build web services is the need to publish a SQL Database via HTTP.
For instance, in a project I am working on, we might expose a list of servers and some information about them, that are stored in a SQL DB . The goal is to allow some management scripts to interact with the DB, to set and retrieve information about the servers, like: “can I use server 12 as a node for application X ?”
Interacting with CURL or a similar tool is simpler and more portable than coding yet another SQL client for this, so the idea is to see how this kind of web service can be done is the minimum pain with Cornice.
What I am thinking about building is a small CRUD interface that glues Cornice and SQLAlchemy. The latter has a way to define a database schema explicitly via mappings meaning that it’s easy to write a generic layer that exposes the database to the web via Cornice definitions. The work consists of transforming POST & PUT requests that contains data to write to the DB into SQLAlchemy objects, and transforming select results asked via GET requests into the proper responses.
Nothing very new, there are tons of existing systems that implement CRUD on the top of ORMs or plain SQL libraries. The only reason to build yet another one is to use it in the context of our current toolset which is composed of Cornice, Pyramid & SQLAlchemy for most projects. The whole code will probably be less than 300 lines at the end anyways.
Oh my.
Turns out this idea is really freaking out some people around me. There’s a strong aversion of some coders against anything that looks a bit like Active Records — in the Rails Context. In other words anything that would completely automate the serialization & deserialization layer and make it hard to tweak some code.
Another criticism is that a CRUD system would not be able to scale in the context of a big database, like Firefox Sync, that uses numerous databases to shard data.
Turns out building a CRUD on tools like SQLAlchemy or Pyramid is not really going to ruin your scalability as long as:
For the latter, Ben Bangert was pointing me at SQLAlchemy horizontal feature, which is basically what I wrote from scratch last year to make the Sync server shard across databases… At this point I sense that Firefox Sync could have been built with a CRUD lib, and be as efficient as it is today, because when I look at the queries produced by the code and the one a CRUD lib would produce, we are one or two tweaks away.
Anyways, here’s a first attempt at such a library.
In SQLAlchemy, you can define the DB model using mappings, which are simple classes containing a description of the tables.
For example, if I have a class “users” with a field “id” and a field “name”, the mapping will look like this:
class Users(_Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(String(256), nullable=False)
What I started to do is write a meta class one can use in a class to publish the mapping via HTTP.
Here’s an example:
from cornicesqla import MetaDBView
from myapp import Users, DBSession
class UsersView(object):
__metaclass__ = MetaDBView
mapping = Users
path = '/users/{id}'
collection_path = '/users'
session = DBSession
What we have here is the definition of a view for the Users mapping. The class defines an URI for the collection (collection_path) and for each user (path). The session attribute is an SQLAlchemy session object you usually define when you work with that tool.
That’s it.
The model gets published, and you can GET, PUT, POST and DELETE on /users and /users/someid.
The code of the prototype is here and you can find a working example in the tests here. It’s called cornice-sqla
By default, cornice-sqla will serialize and deserialize using JSON but you can tweak these steps by providing a custom serializer, or deserializer (or both.)
Let’s say you want to use the Colander libary to validate and serialize the data. To do this, you just have to write your serializer method into the view class
class UsersValidation(colander.MappingSchema):
name = colander.SchemaNode(colander.String())
class UsersView(object):
__metaclass__ = MetaDBView
mapping = Users
path = '/users/{id}'
collection_path = '/users'
session = DBSession
def serialize(self):
"""Unserialize the data from the request, to serialize it for the DB"""
try:
user = json.loads(self.request.body)
except ValueError:
request.errors.add('body', 'item', 'Bad Json data!')
# let's quit
return
schema = UsersValidation()
try:
deserialized = schema.deserialize(user)
except Invalid, e:
# the struct is invalid
request.errors.add('body', 'item', e.message)
return deserialized
Colander is used here to validate the incoming request and create a flat mapping we can push into the DB. Cornice’s error system is in usage here, as explained here.
You can tweak the data that gets back from the DB with unserialize(), and for the collection URI, use collection_serialize() and collection_unserialize().
cornice-sqla is based on a fresh feature Gael added into Cornice lately: resources. A resource is a class where you can define get(), post(), delete() and put() methods for a given URI.
cornice-sqla views are based on resources, meaning that you can override anyone of those methods and do whatever you want if you don’t want the CRUD part.
I need to make sure everything you can do in Cornice (acls various options etc) can still be done in cornice-sqla, and start to work with more complex DB schema that include relations etc. I also need to add basic missing features like batching and some docs.
My hope at the end is that this small library will reduce considerably the code needed in some of our projects that interact with SQL.
If you have been following closely the latest work done by Chris on WebOb, you know that WebOb and eventually Pyramid became Python 3 compatible.
That makes Python 3 a very tempting target for a new web project.
Paste & PasteScript still need to be ported to Python 3 and the Pyramid team has chosen not to. They have created their own paster replacer instead, which can be used to initiate a Pyramid project or run the app using the .ini file.
I am wondering if it would not be simpler at this point to drop Paste and use this replacer for all Python 3 frameworks that are using the Paste script and templates features.
Besides all the features Pyramid and its libs turns out most of the libs you usually need to build a classical web app already support Python 3, like SQLALchemy and PyMysql for MySQL access, Pylibmc for Memcached;
Things I am still missing in Python 3:
If you want to give it a shot, get the latest Python 3.2 and grab more details at : https://github.com/Pylons/pyramid/wiki/Python-3-Porting
And if you miss one lib, add it here
Merry Christmas !
At this stage, I think we’ve added enough helpers in Cornice to get anyone started in building web services in Python.
As a reminder, Cornice provides helpers to build & document REST-ish Web Services with Pyramid, a Python web framework. The main benefits of Cornice are:
This is a small tutorial, extracted from our documentation.
Let’s create a full working application with Cornice. We want to create a light messaging service.
You can find its whole source code at https://github.com/mozilla-services/cornice/blob/master/examples/messaging
Features:
Limitations:
The application provides two services:
On the server, the data is kept in memory.
We’ll provide a single CLI client in Python, using Curses.
To create this application, we’ll use Python 2.7. Make sure you have it on your system, then install virtualenv (see http://pypi.python.org/pypi/virtualenv.)
Create a new directory and a virtualenv in it:
$ mkdir messaging $ cd messaging $ virtualenv --no-site packages .
Once you have it, install Cornice in it with Pip:
$ bin/pip install Cornice
Cornice provides a Paster Template you can use to create a new application:
$ bin/paster create -t cornice messaging Selected and implied templates: cornice#cornice A Cornice application Variables: egg: messaging package: messaging project: messaging Enter appname (Application name) ['']: Messaging Enter description (One-line description of the project) ['']: A simple messaging service. Enter author (Author name) ['']: Tarek Creating template cornice ... Generating Application... Running python2.7 setup.py egg_info
Once your application is generated, go there and call develop against it:
$ cd messaging $ ../bin/python setup.py develop ...
The application can now be launched via Paster, it provides a default “Hello” service, you can check:
$ ../bin/paster serve messaging.ini Starting server in PID 7618. serving on 0.0.0.0:5000 view at http://127.0.0.1:5000
Once the application is running, visit http://127.0.0.1:5000 in your browser or Curl and make sure you get:
{'Hello': 'World'}
Let’s open the file in messaging/views.py, it contains all the Services:
from cornice import Service
hello = Service(name='hello', path='/', description="Simplest app")
@hello.get()
def get_info(request):
"""Returns Hello in JSON."""
return {'Hello': 'World'}
We’re going to get rid of the Hello service, and change this file in order to add our first service – the users managment
_USERS = {}
users = Service(name='users', path='/users', description="Users"0
@users.get(validator=valid_token)
def get_users(request):
"""Returns a list of all users."""
return {'users': _USERS.keys()}
@users.put(validator=unique)
def create_user(request):
"""Adds a new user."""
user = request.validated['user']
_USERS[user['name']] = user['token']
return {'token': '%s-%s' % (user['name'], user['token'])}
@users.delete(validator=valid_token)
def del_user(request):
"""Removes the user."""
user = request.validated['user']
del _USERS[user['name']]
return {'goodbye': user['name']}
What we have here is 3 methods on /users:
Remarks:
Validators are filling the request.validated mapping, the service can then use.
Here’s their code:
import os
import binascii
from webob import exc
def _create_token():
return binascii.b2a_hex(os.urandom(20))
def valid_token(request):
header = 'X-Messaging-Token'
token = request.headers.get(header)
if token is None:
raise exc.HTTPUnauthorized()
token = token.split('-')
if len(token) != 2:
raise exc.HTTPUnauthorized()
user, token = token
valid = user in _USERS and _USERS[user] == token
if not valid:
raise exc.HTTPUnauthorized()
request.validated['user'] = user
def unique(request):
name = request.body
if name in _USERS:
request.errors.add('url', 'name', 'This user exists!')
else:
user = {'name': name, 'token': _create_token()}
request.validated['user'] = user
When the validator finds errors, it adds them to the request.errors mapping, and that will return a 400 with the errors.
Let’s try our application so far with CURL:
$ curl http://localhost:5000/users
{"status": "error", "errors": [{"location": "header",
"name": "X-Messaging-Token",
"description": "No token"}]}
$ curl -X PUT http://localhost:5000/users -d 'tarek'
{"token": "tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"}
$ curl http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'users': ['tarek']}
$ curl -X DELETE http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'Goodbye': 'tarek}
Now that we have users, let’s post and get messages. This is done via two very simple functions we’re adding in the views.py file:
messages = Service(name='messages', path='/', description="Messages")
_MESSAGES = []
@messages.get()
def get_messages(request):
"""Returns the 5 latest messages"""
return _MESSAGES[:5]
@messages.post(validator=(valid_token, valid_message))
def post_message(request):
"""Adds a message"""
_MESSAGES.insert(0, request.validated['message'])
return {'status': 'added'}
The first one simply returns the five first messages in a list, and the second one inserts a new message in the beginning of the list.
The POST uses two validators:
Here’s the valid_message() function:
def valid_message(request):
try:
message = json.loads(request.body)
except ValueError:
request.errors.add('body', 'message', 'Not valid JSON')
return
# make sure we have the fields we want
if 'text' not in message:
request.errors.add('body', 'text', 'Missing text')
return
if 'color' in message and message['color'] not in ('red', 'black'):
request.errors.add('body', 'color', 'only red and black supported')
elif 'color' not in message:
message['color'] = 'black'
message['user'] = request.validated['user']
request.validated['message'] = message
This function extracts the json body, then checks that it contains a text key at least. It adds a color or use the one that was provided, and reuse the user name provided by the previous validator with the token control.
Now that we have a nifty web application, let’s add some doc.
Go back to the root of your project and install Sphinx:
$ bin/pip install Sphinx
Then create a Sphinx structure with sphinx-quickstart:
$ mkdir docs $ sphinx-quickstart Welcome to the Sphinx 1.0.7 quickstart utility. .. Enter the root path for documentation. > Root path for the documentation [.]: docs ... > Separate source and build directories (y/N) [n]: y ... > Project name: Messaging > Author name(s): Tarek ... > Project version: 1.0 ... > Create Makefile? (Y/n) [y]: > Create Windows command file? (Y/n) [y]:
Once the initial structure is created, we need to declare the Cornice extension, by editing the source/conf.py file. We want to change extensions = [] into:
import cornice extensions = ['cornice.sphinxext']
The last step is to document your services by editing the source/index.rst file like this:
Welcome to Messaging's documentation! ===================================== .. services:: :package: messaging
The services directive is told to look at the services in the messaging package. When the documentation is built, you will get a nice output of all the services we’ve described earlier.
A simple client to use against our service can do three things:
Without going into great details, there’s a Python CLI against messaging that uses Curses.
See https://github.com/mozilla-services/cornice/blob/master/examples/messaging/messaging/client.py
If you want to dig deeper, here are a few links:
We’d love feedback & new contributors !
Fetchez le Python 