Making the case for Cliquet?

The Intentions

What JSON structure should I use for my API? Which syntax for filtering the querystring? How do I deal with concurrent writes? And what about synchronising data on the client-side?

Today, when a new project wants to take advantage of a REST API for storing and consuming data, it is entirely reasonable to use the stock HTTP protocol and to concentrate only on the bare essentials. This goes for the clients as well, since the majority of code that handles interaction with the server is reusable.

How can I verify if the service is operational? How about StatsD? Is Sentry configured properly? How can I deploy a new version without breaking client apps?

Cliquet provides everything you need to get you started in production - by design, the transition from prototype to operational service is smooth and rapid. Out of the box, the service will have facilities for monitoring and configuration, as well as aides for deployment and version deprecation. As these realities evolve in the real world, so does the toolkit.

Which storage backend should I use for the JSON documents? What happens if the Ops team mandates that we use PostgreSQL? And what if we want to switch to a Redis, or even memory-based backend, for local testing and development?

One word: abstraction. This concept lies at the heart of our our entire implementation. Those two projects mentioned previously were, at their most simple, HTTP services that exposed a CRUD REST API backed by persistent JSON data stores. Since neither Pyramid nor Cornice (which we'll touch on later) were quite ready to handle this situation, we decided to introduce some classes to abstract the rudiments of both the REST model and of storage backends in general.

An important tenet in all of our projects is that of flexibility. This means that we try to make things optional where we can (and malleable where we can't) - Cliquet is no different. Everything is configurable right from the .ini file! What's more, all of the projects that use the toolkit are deployed in exactly the same way - only the configuration is different.

Technical Choices

Cliquet, the protocol, is currently implemented in Python (2.7, 3.4+, pypy) with Pyramid [3].

Pyramid is a web framework that is responsible for all things HTTP; it is usable by projects of all sizes, great and small.

Cornice is an extension for Pyramid written by Alexis and Tarek that consists of reusable boilerplate code. The goal here is to avoid reinventing the wheel each time when building a REST API with Pyramid. Cornice handles the plumbing for things like HTTP verbs and methods, header validation, content negotiation, HTTP return codes, CORS headers, JSON schema validation, and so forth.

Cliquet, in turn, uses both Pyramid and Cornice to implement the protocol and provide the necessary abstractions; both remain available for expanding and extending from what comes in the box.

Concepts

Conceptually the toolkit reflects the protocol - there are some interesting additional elements, however:

• Backends: Abstractions for storage, caching, and permissions (ex. PostgreSQL, Redis, in-memory, etc.)
• Monitoring: JSON-formatted logs and real-time instrumentation (StatsD) to keep tabs on performance and health.
• Configuration: The configuration itself can be loaded from environment variables or the .ini file - or a combination of both.
• Flexibility: Most of the components can be turned on or off (or substituted completely) via the configuration alone.
• Profiling: Built-in development utilities to help find bottlenecks.

By proportion, the number of lines of code that are necessary to support REST outweighs that of any other component in Cliquet. This is a feature; Cliquet provides a wide range of functionality and can therefore be used in any number of contexts (even without manipulating data).

The primary objective of the toolkit is aid the developer in building their application with a minimum of fuss. You mix and match only what you need, and every component is guaranteed to be production-ready.

For example, the persistence furnished by default is schemaless (e.g. JSONB) but there's no reason a relational storage model couldn't be implemented. Since all of the moving parts can be replaced directly via the configuration, it's entirely feasible to extend Cliquet with business logic or other exotic elements. We've got even more ideas in the "ecosystem" portion of our documentation.

In a few weeks well be introducing the idea of "events" (or "signals") which will allow plugins to interface in a much cleaner way.

We put a high value on code readability, as well as the importance of patterns, tests, and documentation. If you have any feedback, suggestions, accolades, or criticisms, don't hesitate to let us know!

Step 1

Activate Cliquet:

import cliquet
from pyramid.config import Configurator

def main(global_config, **settings):
    config = Configurator(settings=settings)

    cliquet.initialize(config, '1.0')
    return config.make_wsgi_app()

From here, most of the features of Cliquet are active and ready to be used.

For example, the hello (/v1/) or monitoring (/v1/__heartbeat__) URLs, but also the storage and cache backends that are available in the classic views in either Pyramid or Cornice.

Step 2

Add some views:

def main(global_config, **settings):
    config = Configurator(settings=settings)

    cliquet.initialize(config, '1.0')
    config.scan("myproject.views")
    return config.make_wsgi_app()

The first step in defining a CRUD resource is to define the schema with Colander:

from cliquet import resource, schema

class BookmarkSchema(schema.ResourceSchema):
    url = schema.URL()

@resource.register()
class Bookmark(resource.BaseResource):
    mapping = BookmarkSchema()

From here, the CRUD resource is available at /v1/bookmarks with all the synchronisation, filtering, sorting, paging, and timestamp features ready to go. By default the records are private (per user).

$ http GET "http://localhost:8000/v1/bookmarks"
HTTP/1.1 200 OK
...
{
    "data": [
        {
            "url": "http://cliquet.readthedocs.io",
            "id": "cc103eb5-0c80-40ec-b6f5-dad12e7d975e",
            "last_modified": 1437034418940,
        }
    ]
}

Step 3

There's lots of room to change things up, such as the URLs and HTTP verbs, or even modifying the fields before they're saved.

@resource.register(collection_path='/user/bookmarks',
                   record_path='/user/bookmarks/{{id}}',
                   collection_methods=('GET',))
class Bookmark(resource.BaseResource):
    mapping = BookmarkSchema()

    def process_record(self, new, old=None):
        if old is not None and new['device'] != old['device']:
            device = self.request.headers.get('User-Agent')
            new['device'] = device
        return new

For more information, see the resource documentation .

Step 4 (optional)

Let's try using Cliquet abstractions in a Cornice view:

from cliquet import Service

score = Service(name="score",
                path='/score/{game}',
                description="Store game score")

@score.post(schema=ScoreSchema)
def post_score(request):
    collection_id = 'scores-' + request.match_dict['game']
    user_id = request.authenticated_userid
    value = request.validated  # c.f. Cornice.

    storage = request.registry.storage
    record = storage.create(collection_id, user_id, value)
    return record

Areas for improvement

We're quite proud of what we've built in a relatively short time frame, and as we noted in a previous article, there's a lot of potential just waiting to be tapped. Nonetheless, we are aware of a few areas where we could improve things - maybe you'd like to help?

• API Documenatation: Right now we don't have a good solution for a project based on Cliquet to easily integrate all of the API documentation.
• Documentation in general: It's difficult to write, organise, and maintain documentation, especially when it's aimed at a general public that runs the gamut from absolute beginner to expert. We are victims of the "curse of knowledge".
• Protocol: At some point we're going to have to properly version the protocol, which means making a division within versions of Cliquet itself; this will be necessary if we want to move the ecosystem forward.
• Conservatism: We're fans of stability and robustness, but we don't live in a vacuum and must therefore sometimes bend to the realities of the real world. Moving forward with Python 3 is a part of this (async is a good target).
• Releases: The flip side of factoring. Occasionally we add features or otherwise evolve the toolkit as a response to the requirements of a specific project. As a result, we end up having to do waterfall-style project releases.

Common Questions