A toolkit to build HTTP (micro)services.

• PyconFR
• October 2015
• https:/cliquet.readthedocs.org

• Origins
• Protocol
• Toolkit
• Conversation

• http://servicedenuages.fr
• @ametaireau, @leplatrem, @Natim, @n1k0, @phrawzty, @tarek_ziade
• Met around 2010 (Makina Corpus, DjangoCon)
• Same team in 2015 \o/

• We do APIs.
• CANIHAZ the specs
  (Super powers of the newcomer)

• HTTP know-how
• REST API design
• Synchronization protocol
• Production standards

→ We want a protocol.

• Quickstart
• Go live on day #1
• Everything optional and pluggable

→ We want a toolkit.

• API major version prefix (/v1)
• Standard headers (CORS, Content, Authorization, )
• Standard status codes (304, 412, 409, ...)
• REST CRUD (PUT create|replace, PATCH, ...)

• «Hello» endpoint GET /v1/
  (docs URL, settings values)
• Monitoring endpoint ( GET /v1/__heartbeat__ )
• Batch multiple requests ( POST /v1/batch )

• Error channel (JSON responses, error codes)
• Backoff headers (Maintenance, overload)
• End-of-life headers (API Deprecation)

→ Common code in API consumers!

Usual but optional

• JSON payload structure (Bikeshed!)
• Querystring syntax (Filter, sort)
• Concurrency control (ETags headers)
• Polling for changes (Offline sync)

GET /v1/articles?status=3
GET /v1/articles?_sort=-added
GET /v1/articles?_since=11290876567

• Schema validation (Error format)
• Token based pagination
• Permissions
• Transactions
• ...

• Empirical approach (Move fast)
• Towards stabilization (Going live)
• Start with implementation in Python (What else?)

Cliquet is:

• an implementation of the protocol in Python
• with good practices for production
• and pluggability in mind.

• Pyramid (HTTP framework)
• Cornice (Reduce REST boilerplate)

→ Beyond scope of Cliquet: as usual.

import cliquet
from pyramid.config import Configurator

def main(global_config, **settings):
    config = Configurator(settings=settings)

    cliquet.initialize(config, version='1.0')
    return config.make_wsgi_app()

→ Enjoy !

$ http GET "http://localhost:8000/v1/__heartbeat__"
HTTP/1.1 200 OK
...
{
    "cache": true,
    "permission": true,
    "storage": true
}

Use Cliquet abstractions with Pyramid or Cornice !

from cliquet import Service

score = Service(name="score",
                path='/score/{game}',
                description="Store game score")

@score.post(schema=ScoreSchema)
def post_score(request):
    collection_id = 'scores-' + request.match_dict['game']
    user_id = request.authenticated_userid
    value = request.validated  # c.f. Cornice.

    storage = request.registry.storage
    record = storage.create(collection_id, user_id, value)
    return record

from cliquet import resource, schema

class BookmarkSchema(schema.ResourceSchema):
    url = schema.URL()

@resource.register()
class Bookmark(resource.BaseResource):
    mapping = BookmarkSchema()

→ Enjoy !

$ http GET "http://localhost:8000/v1/bookmarks"
HTTP/1.1 200 OK
...
{
    "data": [
        {
            "url": "http://cliquet.readthedocs.org",
            "id": "cc103eb5-0c80-40ec-b6f5-dad12e7d975e",
            "last_modified": 1437034418940,
        }
    ]
}

...of endpoints and behaviour!

@resource.register(collection_path='/user/bookmarks',
                   record_path='/user/bookmarks/{{id}}',
                   collection_methods=('GET',))
class Bookmark(resource.BaseResource):
    mapping = BookmarkSchema()

    def process_record(self, new, old=None):
        if new['device'] != old['device']:
            device = self.request.headers.get('User-Agent')
            new['device'] = device
        return new

...of schema and fields!

import colander

class BookmarkSchema(resource.ResourceSchema):
    url = schema.URL()
    title = colander.SchemaNode(colander.String())
    device = colander.SchemaNode(colander.String(), missing='')

    class Options:
        readonly_fields = ('device',)
        unique_fields = ('url',)

• Unplug | Plug | Override

• From configuration
• Lightweight by default
• Abstractions for backends

• Pyramid eco-system
• Agnostic (multiauth)
• Pluggable (from configuration)
• Firefox Account at Mozilla (extension cliquet-fxa)

Standard deployment:

• Flat configuration (.ini, env vars)
• Monitoring components (StatsD, Sentry, NewRelic)
• Logging renderers (JSON, Heka)

From application.ini:

cliquet.storage_backend = cliquet.storage.redis
cliquet.storage_url = redis://localhost:6379/0
cliquet.statsd_url = udp://localhost:8125
cliquet.logging_renderer = cliquet.logs.CustomRenderer

From environment variables...

# docker-compose.yml
db:
  image: postgres
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: postgres
web:
  links:
   - db
  environment:
    CLIQUET_CACHE_BACKEND: cliquet.cache.postgresql
    CLIQUET_CACHE_URL: postgres://postgres:postgres@db/postgres
    CLIQUET_STORAGE_BACKEND: cliquet.storage.postgresql
    CLIQUET_STORAGE_URL: postgres://postgres:postgres@db/postgres

Using Werkzeug middleware:

def main(global_config, **settings):
    config = Configurator(settings=settings)
    cliquet.initialize(config, __version__)
    app = config.make_wsgi_app()
    return cliquet.install_middlewares(app)

cliquet.profiler_enabled = true
cliquet.profiler_dir = /tmp/profiling

Cliquet brings:

• Standard configuration
• Standard deployment
• Standard monitoring
• Standard service protocol

For developers:

• No boilerplate code
• Focus on business
• Prototypes can go to production :)
• Abstraction for backends
• Reuse API client code

Cliquet is not a «golden hammer».

• Protocol ?
•   Monitoring ?
•     Storage|cache ?
•       CRUD ?

• Reading List, store and synchronize articles
• Kinto, a generic storage service
  (with JavaScript and Python clients)
• SyncTo, a proxy to Firefox Sync API for kinto.js