Merino Contract Tests - Client

Overview

This documentation describes a Python-based test framework for the contract tests. The HTTP client used in the framework supports:

Requests to Kinto (Remote Settings) for record population
Requests for suggestions from Merino, with response checks

For more details on contract test design, refer to the contract-tests documentation.

Scenarios

The client is instructed on request and response check actions via scenarios, recorded in the scenarios.yml file. A scenario is defined by a name, a description and steps.

Name

A test name should identify the use case under test. Names are written in snake case with double-underscores __ for scenario and behavior delineation.

Example: remote_settings__refresh

Description

A test description should outline, in greater detail, the purpose of a test, meaning the feature or common user interaction being verified.

Example: Test that Merino successfully returns refreshed output in the cases of suggestion content updates and additions

Steps

Kinto Service

The Kinto service scenario step populates the records used by the Merino service
Kinto request fields:
- service - Set the value to kinto, to direct requests to the Kinto service
- delay - (optional) Set seconds to pause before execution of request
- record_id - Set the ID that will correspond with the records specified in filename
- filename - Set the file with records to upload. The files are located in ..\volumes\kinto
- data_type - Set to data or offline-expansion-data
A Kinto service scenario step will not check a response defined in the scenario, but will raise an error in the event of an unexpected HTTP response
All records populated by Kinto service scenario steps are deleted as part of the scenario teardown

Example:

- request:
    service: kinto
    record_id: "data-01"
    filename: "data-01.json"
    data_type: "data"

Merino Service

The Merino service scenario step sends queries to merino and checks the validity of the responses
Merino request fields:
- service - Set the value to merino, to direct requests to the Merino service
- delay - (optional) Set seconds to pause before execution of request
- method - Set the HTTP request method
- path - Set the query and parameters for the request method
- headers - Set a list of HTTP request headers
Merino response fields:
- status_code - Set the expected HTTP response status code
- content - Set a list of expected merino suggestion content
  - The request_id is excluded from verification and can be set to null in the scenario

Example:

- name: suggest__apple
  description: Test that Merino successfully returns a suggestion
  steps:
    - request:
        service: merino
        method: GET
        path: '/api/v1/suggest?q=apple'
        headers:
          - name: User-Agent
            value: 'Mozilla/5.0 (Windows NT 10.0; rv:10.0) Gecko/20100101 Firefox/91.0'
          - name: Accept-Language
            value: 'en-US'
      response:
        status_code: 200
        content:
          client_variants: []
          server_variants: []
          request_id: null
          suggestions:
            - block_id: 1
              full_keyword: 'apple'
              title: 'Wikipedia - Apple'
              url: 'https://en.wikipedia.org/wiki/Apple'
              impression_url: 'https://127.0.0.1/'
              click_url: 'https://127.0.0.1/'
              provider: 'test_provider'
              advertiser: 'test_advertiser'
              is_sponsored: false
              icon: 'https://en.wikipedia.org/favicon.ico'
              score: 0.0

To execute the test scenarios outside the client Docker container, create a Python virtual environment, set environment variables, expose the Merino and Kinto API ports in the docker-compose.yml and use a pytest command. It is recommended to execute the tests within a Python virtual environment to prevent dependency cross contamination.

Create a Virtual Environment

The developer documentation on dependences for working on Merino dependencies, provides instruction on creating a virtual environment via pyenv, and installing all requirements via poetry.
Setup Environment Variables

The following environment variables are set in docker-compose.yml, but will require local setup via command line, pytest.ini file or IDE configuration:
- MERINO_URL: The URL of the Merino service
  - Example: MERINO_URL=http://localhost:8000
- KINTO_URL: The URL of the Kinto service
  - Example: KINTO_URL=http://localhost:8888
- KINTO_ATTACHMENTS_URL: The URL of the Kinto Attachments service
  - Example: KINTO_ATTACHMENTS_URL=http://localhost:80
- KINTO_BUCKET: The ID of the Kinto bucket to create
  - Example: KINTO_BUCKET=main
- KINTO_COLLECTION: The ID of the Kinto collection to create
  - Example: KINTO_COLLECTION=quicksuggest
- KINTO_DATA_DIR: The directory containing advertiser data
  - Example: KINTO_DATA_DIR=tests/contract/volumes/kinto
- SCENARIOS_FILE: The directory containing contract test scenario files
  - Example: SCENARIOS_FILE=tests/contract/volumes/client/scenarios.yml
Modify tests/contract/docker-compose.yml

In the merino definition, expose port 8000 by adding the following:
```
ports:
  - "8000:8000"
```
In the kinto definition, expose port 8888 by adding the following:
```
ports:
  - "8888:8888"
```
In the kinto-attachments definition, expose port 80 by adding the following:
```
ports:
  - "80:80"
```

Run merino, kinto-setup, kinto and kinto-attachment docker containers

Execute the following from the project root:

 docker-compose \
   -f tests/contract/docker-compose.yml \
   -p merino-py-contract-tests \
   up merino

Run the contract tests

Execute the following from the project root:
```
pytest tests/contract/client/tests/test_merino.py -vv
```
- Tests can be run individually using -k expr.
  
  Example executing the remote_settings__refresh scenario:
```
pytest tests/contract/client/tests/test_merino.py -vv -k remote_settings__refresh
```

Merino Book