Developer documentation for working on Merino

tl;dr

Here are some useful commands when working on Merino.

Run the main app

$ docker-compose -f dev/docker-compose.yaml up -d
$ cargo run -p merino

Run tests

$ docker-compose -f dev/docker-compose.yaml up -d
$ cargo test

Run dependency servers

$ cd dev
$ docker-compose up

Documentation

You can generate documentation, both code level and book level, for Merino and all related crates by running ./dev/make-all-docs.sh. You'll need mdBook, which you can get with cargo install mdbook.

Pre-built code docs are also available.

Local configuration

The default configuration of Merino is development, which has human-oriented logging and debugging enabled. For settings that you wish to change in the development configuration, you have two options, listed below.

For full details, make sure to check out the documentation for Merino's setting system.

Update the defaults

If the change you want to make makes the system better for most development tasks, consider adding it to config/development.yaml, so that other developers can take advantage of it. You can look at config/base.yaml, which defines all requires configuration, to see an example of the structure.

It is not suitable to put secrets in config/development.yaml.

Create a local override

For local changes to adapt to your machine or tastes, you can put the configuration in config/local.yaml. These file doesn't exist by default. These changes won't be a part of the git history, so it is safe to put secrets here, if needed. Importantly, it should never be required to have a local.yaml to run Merino in a development setting.

Repository structure

This project is structured as a Cargo Workspace that contains one crate for each broad area of behavior for Merino. This structure is advantageous because the crates can be handled either individually or as a group. When compiling, each crate can be compiled in parallel, where dependencies allow, and when running tests, each test suite can be run separately or together. This also provides an advantage if we choose to re-use any of these crates in other projects, or if we publish the crates to Crates.io.

Project crates

This is a brief overview of the crates found in the repository. For more details, see the specific crate docs.

merino

This is the main Merino application, and one of the binary crates in the repository. It brings together and configures the other crates to create a production-like environment for Firefox Suggest.

merino-settings

This defines and documents the settings of the application. These settings should be initialized by one of the binary crates, and passed into the other crates to configure them.

merino-web

This crate provides an HTTP API to access Merino, including providing observability into the running of the application via that API.

merino-suggest

This is a domain crate that defines the data model and traits needed to provide suggestions to Firefox.

merino-cache

This crate contains domain models and behavior for Merino's caching functionality.

merino-adm

This crate provides integration with the AdMarketplace APIs, and implements the traits from merino-suggest.

merino-showroom

This is not a Rust crate, but instead a small Javascript application. It can be used to test Merino during development and demos.

merino-integration-tests

This crate is a separate test system. It works much like merino, in that it brings together the other crates to produce a complete Merino environment. However, this binary crate produces an application that exercise the service as a whole, instead of providing a server to manual test against.

merino-integration-tests-macro

This crate provides a procmacro used in merino-integration-tests. Rust requires that procmacros be in their own crate.

  • rust-analyzer - IDE-like tools for many editors. This provides easy access to type inference and documentation while editing Rust code, which can make the development process much easier.
  • cargo-watch - A Cargo subcommand that re-runs a task when files change. Very useful for things like cargo watch -x clippy or cargo watch -x "test -- merino-adm".

These works have influenced the design of Merino.