Mozilla Security Heavy Hitters

For events matching the message_matcher, this analysis plugin identifies the number of events seen for a given identification field (e.g., the IP address in an nginx log). Can be utilized to trend heavy usage from specific identifiers. Event counts per identifier are stored in a count-min sketch.

During each ticker interval, the plugin calculates the average requests per identifier within the window (event frequency); identifiers that have made requests that exceeed the average + the calculated threshold cap will be captured in the plugin output. Note the threshold is calculated on each interval using the formula shown in the sample configuration.

The plugin by default outputs collected data as TSV; if the send_iprepd configuration option is enabled the plugin will instead output data as iprepd violation messages (for consumption by for example the iprepd output module).

Heavy hitters will be identified on each interval tick, so ensure the ticker_interval parameter is set to a value appropriate for consumption of the intended event stream to ensure the gathered sample is sufficient.

If enable_metrics is true, the module will submit metrics events for collection by the metrics output sandbox. Ensure timer_event_inject_limit is set appropriately, as if enabled timer_event will submit up to 2 messages (the violation notice, and the metric event).

1. Sample Configuration

filename = "moz_security_heavy_hitters.lua"
message_matcher = "Logger == 'input.nginx'"
ticker_interval = 60
preserve_data = false

id_field = "Fields[remote_addr]" -- field to use as the identifier
-- id_field_capture = ",? *([^,]+)$",  -- optional e.g. extract the last entry in a comma delimited list

-- list_max_size = 10000 -- optional, defaults to 10000 (maximum number of heavy hitter IDs to track)
-- send_iprepd = false -- optional, if true plugin will generate iprepd violation messages
-- violation_type = "fxa:heavy_hitter_ip" -- required in violations mode, iprepd violation type

threshold_cap = 10 -- Threshold will be calculated average + (calculated average * cap)
-- threshold_min = 100 -- optional calculated threshold minimum otherwise ignore interval
-- cms_epsilon = 1 / 10000 -- optional CMS value for epsilon
-- cms_delta = 0.0001 -- optional CMS value for delta

-- enable_metrics = false -- optional, if true enable secmetrics submission

source code: moz_security_hh.lua