Mozilla Telemetry docType Monitor

Monitors a specific docType for volume, shape, size, duplicate, ingestion_error and latency issues by normalied channel name.

• volume - monitors for inactivity (no data) and optionally a percent change in the number of submissions compared to the same 24 hour time period of the previous week

• shape - SAX analysis comparing the same 24 hour time period of the previous week for a shape exceeding the configured mindist

• size - monitors the previous 24 hours for a percent change in the average submission size compared to the configured value

• duplicate - monitors the previous 24 hours for a duplicate rate exceeding the configured value

• ingestion_error - monitors the last hour for an ingestion error rate exceeding the configured value

• latency - monitors a one hour tumbling window of latency distributions where X% of submissions are not greater than Y hours latent

1. Sample Configuration

filename = 'moz_telemetry_doctype_monitor.lua'
docType = "main"
message_matcher = 'Fields[docType] == "'.. docType .. '" && (Type == "telemetry" || Type == "telemetry.error"  || Type == "telemetry.duplicate")'
ticker_interval = 60
preserve_data = true

alert = {
  disabled = false,
  prefix = true,
  throttle = 1440, -- default to once a day (inactivity and ingestion_error default to 90)
  modules = {
    email = {recipients = {"trink@mozilla.com"}},
  },

  thresholds = { -- map of normalized channel names
    release = {
      volume = {
        inactivity_timeout = 5, -- minutes
        percent_delta = 10, -- optional
      },
      size = {
        average = 15529, -- bytes
        percent_delta = 10,
      },
      ingestion_error = {
        percent = 0.5,
      },
      duplicate = {
        percent = 2,
      },
      shape = {
        mindist = 0,
      },
      latency = {
        hours = 24,     -- 1 - 192
                        -- everything in the future is added to the first bucket
                        -- everything in the past is added to the last bucket
        percent = 33,   -- alerts if more than 33% of submission are greater than 24 hours latent
      }
    }
  }
}

source code: moz_telemetry_doctype_monitor.lua