Mozilla nginx_moz_ingest Common Decoder Module

Handles the common moz_ingest tasks such as de-duplication, geo lookup and error handling before passing the data off to the correct subdecoder.

1. Decoder Configuration Table

decoders_moz_ingest_common = {
    sub_decoders = { -- required
        -- _namespace_ (string) = Decoder module name
        -- a namespace of "*" can be used to specify a default decoder
        telemetry  = "decoders.moz_ingest.telemetry",
    },

    -- error_on_missing_sub_decoder = false, -- optional

    -- String used to specify GeoIP city database location on disk.
    city_db_file = "/usr/share/geoip/GeoIP2-City.mmdb", -- optional, if not specified no city/country geoip lookup is performed
    geo_report_subdivisions = true, -- optional, if not specified then subdivision1 and subdivision2 aren't reported
    -- geo_ip_keep = false, -- optional, when specified the IP address is saved in the `geoIP` field

    -- String used to specify geoname cities > (1000|5000|15000) csv location on disk.
    -- Cities not in this list will be considered too specific and won't report geoCity.
    city_size_file = "/usr/share/geoip/cities1000.txt", -- optional, if not specified no limiting is performed

    isp_db_file = "/usr/share/geoip/GeoIP2-ISP.mmdb", -- optional
    isp_docTypes = {customStudy = true}, -- docTypes to perform ISP geoip lookups on, must be set if isp_db_file is defined, use `["*"] = true` to enable for everything

    -- WARNING if the cuckoo filter settings are altered the plugin's
    -- `preservation_version` should be incremented
    -- number of items in each de-duping cuckoo filter partition
    cf_items = 32e6, -- optional, if not provided de-duping is disabled

    -- number of partitions, each containing `cf_items`
    -- cf_partitions = 4 -- optional default 4

    -- interval size in minutes for cuckoo filter pruning
    -- cf_interval_size = 6, -- optional, default 6 (25.6 hours)
}

2. Functions

2.1. transform_message

Transform and inject the message using the provided stream reader.

Arguments

• hsr (hsr) - stream reader with the message to process

Return

• throws on error

2.2. decode

Decode and inject the message given as argument, using a module-internal stream reader.

Arguments

• msg (string) - Heka protobuf string to decode

Return

• throws on error

source code: common.lua