Sub Decoder Utility Module

Common funtionality to instantiate an LPeg based sub decoder configuration.

1. Functions

1.1. load_sub_decoders

Returns a table of sub_decoder functions, keyed by sub_decoder_name.


  • sub_decoders (table/nil) sub_decoders configuration table

    sub_decoders = {
    -- sub_decoder_name (string) = (string/array) a sub_decoder_name of "*" is
    -- treated as the default when the name does not exist.
    -- string: decoder or grammar module name
    -- array: (string and/or array) list of specific messages to parse
      -- string: Sample message used to locate the correct grammar
         -- If no grammar matches the sample message then an error is thrown
         -- and another grammar or module must be added to the printf_messages
         -- configuration. If multiple grammars match the message, the first
         -- grammar with the most specific match is selected.
         -- Note: a special token of `<<DROP>>` and `<<FAIL>>` are reserved for
         -- the last entry in the array to handle the no match case; <<DROP>>
         -- silently discards the message and <<FAIL>> reports an error. If
         -- neither is specified the default no match behavior is to inject the
         -- original message produced by the parent decoder.
      -- array:
         -- column 1: (string/array)
            -- string: Sample message (see above) or a decoder module specification
              -- A decoder module specification is used when transformations need
              -- to be applied to a standard decoder output.
                -- Caveats:
                -- 1) the decoder output must be a Lua Heka message table
                -- 2) this must be the only entry in the configuration array
                -- 3) printf_messages must not be defined
            -- array:
              -- printf.build_grammar format specification
              -- module reference to a grammar builder function, any additional columns are passed to the function
              -- module reference to a an LPeg grammar, any additional columns are passed to match see [Carg](
         -- column 2: (table/nil)
            -- Transformation table with Heka message field name keys and a
            -- value of the fully qualified transformation function name
            -- `<module_name>#<function_name>`. The function returns no values but
            -- can error; it receives two arguments: the Heka message table and
            -- the field name to act on. The function can modify the message in
            -- any way.
    nginx  = "decoders.nginx.access", -- decoder module name
    kernel = "lpeg.linux.kernel",     -- grammar module name, must export an lpeg grammar named 'grammar' or 'syslog_grammar'
    -- kernel = "lpeg.linux.kernel#syslog_grammar", -- the above is a shorthand for the explicit grammar specification
    sshd = {
      -- openssh_portable auth message, imported in printf_messages
      {"Accepted publickey for foobar from port 4242 ssh2", {remote_addr = "geoip.heka#add_geoip"}},
    foo = {
      "/tmp/input.tsv:23: invalid line", -- custom log defined in printf_messages
      { {"Status: %s", "status"}, nil},  -- inline printf spec, no transformation
      { {"example#fn", arg1}, nil},      -- use fn(arg1) from the example module to build a grammar
      { {"example#foo"}, nil},           -- use the foo LPeg grammar from the example module
  • printf_messages (table/nil) see:


  • sub_decoders (table)

1.2. load_decoder

Returns the decode function for a single decoder. load_sub_decoder is an alias to this function and has been kept for backwards compatibility.


  • decoder (string/table) decoder configuration entry
  • printf_messages (table/nil) printf_message table (see above)


  • decode (function)

1.3. copy_message

Copies a message for use in decoder/subdecoder


  • src (table) Heka message table. This is a shallow copy of the individual values in the Fields hash and assumes they will be replaced as opposed to modified when they are tables. The main use of this function is to populate a new message with defaults.
  • mutable (bool/nil/none)


  • msg (table) a Heka message hash schema format

1.4. add_fields

Add the fields hash to the msg.Fields overwriting on collision.


  • msg (table) Heka message
  • fields (table) Heka message Fields hash


  • none - msg is modified in place

source code: sub_decoder_util.lua

results matching ""

    No results matching ""