0.1. Hindsight Main Configuration

0.1.1. Configuration Variables

  • output_path - base path where the Heka protobuf streams, checkpoints, state files, and statistics are stored
    • input (directory) - stores the Heka protobuf stream generated by all input plugins
    • analysis (directory) - stores the Heka protobuf stream generated by all analysis plugins
    • hindsight.cp - checkpoint file for all input, analysis and output threads
    • plugins.tsv - performance metrics for all running plugins
    • utilization.tsv - performance metrics for each thread
  • output_size - size at which the output files are rolled (bytes, default 64MiB)
  • sandbox_load_path - base path that Hindsight scans for new cfgs and Lua (string, default "" (dynamic loading disabled)). If the string is not empty the following directory structure must exist under the base path and be writable by Hindsight:
    • input (directory) - input plugin queue
    • analysis (directory) - analysis plugin queue
    • output (directory) - output plugin queue
  • sandbox_run_path - base path containing the running cfgs and dynamically loaded lua. The following directory structure must exist under the base path and be writable by Hindsight when dynamic loading is configured:
    • input (directory) - input plugins
    • analysis (directory) - analysis plugins
    • output (directory) - output plugins
  • sandbox_install_path - base path to the install location of the sandbox packages default: /usr/share/luasandbox/sandboxes/heka. This path should contain the input, analysis, and output subdirectory structure.
  • analysis_threads - number of analysis threads to run (count, max 64)
  • analysis_utilization_limit - percent utilization where dynamic loading is blocked (0-100 default 95 (0 to disable))
  • analysis_lua_path - path used by the analysis plugins to look for Lua modules
  • analysis_lua_cpath - path used by the analysis plugins to look for Lua C modules
  • io_lua_path - path used by the input and output plugins to look for Lua modules
  • io_lua_cpath - path used by the input and output plugins to look for Lua C modules
  • max_message_size - maximum size of a valid message (bytes, default 64KiB)
  • backpressure - delta between the writer queue file and the slowest reader, when exceeded backpressure is applied (message injection will be slowed) until the writer and reader are both on the same file (count, default 0 (no backpressure)) e.g.
    backpressure = 10 -- [writer = 100.log, slowest reader = 89.log, delta = 11]
  • backpressure_disk_free - Number of output file units (N * <output_size>). Backpressure is applied (message injection will be slowed) until the free space rises above this threshold (count, default 4 (0 to disable)) e.g.
    backpressure_disk_free = 4 -- [256MiB when using the defaults]
  • hostname - hostname used in logging/messages (default gethostname())
output_path             = "output"
output_size             = 64 * 1024 * 1024
sandbox_load_path       = "load"
sandbox_run_path        = "run"
sandbox_install_path    = "/usr/share/luasandbox/sandboxes/heka"
analysis_threads        = 1
analysis_lua_path       = "/usr/lib/luasandbox/modules/?.lua"
analysis_lua_cpath      = "/usr/lib/luasandbox/modules/?.so"
io_lua_path             = analysis_lua_path ..  ";/usr/lib/luasandbox/io_modules/?.lua"
io_lua_cpath            = analysis_lua_cpath .. ";/usr/lib/luasandbox/io_modules/?.so"
max_message_size        = 64 * 1024
backpressure            = 0
backpressure_disk_free  = 4
-- hostname                = "hindsight.example.com"

input_defaults = {
  -- see: Default Sandbox Configuration Variables
  -- output_limit           = 64 * 1024
  -- memory_limit           = 8 * 1024 * 1024
  -- instruction_limit      = 1e6
  -- preserve_data          = false
  -- restricted_headers     = false
  -- ticker_interval        = 0
  -- shutdown_on_terminate  = false
}

analysis_defaults = {
  -- see: Default Sandbox Configuration Variables
}

output_defaults = {
  -- see: Default Sandbox Configuration Variables
  -- remove_checkpoints_on_terminate = false
  -- read_queue = "both"
}

0.2. Hindsight Sandbox Configuration

0.2.1. Default Sandbox Configuration Variables

  • output_limit - the largest message an input or analysis plugin can inject into Hindsight (bytes, default 64KiB)
  • memory_limit - the maximum amount of memory a plugin can use before being terminated (bytes, default 8MiB)
  • instruction_limit - the maximum number of Lua instructions a plugin can execute in a single process_message or timer_event function call (count, default 1MM)
  • preserve_data - flag indicating if all global data should be saved on shutdown (bool, default false)
  • restricted_headers - flag indicating that the following header fields are not user modifiable Hostname and Logger (bool, default true (analysis), false (input/output))
  • ticker_interval (seconds, default 0)
    • For input plugins it is the poll interval when process_message is called see: polling input plugins
    • For analysis and output plugins it is the amount of time between timer_event function calls
  • shutdown_on_terminate - cleanly shuts down Hindsight if this plugin is terminated (bool, default false)

0.2.2. Default Analysis Sandbox Configuration Variables

  • process_message_inject_limit - the maximum number of messages the process_message function can inject in a single invocation (count, default 0 (none)).

  • timer_event_inject_limit - the maximum number of messages the timer_event function can inject in a single invocation (count, default 10).

0.2.3. Default Output Sandbox Configuration Variables

  • remove_checkpoints_on_terminate - removes the checkpoint entries when the plugin is terminated (bool, default false). This prevents the data from being pruned until the plugin can be fixed but it can also cause the system to back pressure as it will start filling the disk.
  • read_queue - specifies which queue the output plugin consumes (both|input|analysis) defaults to both.

0.2.4. Common Plugin Configuration Variables

  • filename - plugin source file (must have no path component and a .lua extension). Search order: sandbox_run_path, sandbox_install_path
Reserved (Hindsight populates the following variables in the configuration)
  • Logger - configuration filename without the .cfg extension prefixed with the sandbox type e.g., analysis.myplugin
  • Pid - process ID
  • Hostname - configuration hostname.
  • path - Lua module search path (corresponding to the type of plugin)
  • cpath - Lua C module search path (corresponding to the type of plugin)
  • log_level - Integer specifying the syslog severity level, when set to debug (7) the sandbox print function will be wired into the Hindsight logger
For input/output plugins these additional Hindsight configuration options are available from read_config()
  • output_path
  • output_size
  • max_message_size
  • sandbox_load_path
  • sandbox_run_path
  • sandbox_install_path
User Defined
  • key (string | number) - user defined variable with a (string | number | boolean | table) value
User Defined Private Variable
  • _*key* (string) - user defined variable beginning with an underscore (allows filtering in UI displays)

0.2.5. Input Plugin Configuration Variables

0.2.6. Analysis Plugin Configuration Variables

0.2.7. Output Plugin Configuration Variables

results matching ""

    No results matching ""