Hindsight
1. Overview
Hindsight is a C based data processing infrastructure based on the lua sandbox project. I have received several inquiries about a lighter weight and faster data pipeline with delivery guarantees to replace Heka. Hindsight is that light weight skeleton around the same lua sandbox offering 'at least once' delivery semantics.
So how much lighter and faster? see: Performance
2. Table of Contents
- Architecture
- Configuration
- Command Line Version
- Heka Sandbox Documentation
- Sandbox Extensions Documentation
2.1. Differences from Heka
- Sandbox API Differences
- The message matcher now uses Lua string match patterns instead of RE2 expressions.
- Looping messages in Heka (injecting messages back to an earlier point in the pipeline) has always been a bad idea so it was removed. Most looping requirements can be flattened out: i.e., alerting can be handled in the output plugins, aggregation/sessionization can be handled in the inputs.
- Splitters/Decoders/Encoders no longer exist they simply become part of the input/output plugins respectively. The common functionality should now be broken out into Lua modules to allow for re-use in different plugins.
- Checkpoint are all managed by the Hindsight infrastructure (so much of the burden is removed from the plugin writer, this also alters the plugin API slightly and should greatly lower the checkpoint related IOPS when compared to Heka).