com.mozilla.secops.input

Class Input

java.lang.Object

com.mozilla.secops.input.Input

All Implemented Interfaces:

Serializable

public class Input
extends Object
implements Serializable

Standard data ingestion

The Input class can be used to configured and execute various forms of pipeline raw event ingestion from external sources.

The class supports two different modes of operation; simplex mode and multiplex mode.

In simplex mode, raw events are read from one or more input sources, and the resulting data set is output in a single flat collection as is.

In multiplex mode, raw events are read from one or more elements, where each element can have one or more input sources. The resulting data set is output in key value pairs, with the key being the element name and the values being data read from the input streams belonging to that particular element.

In both simplex and multiplex mode, the reads can occur in either raw mode or a parsed mode. In raw mode, no attempted parsing of a raw entry will occur, and the resulting data set will simply consist of the raw string data.

To enable parsed mode a parser configuration must be supplied. In this mode, the raw data set will be passed through the parser with the resulting data set consistent of Event objects instead of raw strings.

A filter can be associated with parsed read operations to filter events based on what is desired for a particular element.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Input.MultiplexReader Read raw events from configured sources, returning a key value collection with the key being the element name and the value being a parsed event
static class	Input.MultiplexReaderRaw Read raw events from configured sources, returning a key value collection with the key being the element name and the value being a raw string
static class	Input.SimplexReader Read raw events from configured sources, combining all events into a single output collection as Event objects
static class	Input.SimplexReaderRaw Read raw events from configured sources, combining all events into a single output collection as strings

Field Summary

Fields

Modifier and Type	Field and Description
static String	SIMPLEX_DEFAULT_ELEMENT Default simplex element name

Constructor Summary

Constructors

Constructor and Description
Input() Create new input object
Input(String project) Create new input object

Method Summary

Modifier and Type	Method and Description
static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>>	compositeInputAdapter(InputOptions options, String cfgTick) Adapter to simplify Input usage for pipelines that used previous composite input tranform
Input	fromPipelineOptions(InputOptions options, String cfgTickMessage) Configure input using specified InputOptions
InputCollectionCache	getCache() Request input collection cache
InputElement	getInputElementByName(String name) Get an input element by name
ArrayList<InputElement>	getInputElements() Get input elements
com.mozilla.secops.input.Input.OperatingMode	getOperatingMode() Get operating mode
String	getProject() Get project
Input	multiplex() Enable multiplex mode
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<String,Event>>>	multiplexRead() Return a transform that will ingest data, and emit parsed events in multiplex mode
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<String,String>>>	multiplexReadRaw() Return a transform that will ingest data, and emit raw events in multiplex mode
void	setInputElements(ArrayList<InputElement> elements) Set input elements
void	setOperatingMode(com.mozilla.secops.input.Input.OperatingMode mode) Set operating mode
void	setProject(String project) Set project
Input	simplex() Enable simplex input mode
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<Event>>	simplexRead() Return a transform that will ingest data, and emit parsed events in simplex mode
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>>	simplexReadRaw() Return a transform that will ingest data, and emit raw strings in simplex mode
Input	withInputElement(InputElement el) Add input element

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SIMPLEX_DEFAULT_ELEMENT

public static final String SIMPLEX_DEFAULT_ELEMENT

Default simplex element name

See Also:

Constant Field Values

Constructor Detail

Input

public Input()

Create new input object

Input

public Input(String project)

Create new input object

This variant should be used if any input options require KMS decryption, typically the case for pipelines executing in Dataflow.

Parameters:

project - GCP project

Method Detail

getCache

public InputCollectionCache getCache()

Request input collection cache

This is an internal method and should not generally be called directly.

Returns:

InputCollectionCache

compositeInputAdapter

public static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>> compositeInputAdapter(InputOptions options,
                                                                                                                                                                String cfgTick)
                                                                                                                                                         throws IOException

Adapter to simplify Input usage for pipelines that used previous composite input tranform

Parameters:

options - InputOptions

cfgTick - Configuration tick message, null if not enabled

Returns:

PTransform

Throws:

IOException - IOException

setOperatingMode

public void setOperatingMode(com.mozilla.secops.input.Input.OperatingMode mode)

Set operating mode

Parameters:

mode - Operating mode

getOperatingMode

public com.mozilla.secops.input.Input.OperatingMode getOperatingMode()

Get operating mode

Returns:

Operating mode

setProject

public void setProject(String project)

Set project

Parameters:

project - Project string

getProject

public String getProject()

Get project

Returns:

Project string

setInputElements

public void setInputElements(ArrayList<InputElement> elements)

Set input elements

Parameters:

elements - Input element array

getInputElements

public ArrayList<InputElement> getInputElements()

Get input elements

Returns:

Input elements

getInputElementByName

public InputElement getInputElementByName(String name)

Get an input element by name

Parameters:

name - Input element name

Returns:

Element or null if not found

simplex

public Input simplex()

Enable simplex input mode

Returns:

this for chaining

fromPipelineOptions

public Input fromPipelineOptions(InputOptions options,
                                 String cfgTickMessage)
                          throws IOException

Configure input using specified InputOptions

This method is only valid in simplex operating mode.

Parameters:

options - Pipeline input options

cfgTickMessage - Configuration tick message, null if not enabled

Returns:

this for chaining

Throws:

IOException - IOException

multiplex

public Input multiplex()

Enable multiplex mode

Returns:

this for chaining

withInputElement

public Input withInputElement(InputElement el)
                       throws IOException

Add input element

Parameters:

el - Input element

Returns:

this for chaining

Throws:

IOException - IOException

simplexReadRaw

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>> simplexReadRaw()

Return a transform that will ingest data, and emit raw strings in simplex mode

Returns:

PTransform

simplexRead

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<Event>> simplexRead()

Return a transform that will ingest data, and emit parsed events in simplex mode

Returns:

PTransform

multiplexReadRaw

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<String,String>>> multiplexReadRaw()

Return a transform that will ingest data, and emit raw events in multiplex mode

Returns:

PTransform

multiplexRead

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<String,Event>>> multiplexRead()

Return a transform that will ingest data, and emit parsed events in multiplex mode

Returns:

PTransform