com.mozilla.secops.input

Class InputElement

java.lang.Object

com.mozilla.secops.input.InputElement

All Implemented Interfaces:

Serializable

public class InputElement
extends Object
implements Serializable

InputElement represents a set of input sources that will always result in a single output collection.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InputElement(String name) Create new InputElement

Method Summary

Modifier and Type	Method and Description
InputElement	addFileInput(String input) Add a new file input
InputElement	addKinesisInput(String input) Add new Kinesis input
InputElement	addPubsubInput(String input) Add new Pubsub input
InputElement	addWiredStream(org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>> s) Add wired stream
org.apache.beam.sdk.values.PCollection<Event>	expandElement(org.apache.beam.sdk.values.PBegin begin, String project) Expand configured input types into a resulting collection of parsed events
org.apache.beam.sdk.values.PCollection<String>	expandElementRaw(org.apache.beam.sdk.values.PBegin begin, String project) Expand configured input types into a resulting collection of strings
static InputElement	fromPipelineOptions(String name, InputOptions options, String cfgTickMessage) Return an InputElement given pipeline options
EventFilter	getEventFilter() Get event filter
ArrayList<String>	getFileInputs() Get file inputs
ArrayList<String>	getKinesisInputs() Get Kinesis inputs
String	getName() Get element name
ParserCfg	getParserConfiguration() Get parser configuration
ArrayList<String>	getPubsubInputs() Get Pubsub inputs
InputElement	setConfigurationTicks(String cfgTickMessage, Integer cfgTickInterval, long cfgTickMax) Set configuration ticks for input element
InputElement	setEventFilter(EventFilter filter) Set event filter to use with parsed reads
void	setFileInputs(ArrayList<String> fileInputs) Set file inputs
void	setKinesisInputs(ArrayList<String> kinesisInputs) Set Kinesis inputs
void	setParentInput(Input parent) Set parent Input object
InputElement	setParserConfiguration(ParserCfg parserCfg) Set the parser configuration to use with parsed reads
void	setPubsubInputs(ArrayList<String> pubsubInputs) Set Pubsub inputs

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputElement

public InputElement(String name)

Create new InputElement

Parameters:

name - Name to associate with element

Method Detail

fromPipelineOptions

public static InputElement fromPipelineOptions(String name,
                                               InputOptions options,
                                               String cfgTickMessage)

Return an InputElement given pipeline options

This can be used to configure an input element using settings present in InputOptions.

Parameters:

name - Name to associate with element

options - Input options

cfgTickMessage - Configuration tick message, null if not enabled

Returns:

New element

getName

public String getName()

Get element name

Returns:

Element name

expandElementRaw

public org.apache.beam.sdk.values.PCollection<String> expandElementRaw(org.apache.beam.sdk.values.PBegin begin,
                                                                       String project)

Expand configured input types into a resulting collection of strings

Parameters:

begin - Pipeline begin using PBegin

project - GCP project, set if using RuntimeSecrets

Returns:

PCollection of raw events

expandElement

public org.apache.beam.sdk.values.PCollection<Event> expandElement(org.apache.beam.sdk.values.PBegin begin,
                                                                   String project)

Expand configured input types into a resulting collection of parsed events

Parameters:

begin - Pipeline begin using PBegin

project - GCP project, set if using RuntimeSecrets

Returns:

PCollection of Event objects

setParserConfiguration

public InputElement setParserConfiguration(ParserCfg parserCfg)

Set the parser configuration to use with parsed reads

Specifies the parser configuration that will be used with the parser when the resulting collection returned from the element is of type Event.

Parameters:

parserCfg - Parser configuration

Returns:

this for chaining

getParserConfiguration

public ParserCfg getParserConfiguration()

Get parser configuration

Returns:

Parser configuration or null if unset

setEventFilter

public InputElement setEventFilter(EventFilter filter)

Set event filter to use with parsed reads

A valid parser configuration must be installed prior to calling this method.

Parameters:

filter - EventFilter

Returns:

this for chaining

getEventFilter

public EventFilter getEventFilter()

Get event filter

Returns:

Configured event filter or null

setConfigurationTicks

public InputElement setConfigurationTicks(String cfgTickMessage,
                                          Integer cfgTickInterval,
                                          long cfgTickMax)

Set configuration ticks for input element

Note that although InputElement is serializable to/from JSON, configuration ticks are specifically excluded from this. If an input element is initialized from JSON and configuration ticks are desired, these must be set manually.

Parameters:

cfgTickMessage - JSON message string to use

cfgTickInterval - Tick interval in seconds

cfgTickMax - Maximum number of ticks to generate before exiting

Returns:

this for chaining

addWiredStream

public InputElement addWiredStream(org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<String>> s)

Add wired stream

Parameters:

s - Wired input transform

Returns:

this for chaining

addFileInput

public InputElement addFileInput(String input)

Add a new file input

Parameters:

input - File input path

Returns:

this for chaining

setFileInputs

public void setFileInputs(ArrayList<String> fileInputs)

Set file inputs

Parameters:

fileInputs - File inputs

getFileInputs

public ArrayList<String> getFileInputs()

Get file inputs

Returns:

File inputs

addPubsubInput

public InputElement addPubsubInput(String input)

Add new Pubsub input

Parameters:

input - Pubsub topic

Returns:

this for chaining

setPubsubInputs

public void setPubsubInputs(ArrayList<String> pubsubInputs)

Set Pubsub inputs

Parameters:

pubsubInputs - Pubsub inputs

getPubsubInputs

public ArrayList<String> getPubsubInputs()

Get Pubsub inputs

Returns:

Pubsub inputs

addKinesisInput

public InputElement addKinesisInput(String input)

Add new Kinesis input

Parameters:

input - Kinesis input specification

Returns:

this for chaining

setKinesisInputs

public void setKinesisInputs(ArrayList<String> kinesisInputs)

Set Kinesis inputs

Parameters:

kinesisInputs - Kinesis inputs

getKinesisInputs

public ArrayList<String> getKinesisInputs()

Get Kinesis inputs

Returns:

Kinesis inputs

setParentInput

public void setParentInput(Input parent)

Set parent Input object

This is an internal method and should not generally be called directly.

Parameters:

parent - Parent input object