com.mozilla.secops.postprocessing

Class PostProcessing.WatchlistAnalyze

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<Boolean,Alert>,Alert>

com.mozilla.secops.postprocessing.PostProcessing.WatchlistAnalyze

All Implemented Interfaces:

DocumentingTransform, Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

PostProcessing

public static class PostProcessing.WatchlistAnalyze
extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<Boolean,Alert>,Alert>
implements DocumentingTransform

Check incoming alert events against a watchlist of various identifiers.

Uses Watchlist to retrieve the watchlist from datastore and check these entries against alert metadata keys.

Since state based DoFn requires keyed input, we just expect a dummy key here to satisfy that requirement. Elements passed to the function can all be keyed with the same value.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
WatchlistAnalyze(PostProcessing.PostProcessingOptions options) Initialize WatchlistAnalyze with PostProcessing.PostProcessingOptions

Method Summary

Modifier and Type	Method and Description
String	getTransformDoc() Get documentation string from transform based on it's current configuration
void	onExpiry(org.apache.beam.sdk.transforms.DoFn.OnTimerContext c, org.apache.beam.sdk.state.BagState<Alert> alertBuffer, org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount)
void	onStale(org.apache.beam.sdk.transforms.DoFn.OnTimerContext c, org.apache.beam.sdk.state.BagState<Alert> alertBuffer, org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount)
void	processElement(org.apache.beam.sdk.transforms.DoFn.ProcessContext c, org.apache.beam.sdk.transforms.windowing.BoundedWindow w, org.apache.beam.sdk.state.BagState<Alert> alertBuffer, org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount, org.apache.beam.sdk.state.Timer alertExpiry, org.apache.beam.sdk.state.Timer alertStale)
void	setup()
void	teardown()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WatchlistAnalyze

public WatchlistAnalyze(PostProcessing.PostProcessingOptions options)

Initialize WatchlistAnalyze with PostProcessing.PostProcessingOptions

Parameters:

options - PostProcessing.PostProcessingOptions

Method Detail

getTransformDoc

public String getTransformDoc()

Get documentation string from transform based on it's current configuration

Specified by:

getTransformDoc in interface DocumentingTransform

Returns:

String

setup

@DoFn.Setup
public void setup()
                       throws IOException

Throws:

IOException

teardown

@DoFn.Teardown
public void teardown()
                             throws IOException

Throws:

IOException

processElement

@DoFn.ProcessElement
public void processElement(org.apache.beam.sdk.transforms.DoFn.ProcessContext c,
                                                org.apache.beam.sdk.transforms.windowing.BoundedWindow w,
                                                @DoFn.StateId(value="alertBuffer")
                                                org.apache.beam.sdk.state.BagState<Alert> alertBuffer,
                                                @DoFn.StateId(value="alertBufferCount")
                                                org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount,
                                                @DoFn.TimerId(value="alertExpiry")
                                                org.apache.beam.sdk.state.Timer alertExpiry,
                                                @DoFn.TimerId(value="alertStale")
                                                org.apache.beam.sdk.state.Timer alertStale)

onStale

@DoFn.OnTimer(value="alertStale")
public void onStale(org.apache.beam.sdk.transforms.DoFn.OnTimerContext c,
                                                      @DoFn.StateId(value="alertBuffer")
                                                      org.apache.beam.sdk.state.BagState<Alert> alertBuffer,
                                                      @DoFn.StateId(value="alertBufferCount")
                                                      org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount)

onExpiry

@DoFn.OnTimer(value="alertExpiry")
public void onExpiry(org.apache.beam.sdk.transforms.DoFn.OnTimerContext c,
                                                        @DoFn.StateId(value="alertBuffer")
                                                        org.apache.beam.sdk.state.BagState<Alert> alertBuffer,
                                                        @DoFn.StateId(value="alertBufferCount")
                                                        org.apache.beam.sdk.state.ValueState<Integer> alertBufferCount)