BaseIterativeProcessor (stumps 1.0-SNAPSHOT API)

java.lang.Object
- com.neurosys.etl.process.EventAwareProcess
- - com.neurosys.etl.readers.BaseIterativeProcessor<TUPLE_IN,RECORD_OUT,STATE>

All Implemented Interfaces:

FailFastAware, Identifiable, ParamsAware, BatchProcess, ControllableProcess<STATE>, Process, RealTimeLineReader<RECORD_OUT>, ResourceAwareRealTimeLineReader<RECORD_OUT>, TransformAwareReader, TupleContextAwareReader<TUPLE_IN,RECORD_OUT>, Serializable, Iterator<RECORD_OUT>

Direct Known Subclasses:

AbstractProcessBasedReporter, AbstractReader
```
public abstract class BaseIterativeProcessor<TUPLE_IN,RECORD_OUT,STATE>
extends EventAwareProcess
implements Iterator<RECORD_OUT>, TupleContextAwareReader<TUPLE_IN,RECORD_OUT>, ResourceAwareRealTimeLineReader<RECORD_OUT>, TransformAwareReader, Process, FailFastAware, ParamsAware, ControllableProcess<STATE>, BatchProcess, Identifiable
```
Support generic Process Aware aspects for Read operations and their boiler plate management

Author:

Arjun Dhar

See Also:

Serialized Form

Nested Class Summary
- Nested classes/interfaces inherited from interface com.neurosys.etl.process.Process
  Process.Event, Process.EventPoint

Field Summary

Fields
Modifier and Type	Field and Description
`protected Collection<Throwable>`	`exceptions` Used if failFast is off (false)
`protected boolean`	`failFast` If true will exit on first exception else will collect exceptions
`protected org.apache.commons.collections.Predicate`	`filter` Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps. If the filter predicate returns false, then it will not be available in `next()` The `Predicate.evaluate(Object)` may return false or even throw a `RuntimeException`.
`protected boolean`	`hasNext` If turned off then hasNext wont work and processes will terminate
`protected long`	`index`
`protected Collection<String>`	`messages` Messages generated during the process
`protected Object[]`	`params` Additional Params for the ETL Process
`protected boolean`	`paused`
`protected org.springframework.core.io.Resource`	`resource`
`protected org.apache.commons.collections.Transformer`	`transformer` Converts raw record TUPLE_IN Object to type RECORD_OUT

Fields inherited from class com.neurosys.etl.process.EventAwareProcess
listeners, processState

Constructor Summary

Constructors
Constructor and Description

BaseIterativeProcessor(org.springframework.core.io.Resource resource)

Constructors
Constructor and Description
`BaseIterativeProcessor(org.springframework.core.io.Resource resource)`

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()` Any cleanup
`protected boolean`	`filter(TUPLE_IN tuple)` A false will filter out the record, i.e.
`Long`	`getCurrentIndex()` Depending on the resource the `ResourceAwareRealTimeLineReader.size()` may factor in redundant rows also that maybe filtered by the reader if it uses some sort of filter.
`abstract Long`	`getCurrentRawIndex(TUPLE_IN row)` Get the raw index/position on the source; `getCurrentIndex()` only tracks the relevant position.
`String`	`getCurrentTupleIdentifier()` {`getCurrentIndex()` keeps track of the number or index, but for many readers it maybe useful to have some Name or description define the current Tuple.
`Collection<Throwable>`	`getExceptions()` Used if failFast is off (false) to get the exceptions that occurred in the run/load
`org.apache.commons.collections.Predicate`	`getFilter()` Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps. If the filter predicate returns false, then it will not be available in `next()` The `Predicate.evaluate(Object)` may return false or even throw a `RuntimeException`.
`Object`	`getId()` An optional id to identify the Process.
`Collection<String>`	`getMessages()` Messages generated during the process
`String`	`getMeta()` Meta Data like Description etc.
`Object[]`	`getParams()` Additional Params for the ETL Process
`org.springframework.core.io.Resource`	`getResource()`
`String`	`getSourceLocation()`
`org.apache.commons.collections.Transformer`	`getTransformer()` Converts raw record TUPLE_IN Object to type RECORD_OUT
`protected void`	`handleMiscException(RuntimeException e)` Can override if we want to insert audits and other handlers for misc exception handling
`boolean`	`hasNext()` True if there is another record that can be read by the parser, false implies end of parse/read
`void`	`init()` Allow lazy initialization of Reader in cases where the resource may not be known @ `PostConstruct` time
`boolean`	`isFailFast()` If true will exit on first exception else will collect exceptions
`boolean`	`isPaused()`
`void`	`kill()` Kill a process
`RECORD_OUT`	`next()` Fetch only next Valid filtered record else it may/will return a RuntimeException This method has been marked final by the public API, please refer to if you wish to override its functionality
`protected abstract TUPLE_IN`	`nextRecord()`
`Map.Entry<TUPLE_IN,RECORD_OUT>`	`nextTupleResult()` Get the next Output entity but also return the original Input tuple that resulted in the Output.
`void`	`pause()` Pause till resume
`void`	`remove()`
`void`	`resume()`
`void`	`setExceptions(Collection<Throwable> exceptions)` Used if failFast is off (false) to get the exceptions that occurred in the run/load
`void`	`setFailFast(boolean failFast)` If true will exit on first exception else will collect exceptions
`void`	`setFilter(org.apache.commons.collections.Predicate filter)` Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps. If the filter predicate returns false, then it will not be available in `next()` The `Predicate.evaluate(Object)` may return false or even throw a `RuntimeException`.
`void`	`setId(Object id)` An optional id to identify the Process.
`void`	`setMessages(Collection<String> messages)` Messages generated during the process
`void`	`setMeta(String meta)` Meta Data like Description etc.
`void`	`setParams(Object[] params)` Additional Params for the ETL Process
`void`	`setResource(org.springframework.core.io.Resource resource)`
`void`	`setTransformer(org.apache.commons.collections.Transformer transformer)` Converts raw record TUPLE_IN Object to type RECORD_OUT
`void`	`stop()` Request graceful Shutdown
`protected RECORD_OUT`	`transform(TUPLE_IN row)`

Methods inherited from class com.neurosys.etl.process.EventAwareProcess
addListener, getListeners, onChangeHandler, setListeners

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator
forEachRemaining

Methods inherited from interface com.neurosys.etl.readers.ResourceAwareRealTimeLineReader
size

Methods inherited from interface com.neurosys.etl.readers.RealTimeLineReader
dispose, isInitialized

Methods inherited from interface com.neurosys.etl.process.ControllableProcess
getState, resumeFromState

- Field Detail
  - resource
```
protected org.springframework.core.io.Resource resource
```
  - index
```
protected long index
```
  - failFast
```
protected boolean failFast
```
    If true will exit on first exception else will collect exceptions
  - exceptions
```
protected Collection<Throwable> exceptions
```
    Used if failFast is off (false)
  - messages
```
protected Collection<String> messages
```
    Messages generated during the process
  - params
```
protected Object[] params
```
    Additional Params for the ETL Process
  - transformer
```
protected org.apache.commons.collections.Transformer transformer
```
    Converts raw record TUPLE_IN Object to type RECORD_OUT
    
    Can be null:
    
    true, if null then TUPLE_IN = RECORD_OUT
  - filter
```
protected org.apache.commons.collections.Predicate filter
```
    Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps.
    If the filter predicate returns false, then it will not be available in next()
    
    The Predicate.evaluate(Object) may return false or even throw a RuntimeException.
    
    Can be null:
    
    true
  - hasNext
```
protected boolean hasNext
```
    If turned off then hasNext wont work and processes will terminate
  - paused
```
protected boolean paused
```
- Constructor Detail
  - BaseIterativeProcessor
```
public BaseIterativeProcessor(org.springframework.core.io.Resource resource)
```
- Method Detail
  - init
```
public void init()
          throws Exception
```
    Description copied from interface: RealTimeLineReader
    
    Allow lazy initialization of Reader in cases where the resource may not be known @ PostConstruct time
    
    Specified by:
    
    init in interface RealTimeLineReader<RECORD_OUT>
    
    Throws:
    
    Exception
  - transform
```
protected RECORD_OUT transform(TUPLE_IN row)
```
  - nextRecord
```
protected abstract TUPLE_IN nextRecord()
```
  - filter
```
protected boolean filter(TUPLE_IN tuple)
```
    A false will filter out the record, i.e. not consider it.
    
    Parameters:
    
    tuple - as
    
    Returns:
    
    boolean
  - nextTupleResult
```
public Map.Entry<TUPLE_IN,RECORD_OUT> nextTupleResult()
```
    Description copied from interface: TupleContextAwareReader
    
    Get the next Output entity but also return the original Input tuple that resulted in the Output.
    
    Specified by:
    
    nextTupleResult in interface TupleContextAwareReader<TUPLE_IN,RECORD_OUT>
    
    Returns:
    
    Map.Entry of Input to Output key-value pair
  - next
```
public final RECORD_OUT next()
```
    Fetch only next Valid filtered record else it may/will return a RuntimeException
    This method has been marked final by the public API, please refer to if you wish to override its functionality
    
    Specified by:
    
    next in interface RealTimeLineReader<RECORD_OUT>
    
    Specified by:
    
    next in interface Iterator<RECORD_OUT>
    
    Returns:
    
    Record
  - handleMiscException
```
protected void handleMiscException(RuntimeException e)
```
    Can override if we want to insert audits and other handlers for misc exception handling
    
    Parameters:
    
    e - as RuntimeException
  - close
```
public void close()
           throws Exception
```
    Description copied from interface: RealTimeLineReader
    
    Any cleanup
    
    Specified by:
    
    close in interface RealTimeLineReader<RECORD_OUT>
    
    Throws:
    
    Exception
  - setResource
```
public void setResource(org.springframework.core.io.Resource resource)
```
  - getResource
```
public org.springframework.core.io.Resource getResource()
```
  - getSourceLocation
```
public String getSourceLocation()
```
    Specified by:
    
    getSourceLocation in interface RealTimeLineReader<RECORD_OUT>
  - remove
```
public void remove()
```
    Specified by:
    
    remove in interface Iterator<RECORD_OUT>
  - getTransformer
```
public org.apache.commons.collections.Transformer getTransformer()
```
    Converts raw record TUPLE_IN Object to type RECORD_OUT
    
    Specified by:
    
    getTransformer in interface TransformAwareReader
    
    Can be null:
    
    true, if null then TUPLE_IN = RECORD_OUT
  - setTransformer
```
public void setTransformer(org.apache.commons.collections.Transformer transformer)
```
    Converts raw record TUPLE_IN Object to type RECORD_OUT
    
    Specified by:
    
    setTransformer in interface TransformAwareReader
    
    Can be null:
    
    true, if null then TUPLE_IN = RECORD_OUT
  - getFilter
```
public org.apache.commons.collections.Predicate getFilter()
```
    Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps.
    If the filter predicate returns false, then it will not be available in next()
    
    The Predicate.evaluate(Object) may return false or even throw a RuntimeException.
    
    Can be null:
    
    true
  - setFilter
```
public void setFilter(org.apache.commons.collections.Predicate filter)
```
    Filter lines or records that should be ignored perhaps; Records that the user may have used for formatting etc perhaps.
    If the filter predicate returns false, then it will not be available in next()
    
    The Predicate.evaluate(Object) may return false or even throw a RuntimeException.
    
    Can be null:
    
    true
  - getCurrentIndex
```
public Long getCurrentIndex()
```
    Description copied from interface: ResourceAwareRealTimeLineReader
    
    Depending on the resource the ResourceAwareRealTimeLineReader.size() may factor in redundant rows also that maybe filtered by the reader if it uses some sort of filter.
    This method allows to keep track of the real position, since the calling API may/will not be notified of skipped rows. Numbering starts from 0.
    
    Size and getCurrentIndex reflect on valid rows being processed. But there maybe noise and extra un-processable rows due to which getCurrentIndex() on the last record maybe < what size() gives.
    Hence @ the end of your process normalize to 100% when we are sure its done without errors/abort.
    
    Specified by:
    
    getCurrentIndex in interface ResourceAwareRealTimeLineReader<RECORD_OUT>
    
    Returns:
  - getCurrentTupleIdentifier
```
public String getCurrentTupleIdentifier()
```
    {getCurrentIndex() keeps track of the number or index, but for many readers it maybe useful to have some Name or description define the current Tuple.
    Implementation dependent. by default value is same as {getCurrentIndex()
    
    Returns:
  - getCurrentRawIndex
```
public abstract Long getCurrentRawIndex(TUPLE_IN row)
```
    Get the raw index/position on the source; getCurrentIndex() only tracks the relevant position. This method allows us to get the position on the source.
    In many cases/implementations this maybe used internally to derive the getCurrentIndex()
    
    Parameters:
    
    -
    
    Returns:
    
    Long
  - setFailFast
```
public void setFailFast(boolean failFast)
```
    If true will exit on first exception else will collect exceptions
    
    Specified by:
    
    setFailFast in interface FailFastAware
  - isFailFast
```
public boolean isFailFast()
```
    If true will exit on first exception else will collect exceptions
    
    Specified by:
    
    isFailFast in interface FailFastAware
  - getExceptions
```
public Collection<Throwable> getExceptions()
```
    Used if failFast is off (false) to get the exceptions that occurred in the run/load
    
    Specified by:
    
    getExceptions in interface FailFastAware
    
    Returns:
  - setExceptions
```
public void setExceptions(Collection<Throwable> exceptions)
```
    Used if failFast is off (false) to get the exceptions that occurred in the run/load
    
    Specified by:
    
    setExceptions in interface FailFastAware
  - getMessages
```
public Collection<String> getMessages()
```
    Messages generated during the process
    
    Specified by:
    
    getMessages in interface FailFastAware
    
    Returns:
  - setMessages
```
public void setMessages(Collection<String> messages)
```
    Messages generated during the process
    
    Specified by:
    
    setMessages in interface FailFastAware
  - getParams
```
public Object[] getParams()
```
    Additional Params for the ETL Process
    
    Specified by:
    
    getParams in interface ParamsAware
  - setParams
```
public void setParams(Object[] params)
```
    Additional Params for the ETL Process
    
    Specified by:
    
    setParams in interface ParamsAware
  - getId
```
public Object getId()
```
    An optional id to identify the Process. This is only required usually when there are multiple instances of the same Reader/Processor and we have to identify a reader of the same class from another instance.
    
    Specified by:
    
    getId in interface Identifiable
    
    Can be null:
    
    true
  - setId
```
public void setId(Object id)
```
    An optional id to identify the Process. This is only required usually when there are multiple instances of the same Reader/Processor and we have to identify a reader of the same class from another instance.
    
    Specified by:
    
    setId in interface Identifiable
    
    Can be null:
    
    true
  - getMeta
```
public String getMeta()
```
    Meta Data like Description etc. Can be plain text or JSON.
    Application defined.
    
    Can be null:
    
    true
  - setMeta
```
public void setMeta(String meta)
```
    Meta Data like Description etc. Can be plain text or JSON.
    Application defined.
    
    Can be null:
    
    true
  - hasNext
```
public boolean hasNext()
```
    Description copied from interface: RealTimeLineReader
    
    True if there is another record that can be read by the parser, false implies end of parse/read
    
    Specified by:
    
    hasNext in interface RealTimeLineReader<RECORD_OUT>
    
    Specified by:
    
    hasNext in interface Iterator<RECORD_OUT>
    
    Returns:
    
    boolean
  - stop
```
public void stop()
          throws Exception
```
    Description copied from interface: ControllableProcess
    
    Request graceful Shutdown
    
    Specified by:
    
    stop in interface ControllableProcess<STATE>
    
    Throws:
    
    Exception
  - pause
```
public void pause()
           throws Exception
```
    Description copied from interface: ControllableProcess
    
    Pause till resume
    
    Specified by:
    
    pause in interface ControllableProcess<STATE>
    
    Throws:
    
    Exception
  - isPaused
```
public boolean isPaused()
```
    Specified by:
    
    isPaused in interface ControllableProcess<STATE>
  - resume
```
public void resume()
            throws Exception
```
    Specified by:
    
    resume in interface ControllableProcess<STATE>
    
    Throws:
    
    Exception
  - kill
```
public void kill()
          throws Exception
```
    Description copied from interface: ControllableProcess
    
    Kill a process
    
    Specified by:
    
    kill in interface ControllableProcess<STATE>
    
    Throws:
    
    Exception

Class BaseIterativeProcessor<TUPLE_IN,RECORD_OUT,STATE>

Nested Class Summary

Nested classes/interfaces inherited from interface com.neurosys.etl.process.Process

Field Summary

Fields inherited from class com.neurosys.etl.process.EventAwareProcess

Constructor Summary

Method Summary

Methods inherited from class com.neurosys.etl.process.EventAwareProcess

Methods inherited from class java.lang.Object

Methods inherited from interface java.util.Iterator

Methods inherited from interface com.neurosys.etl.readers.ResourceAwareRealTimeLineReader

Methods inherited from interface com.neurosys.etl.readers.RealTimeLineReader

Methods inherited from interface com.neurosys.etl.process.ControllableProcess

Field Detail

resource

index

failFast

exceptions

messages

params

transformer

filter

hasNext

paused

Constructor Detail

BaseIterativeProcessor

Method Detail

init

transform

nextRecord

filter

nextTupleResult

next

handleMiscException

close

setResource

getResource

getSourceLocation

remove

getTransformer

setTransformer

getFilter

setFilter

getCurrentIndex

getCurrentTupleIdentifier

getCurrentRawIndex

setFailFast

isFailFast

getExceptions

setExceptions

getMessages

setMessages

getParams

setParams

getId

setId

getMeta

setMeta

hasNext

stop

pause

isPaused

resume

kill