public abstract class BaseIterativeProcessor<TUPLE_IN,RECORD_OUT,STATE> extends EventAwareProcess implements Iterator<RECORD_OUT>, TupleContextAwareReader<TUPLE_IN,RECORD_OUT>, ResourceAwareRealTimeLineReader<RECORD_OUT>, TransformAwareReader, Process, FailFastAware, ParamsAware, ControllableProcess<STATE>, BatchProcess, Identifiable
Process.Event, Process.EventPoint
Modifier and Type | Field and Description |
---|---|
protected Collection<Throwable> |
exceptions
Used if failFast is off (false)
|
protected boolean |
failFast
If true will exit on first exception else will collect exceptions
|
protected org.apache.commons.collections.Predicate |
filter
Filter lines or records that should be ignored perhaps;
Records that the user may have used for formatting etc perhaps.
If the filter predicate returns false, then it will not be available in next() The Predicate.evaluate(Object) may return false or even throw a RuntimeException . |
protected boolean |
hasNext
If turned off then hasNext wont work and processes will terminate
|
protected long |
index |
protected Collection<String> |
messages
Messages generated during the process
|
protected Object[] |
params
Additional Params for the ETL Process
|
protected boolean |
paused |
protected org.springframework.core.io.Resource |
resource |
protected org.apache.commons.collections.Transformer |
transformer
Converts raw record TUPLE_IN Object to type RECORD_OUT
|
listeners, processState
Constructor and Description |
---|
BaseIterativeProcessor(org.springframework.core.io.Resource resource) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Any cleanup
|
protected boolean |
filter(TUPLE_IN tuple)
A false will filter out the record, i.e.
|
Long |
getCurrentIndex()
Depending on the resource the
ResourceAwareRealTimeLineReader.size() may factor in redundant rows also that maybe filtered by the reader if it uses some sort of filter. |
abstract Long |
getCurrentRawIndex(TUPLE_IN row)
Get the raw index/position on the source;
getCurrentIndex() only tracks the relevant position. |
String |
getCurrentTupleIdentifier()
{
getCurrentIndex() keeps track of the number or index, but for many readers
it maybe useful to have some Name or description define the current Tuple. |
Collection<Throwable> |
getExceptions()
Used if failFast is off (false) to get the exceptions that occurred in the run/load
|
org.apache.commons.collections.Predicate |
getFilter()
Filter lines or records that should be ignored perhaps;
Records that the user may have used for formatting etc perhaps.
If the filter predicate returns false, then it will not be available in next() The Predicate.evaluate(Object) may return false or even throw a RuntimeException . |
Object |
getId()
An optional id to identify the Process.
|
Collection<String> |
getMessages()
Messages generated during the process
|
String |
getMeta()
Meta Data like Description etc.
|
Object[] |
getParams()
Additional Params for the ETL Process
|
org.springframework.core.io.Resource |
getResource() |
String |
getSourceLocation() |
org.apache.commons.collections.Transformer |
getTransformer()
Converts raw record TUPLE_IN Object to type RECORD_OUT
|
protected void |
handleMiscException(RuntimeException e)
Can override if we want to insert audits and other handlers for misc exception handling
|
boolean |
hasNext()
True if there is another record that can be read by the parser, false implies end of parse/read
|
void |
init()
Allow lazy initialization of Reader in cases where the resource may not be known @
PostConstruct time |
boolean |
isFailFast()
If true will exit on first exception else will collect exceptions
|
boolean |
isPaused() |
void |
kill()
Kill a process
|
RECORD_OUT |
next()
Fetch only next Valid filtered record
else it may/will return a RuntimeException
This method has been marked final by the public API, please refer to if you wish to override its functionality |
protected abstract TUPLE_IN |
nextRecord() |
Map.Entry<TUPLE_IN,RECORD_OUT> |
nextTupleResult()
Get the next Output entity but also return the original Input tuple that resulted in the Output.
|
void |
pause()
Pause till resume
|
void |
remove() |
void |
resume() |
void |
setExceptions(Collection<Throwable> exceptions)
Used if failFast is off (false) to get the exceptions that occurred in the run/load
|
void |
setFailFast(boolean failFast)
If true will exit on first exception else will collect exceptions
|
void |
setFilter(org.apache.commons.collections.Predicate filter)
Filter lines or records that should be ignored perhaps;
Records that the user may have used for formatting etc perhaps.
If the filter predicate returns false, then it will not be available in next() The Predicate.evaluate(Object) may return false or even throw a RuntimeException . |
void |
setId(Object id)
An optional id to identify the Process.
|
void |
setMessages(Collection<String> messages)
Messages generated during the process
|
void |
setMeta(String meta)
Meta Data like Description etc.
|
void |
setParams(Object[] params)
Additional Params for the ETL Process
|
void |
setResource(org.springframework.core.io.Resource resource) |
void |
setTransformer(org.apache.commons.collections.Transformer transformer)
Converts raw record TUPLE_IN Object to type RECORD_OUT
|
void |
stop()
Request graceful Shutdown
|
protected RECORD_OUT |
transform(TUPLE_IN row) |
addListener, getListeners, onChangeHandler, setListeners
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
forEachRemaining
size
dispose, isInitialized
getState, resumeFromState
protected org.springframework.core.io.Resource resource
protected long index
protected boolean failFast
protected Collection<Throwable> exceptions
protected Collection<String> messages
protected Object[] params
protected org.apache.commons.collections.Transformer transformer
protected org.apache.commons.collections.Predicate filter
next()
Predicate.evaluate(Object)
may return false or even throw a RuntimeException
.protected boolean hasNext
protected boolean paused
public BaseIterativeProcessor(org.springframework.core.io.Resource resource)
public void init() throws Exception
RealTimeLineReader
PostConstruct
timeinit
in interface RealTimeLineReader<RECORD_OUT>
Exception
protected RECORD_OUT transform(TUPLE_IN row)
protected abstract TUPLE_IN nextRecord()
protected boolean filter(TUPLE_IN tuple)
tuple
- as public Map.Entry<TUPLE_IN,RECORD_OUT> nextTupleResult()
TupleContextAwareReader
nextTupleResult
in interface TupleContextAwareReader<TUPLE_IN,RECORD_OUT>
Map.Entry
of Input to Output key-value pairpublic final RECORD_OUT next()
next
in interface RealTimeLineReader<RECORD_OUT>
next
in interface Iterator<RECORD_OUT>
protected void handleMiscException(RuntimeException e)
e
- as RuntimeException
public void close() throws Exception
RealTimeLineReader
close
in interface RealTimeLineReader<RECORD_OUT>
Exception
public void setResource(org.springframework.core.io.Resource resource)
public org.springframework.core.io.Resource getResource()
public String getSourceLocation()
getSourceLocation
in interface RealTimeLineReader<RECORD_OUT>
public void remove()
remove
in interface Iterator<RECORD_OUT>
public org.apache.commons.collections.Transformer getTransformer()
getTransformer
in interface TransformAwareReader
public void setTransformer(org.apache.commons.collections.Transformer transformer)
setTransformer
in interface TransformAwareReader
public org.apache.commons.collections.Predicate getFilter()
next()
Predicate.evaluate(Object)
may return false or even throw a RuntimeException
.public void setFilter(org.apache.commons.collections.Predicate filter)
next()
Predicate.evaluate(Object)
may return false or even throw a RuntimeException
.public Long getCurrentIndex()
ResourceAwareRealTimeLineReader
ResourceAwareRealTimeLineReader.size()
may factor in redundant rows also that maybe filtered by the reader if it uses some sort of filter.
getCurrentIndex
in interface ResourceAwareRealTimeLineReader<RECORD_OUT>
public String getCurrentTupleIdentifier()
getCurrentIndex()
keeps track of the number or index, but for many readers
it maybe useful to have some Name or description define the current Tuple.
getCurrentIndex()
public abstract Long getCurrentRawIndex(TUPLE_IN row)
getCurrentIndex()
only tracks the relevant position.
This method allows us to get the position on the source.
getCurrentIndex()
public void setFailFast(boolean failFast)
setFailFast
in interface FailFastAware
public boolean isFailFast()
isFailFast
in interface FailFastAware
public Collection<Throwable> getExceptions()
getExceptions
in interface FailFastAware
public void setExceptions(Collection<Throwable> exceptions)
setExceptions
in interface FailFastAware
public Collection<String> getMessages()
getMessages
in interface FailFastAware
public void setMessages(Collection<String> messages)
setMessages
in interface FailFastAware
public Object[] getParams()
getParams
in interface ParamsAware
public void setParams(Object[] params)
setParams
in interface ParamsAware
public Object getId()
getId
in interface Identifiable
public void setId(Object id)
setId
in interface Identifiable
public String getMeta()
public void setMeta(String meta)
public boolean hasNext()
RealTimeLineReader
hasNext
in interface RealTimeLineReader<RECORD_OUT>
hasNext
in interface Iterator<RECORD_OUT>
public void stop() throws Exception
ControllableProcess
stop
in interface ControllableProcess<STATE>
Exception
public void pause() throws Exception
ControllableProcess
pause
in interface ControllableProcess<STATE>
Exception
public boolean isPaused()
isPaused
in interface ControllableProcess<STATE>
public void resume() throws Exception
resume
in interface ControllableProcess<STATE>
Exception
public void kill() throws Exception
ControllableProcess
kill
in interface ControllableProcess<STATE>
Exception
Copyright © 2018. All rights reserved.