com.neurosys.etl.readers

Class FileReader<T,S>

java.lang.Object

com.neurosys.etl.process.EventAwareProcess

com.neurosys.etl.readers.BaseIterativeProcessor<TUPLE_IN,RECORD_OUT,STATE>

com.neurosys.etl.readers.AbstractReader<File,T,S>

com.neurosys.etl.readers.FileReader<T,S>

All Implemented Interfaces:

FailFastAware, Identifiable, ParamsAware, BatchProcess, ControllableProcess<S>, Process, RealTimeLineReader<T>, ResourceAwareRealTimeLineReader<T>, TransformAwareReader, TupleContextAwareReader<File,T>, Serializable, Iterator<T>

public class FileReader<T,S>
extends AbstractReader<File,T,S>

For a given Root Folder (File as a Resource), read each file in all the directories (nested optional); and for each file return a n entity of type T.

Author:

Arjun Dhar


Bean hierarchy

 FileReader
        - transformer : FileToProductsTransformer
                - Predicate filter : NULL
                - Lookup entitiesLookup : FileToProductLookup
                        - productService : xxx
                        - Pattern extractionPattern : xxx
                        - relatedFieldName : xxx
                - Transformer pathTransformer : FileToPathTransformer
                - FacetMeta contentFacet : xxx
        - Predicate filter : FileResourceFilter
                - Pattern fileNamePattern : xxx
        

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface com.neurosys.etl.process.Process

Process.Event, Process.EventPoint

Field Summary

Fields

Modifier and Type	Field and Description
protected Iterator<File>	fileIter
protected boolean	recurse If true, will recurse through sub-folders and drill down

Fields inherited from class com.neurosys.etl.readers.BaseIterativeProcessor

exceptions, failFast, filter, hasNext, index, messages, params, paused, resource, transformer

Fields inherited from class com.neurosys.etl.process.EventAwareProcess

listeners, processState

Constructor Summary

Constructors

Constructor and Description
FileReader() Default Constructor to allow lazy initialization If so, init() must be called by the calling API explicitly after calling BaseIterativeProcessor.setResource(Resource)
FileReader(org.springframework.core.io.Resource resource) The resource is the root File path to scan/read from

Method Summary

Modifier and Type	Method and Description
void	dispose() Un-initialize reader, so that it maybe re consumed Except for the main Resource & configuration set by Spring context all other instance variables used will be re-set/disposed/cleared.
Long	getCurrentRawIndex(File file) Get the raw index/position on the source; BaseIterativeProcessor.getCurrentIndex() only tracks the relevant position.
String	getCurrentTupleIdentifier() {BaseIterativeProcessor.getCurrentIndex() keeps track of the number or index, but for many readers it maybe useful to have some Name or description define the current Tuple.
S	getState() Get current state of process
boolean	hasNext() True if there is another record that can be read by the parser, false implies end of parse/read
void	init() Allow lazy initialization of Reader in cases where the resource may not be known @ PostConstruct time
boolean	isInitialized() Allow the calling API to determine if the reader has been initialized via RealTimeLineReader.init() or PostConstruct phase If not RealTimeLineReader.init() should be called upon the instance.
boolean	isRecurse() If true, will recurse through sub-folders and drill down
protected File	nextRecord()
void	resume()
void	resumeFromState(S state) Resume process from the given State provided
void	setRecurse(boolean recurse) If true, will recurse through sub-folders and drill down
Long	size() Get number of tuples/records in resource

Methods inherited from class com.neurosys.etl.readers.BaseIterativeProcessor

close, filter, getCurrentIndex, getExceptions, getFilter, getId, getMessages, getMeta, getParams, getResource, getSourceLocation, getTransformer, handleMiscException, isFailFast, isPaused, kill, next, nextTupleResult, pause, remove, setExceptions, setFailFast, setFilter, setId, setMessages, setMeta, setParams, setResource, setTransformer, stop, transform

Methods inherited from class com.neurosys.etl.process.EventAwareProcess

addListener, getListeners, onChangeHandler, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

recurse

protected boolean recurse

If true, will recurse through sub-folders and drill down

Default value:

true

fileIter

protected Iterator<File> fileIter

Constructor Detail

FileReader

public FileReader(org.springframework.core.io.Resource resource)

The resource is the root File path to scan/read from

FileReader

public FileReader()

Default Constructor to allow lazy initialization
If so, init() must be called by the calling API explicitly after calling BaseIterativeProcessor.setResource(Resource)

Method Detail

init

@PostConstruct
public void init()
                         throws IOException,
                                Exception

Description copied from interface: RealTimeLineReader

Allow lazy initialization of Reader in cases where the resource may not be known @ PostConstruct time

Specified by:

init in interface RealTimeLineReader<T>

Overrides:

init in class BaseIterativeProcessor<File,T,S>

Throws:

Exception

IOException

dispose

public void dispose()

Description copied from interface: RealTimeLineReader

Un-initialize reader, so that it maybe re consumed
Except for the main Resource & configuration set by Spring context all other instance variables used will be re-set/disposed/cleared.

isInitialized

public boolean isInitialized()
                      throws Exception

Description copied from interface: RealTimeLineReader

Allow the calling API to determine if the reader has been initialized via RealTimeLineReader.init() or PostConstruct phase
If not RealTimeLineReader.init() should be called upon the instance.

Throws:

Exception

hasNext

public boolean hasNext()

Description copied from interface: RealTimeLineReader

True if there is another record that can be read by the parser, false implies end of parse/read

Specified by:

hasNext in interface RealTimeLineReader<T>

Specified by:

hasNext in interface Iterator<T>

Overrides:

hasNext in class BaseIterativeProcessor<File,T,S>

Returns:

boolean

resume

public void resume()
            throws Exception

Specified by:

resume in interface ControllableProcess<S>

Overrides:

resume in class BaseIterativeProcessor<File,T,S>

Throws:

Exception

nextRecord

protected File nextRecord()

Specified by:

nextRecord in class BaseIterativeProcessor<File,T,S>

size

public Long size()

Description copied from interface: ResourceAwareRealTimeLineReader

Get number of tuples/records in resource

Returns:

Long

resumeFromState

public void resumeFromState(S state)
                     throws Exception

Description copied from interface: ControllableProcess

Resume process from the given State provided

Throws:

Exception

getCurrentRawIndex

public Long getCurrentRawIndex(File file)

Description copied from class: BaseIterativeProcessor

Get the raw index/position on the source; BaseIterativeProcessor.getCurrentIndex() only tracks the relevant position. This method allows us to get the position on the source.
In many cases/implementations this maybe used internally to derive the BaseIterativeProcessor.getCurrentIndex()

Specified by:

getCurrentRawIndex in class BaseIterativeProcessor<File,T,S>

Returns:

Long

getState

public S getState()
           throws Exception

Description copied from interface: ControllableProcess

Get current state of process

Throws:

Exception

isRecurse

public boolean isRecurse()

If true, will recurse through sub-folders and drill down

Default value:

true

setRecurse

public void setRecurse(boolean recurse)

If true, will recurse through sub-folders and drill down

Default value:

true

getCurrentTupleIdentifier

public String getCurrentTupleIdentifier()

Description copied from class: BaseIterativeProcessor

{BaseIterativeProcessor.getCurrentIndex() keeps track of the number or index, but for many readers it maybe useful to have some Name or description define the current Tuple.
Implementation dependent. by default value is same as {BaseIterativeProcessor.getCurrentIndex()

Overrides:

getCurrentTupleIdentifier in class BaseIterativeProcessor<File,T,S>

Returns: