weka.dl4j.iterators.instance.sequence.text

Class AbstractTextEmbeddingIterator

java.lang.Object

weka.dl4j.iterators.instance.AbstractInstanceIterator

weka.dl4j.iterators.instance.sequence.AbstractSequenceInstanceIterator

weka.dl4j.iterators.instance.sequence.text.AbstractTextEmbeddingIterator

All Implemented Interfaces:

java.io.Serializable, weka.core.OptionHandler

Direct Known Subclasses:

CnnTextEmbeddingInstanceIterator, RnnTextEmbeddingInstanceIterator

public abstract class AbstractTextEmbeddingIterator
extends AbstractSequenceInstanceIterator

Abstract text iterator that provides variables and methods for text processing.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
org.deeplearning4j.models.embeddings.wordvectors.WordVectors	wordVectors Loaded word vectors

Constructor Summary

Constructors

Constructor and Description
AbstractTextEmbeddingIterator()

Method Summary

Modifier and Type	Method and Description
org.deeplearning4j.iterator.LabeledSentenceProvider	getSentenceProvider(weka.core.Instances data) Create a sentence provider from the given data.
Dl4jAbstractStopwords	getStopwords()
TokenizerFactory	getTokenizerFactory()
TokenPreProcess	getTokenPreProcess()
int	getTruncateLength()
java.io.File	getWordVectorLocation()
org.deeplearning4j.models.embeddings.wordvectors.WordVectors	getWordVectors()
void	initialize() Initialize the iterator
void	initWordVectors() Initialize the word vectors from the given file
void	setStopwords(Dl4jAbstractStopwords stopwords)
void	setTokenizerFactory(TokenizerFactory tokenizerFactory)
void	setTokenPreProcess(TokenPreProcess tokenPreProcess)
void	setTruncateLength(int truncateLength)
void	setWordVectorLocation(java.io.File file) Set the word vector location and try to initialize it
void	setWordVectors(org.deeplearning4j.models.embeddings.wordvectors.WordVectors wordVectors)

Methods inherited from class weka.dl4j.iterators.instance.AbstractInstanceIterator

getDataSetIterator, getDataSetIterator, getOptions, getTrainBatchSize, listOptions, setOptions, setTrainBatchSize, validate

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface weka.core.OptionHandler

makeCopy

Field Detail

wordVectors

public transient org.deeplearning4j.models.embeddings.wordvectors.WordVectors wordVectors

Loaded word vectors

Constructor Detail

AbstractTextEmbeddingIterator

public AbstractTextEmbeddingIterator()

Method Detail

initWordVectors

public void initWordVectors()

Initialize the word vectors from the given file

getTruncateLength

@OptionMetadata(displayName="truncation length",
                description="The maximum number of tokens per document (default = 100).",
                commandLineParamName="truncationLength",
                commandLineParamSynopsis="-truncationLength <int>",
                displayOrder=2)
public int getTruncateLength()

setTruncateLength

public void setTruncateLength(int truncateLength)

getWordVectorLocation

@OptionMetadata(displayName="location of word vectors",
                description="The word vectors location.",
                commandLineParamName="wordVectorLocation",
                commandLineParamSynopsis="-wordVectorLocation <string>",
                displayOrder=3)
public java.io.File getWordVectorLocation()

setWordVectorLocation

public void setWordVectorLocation(java.io.File file)

Set the word vector location and try to initialize it

Parameters:

file - Word vector location

getTokenPreProcess

@OptionMetadata(displayName="token pre processor",
                description="The token pre processor.",
                commandLineParamName="tokenPreProcessor",
                commandLineParamSynopsis="-tokenPreProcessor <string>",
                displayOrder=4)
public TokenPreProcess getTokenPreProcess()

setTokenPreProcess

public void setTokenPreProcess(TokenPreProcess tokenPreProcess)

getTokenizerFactory

@OptionMetadata(displayName="tokenizer factory",
                description="The tokenizer factory.",
                commandLineParamName="tokenizerFactory",
                commandLineParamSynopsis="-tokenizerFactory <string>",
                displayOrder=5)
public TokenizerFactory getTokenizerFactory()

setTokenizerFactory

public void setTokenizerFactory(TokenizerFactory tokenizerFactory)

getStopwords

@OptionMetadata(displayName="stop words",
                description="The stop words to use.",
                commandLineParamName="stopWords",
                commandLineParamSynopsis="-stopWords <string>",
                displayOrder=5)
public Dl4jAbstractStopwords getStopwords()

setStopwords

public void setStopwords(Dl4jAbstractStopwords stopwords)

getWordVectors

@ProgrammaticProperty
public org.deeplearning4j.models.embeddings.wordvectors.WordVectors getWordVectors()

setWordVectors

@ProgrammaticProperty
public void setWordVectors(org.deeplearning4j.models.embeddings.wordvectors.WordVectors wordVectors)

initialize

public void initialize()

Description copied from class: AbstractInstanceIterator

Initialize the iterator

Overrides:

initialize in class AbstractInstanceIterator

getSentenceProvider

public org.deeplearning4j.iterator.LabeledSentenceProvider getSentenceProvider(weka.core.Instances data)

Create a sentence provider from the given data.

Parameters:

data - Data

Returns:

Sentence provider