weka.dl4j.text.tokenization.tokenizer.factory.impl

Class NGramTokenizerFactoryImpl

java.lang.Object

weka.dl4j.text.tokenization.tokenizer.factory.impl.NGramTokenizerFactoryImpl

All Implemented Interfaces:

java.io.Serializable, org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory, weka.core.OptionHandler

public class NGramTokenizerFactoryImpl
extends java.lang.Object
implements org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory, java.io.Serializable, weka.core.OptionHandler

A DeepLearning4j's TokenizerFactory interface for Weka core tokenizers.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
NGramTokenizerFactoryImpl()

Method Summary

Modifier and Type	Method and Description
org.deeplearning4j.text.tokenization.tokenizer.Tokenizer	create(java.io.InputStream toTokenize)
org.deeplearning4j.text.tokenization.tokenizer.Tokenizer	create(java.lang.String toTokenize)
java.lang.String	getDelimiters()
int	getNMax()
int	getNMin()
java.lang.String[]	getOptions() Gets the current settings of the Classifier.
org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess	getTokenPreProcessor()
java.lang.String	globalInfo() Returns a string describing this object.
java.util.Enumeration<weka.core.Option>	listOptions() Returns an enumeration describing the available options.
void	setDelimiters(java.lang.String delimiters)
void	setNMax(int nMax)
void	setNMin(int nMin)
void	setOptions(java.lang.String[] options) Parses a given list of options.
void	setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess preProcessor)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface weka.core.OptionHandler

makeCopy

Constructor Detail

NGramTokenizerFactoryImpl

public NGramTokenizerFactoryImpl()

Method Detail

create

public org.deeplearning4j.text.tokenization.tokenizer.Tokenizer create(java.lang.String toTokenize)

Specified by:

create in interface org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory

create

public org.deeplearning4j.text.tokenization.tokenizer.Tokenizer create(java.io.InputStream toTokenize)

Specified by:

create in interface org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory

getTokenPreProcessor

public org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess getTokenPreProcessor()

Specified by:

getTokenPreProcessor in interface org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory

setTokenPreProcessor

public void setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess preProcessor)

Specified by:

setTokenPreProcessor in interface org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory

globalInfo

public java.lang.String globalInfo()

Returns a string describing this object.

Returns:

a description of the object suitable for displaying in the explorer/experimenter gui

listOptions

public java.util.Enumeration<weka.core.Option> listOptions()

Returns an enumeration describing the available options.

Specified by:

listOptions in interface weka.core.OptionHandler

Returns:

an enumeration of all the available options.

getOptions

public java.lang.String[] getOptions()

Gets the current settings of the Classifier.

Specified by:

getOptions in interface weka.core.OptionHandler

Returns:

an array of strings suitable for passing to setOptions

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Parses a given list of options.

Specified by:

setOptions in interface weka.core.OptionHandler

Parameters:

options - the list of options as an array of strings

Throws:

java.lang.Exception - if an option is not supported

getNMax

@OptionMetadata(displayName="NMax",
                description="NGram max size.",
                commandLineParamName="NMax",
                commandLineParamSynopsis="-NMax <int>",
                displayOrder=0)
public int getNMax()

setNMax

public void setNMax(int nMax)

getNMin

@OptionMetadata(displayName="NMin",
                description="NGram min size.",
                commandLineParamName="NMin",
                commandLineParamSynopsis="-NMin <int>",
                displayOrder=1)
public int getNMin()

setNMin

public void setNMin(int nMin)

getDelimiters

@OptionMetadata(displayName="delimiters",
                description="Set of delimiter characters to use in tokenizing (\\r, \\n and \\t can be used for carriage-return, line-feed and tab).",
                commandLineParamName="delimiters",
                commandLineParamSynopsis="-delimiters <int>",
                displayOrder=2)
public java.lang.String getDelimiters()

setDelimiters

public void setDelimiters(java.lang.String delimiters)