weka.dl4j.text.tokenization.tokenizer

Class WekaTokenizer

java.lang.Object

weka.dl4j.text.tokenization.tokenizer.WekaTokenizer

All Implemented Interfaces:

java.io.Serializable, org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

public class WekaTokenizer
extends java.lang.Object
implements org.deeplearning4j.text.tokenization.tokenizer.Tokenizer, java.io.Serializable

A DeepLearning4j's Tokenizer interface to Weka Tokenizer.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
WekaTokenizer(java.lang.String content, weka.core.tokenizers.Tokenizer wekaTokenizer) initializes the Object

Method Summary

Modifier and Type	Method and Description
int	countTokens()
java.util.List<java.lang.String>	getTokens()
boolean	hasMoreTokens()
java.lang.String	nextToken()
void	setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess tokenPreProcessor)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WekaTokenizer

public WekaTokenizer(java.lang.String content,
                     weka.core.tokenizers.Tokenizer wekaTokenizer)

initializes the Object

Parameters:

content - the String to tokenize

wekaTokenizer - the WekaTokenizer Object

Method Detail

countTokens

public int countTokens()

Specified by:

countTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

getTokens

public java.util.List<java.lang.String> getTokens()

Specified by:

getTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

setTokenPreProcessor

public void setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess tokenPreProcessor)

Specified by:

setTokenPreProcessor in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

hasMoreTokens

public boolean hasMoreTokens()

Specified by:

hasMoreTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

nextToken

public java.lang.String nextToken()

Specified by:

nextToken in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer