weka.dl4j.text.tokenization.tokenizer

Class TweetNLPTokenizer

java.lang.Object

weka.dl4j.text.tokenization.tokenizer.TweetNLPTokenizer

All Implemented Interfaces:

java.io.Serializable, org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

public class TweetNLPTokenizer
extends java.lang.Object
implements org.deeplearning4j.text.tokenization.tokenizer.Tokenizer, java.io.Serializable

A DeepLearning4j's Tokenizer interface for the CMU TweetNLP tokenizer.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
TweetNLPTokenizer(java.lang.String content) initializes the Object

Method Summary

Modifier and Type	Method and Description
int	countTokens()
java.util.List<java.lang.String>	getTokens()
boolean	hasMoreTokens()
java.lang.String	nextToken()
void	setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess tokenPreProcessor)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TweetNLPTokenizer

public TweetNLPTokenizer(java.lang.String content)

initializes the Object

Parameters:

content - the String to tokenize

Method Detail

countTokens

public int countTokens()

Specified by:

countTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

getTokens

public java.util.List<java.lang.String> getTokens()

Specified by:

getTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

setTokenPreProcessor

public void setTokenPreProcessor(org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess tokenPreProcessor)

Specified by:

setTokenPreProcessor in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

hasMoreTokens

public boolean hasMoreTokens()

Specified by:

hasMoreTokens in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

nextToken

public java.lang.String nextToken()

Specified by:

nextToken in interface org.deeplearning4j.text.tokenization.tokenizer.Tokenizer