LowerCaseTokenizer

Class Overview

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

Summary

[Expand] Inherited Fields
From class org.apache.lucene.analysis.Tokenizer protected Reader input The text source for this Tokenizer.

[Expand] Inherited Methods
From class org.apache.lucene.analysis.LetterTokenizer boolean isTokenChar(char c) Collects only characters which satisfy isLetter(char).
From class org.apache.lucene.analysis.CharTokenizer final void end() This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). final boolean incrementToken() Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. abstract boolean isTokenChar(char c) Returns true iff a character should be included in a token. char normalize(char c) Called on each token character to normalize it before it is added to the token. void reset(Reader input) Expert: Reset the tokenizer to a new reader.
From class org.apache.lucene.analysis.Tokenizer void close() By default, closes the input Reader. final int correctOffset(int currentOff) Return the corrected offset. void reset(Reader input) Expert: Reset the tokenizer to a new reader.
From class org.apache.lucene.analysis.TokenStream void close() Releases resources associated with this stream. void end() This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). abstract boolean incrementToken() Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. void reset() Resets this stream to the beginning.
From class org.apache.lucene.util.AttributeSource <A extends Attribute> A addAttribute(Class<A> attClass) The caller must pass in a Class<? extends Attribute> value. void addAttributeImpl(AttributeImpl att) Expert: Adds a custom AttributeImpl instance with one or more Attribute interfaces. AttributeSource.State captureState() Captures the state of all Attributes. void clearAttributes() Resets all Attributes in this AttributeSource by calling clear() on each Attribute implementation. AttributeSource cloneAttributes() Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. boolean equals(Object obj) <A extends Attribute> A getAttribute(Class<A> attClass) The caller must pass in a Class<? extends Attribute> value. Iterator<Class<? extends Attribute>> getAttributeClassesIterator() Returns a new iterator that iterates the attribute classes in the same order they were added in. AttributeSource.AttributeFactory getAttributeFactory() returns the used AttributeFactory. Iterator<AttributeImpl> getAttributeImplsIterator() Returns a new iterator that iterates all unique Attribute implementations. boolean hasAttribute(Class<? extends Attribute> attClass) The caller must pass in a Class<? extends Attribute> value. boolean hasAttributes() Returns true, iff this AttributeSource has any attributes int hashCode() void restoreState(AttributeSource.State state) Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. String toString()
From class java.lang.Object Object clone() boolean equals(Object arg0) void finalize() final Class<?> getClass() int hashCode() final void notify() final void notifyAll() String toString() final void wait() final void wait(long arg0, int arg1) final void wait(long arg0)
From interface java.io.Closeable abstract void close()