java.lang.Object | ||
↳ | org.apache.lucene.util.AttributeSource | |
↳ | org.apache.lucene.analysis.TokenStream |
Known Direct Subclasses |
Known Indirect Subclasses |
A TokenStream
enumerates the sequence of tokens, either from
Field
s of a Document
or from query text.
This is an abstract class; concrete subclasses are:
Tokenizer
, a TokenStream
whose input is a Reader; and
TokenFilter
, a TokenStream
whose input is another
TokenStream
.
TokenStream
API has been introduced with Lucene 2.9. This API
has moved from being Token
-based to Attribute
-based. While
Token
still exists in 2.9 as a convenience class, the preferred way
to store the information of a Token
is to use AttributeImpl
s.
TokenStream
now extends AttributeSource
, which provides
access to all of the token Attribute
s for the TokenStream
.
Note that only one instance per AttributeImpl
is created and reused
for every token. This approach reduces object creation and allows local
caching of references to the AttributeImpl
s. See
incrementToken()
for further details.
The workflow of the new TokenStream
API is as follows:
TokenStream
/TokenFilter
s which add/get
attributes to/from the AttributeSource
.
reset()
.
incrementToken()
until it returns false
consuming the attributes after each call.
end()
so that any end-of-stream operations
can be performed.
close()
to release any resource when finished
using the TokenStream
.
incrementToken()
.
You can find some example code for the new API in the analysis package level Javadoc.
Sometimes it is desirable to capture a current state of a TokenStream
,
e.g., for buffering purposes (see CachingTokenFilter
,
TeeSinkTokenFilter
). For this usecase
captureState()
and restoreState(AttributeSource.State)
can be used.
Protected Constructors | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
A TokenStream using the default attribute factory.
| |||||||||||
A TokenStream that uses the same attributes as the supplied one.
| |||||||||||
A TokenStream using the supplied AttributeFactory for creating new
Attribute instances. |
Public Methods | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Releases resources associated with this stream.
| |||||||||||
This method is called by the consumer after the last token has been
consumed, after
incrementToken() returned false
(using the new TokenStream API). | |||||||||||
Consumers (i.e.,
IndexWriter ) use this method to advance the stream to
the next token. | |||||||||||
Resets this stream to the beginning.
|
[Expand]
Inherited Methods | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
From class
org.apache.lucene.util.AttributeSource
| |||||||||||
From class
java.lang.Object
| |||||||||||
From interface
java.io.Closeable
|
A TokenStream using the default attribute factory.
A TokenStream that uses the same attributes as the supplied one.
A TokenStream using the supplied AttributeFactory for creating new Attribute
instances.
This method is called by the consumer after the last token has been
consumed, after incrementToken()
returned false
(using the new TokenStream
API). Streams implementing the old API
should upgrade to use this feature.
WhitespaceTokenizer
was used.IOException |
---|
Consumers (i.e., IndexWriter
) use this method to advance the stream to
the next token. Implementing classes must implement this method and update
the appropriate AttributeImpl
s with the attributes of the next
token.
The producer must make no assumptions about the attributes after the method
has been returned: the caller may arbitrarily change it. If the producer
needs to preserve the state for subsequent calls, it can use
captureState()
to create a copy of the current attribute state.
This method is called for every token of a document, so an efficient
implementation is crucial for good performance. To avoid calls to
addAttribute(Class)
and getAttribute(Class)
,
references to all AttributeImpl
s that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available,
the attributes must be added during instantiation. Filters and consumers
are not required to check for availability of attributes in
incrementToken()
.
IOException |
---|
Resets this stream to the beginning. This is an optional operation, so
subclasses may or may not implement this method. reset()
is not needed for
the standard indexing process. However, if the tokens of a
TokenStream
are intended to be consumed more than once, it is
necessary to implement reset()
. Note that if your TokenStream
caches tokens and feeds them back again after a reset, it is imperative
that you clone the tokens when you store them away (on the first pass) as
well as when you return them (on future passes after reset()
).
IOException |
---|