| /* |
| * Licensed to the Apache Software Foundation (ASF) under one or more |
| * contributor license agreements. See the NOTICE file distributed with |
| * this work for additional information regarding copyright ownership. |
| * The ASF licenses this file to You under the Apache License, Version 2.0 |
| * (the "License"); you may not use this file except in compliance with |
| * the License. You may obtain a copy of the License at |
| * |
| * http://www.apache.org/licenses/LICENSE-2.0 |
| * |
| * Unless required by applicable law or agreed to in writing, software |
| * distributed under the License is distributed on an "AS IS" BASIS, |
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| * See the License for the specific language governing permissions and |
| * limitations under the License. |
| */ |
| |
| /** |
| * Text analysis. |
| * <p>API and code to convert text into indexable/searchable tokens. Covers {@link org.apache.lucene.analysis.Analyzer} and related classes.</p> |
| * <h2>Parsing? Tokenization? Analysis!</h2> |
| * <p> |
| * Lucene, an indexing and search library, accepts only plain text input. |
| * <h2>Parsing</h2> |
| * <p> |
| * Applications that build their search capabilities upon Lucene may support documents in various formats – HTML, XML, PDF, Word – just to name a few. |
| * Lucene does not care about the <i>Parsing</i> of these and other document formats, and it is the responsibility of the |
| * application using Lucene to use an appropriate <i>Parser</i> to convert the original format into plain text before passing that plain text to Lucene. |
| * <h2>Tokenization</h2> |
| * <p> |
| * Plain text passed to Lucene for indexing goes through a process generally called tokenization. Tokenization is the process |
| * of breaking input text into small indexing elements – tokens. |
| * The way input text is broken into tokens heavily influences how people will then be able to search for that text. |
| * For instance, sentences beginnings and endings can be identified to provide for more accurate phrase |
| * and proximity searches (though sentence identification is not provided by Lucene). |
| * <p> |
| * In some cases simply breaking the input text into tokens is not enough |
| * – a deeper <i>Analysis</i> may be needed. Lucene includes both |
| * pre- and post-tokenization analysis facilities. |
| * </p> |
| * <p> |
| * Pre-tokenization analysis can include (but is not limited to) stripping |
| * HTML markup, and transforming or removing text matching arbitrary patterns |
| * or sets of fixed strings. |
| * </p> |
| * <p> |
| * There are many post-tokenization steps that can be done, including |
| * (but not limited to): |
| * </p> |
| * <ul> |
| * <li><a href="http://en.wikipedia.org/wiki/Stemming">Stemming</a> – |
| * Replacing words with their stems. |
| * For instance with English stemming "bikes" is replaced with "bike"; |
| * now query "bike" can find both documents containing "bike" and those containing "bikes". |
| * </li> |
| * <li><a href="http://en.wikipedia.org/wiki/Stop_words">Stop Words Filtering</a> – |
| * Common words like "the", "and" and "a" rarely add any value to a search. |
| * Removing them shrinks the index size and increases performance. |
| * It may also reduce some "noise" and actually improve search quality. |
| * </li> |
| * <li><a href="http://en.wikipedia.org/wiki/Text_normalization">Text Normalization</a> – |
| * Stripping accents and other character markings can make for better searching. |
| * </li> |
| * <li><a href="http://en.wikipedia.org/wiki/Synonym">Synonym Expansion</a> – |
| * Adding in synonyms at the same token position as the current word can mean better |
| * matching when users search with words in the synonym set. |
| * </li> |
| * </ul> |
| * <h2>Core Analysis</h2> |
| * <p> |
| * The analysis package provides the mechanism to convert Strings and Readers |
| * into tokens that can be indexed by Lucene. There are four main classes in |
| * the package from which all analysis processes are derived. These are: |
| * </p> |
| * <ul> |
| * <li> |
| * {@link org.apache.lucene.analysis.Analyzer} – An <code>Analyzer</code> is |
| * responsible for supplying a |
| * {@link org.apache.lucene.analysis.TokenStream} which can be consumed |
| * by the indexing and searching processes. See below for more information |
| * on implementing your own {@link org.apache.lucene.analysis.Analyzer}. Most of the time, you can use |
| * an anonymous subclass of {@link org.apache.lucene.analysis.Analyzer}. |
| * </li> |
| * <li> |
| * {@link org.apache.lucene.analysis.CharFilter} – <code>CharFilter</code> extends |
| * {@link java.io.Reader} to transform the text before it is |
| * tokenized, while providing |
| * corrected character offsets to account for these modifications. This |
| * capability allows highlighting to function over the original text when |
| * indexed tokens are created from <code>CharFilter</code>-modified text with offsets |
| * that are not the same as those in the original text. {@link org.apache.lucene.analysis.Tokenizer#setReader(java.io.Reader)} |
| * accept <code>CharFilter</code>s. <code>CharFilter</code>s may |
| * be chained to perform multiple pre-tokenization modifications. |
| * </li> |
| * <li> |
| * {@link org.apache.lucene.analysis.Tokenizer} – A <code>Tokenizer</code> is a |
| * {@link org.apache.lucene.analysis.TokenStream} and is responsible for |
| * breaking up incoming text into tokens. In many cases, an {@link org.apache.lucene.analysis.Analyzer} will |
| * use a {@link org.apache.lucene.analysis.Tokenizer} as the first step in the analysis process. However, |
| * to modify text prior to tokenization, use a {@link org.apache.lucene.analysis.CharFilter} subclass (see |
| * above). |
| * </li> |
| * <li> |
| * {@link org.apache.lucene.analysis.TokenFilter} – A <code>TokenFilter</code> is |
| * a {@link org.apache.lucene.analysis.TokenStream} and is responsible |
| * for modifying tokens that have been created by the <code>Tokenizer</code>. Common |
| * modifications performed by a <code>TokenFilter</code> are: deletion, stemming, synonym |
| * injection, and case folding. Not all <code>Analyzer</code>s require <code>TokenFilter</code>s. |
| * </li> |
| * </ul> |
| * <h2>Hints, Tips and Traps</h2> |
| * <p> |
| * The relationship between {@link org.apache.lucene.analysis.Analyzer} and |
| * {@link org.apache.lucene.analysis.CharFilter}s, |
| * {@link org.apache.lucene.analysis.Tokenizer}s, |
| * and {@link org.apache.lucene.analysis.TokenFilter}s is sometimes confusing. To ease |
| * this confusion, here is some clarifications: |
| * </p> |
| * <ul> |
| * <li> |
| * The {@link org.apache.lucene.analysis.Analyzer} is a |
| * <strong>factory</strong> for analysis chains. <code>Analyzer</code>s don't |
| * process text, <code>Analyzer</code>s construct <code>CharFilter</code>s, <code>Tokenizer</code>s, and/or |
| * <code>TokenFilter</code>s that process text. An <code>Analyzer</code> has two tasks: |
| * to produce {@link org.apache.lucene.analysis.TokenStream}s that accept a |
| * reader and produces tokens, and to wrap or otherwise |
| * pre-process {@link java.io.Reader} objects. |
| * </li> |
| * <li> |
| * The {@link org.apache.lucene.analysis.CharFilter} is a subclass of |
| * {@link java.io.Reader} that supports offset tracking. |
| * </li> |
| * <li>The{@link org.apache.lucene.analysis.Tokenizer} |
| * is only responsible for <u>breaking</u> the input text into tokens. |
| * </li> |
| * <li>The{@link org.apache.lucene.analysis.TokenFilter} modifies a |
| * stream of tokens and their contents. |
| * </li> |
| * <li> |
| * {@link org.apache.lucene.analysis.Tokenizer} is a {@link org.apache.lucene.analysis.TokenStream}, |
| * but {@link org.apache.lucene.analysis.Analyzer} is not. |
| * </li> |
| * <li> |
| * {@link org.apache.lucene.analysis.Analyzer} is "field aware", but |
| * {@link org.apache.lucene.analysis.Tokenizer} is not. {@link org.apache.lucene.analysis.Analyzer}s may |
| * take a field name into account when constructing the {@link org.apache.lucene.analysis.TokenStream}. |
| * </li> |
| * </ul> |
| * <p> |
| * If you want to use a particular combination of <code>CharFilter</code>s, a |
| * <code>Tokenizer</code>, and some <code>TokenFilter</code>s, the simplest thing is often an |
| * create an anonymous subclass of {@link org.apache.lucene.analysis.Analyzer}, provide {@link |
| * org.apache.lucene.analysis.Analyzer#createComponents(String)} and perhaps also |
| * {@link org.apache.lucene.analysis.Analyzer#initReader(String, |
| * java.io.Reader)}. However, if you need the same set of components |
| * over and over in many places, you can make a subclass of |
| * {@link org.apache.lucene.analysis.Analyzer}. In fact, Apache Lucene |
| * supplies a large family of <code>Analyzer</code> classes that deliver useful |
| * analysis chains. The most common of these is the <a href="{@docRoot}/org/apache/lucene/analysis/standard/StandardAnalyzer.html">StandardAnalyzer</a>. |
| * Many applications will have a long and industrious life with nothing more |
| * than the <code>StandardAnalyzer</code>. The <a href="{@docRoot}/../analyzers-common/overview-summary.html">analyzers-common</a> |
| * library provides many pre-existing analyzers for various languages. |
| * The analysis-common library also allows to configure a custom Analyzer without subclassing using the |
| * <a href="{@docRoot}/../analyzers-common/org/apache/lucene/analysis/custom/CustomAnalyzer.html">CustomAnalyzer</a> |
| * class. |
| * </p> |
| * <p> |
| * Aside from the <code>StandardAnalyzer</code>, |
| * Lucene includes several components containing analysis components, |
| * all under the 'analysis' directory of the distribution. Some of |
| * these support particular languages, others integrate external |
| * components. The 'common' subdirectory has some noteworthy |
| * general-purpose analyzers, including the <a href="{@docRoot}/../analyzers-common/org/apache/lucene/analysis/miscellaneous/PerFieldAnalyzerWrapper.html">PerFieldAnalyzerWrapper</a>. Most <code>Analyzer</code>s perform the same operation on all |
| * {@link org.apache.lucene.document.Field}s. The PerFieldAnalyzerWrapper can be used to associate a different <code>Analyzer</code> with different |
| * {@link org.apache.lucene.document.Field}s. There is a great deal of |
| * functionality in the analysis area, you should study it carefully to |
| * find the pieces you need. |
| * </p> |
| * <p> |
| * Analysis is one of the main causes of slow indexing. Simply put, the more you analyze the slower the indexing (in most cases). |
| * Perhaps your application would be just fine using the simple WhitespaceTokenizer combined with a StopFilter. The benchmark/ library can be useful |
| * for testing out the speed of the analysis process. |
| * </p> |
| * <h2>Invoking the Analyzer</h2> |
| * <p> |
| * Applications usually do not invoke analysis – Lucene does it |
| * for them. Applications construct <code>Analyzer</code>s and pass then into Lucene, |
| * as follows: |
| * </p> |
| * <ul> |
| * <li> |
| * At indexing, as a consequence of |
| * {@link org.apache.lucene.index.IndexWriter#addDocument(Iterable) addDocument(doc)}, |
| * the <code>Analyzer</code> in effect for indexing is invoked for each indexed field of the added document. |
| * </li> |
| * <li> |
| * At search, a <code>QueryParser</code> may invoke the Analyzer during parsing. Note that for some queries, analysis does not |
| * take place, e.g. wildcard queries. |
| * </li> |
| * </ul> |
| * <p> |
| * However an application might invoke Analysis of any text for testing or for any other purpose, something like: |
| * </p> |
| * <PRE class="prettyprint" id="analysis-workflow"> |
| * Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY |
| * Analyzer analyzer = new StandardAnalyzer(matchVersion); // or any other analyzer |
| * TokenStream ts = analyzer.tokenStream("myfield", new StringReader("some text goes here")); |
| * // The Analyzer class will construct the Tokenizer, TokenFilter(s), and CharFilter(s), |
| * // and pass the resulting Reader to the Tokenizer. |
| * OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class); |
| * |
| * try { |
| * ts.reset(); // Resets this stream to the beginning. (Required) |
| * while (ts.incrementToken()) { |
| * // Use {@link org.apache.lucene.util.AttributeSource#reflectAsString(boolean)} |
| * // for token stream debugging. |
| * System.out.println("token: " + ts.reflectAsString(true)); |
| * |
| * System.out.println("token start offset: " + offsetAtt.startOffset()); |
| * System.out.println(" token end offset: " + offsetAtt.endOffset()); |
| * } |
| * ts.end(); // Perform end-of-stream operations, e.g. set the final offset. |
| * } finally { |
| * ts.close(); // Release resources associated with this stream. |
| * } |
| * </PRE> |
| * <h2>Indexing Analysis vs. Search Analysis</h2> |
| * <p> |
| * Selecting the "correct" analyzer is crucial |
| * for search quality, and can also affect indexing and search performance. |
| * The "correct" analyzer for your application will depend on what your input text |
| * looks like and what problem you are trying to solve. |
| * Lucene java's wiki page |
| * <a href="http://wiki.apache.org/lucene-java/AnalysisParalysis">AnalysisParalysis</a> |
| * provides some data on "analyzing your analyzer". |
| * Here are some rules of thumb: |
| * <ol> |
| * <li>Test test test... (did we say test?)</li> |
| * <li>Beware of too much analysis – it might hurt indexing performance.</li> |
| * <li>Start with the same analyzer for indexing and search, otherwise searches would not find what they are supposed to...</li> |
| * <li>In some cases a different analyzer is required for indexing and search, for instance: |
| * <ul> |
| * <li>Certain searches require more stop words to be filtered. (i.e. more than those that were filtered at indexing.)</li> |
| * <li>Query expansion by synonyms, acronyms, auto spell correction, etc.</li> |
| * </ul> |
| * This might sometimes require a modified analyzer – see the next section on how to do that. |
| * </li> |
| * </ol> |
| * <h2>Implementing your own Analyzer and Analysis Components</h2> |
| * <p> |
| * Creating your own Analyzer is straightforward. Your Analyzer should subclass {@link org.apache.lucene.analysis.Analyzer}. It can use |
| * existing analysis components — CharFilter(s) <i>(optional)</i>, a |
| * Tokenizer, and TokenFilter(s) <i>(optional)</i> — or components you |
| * create, or a combination of existing and newly created components. Before |
| * pursuing this approach, you may find it worthwhile to explore the |
| * <a href="{@docRoot}/../analyzers-common/overview-summary.html">analyzers-common</a> library and/or ask on the |
| * <a href="http://lucene.apache.org/core/discussion.html">java-user@lucene.apache.org mailing list</a> first to see if what you |
| * need already exists. If you are still committed to creating your own |
| * Analyzer, have a look at the source code of any one of the many samples |
| * located in this package. |
| * </p> |
| * <p> |
| * The following sections discuss some aspects of implementing your own analyzer. |
| * </p> |
| * <h3>Field Section Boundaries</h3> |
| * <p> |
| * When {@link org.apache.lucene.document.Document#add(org.apache.lucene.index.IndexableField) document.add(field)} |
| * is called multiple times for the same field name, we could say that each such call creates a new |
| * section for that field in that document. |
| * In fact, a separate call to |
| * {@link org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader) tokenStream(field,reader)} |
| * would take place for each of these so called "sections". |
| * However, the default Analyzer behavior is to treat all these sections as one large section. |
| * This allows phrase search and proximity search to seamlessly cross |
| * boundaries between these "sections". |
| * In other words, if a certain field "f" is added like this: |
| * </p> |
| * <PRE class="prettyprint"> |
| * document.add(new Field("f","first ends",...); |
| * document.add(new Field("f","starts two",...); |
| * indexWriter.addDocument(document); |
| * </PRE> |
| * <p> |
| * Then, a phrase search for "ends starts" would find that document. |
| * Where desired, this behavior can be modified by introducing a "position gap" between consecutive field "sections", |
| * simply by overriding |
| * {@link org.apache.lucene.analysis.Analyzer#getPositionIncrementGap(java.lang.String) Analyzer.getPositionIncrementGap(fieldName)}: |
| * </p> |
| * <PRE class="prettyprint"> |
| * Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY |
| * Analyzer myAnalyzer = new StandardAnalyzer(matchVersion) { |
| * public int getPositionIncrementGap(String fieldName) { |
| * return 10; |
| * } |
| * }; |
| * </PRE> |
| * <h3>End of Input Cleanup</h3> |
| * <p> |
| * At the ends of each field, Lucene will call the {@link org.apache.lucene.analysis.TokenStream#end()}. |
| * The components of the token stream (the tokenizer and the token filters) <strong>must</strong> |
| * put accurate values into the token attributes to reflect the situation at the end of the field. |
| * The Offset attribute must contain the final offset (the total number of characters processed) |
| * in both start and end. Attributes like PositionLength must be correct. |
| * </p> |
| * <p> |
| * The base method{@link org.apache.lucene.analysis.TokenStream#end()} sets PositionIncrement to 0, which is required. |
| * Other components must override this method to fix up the other attributes. |
| * </p> |
| * <h3>Token Position Increments</h3> |
| * <p> |
| * By default, TokenStream arranges for the |
| * {@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute#getPositionIncrement() position increment} of all tokens to be one. |
| * This means that the position stored for that token in the index would be one more than |
| * that of the previous token. |
| * Recall that phrase and proximity searches rely on position info. |
| * </p> |
| * <p> |
| * If the selected analyzer filters the stop words "is" and "the", then for a document |
| * containing the string "blue is the sky", only the tokens "blue", "sky" are indexed, |
| * with position("sky") = 3 + position("blue"). Now, a phrase query "blue is the sky" |
| * would find that document, because the same analyzer filters the same stop words from |
| * that query. But the phrase query "blue sky" would not find that document because the |
| * position increment between "blue" and "sky" is only 1. |
| * </p> |
| * <p> |
| * If this behavior does not fit the application needs, the query parser needs to be |
| * configured to not take position increments into account when generating phrase queries. |
| * </p> |
| * <p> |
| * Note that a filter that filters <strong>out</strong> tokens <strong>must</strong> increment the position increment in order not to generate corrupt |
| * tokenstream graphs. Here is the logic used by StopFilter to increment positions when filtering out tokens: |
| * </p> |
| * <PRE class="prettyprint"> |
| * public TokenStream tokenStream(final String fieldName, Reader reader) { |
| * final TokenStream ts = someAnalyzer.tokenStream(fieldName, reader); |
| * TokenStream res = new TokenStream() { |
| * CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); |
| * PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class); |
| * |
| * public boolean incrementToken() throws IOException { |
| * int extraIncrement = 0; |
| * while (true) { |
| * boolean hasNext = ts.incrementToken(); |
| * if (hasNext) { |
| * if (stopWords.contains(termAtt.toString())) { |
| * extraIncrement += posIncrAtt.getPositionIncrement(); // filter this word |
| * continue; |
| * } |
| * if (extraIncrement > 0) { |
| * posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement()+extraIncrement); |
| * } |
| * } |
| * return hasNext; |
| * } |
| * } |
| * }; |
| * return res; |
| * } |
| * </PRE> |
| * <p> |
| * A few more use cases for modifying position increments are: |
| * </p> |
| * <ol> |
| * <li>Inhibiting phrase and proximity matches in sentence boundaries – for this, a tokenizer that |
| * identifies a new sentence can add 1 to the position increment of the first token of the new sentence.</li> |
| * <li>Injecting synonyms – synonyms of a token should be created at the same position as the |
| * original token, and the output order of the original token and the injected synonym is undefined |
| * as long as they both leave from the same position. As result, all synonyms of a token would be |
| * considered to appear in exactly the same position as that token, and so would they be seen by |
| * phrase and proximity searches. For multi-token synonyms to work correctly, you should use |
| * {@code SynoymGraphFilter} at search time only.</li> |
| * </ol> |
| * |
| * <h3>Token Position Length</h3> |
| * <p> |
| * By default, all tokens created by Analyzers and Tokenizers have a |
| * {@link org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute#getPositionLength() position length} of one. |
| * This means that the token occupies a single position. This attribute is not indexed |
| * and thus not taken into account for positional queries, but is used by eg. suggesters. |
| * </p> |
| * <p> |
| * The main use case for positions lengths is multi-word synonyms. With single-word |
| * synonyms, setting the position increment to 0 is enough to denote the fact that two |
| * words are synonyms, for example: |
| * </p> |
| * <table summary="table showing position increments of 1 and 0 for red and magenta, respectively"> |
| * <tr><td>Term</td><td>red</td><td>magenta</td></tr> |
| * <tr><td>Position increment</td><td>1</td><td>0</td></tr> |
| * </table> |
| * <p> |
| * Given that position(magenta) = 0 + position(red), they are at the same position, so anything |
| * working with analyzers will return the exact same result if you replace "magenta" with "red" |
| * in the input. However, multi-word synonyms are more tricky. Let's say that you want to build |
| * a TokenStream where "IBM" is a synonym of "Internal Business Machines". Position increments |
| * are not enough anymore: |
| * </p> |
| * <table summary="position increments where international is zero"> |
| * <tr><td>Term</td><td>IBM</td><td>International</td><td>Business</td><td>Machines</td></tr> |
| * <tr><td>Position increment</td><td>1</td><td>0</td><td>1</td><td>1</td></tr> |
| * </table> |
| * <p> |
| * The problem with this token stream is that "IBM" is at the same position as "International" |
| * although it is a synonym with "International Business Machines" as a whole. Setting |
| * the position increment of "Business" and "Machines" to 0 wouldn't help as it would mean |
| * than "International" is a synonym of "Business". The only way to solve this issue is to |
| * make "IBM" span across 3 positions, this is where position lengths come to rescue. |
| * </p> |
| * <table summary="position lengths where IBM is three"> |
| * <tr><td>Term</td><td>IBM</td><td>International</td><td>Business</td><td>Machines</td></tr> |
| * <tr><td>Position increment</td><td>1</td><td>0</td><td>1</td><td>1</td></tr> |
| * <tr><td>Position length</td><td>3</td><td>1</td><td>1</td><td>1</td></tr> |
| * </table> |
| * <p> |
| * This new attribute makes clear that "IBM" and "International Business Machines" start and end |
| * at the same positions. |
| * </p> |
| * <a name="corrupt"></a> |
| * <h3>How to not write corrupt token streams</h3> |
| * <p> |
| * There are a few rules to observe when writing custom Tokenizers and TokenFilters: |
| * </p> |
| * <ul> |
| * <li>The first position increment must be > 0.</li> |
| * <li>Positions must not go backward.</li> |
| * <li>Tokens that have the same start position must have the same start offset.</li> |
| * <li>Tokens that have the same end position (taking into account the |
| * position length) must have the same end offset.</li> |
| * <li>Tokenizers must call {@link |
| * org.apache.lucene.util.AttributeSource#clearAttributes()} in |
| * incrementToken().</li> |
| * <li>Tokenizers must override {@link |
| * org.apache.lucene.analysis.TokenStream#end()}, and pass the final |
| * offset (the total number of input characters processed) to both |
| * parameters of {@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute#setOffset(int, int)}.</li> |
| * </ul> |
| * <p> |
| * Although these rules might seem easy to follow, problems can quickly happen when chaining |
| * badly implemented filters that play with positions and offsets, such as synonym or n-grams |
| * filters. Here are good practices for writing correct filters: |
| * </p> |
| * <ul> |
| * <li>Token filters should not modify offsets. If you feel that your filter would need to modify offsets, then it should probably be implemented as a tokenizer.</li> |
| * <li>Token filters should not insert positions. If a filter needs to add tokens, then they should all have a position increment of 0.</li> |
| * <li>When they add tokens, token filters should call {@link org.apache.lucene.util.AttributeSource#clearAttributes()} first.</li> |
| * <li>When they remove tokens, token filters should increment the position increment of the following token.</li> |
| * <li>Token filters should preserve position lengths.</li> |
| * </ul> |
| * <h2>TokenStream API</h2> |
| * <p> |
| * "Flexible Indexing" summarizes the effort of making the Lucene indexer |
| * pluggable and extensible for custom index formats. A fully customizable |
| * indexer means that users will be able to store custom data structures on |
| * disk. Therefore the analysis API must transport custom types of |
| * data from the documents to the indexer. (It also supports communications |
| * amongst the analysis components.) |
| * </p> |
| * <h3>Attribute and AttributeSource</h3> |
| * <p> |
| * Classes {@link org.apache.lucene.util.Attribute} and |
| * {@link org.apache.lucene.util.AttributeSource} serve as the basis upon which |
| * the analysis elements of "Flexible Indexing" are implemented. An Attribute |
| * holds a particular piece of information about a text token. For example, |
| * {@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute} |
| * contains the term text of a token, and |
| * {@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute} contains |
| * the start and end character offsets of a token. An AttributeSource is a |
| * collection of Attributes with a restriction: there may be only one instance |
| * of each attribute type. TokenStream now extends AttributeSource, which means |
| * that one can add Attributes to a TokenStream. Since TokenFilter extends |
| * TokenStream, all filters are also AttributeSources. |
| * </p> |
| * <p> |
| * Lucene provides seven Attributes out of the box: |
| * </p> |
| * <table rules="all" frame="box" cellpadding="3" summary="common bundled attributes"> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute}</td> |
| * <td> |
| * The term text of a token. Implements {@link java.lang.CharSequence} |
| * (providing methods length() and charAt(), and allowing e.g. for direct |
| * use with regular expression {@link java.util.regex.Matcher}s) and |
| * {@link java.lang.Appendable} (allowing the term text to be appended to.) |
| * </td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute}</td> |
| * <td>The start and end offset of a token in characters.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute}</td> |
| * <td>See above for detailed information about position increment.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute}</td> |
| * <td>The number of positions occupied by a token.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.PayloadAttribute}</td> |
| * <td>The payload that a Token can optionally have.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.TypeAttribute}</td> |
| * <td>The type of the token. Default is 'word'.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.FlagsAttribute}</td> |
| * <td>Optional flags a token can have.</td> |
| * </tr> |
| * <tr> |
| * <td>{@link org.apache.lucene.analysis.tokenattributes.KeywordAttribute}</td> |
| * <td> |
| * Keyword-aware TokenStreams/-Filters skip modification of tokens that |
| * return true from this attribute's isKeyword() method. |
| * </td> |
| * </tr> |
| * </table> |
| * <h3>More Requirements for Analysis Component Classes</h3> |
| * Due to the historical development of the API, there are some perhaps |
| * less than obvious requirements to implement analysis components |
| * classes. |
| * <h4 id="analysis-lifetime">Token Stream Lifetime</h4> |
| * The code fragment of the <a href="#analysis-workflow">analysis workflow |
| * protocol</a> above shows a token stream being obtained, used, and then |
| * left for garbage. However, that does not mean that the components of |
| * that token stream will, in fact, be discarded. The default is just the |
| * opposite. {@link org.apache.lucene.analysis.Analyzer} applies a reuse |
| * strategy to the tokenizer and the token filters. It will reuse |
| * them. For each new input, it calls {@link org.apache.lucene.analysis.Tokenizer#setReader(java.io.Reader)} |
| * to set the input. Your components must be prepared for this scenario, |
| * as described below. |
| * <h4>Tokenizer</h4> |
| * <ul> |
| * <li> |
| * You should create your tokenizer class by extending {@link org.apache.lucene.analysis.Tokenizer}. |
| * </li> |
| * <li> |
| * Your tokenizer <strong>must</strong> override {@link org.apache.lucene.analysis.TokenStream#end()}. |
| * Your implementation <strong>must</strong> call |
| * <code>super.end()</code>. It must set a correct final offset into |
| * the offset attribute, and finish up and other attributes to reflect |
| * the end of the stream. |
| * </li> |
| * <li> |
| * If your tokenizer overrides {@link org.apache.lucene.analysis.TokenStream#reset()} |
| * or {@link org.apache.lucene.analysis.TokenStream#close()}, it |
| * <strong>must</strong> call the corresponding superclass method. |
| * </li> |
| * </ul> |
| * <h4>Token Filter</h4> |
| * You should create your token filter class by extending {@link org.apache.lucene.analysis.TokenFilter}. |
| * If your token filter overrides {@link org.apache.lucene.analysis.TokenStream#reset()}, |
| * {@link org.apache.lucene.analysis.TokenStream#end()} |
| * or {@link org.apache.lucene.analysis.TokenStream#close()}, it |
| * <strong>must</strong> call the corresponding superclass method. |
| * <h4>Creating delegates</h4> |
| * Forwarding classes (those which extend {@link org.apache.lucene.analysis.Tokenizer} but delegate |
| * selected logic to another tokenizer) must also set the reader to the delegate in the overridden |
| * {@link org.apache.lucene.analysis.Tokenizer#reset()} method, e.g.: |
| * <pre class="prettyprint"> |
| * public class ForwardingTokenizer extends Tokenizer { |
| * private Tokenizer delegate; |
| * ... |
| * {@literal @Override} |
| * public void reset() { |
| * super.reset(); |
| * delegate.setReader(this.input); |
| * delegate.reset(); |
| * } |
| * } |
| * </pre> |
| * <h3>Testing Your Analysis Component</h3> |
| * <p> |
| * The lucene-test-framework component defines |
| * <a href="{@docRoot}/../test-framework/org/apache/lucene/analysis/BaseTokenStreamTestCase.html">BaseTokenStreamTestCase</a>. By extending |
| * this class, you can create JUnit tests that validate that your |
| * Analyzer and/or analysis components correctly implement the |
| * protocol. The checkRandomData methods of that class are particularly effective in flushing out errors. |
| * </p> |
| * <h3>Using the TokenStream API</h3> |
| * There are a few important things to know in order to use the new API efficiently which are summarized here. You may want |
| * to walk through the example below first and come back to this section afterwards. |
| * <ol><li> |
| * Please keep in mind that an AttributeSource can only have one instance of a particular Attribute. Furthermore, if |
| * a chain of a TokenStream and multiple TokenFilters is used, then all TokenFilters in that chain share the Attributes |
| * with the TokenStream. |
| * </li> |
| * <li> |
| * Attribute instances are reused for all tokens of a document. Thus, a TokenStream/-Filter needs to update |
| * the appropriate Attribute(s) in incrementToken(). The consumer, commonly the Lucene indexer, consumes the data in the |
| * Attributes and then calls incrementToken() again until it returns false, which indicates that the end of the stream |
| * was reached. This means that in each call of incrementToken() a TokenStream/-Filter can safely overwrite the data in |
| * the Attribute instances. |
| * </li> |
| * <li> |
| * For performance reasons a TokenStream/-Filter should add/get Attributes during instantiation; i.e., create an attribute in the |
| * constructor and store references to it in an instance variable. Using an instance variable instead of calling addAttribute()/getAttribute() |
| * in incrementToken() will avoid attribute lookups for every token in the document. |
| * </li> |
| * <li> |
| * All methods in AttributeSource are idempotent, which means calling them multiple times always yields the same |
| * result. This is especially important to know for addAttribute(). The method takes the <b>type</b> (<code>Class</code>) |
| * of an Attribute as an argument and returns an <b>instance</b>. If an Attribute of the same type was previously added, then |
| * the already existing instance is returned, otherwise a new instance is created and returned. Therefore TokenStreams/-Filters |
| * can safely call addAttribute() with the same Attribute type multiple times. Even consumers of TokenStreams should |
| * normally call addAttribute() instead of getAttribute(), because it would not fail if the TokenStream does not have this |
| * Attribute (getAttribute() would throw an IllegalArgumentException, if the Attribute is missing). More advanced code |
| * could simply check with hasAttribute(), if a TokenStream has it, and may conditionally leave out processing for |
| * extra performance. |
| * </li></ol> |
| * <h3>Example</h3> |
| * <p> |
| * In this example we will create a WhiteSpaceTokenizer and use a LengthFilter to suppress all words that have |
| * only two or fewer characters. The LengthFilter is part of the Lucene core and its implementation will be explained |
| * here to illustrate the usage of the TokenStream API. |
| * </p> |
| * <p> |
| * Then we will develop a custom Attribute, a PartOfSpeechAttribute, and add another filter to the chain which |
| * utilizes the new custom attribute, and call it PartOfSpeechTaggingFilter. |
| * </p> |
| * <h4>Whitespace tokenization</h4> |
| * <pre class="prettyprint"> |
| * public class MyAnalyzer extends Analyzer { |
| * |
| * private Version matchVersion; |
| * |
| * public MyAnalyzer(Version matchVersion) { |
| * this.matchVersion = matchVersion; |
| * } |
| * |
| * {@literal @Override} |
| * protected TokenStreamComponents createComponents(String fieldName) { |
| * return new TokenStreamComponents(new WhitespaceTokenizer(matchVersion)); |
| * } |
| * |
| * public static void main(String[] args) throws IOException { |
| * // text to tokenize |
| * final String text = "This is a demo of the TokenStream API"; |
| * |
| * Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY |
| * MyAnalyzer analyzer = new MyAnalyzer(matchVersion); |
| * TokenStream stream = analyzer.tokenStream("field", new StringReader(text)); |
| * |
| * // get the CharTermAttribute from the TokenStream |
| * CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class); |
| * |
| * try { |
| * stream.reset(); |
| * |
| * // print all tokens until stream is exhausted |
| * while (stream.incrementToken()) { |
| * System.out.println(termAtt.toString()); |
| * } |
| * |
| * stream.end(); |
| * } finally { |
| * stream.close(); |
| * } |
| * } |
| * } |
| * </pre> |
| * In this easy example a simple white space tokenization is performed. In main() a loop consumes the stream and |
| * prints the term text of the tokens by accessing the CharTermAttribute that the WhitespaceTokenizer provides. |
| * Here is the output: |
| * <pre> |
| * This |
| * is |
| * a |
| * demo |
| * of |
| * the |
| * new |
| * TokenStream |
| * API |
| * </pre> |
| * <h4>Adding a LengthFilter</h4> |
| * We want to suppress all tokens that have 2 or less characters. We can do that |
| * easily by adding a LengthFilter to the chain. Only the |
| * <code>createComponents()</code> method in our analyzer needs to be changed: |
| * <pre class="prettyprint"> |
| * {@literal @Override} |
| * protected TokenStreamComponents createComponents(String fieldName) { |
| * final Tokenizer source = new WhitespaceTokenizer(matchVersion); |
| * TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE); |
| * return new TokenStreamComponents(source, result); |
| * } |
| * </pre> |
| * Note how now only words with 3 or more characters are contained in the output: |
| * <pre> |
| * This |
| * demo |
| * the |
| * new |
| * TokenStream |
| * API |
| * </pre> |
| * Now let's take a look how the LengthFilter is implemented: |
| * <pre class="prettyprint"> |
| * public final class LengthFilter extends FilteringTokenFilter { |
| * |
| * private final int min; |
| * private final int max; |
| * |
| * private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); |
| * |
| * /** |
| * * Create a new LengthFilter. This will filter out tokens whose |
| * * CharTermAttribute is either too short |
| * * (< min) or too long (> max). |
| * * {@literal @param} version the Lucene match version |
| * * {@literal @param} in the TokenStream to consume |
| * * {@literal @param} min the minimum length |
| * * {@literal @param} max the maximum length |
| * */ |
| * public LengthFilter(Version version, TokenStream in, int min, int max) { |
| * super(version, in); |
| * this.min = min; |
| * this.max = max; |
| * } |
| * |
| * {@literal @Override} |
| * public boolean accept() { |
| * final int len = termAtt.length(); |
| * return (len >= min && len <= max); |
| * } |
| * |
| * } |
| * </pre> |
| * <p> |
| * In LengthFilter, the CharTermAttribute is added and stored in the instance |
| * variable <code>termAtt</code>. Remember that there can only be a single |
| * instance of CharTermAttribute in the chain, so in our example the |
| * <code>addAttribute()</code> call in LengthFilter returns the |
| * CharTermAttribute that the WhitespaceTokenizer already added. |
| * </p> |
| * <p> |
| * The tokens are retrieved from the input stream in FilteringTokenFilter's |
| * <code>incrementToken()</code> method (see below), which calls LengthFilter's |
| * <code>accept()</code> method. By looking at the term text in the |
| * CharTermAttribute, the length of the term can be determined and tokens that |
| * are either too short or too long are skipped. Note how |
| * <code>accept()</code> can efficiently access the instance variable; no |
| * attribute lookup is necessary. The same is true for the consumer, which can |
| * simply use local references to the Attributes. |
| * </p> |
| * <p> |
| * LengthFilter extends FilteringTokenFilter: |
| * </p> |
| * |
| * <pre class="prettyprint"> |
| * public abstract class FilteringTokenFilter extends TokenFilter { |
| * |
| * private final PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class); |
| * |
| * /** |
| * * Create a new FilteringTokenFilter. |
| * * {@literal @param} in the TokenStream to consume |
| * */ |
| * public FilteringTokenFilter(Version version, TokenStream in) { |
| * super(in); |
| * } |
| * |
| * /** Override this method and return if the current input token should be returned by incrementToken. */ |
| * protected abstract boolean accept() throws IOException; |
| * |
| * {@literal @Override} |
| * public final boolean incrementToken() throws IOException { |
| * int skippedPositions = 0; |
| * while (input.incrementToken()) { |
| * if (accept()) { |
| * if (skippedPositions != 0) { |
| * posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions); |
| * } |
| * return true; |
| * } |
| * skippedPositions += posIncrAtt.getPositionIncrement(); |
| * } |
| * // reached EOS -- return false |
| * return false; |
| * } |
| * |
| * {@literal @Override} |
| * public void reset() throws IOException { |
| * super.reset(); |
| * } |
| * |
| * } |
| * </pre> |
| * |
| * <h4>Adding a custom Attribute</h4> |
| * Now we're going to implement our own custom Attribute for part-of-speech tagging and call it consequently |
| * <code>PartOfSpeechAttribute</code>. First we need to define the interface of the new Attribute: |
| * <pre class="prettyprint"> |
| * public interface PartOfSpeechAttribute extends Attribute { |
| * public static enum PartOfSpeech { |
| * Noun, Verb, Adjective, Adverb, Pronoun, Preposition, Conjunction, Article, Unknown |
| * } |
| * |
| * public void setPartOfSpeech(PartOfSpeech pos); |
| * |
| * public PartOfSpeech getPartOfSpeech(); |
| * } |
| * </pre> |
| * <p> |
| * Now we also need to write the implementing class. The name of that class is important here: By default, Lucene |
| * checks if there is a class with the name of the Attribute with the suffix 'Impl'. In this example, we would |
| * consequently call the implementing class <code>PartOfSpeechAttributeImpl</code>. |
| * </p> |
| * <p> |
| * This should be the usual behavior. However, there is also an expert-API that allows changing these naming conventions: |
| * {@link org.apache.lucene.util.AttributeFactory}. The factory accepts an Attribute interface as argument |
| * and returns an actual instance. You can implement your own factory if you need to change the default behavior. |
| * </p> |
| * <p> |
| * Now here is the actual class that implements our new Attribute. Notice that the class has to extend |
| * {@link org.apache.lucene.util.AttributeImpl}: |
| * </p> |
| * <pre class="prettyprint"> |
| * public final class PartOfSpeechAttributeImpl extends AttributeImpl |
| * implements PartOfSpeechAttribute { |
| * |
| * private PartOfSpeech pos = PartOfSpeech.Unknown; |
| * |
| * public void setPartOfSpeech(PartOfSpeech pos) { |
| * this.pos = pos; |
| * } |
| * |
| * public PartOfSpeech getPartOfSpeech() { |
| * return pos; |
| * } |
| * |
| * {@literal @Override} |
| * public void clear() { |
| * pos = PartOfSpeech.Unknown; |
| * } |
| * |
| * {@literal @Override} |
| * public void copyTo(AttributeImpl target) { |
| * ((PartOfSpeechAttribute) target).setPartOfSpeech(pos); |
| * } |
| * } |
| * </pre> |
| * <p> |
| * This is a simple Attribute implementation has only a single variable that |
| * stores the part-of-speech of a token. It extends the |
| * <code>AttributeImpl</code> class and therefore implements its abstract methods |
| * <code>clear()</code> and <code>copyTo()</code>. Now we need a TokenFilter that |
| * can set this new PartOfSpeechAttribute for each token. In this example we |
| * show a very naive filter that tags every word with a leading upper-case letter |
| * as a 'Noun' and all other words as 'Unknown'. |
| * </p> |
| * <pre class="prettyprint"> |
| * public static class PartOfSpeechTaggingFilter extends TokenFilter { |
| * PartOfSpeechAttribute posAtt = addAttribute(PartOfSpeechAttribute.class); |
| * CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); |
| * |
| * protected PartOfSpeechTaggingFilter(TokenStream input) { |
| * super(input); |
| * } |
| * |
| * public boolean incrementToken() throws IOException { |
| * if (!input.incrementToken()) {return false;} |
| * posAtt.setPartOfSpeech(determinePOS(termAtt.buffer(), 0, termAtt.length())); |
| * return true; |
| * } |
| * |
| * // determine the part of speech for the given term |
| * protected PartOfSpeech determinePOS(char[] term, int offset, int length) { |
| * // naive implementation that tags every uppercased word as noun |
| * if (length > 0 && Character.isUpperCase(term[0])) { |
| * return PartOfSpeech.Noun; |
| * } |
| * return PartOfSpeech.Unknown; |
| * } |
| * } |
| * </pre> |
| * <p> |
| * Just like the LengthFilter, this new filter stores references to the |
| * attributes it needs in instance variables. Notice how you only need to pass |
| * in the interface of the new Attribute and instantiating the correct class |
| * is automatically taken care of. |
| * </p> |
| * <p>Now we need to add the filter to the chain in MyAnalyzer:</p> |
| * <pre class="prettyprint"> |
| * {@literal @Override} |
| * protected TokenStreamComponents createComponents(String fieldName) { |
| * final Tokenizer source = new WhitespaceTokenizer(matchVersion); |
| * TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE); |
| * result = new PartOfSpeechTaggingFilter(result); |
| * return new TokenStreamComponents(source, result); |
| * } |
| * </pre> |
| * Now let's look at the output: |
| * <pre> |
| * This |
| * demo |
| * the |
| * new |
| * TokenStream |
| * API |
| * </pre> |
| * Apparently it hasn't changed, which shows that adding a custom attribute to a TokenStream/Filter chain does not |
| * affect any existing consumers, simply because they don't know the new Attribute. Now let's change the consumer |
| * to make use of the new PartOfSpeechAttribute and print it out: |
| * <pre class="prettyprint"> |
| * public static void main(String[] args) throws IOException { |
| * // text to tokenize |
| * final String text = "This is a demo of the TokenStream API"; |
| * |
| * MyAnalyzer analyzer = new MyAnalyzer(); |
| * TokenStream stream = analyzer.tokenStream("field", new StringReader(text)); |
| * |
| * // get the CharTermAttribute from the TokenStream |
| * CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class); |
| * |
| * // get the PartOfSpeechAttribute from the TokenStream |
| * PartOfSpeechAttribute posAtt = stream.addAttribute(PartOfSpeechAttribute.class); |
| * |
| * try { |
| * stream.reset(); |
| * |
| * // print all tokens until stream is exhausted |
| * while (stream.incrementToken()) { |
| * System.out.println(termAtt.toString() + ": " + posAtt.getPartOfSpeech()); |
| * } |
| * |
| * stream.end(); |
| * } finally { |
| * stream.close(); |
| * } |
| * } |
| * </pre> |
| * The change that was made is to get the PartOfSpeechAttribute from the TokenStream and print out its contents in |
| * the while loop that consumes the stream. Here is the new output: |
| * <pre> |
| * This: Noun |
| * demo: Unknown |
| * the: Unknown |
| * new: Unknown |
| * TokenStream: Noun |
| * API: Noun |
| * </pre> |
| * Each word is now followed by its assigned PartOfSpeech tag. Of course this is a naive |
| * part-of-speech tagging. The word 'This' should not even be tagged as noun; it is only spelled capitalized because it |
| * is the first word of a sentence. Actually this is a good opportunity for an exercise. To practice the usage of the new |
| * API the reader could now write an Attribute and TokenFilter that can specify for each word if it was the first token |
| * of a sentence or not. Then the PartOfSpeechTaggingFilter can make use of this knowledge and only tag capitalized words |
| * as nouns if not the first word of a sentence (we know, this is still not a correct behavior, but hey, it's a good exercise). |
| * As a small hint, this is how the new Attribute class could begin: |
| * <pre class="prettyprint"> |
| * public class FirstTokenOfSentenceAttributeImpl extends AttributeImpl |
| * implements FirstTokenOfSentenceAttribute { |
| * |
| * private boolean firstToken; |
| * |
| * public void setFirstToken(boolean firstToken) { |
| * this.firstToken = firstToken; |
| * } |
| * |
| * public boolean getFirstToken() { |
| * return firstToken; |
| * } |
| * |
| * {@literal @Override} |
| * public void clear() { |
| * firstToken = false; |
| * } |
| * |
| * ... |
| * </pre> |
| * <h4>Adding a CharFilter chain</h4> |
| * Analyzers take Java {@link java.io.Reader}s as input. Of course you can wrap your Readers with {@link java.io.FilterReader}s |
| * to manipulate content, but this would have the big disadvantage that character offsets might be inconsistent with your original |
| * text. |
| * <p> |
| * {@link org.apache.lucene.analysis.CharFilter} is designed to allow you to pre-process input like a FilterReader would, but also |
| * preserve the original offsets associated with those characters. This way mechanisms like highlighting still work correctly. |
| * CharFilters can be chained. |
| * <p> |
| * Example: |
| * <pre class="prettyprint"> |
| * public class MyAnalyzer extends Analyzer { |
| * |
| * {@literal @Override} |
| * protected TokenStreamComponents createComponents(String fieldName) { |
| * return new TokenStreamComponents(new MyTokenizer()); |
| * } |
| * |
| * {@literal @Override} |
| * protected Reader initReader(String fieldName, Reader reader) { |
| * // wrap the Reader in a CharFilter chain. |
| * return new SecondCharFilter(new FirstCharFilter(reader)); |
| * } |
| * } |
| * </pre> |
| */ |
| package org.apache.lucene.analysis; |
