Lucene contrib change Log

======================= Release 2.9.0 2009-09-23 =======================

Changes in runtime behavior

 * LUCENE-1505: Local lucene now uses org.apache.lucene.util.NumericUtils for all
    number conversion.  You'll need to fully re-index any previously created indexes.
    This isn't a break in back-compatibility because local Lucene has not yet
    been released.  (Mike McCandless)
 
 * LUCENE-1758: ArabicAnalyzer now uses the light10 algorithm, has a refined
    default stopword list, and lowercases non-Arabic text.  
    You'll need to fully re-index any previously created indexes. This isn't a 
    break in back-compatibility because ArabicAnalyzer has not yet been 
    released.  (Robert Muir)


API Changes

 * LUCENE-1695: Update the Highlighter to use the new TokenStream API. This issue breaks backwards
    compatibility with some public classes. If you have implemented custom Fragmenters or Scorers, 
    you will need to adjust them to work with the new TokenStream API. Rather than getting passed a 
    Token at a time, you will be given a TokenStream to init your impl with - store the Attributes 
    you are interested in locally and access them on each call to the method that used to pass a new 
    Token. Look at the included updated impls for examples.  (Mark Miller)

 * LUCENE-1460: Change contrib TokenStreams/Filters to use the new
    TokenStream API. (Robert Muir, Michael Busch)

 * LUCENE-1775, LUCENE-1903: Change remaining TokenFilters (shingle, prefix-suffix)
    to use the new TokenStream API. ShingleFilter is much more efficient now,
    it clones much less often and computes the tokens mostly on the fly now.
    Also added more tests. (Robert Muir, Michael Busch, Uwe Schindler, Chris Harris)
    
 * LUCENE-1685: The position aware SpanScorer has become the default scorer
    for Highlighting. The SpanScorer implementation has replaced QueryScorer
    and the old term highlighting QueryScorer has been renamed to 
    QueryTermScorer. Multi-term queries are also now expanded by default. If
    you were previously rewriting the query for multi-term query highlighting,
    you should no longer do that (unless you switch to using QueryTermScorer).
    The SpanScorer API (now QueryScorer) has also been improved to more closely
    match the API of the previous QueryScorer implementation.  (Mark Miller)  

 * LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
    Analyzers. If you need to index text in these encodings, please use Java's
    character set conversion facilities (InputStreamReader, etc) during I/O, 
    so that Lucene can analyze this text as Unicode instead.  (Robert Muir)  

Bug fixes

 * LUCENE-1423: InstantiatedTermEnum#skipTo(Term) throws ArrayIndexOutOfBounds on empty index.
    (Karl Wettin) 

 * LUCENE-1462: InstantiatedIndexWriter did not reset pre analyzed TokenStreams the
    same way IndexWriter does. Parts of InstantiatedIndex was not Serializable.
    (Karl Wettin)

 * LUCENE-1510: InstantiatedIndexReader#norms methods throws NullPointerException on empty index.
    (Karl Wettin, Robert Newson)

 * LUCENE-1514: ShingleMatrixFilter#next(Token) easily throws a StackOverflowException
    due to recursive invocation. (Karl Wettin)

 * LUCENE-1548: Fix distance normalization in LevenshteinDistance to
    not produce negative distances (Thomas Morton via Mike McCandless)

 * LUCENE-1490: Fix latin1 conversion of HALFWIDTH_AND_FULLWIDTH_FORMS
    characters to only apply to the correct subset (Daniel Cheng via
    Mike McCandless)

 * LUCENE-1576: Fix BrazilianAnalyzer to downcase tokens after
    StandardTokenizer so that stop words with mixed case are filtered
    out.  (Rafael Cunha de Almeida, Douglas Campos via Mike McCandless)

 * LUCENE-1491: EdgeNGramTokenFilter no longer stops on tokens shorter than minimum n-gram size.
    (Todd Teak via Otis Gospodnetic)

 * LUCENE-1683: Fixed JavaUtilRegexCapabilities (an impl used by
    RegexQuery) to use Matcher.matches() not Matcher.lookingAt() so
    that the regexp must match the entire string, not just a prefix.
    (Trejkaz via Mike McCandless)

 * LUCENE-1792: Fix new query parser to set rewrite method for
    multi-term queries. (Luis Alves, Mike McCandless via Michael Busch)

 * LUCENE-1828: Fix memory index to call TokenStream.reset() and
    TokenStream.end(). (Tim Smith via Michael Busch)

 * LUCENE-1912: Fix fast-vector-highlighter issue when two or more
   terms are concatenated (Koji Sekiguchi via Mike McCandless)

New features

 * LUCENE-1531: Added support for BoostingTermQuery to XML query parser. (Karl Wettin)

 * LUCENE-1435: Added contrib/collation, a CollationKeyFilter
    allowing you to convert tokens into CollationKeys encoded using
    IndexableBinaryStringTools.  This allows for faster RangeQuery when
    a field needs to use a custom Collator.  (Steven Rowe via Mike
    McCandless)

 * LUCENE-1591: EnWikiDocMaker, LineDocMaker, WriteLineDoc can now
    read/write bz2 using Apache commons compress library.  This means
    you can download the .bz2 export from http://wikipedia.org and
    immediately index it.  (Shai Erera via Mike McCandless)

 * LUCENE-1629: Add SmartChineseAnalyzer to contrib/analyzers.  It
    improves on CJKAnalyzer and ChineseAnalyzer by handling Chinese
    sentences properly.  SmartChineseAnalyzer uses a Hidden Markov
    Model to tokenize Chinese words in a more intelligent way.
    (Xiaoping Gao via Mike McCandless)

 * LUCENE-1676: Added DelimitedPayloadTokenFilter class for automatically adding payloads "in-stream" (Grant Ingersoll)    
 
 * LUCENE-1578: Support for loading unoptimized readers to the
    constructor of InstantiatedIndex. (Karl Wettin)

 * LUCENE-1704: Allow specifying the Tidy configuration file when
    parsing HTML docs with contrib/ant.  (Keith Sprochi via Mike
    McCandless)

 * LUCENE-1522: Added contrib/fast-vector-highlighter, a new alternative
    highlighter.  (Koji Sekiguchi via Mike McCandless)

 * LUCENE-1740: Added "analyzer" command to Lucli, enabling changing
    the analyzer from the default StandardAnalyzer.  (Bernd Fondermann
    via Mike McCandless)

 * LUCENE-1272: Add get/setBoost to MoreLikeThis. (Jonathan
    Leibiusky via Mike McCandless)
 
 * LUCENE-1745: Added constructors to JakartaRegexpCapabilities and
    JavaUtilRegexCapabilities as well as static flags to support
    configuring a RegexCapabilities implementation with the
    implementation-specific modifier flags. Allows for callers to
    customize the RegexQuery using the implementation-specific options
    and fine tune how regular expressions are compiled and
    matched. (Marc Zampetti zampettim@aim.com via Mike McCandless)
 
 * LUCENE-1567: Added a new QueryParser framework, that allows 
    implementing a new query syntax in a flexible and efficient way.
    This new QueryParser will be moved to Lucene's core in release
    3.0 and will then replace the current core QueryParser, which
    has been deprecated with this patch.
    (Luis Alves and Adriano Campos via Michael Busch)
    
 * LUCENE-1486: Added ComplexPhraseQueryParser, an extension of QueryParser 
    that allows a subset of the Lucene query language to be embedded in
    PhraseQuerys. Wildcard, Range, and Fuzzy queries, as well as limited 
    boolean logic, can be used within quote operators with this parser, ie: 
    "(jo* -john) smyth~". (Mark Harwood via Mark Miller)
    
 * Added web-based demo of functionality in contrib's XML Query Parser
    packaged as War file (Mark Harwood)

 * LUCENE-1406: Added Arabic analyzer.  (Robert Muir via Grant Ingersoll)

 * LUCENE-1628: Added Persian analyzer.  (Robert Muir)

 * LUCENE-1813: Add option to ReverseStringFilter to mark reversed tokens.
    (Andrzej Bialecki via Robert Muir)

Optimizations

 * LUCENE-1643: Re-use the collation key (RawCollationKey) for
     better performance, in ICUCollationKeyFilter.  (Robert Muir via
     Mike McCandless)

 * LUCENE-1794: Implement TokenStream reuse for contrib Analyzers, 
     and implement reset() for TokenStreams to support reuse.  (Robert Muir)

Documentation

 * LUCENE-1876: added missing package level documentation for numerous
     contrib packages.
     (Steven Rowe & Robert Muir)

Build

 * LUCENE-1728: Split contrib/analyzers into common and smartcn modules. 
   Contrib/analyzers now builds an additional lucene-smartcn Jar file. All
   smartcn classes are not included in the lucene-analyzers JAR file.
   (Robert Muir via Simon Willnauer)
 
 * LUCENE-1829: Fix contrib query parser to properly create javacc files.
   (Jan-Pascal and Luis Alves via Michael Busch)      

Test Cases


======================= Release 2.4.0 2008-10-06 =======================

Changes in runtime behavior

 (None)

API Changes

 1. 

 (None)

Bug fixes

 1. LUCENE-1312: Added full support for InstantiatedIndexReader#getFieldNames()
    and tests that assert that deleted documents behaves as they should (they did).
    (Jason Rutherglen, Karl Wettin)

 2. LUCENE-1318: InstantiatedIndexReader.norms(String, b[], int) didn't treat
    the array offset right. (Jason Rutherglen via Karl Wettin)

New features

 1. LUCENE-1320: ShingleMatrixFilter, multidimensional shingle token filter. (Karl Wettin)

 2. LUCENE-1142: Updated Snowball package, org.tartarus distribution revision 500.
    Introducing Hungarian, Turkish and Romanian support, updated older stemmers
    and optimized (reflectionless) SnowballFilter.
    IMPORTANT NOTICE ON BACKWARDS COMPATIBILITY: an index created using the 2.3.2 (or older)
    might not be compatible with these updated classes as some algorithms have changed.
    (Karl Wettin)

 3. LUCENE-1016: TermVectorAccessor, transparent vector space access via stored vectors
    or by resolving the inverted index. (Karl Wettin) 

Documentation

 (None)

Build

 (None)

Test Cases

 (None)