Java 11 supports to use Direct IO without native wrappers from Java code. NativeUnixDirectory in the misc module was therefore removed and replaced by DirectIODirectory. To use it, you need a JVM and operating system that supports Direct IO.
The discount discountOverlaps parameter for both BM25Similarity and LegacyBM25Similarity is now set by the constructor of those classes.
Following package names in misc module are renamed.
Also, o.a.l.document.InetAddressPoint and o.a.l.document.InetAddressRange are moved to core module.
Following package names in sandbox module are renamed.
o.a.l.codecs package in lucene-backward-codecs module is renamed to o.a.l.backward_codecs.
Previously, JapanesePartOfSpeechStopFilterFactory added no filter if args didn't include “tags”. Now, it will load the default stop tags returned by JapaneseAnalyzer.getDefaultStopTags() (i.e. the tags fromstoptags.txt in the lucene-analyzers-kuromoji jar.)
o.a.l.collation.ICUCollationAnalyzer is renamed to o.a.l.a.icu.ICUCollationKeyAnalyzer. Also, its dependant classes are renamed in the same way.
All binary analysis packages (and corresponding Maven artifacts) have been renamed and are now consistent with repository module ‘analysis’.
lucene-core, also their package names are renamed.The service provider files placed in META-INF/services for custom analysis factories should be renamed as follows:
o.a.l.a.standard.StandardTokenizerFactory is moved to lucene-core module.
o.a.l.a.standard package in lucene-analysis-common module is split into o.a.l.a.classic and o.a.l.a.email.
We now follow the Java rules for accepting backslashes. Alphabetic characters other than s, S, w, W, d or D that are preceded by a backslash are considered illegal syntax and will throw an exception.
The commonly used regular expressions \w \W \d \D \s and \S now work the same way Java Pattern matching works. Previously these expressions were (mis)interpreted as searches for the literal characters w, d, s etc.
The factory option name to output the original term was corrected in accordance with its Javadoc.
This command-line tool no longer forceMerges to a single segment. Instead, by default it just follows (configurable) merge policy. If you really want to merge to a single segment, you can pass -max-segments 1.
Simply use FSTCompiler instead of the previous Builder. Use either the simple constructor with default settings, or the FSTCompiler.Builder to tune and tweak any parameter.
User dictionary now strictly validates if the (concatenated) segment is the same as the surface form. This change avoids unexpected runtime exceptions or behaviours. For example, these entries are not allowed at all and an exception is thrown when loading the dictionary file.
# concatenated "日本経済新聞" does not match the surface form "日経新聞" 日経新聞,日本 経済 新聞,ニホン ケイザイ シンブン,カスタム名詞 # concatenated "日経新聞" does not match the surface form "日本経済新聞" 日本経済新聞,日経 新聞,ニッケイ シンブン,カスタム名詞
JapaneseTokenizer and JapaneseAnalyzer no longer emits original tokens when discardCompoundToken option is not specified. The constructor option has been introduced since Lucene 8.5.0, and the default value is changed to true.
When given the text “株式会社”, JapaneseTokenizer (mode != NORMAL) emits decompounded tokens “株式” and “会社” only and no longer outputs the original token “株式会社” by default. To output original tokens, discardCompoundToken option should be explicitly set to false. Be aware that if this option is set to false SynonymFilter or SynonymGraphFilter does not work correctly (see LUCENE-9173).
The SPI names for concrete subclasses of TokenizerFactory, TokenFilterFactory, and CharfilterFactory are no longer derived from their class name. Instead, each factory must have a static “NAME” field like this:
/** o.a.l.a.standard.StandardTokenizerFactory's SPI name */
public static final String NAME = "standard";
A factory can be resolved/instantiated with its NAME by using methods such as TokenizerFactory#lookupClass(String) or TokenizerFactory#forName(String, Map<String,String>).
If there are any user-defined factory classes that don't have proper NAME field, an exception will be thrown when (re)loading factories. e.g., when calling TokenizerFactory#reloadTokenizers(ClassLoader).
In addition starting all factories need to implement a public no-arg constructor, too. The reason for this change comes from the fact that Lucene now uses java.util.ServiceLoader instead its own implementation to load the factory classes to be compatible with Java Module System changes (e.g., load factories from modules). In the future, extensions to Lucene developed on the Java Module System may expose the factories from their module-info.java file instead of META-INF/services.
This constructor is never called by Lucene, so by default it throws a UnsupportedOperationException. User-defined factory classes should implement it in the following way:
/** Default ctor for compatibility with SPI */
public StandardTokenizerFactory() {
throw defaultCtorException();
}
(defaultCtorException() is a protected static helper method)
TermsEnum has been changed to be fully abstract, so non-abstract subclass must implement all it's methods. Non-Performance critical TermsEnums can use BaseTermsEnum as a base class instead. The change was motivated by several performance issues with FilterTermsEnum that caused significant slowdowns and massive memory consumption due to not delegating all method from TermsEnum. See LUCENE-8292 and LUCENE-8662
RAM-based directory implementation have been removed. (LUCENE-8474). ByteBuffersDirectory can be used as a RAM-resident replacement, although it is discouraged in favor of the default memory-mapped directory.
SpanQuery and PhraseQuery now always calculate their slops as (1.0 / (1.0 + distance)). Payload factor calculation is performed by PayloadDecoder in the queries module
Scorers are no longer allowed to produce negative scores. If you have custom query implementations, you should make sure their score formula may never produce negative scores.
As a side-effect of this change, negative boosts are now rejected and FunctionScoreQuery maps negative values to 0.
Instead use FunctionScoreQuery and a DoubleValuesSource implementation. BoostedQuery and BoostingQuery may be replaced by calls to FunctionScoreQuery.boostByValue() and FunctionScoreQuery.boostByQuery(). To replace more complex calculations in CustomScoreQuery, use the lucene-expressions module:
SimpleBindings bindings = new SimpleBindings();
bindings.add("score", DoubleValuesSource.SCORES);
bindings.add("boost1", DoubleValuesSource.fromIntField("myboostfield"));
bindings.add("boost2", DoubleValuesSource.fromIntField("myotherboostfield"));
Expression expr = JavascriptCompiler.compile("score * (boost1 + ln(boost2))");
FunctionScoreQuery q = new FunctionScoreQuery(inputQuery, expr.getDoubleValuesSource(bindings));
Changing index options on the fly is now going to result into an IllegalArgumentException. If a field is indexed (FieldType.indexOptions() != IndexOptions.NONE) then all documents must have the same index options for that field.
Instead use IndexSearcher.createWeight(), rewriting the query first, and using a boost of 1f.
Memory codecs have been removed from the codebase (MemoryPostings, MemoryDocValues).
The “Direct” doc-value format has been removed from the codebase.
Caching everything is discouraged as it disables the ability to skip non-interesting documents. ALWAYS_CACHE can be replaced by a UsageTrackingQueryCachingPolicy with an appropriate config.
To retain the old behaviour, pass EnglishAnalyzer.ENGLISH_STOP_WORDS_SET as an argument to the constructor
English stop words are now defined in EnglishAnalyzer#ENGLISH_STOP_WORDS_SET in the analysis-common module
TopDocs.maxScore is removed. IndexSearcher and TopFieldCollector no longer have an option to compute the maximum score when sorting by field. If you need to know the maximum score for a query, the recommended approach is to run a separate query:
TopDocs topHits = searcher.search(query, 1); float maxScore = topHits.scoreDocs.length == 0 ? Float.NaN : topHits.scoreDocs[0].score;
Thanks to other optimizations that were added to Lucene 8, this query will be able to efficiently select the top-scoring document without having to visit all matches.
Because filling sort values doesn't have a significant overhead, the fillFields option has been removed from TopFieldCollector factory methods. Everything behaves as if it was set to true.
Computing scores at collection time is less efficient than running a second request in order to only compute scores for documents that made it to the top hits. As a consequence, the trackDocScores option has been removed and can be replaced with the new TopFieldCollector#populateScores helper method.
Lucene 8 received optimizations for collection of top-k matches by not visiting all matches. However these optimizations won‘t help if all matches still need to be visited in order to compute the total number of hits. As a consequence, IndexSearcher’s search and searchAfter methods were changed to only count hits accurately up to 1,000, and Topdocs.totalHits was changed from a long to an object that says whether the hit count is accurate or a lower bound of the actual hit count.
This RAM-based directory implementation is an old piece of code that uses inefficient thread synchronization primitives and can be confused as “faster” than the NIO-based MMapDirectory. It is deprecated and scheduled for removal in future versions of Lucene. (LUCENE-8467, LUCENE-8438)
Scorer has a number of methods that should never be called from Collectors, for example those that advance the underlying iterators. To hide these, LeafCollector.setScorer() now takes a Scorable, an abstract class that Scorers can extend, with methods docId() and score() (LUCENE-6228)
If a custom Scorer implementation does not have an associated Weight, it can probably be replaced with a Scorable instead.
Most code should just require recompilation, though possibly requiring some added casts.
Instead of overriding TokenStreamComponents#setReader() to customise analyzer initialisation, you should now pass a Consumer<Reader> instance to the TokenStreamComponents constructor.
LowerCaseTokenizer combined tokenization and filtering in a way that broke token normalization, so they have been removed. Instead, use a LetterTokenizer followed by a LowerCaseFilter
CharTokenizer now only performs tokenization. To perform any type of filtering use a TokenFilter chain as you would with any other Tokenizer.
Both Highlighter and FastVectorHighlighter need a custom WeightedSpanTermExtractor or FieldQuery respectively in order to support ToParent/ToChildBlockJoinQuery.
Normalization is now type-safe, with CharFilterFactory#normalize() returning a Reader and TokenFilterFactory#normalize() returning a TokenFilter.
Scores computed by the BM25 similarity are lower than previously as the k1+1 constant factor was removed from the numerator of the scoring formula. Ordering of results is preserved unless scores are computed from multiple fields using different similarities. The previous behaviour is now exposed by the LegacyBM25Similarity class which can be found in the lucene-misc jar.
IndexWriter#getDocStats() should be used instead of #maxDoc() / #numDocs() which offers a consistent view on document stats. Previously calling two methods in order ot get point in time stats was subject to concurrent changes.
IndexSearcher now performs max clause count checks on all types of queries (including BooleanQueries). This led to a logical move of the clauses count from BooleanQuery to IndexSearcher.
TopDocs.merge's API has been changed to stop allowing passing in a parameter to indicate if it should set shard indices for hits as they are seen during the merge process. This is done to simplify the API to be more dynamic in terms of passing in custom tie breakers. If shard indices are to be used for tie breaking docs with equal scores during TopDocs.merge, then it is mandatory that the input ScoreDocs have their shard indices set to valid values prior to calling TopDocs.merge
TopDocsCollector shall no longer return an empty TopDocs for malformed arguments. Rather, an IllegalArgumentException shall be thrown. This is introduced for better defence and to ensure that there is no bubbling up of errors when Lucene is used in multi level applications
Sorting on a numeric field that is indexed with both doc values and points may use an optimization to skip non-competitive documents. This optimization relies on the assumption that the same data is stored in these points and doc values.