blob: 2cfa45e73b4c678b412bf51affc24141015b99ca [file] [log] [blame]
<!DOCTYPE html>
<!--[if IE]><![endif]-->
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Namespace Lucene.Net.Analysis.Core
| Apache Lucene.NET 4.8.0-beta00010 Documentation </title>
<meta name="viewport" content="width=device-width">
<meta name="title" content="Namespace Lucene.Net.Analysis.Core
| Apache Lucene.NET 4.8.0-beta00010 Documentation ">
<meta name="generator" content="docfx 2.56.0.0">
<link rel="shortcut icon" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/favicon.ico">
<link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.css">
<link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.css">
<link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.css">
<meta property="docfx:navrel" content="toc.html">
<meta property="docfx:tocrel" content="analysis-common/toc.html">
<meta property="docfx:rel" content="https://lucenenet.apache.org/docs/4.8.0-beta00009/">
</head>
<body data-spy="scroll" data-target="#affix" data-offset="120">
<div id="wrapper">
<header>
<nav id="autocollapse" class="navbar ng-scope" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="/">
<img id="logo" class="svg" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/lucene-net-color.png" alt="">
</a>
</div>
<div class="collapse navbar-collapse" id="navbar">
<form class="navbar-form navbar-right" role="search" id="search">
<div class="form-group">
<input type="text" class="form-control" id="search-query" placeholder="Search" autocomplete="off">
</div>
</form>
</div>
</div>
</nav>
<div class="subnav navbar navbar-default">
<div class="container hide-when-search">
<ul class="level0 breadcrumb">
<li>
<a href="https://lucenenet.apache.org/docs/4.8.0-beta00009/">API</a>
<span id="breadcrumb">
<ul class="breadcrumb">
<li></li>
</ul>
</span>
</li>
</ul>
</div>
</div>
</header>
<div class="container body-content">
<div id="search-results">
<div class="search-list"></div>
<div class="sr-items">
<p><i class="glyphicon glyphicon-refresh index-loading"></i></p>
</div>
<ul id="pagination"></ul>
</div>
</div>
<div role="main" class="container body-content hide-when-search">
<div class="sidenav hide-when-search">
<a class="btn toc-toggle collapse" data-toggle="collapse" href="#sidetoggle" aria-expanded="false" aria-controls="sidetoggle">Show / Hide Table of Contents</a>
<div class="sidetoggle collapse" id="sidetoggle">
<div id="sidetoc"></div>
</div>
</div>
<div class="article row grid-right">
<div class="col-md-10">
<article class="content wrap" id="_content" data-uid="Lucene.Net.Analysis.Core">
<h1 id="Lucene_Net_Analysis_Core" data-uid="Lucene.Net.Analysis.Core" class="text-break">Namespace Lucene.Net.Analysis.Core
</h1>
<div class="markdown level0 summary"><!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>Basic, general-purpose analysis components.</p>
</div>
<div class="markdown level0 conceptual"></div>
<div class="markdown level0 remarks"></div>
<h3 id="classes">Classes
</h3>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.KeywordAnalyzer.html">KeywordAnalyzer</a></h4>
<section><p>&quot;Tokenizes&quot; the entire stream as a single token. This is useful
for data like zip codes, ids, and some product names.</p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.KeywordTokenizer.html">KeywordTokenizer</a></h4>
<section><p>Emits the entire input as a single token.</p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.KeywordTokenizerFactory.html">KeywordTokenizerFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.KeywordTokenizer.html">KeywordTokenizer</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_keyword&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.KeywordTokenizerFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a></h4>
<section><p>A <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a> is a tokenizer that divides text at non-letters. That&apos;s to
say, it defines tokens as maximal strings of adjacent letters, as defined by
<span class="xref">System.Char.IsLetter(System.Char)</span> predicate.
<p>
Note: this does a decent job for most European languages, but does a terrible
job for some Asian languages, where words are not separated by spaces.
</p>
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span> compatibility when creating
<a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a>:
<ul><li>As of 3.1, <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html">CharTokenizer</a> uses an <span class="xref">System.Int32</span> based API to normalize and
detect token characters. See <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_IsTokenChar_System_Int32_">IsTokenChar(Int32)</a> and
<a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_Normalize_System_Int32_">Normalize(Int32)</a> for details.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizerFactory.html">LetterTokenizerFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_letter&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.LetterTokenizerFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a></h4>
<section><p>Normalizes token text to lower case.
<p>You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span>
compatibility when creating LowerCaseFilter:
<ul><li> As of 3.1, supplementary characters are properly lowercased.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilterFactory.html">LowerCaseFilterFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_lwrcase&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/>
&lt;filter class=&quot;solr.LowerCaseFilterFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizer.html">LowerCaseTokenizer</a></h4>
<section><p><a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizer.html">LowerCaseTokenizer</a> performs the function of <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a>
and <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a> together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the combination
of <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a> and <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a>, there is a performance advantage
to doing the two tasks at once, hence this (redundant) implementation.
<p>
Note: this does a decent job for most European languages, but does a terrible
job for some Asian languages, where words are not separated by spaces.
</p>
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span> compatibility when creating
<a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizer.html">LowerCaseTokenizer</a>:
<ul><li>As of 3.1, <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html">CharTokenizer</a> uses an int based API to normalize and
detect token characters. See <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_IsTokenChar_System_Int32_">IsTokenChar(Int32)</a> and
<a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_Normalize_System_Int32_">Normalize(Int32)</a> for details.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizerFactory.html">LowerCaseTokenizerFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizer.html">LowerCaseTokenizer</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_lwrcase&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.LowerCaseTokenizerFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.SimpleAnalyzer.html">SimpleAnalyzer</a></h4>
<section><p>An <span class="xref">Lucene.Net.Analysis.Analyzer</span> that filters <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a>
with <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a>
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span> compatibility
when creating <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html">CharTokenizer</a>:
<ul><li>As of 3.1, <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseTokenizer.html">LowerCaseTokenizer</a> uses an int based API to normalize and
detect token codepoints. See <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_IsTokenChar_System_Int32_">IsTokenChar(Int32)</a> and
<a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_Normalize_System_Int32_">Normalize(Int32)</a> for details.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.StopAnalyzer.html">StopAnalyzer</a></h4>
<section><p>Filters <a class="xref" href="Lucene.Net.Analysis.Core.LetterTokenizer.html">LetterTokenizer</a> with <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a> and <a class="xref" href="Lucene.Net.Analysis.Core.StopFilter.html">StopFilter</a>.
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span>
compatibility when creating <a class="xref" href="Lucene.Net.Analysis.Core.StopAnalyzer.html">StopAnalyzer</a>:
<ul><li> As of 3.1, StopFilter correctly handles Unicode 4.0
supplementary characters in stopwords</li><li> As of 2.9, position increments are preserved</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.StopFilter.html">StopFilter</a></h4>
<section><p>Removes stop words from a token stream.
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span>
compatibility when creating <a class="xref" href="Lucene.Net.Analysis.Core.StopFilter.html">StopFilter</a>:
<ul><li>As of 3.1, StopFilter correctly handles Unicode 4.0
supplementary characters in stopwords and position
increments are preserved</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.StopFilterFactory.html">StopFilterFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.StopFilter.html">StopFilter</a>.</p>
<pre><code>&lt;fieldType name=&quot;text_stop&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot; autoGeneratePhraseQueries=&quot;true&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/>
&lt;filter class=&quot;solr.StopFilterFactory&quot; ignoreCase=&quot;true&quot;
words=&quot;stopwords.txt&quot; format=&quot;wordset&quot; />
&lt;/analyzer>
&lt;/fieldType></code></pre>
<p><p>
All attributes are optional:
</p>
<ul><li><code>ignoreCase</code> defaults to <code>false</code></li><li><code>words</code> should be the name of a stopwords file to parse, if not
specified the factory will use <a class="xref" href="Lucene.Net.Analysis.Core.StopAnalyzer.html#Lucene_Net_Analysis_Core_StopAnalyzer_ENGLISH_STOP_WORDS_SET">ENGLISH_STOP_WORDS_SET</a>
</li><li><code>format</code> defines how the <code>words</code> file will be parsed,
and defaults to <code>wordset</code>. If <code>words</code> is not specified,
then <code>format</code> must not be specified.
</li></ul>
<p>
The valid values for the <code>format</code> option are:
</p>
<ul><li><code>wordset</code> - This is the default format, which supports one word per
line (including any intra-word whitespace) and allows whole line comments
begining with the &quot;#&quot; character. Blank lines are ignored. See
<a class="xref" href="Lucene.Net.Analysis.Util.WordlistLoader.html#Lucene_Net_Analysis_Util_WordlistLoader_GetLines_System_IO_Stream_System_Text_Encoding_">GetLines(Stream, Encoding)</a> for details.
</li><li><code>snowball</code> - This format allows for multiple words specified on each
line, and trailing comments may be specified using the vertical line (&quot;|&quot;).
Blank lines are ignored. See
<a class="xref" href="Lucene.Net.Analysis.Util.WordlistLoader.html#Lucene_Net_Analysis_Util_WordlistLoader_GetSnowballWordSet_System_IO_TextReader_Lucene_Net_Util_LuceneVersion_">GetSnowballWordSet(TextReader, LuceneVersion)</a>
for details.
</li></ul></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.TypeTokenFilter.html">TypeTokenFilter</a></h4>
<section><p>Removes tokens whose types appear in a set of blocked types from a token stream.</p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.TypeTokenFilterFactory.html">TypeTokenFilterFactory</a></h4>
<section><p>Factory class for <a class="xref" href="Lucene.Net.Analysis.Core.TypeTokenFilter.html">TypeTokenFilter</a>.</p>
<pre><code>&lt;fieldType name=&quot;chars&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.StandardTokenizerFactory&quot;/>
&lt;filter class=&quot;solr.TypeTokenFilterFactory&quot; types=&quot;stoptypes.txt&quot;
useWhitelist=&quot;false&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.UpperCaseFilter.html">UpperCaseFilter</a></h4>
<section><p>Normalizes token text to UPPER CASE.
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span>
compatibility when creating <a class="xref" href="Lucene.Net.Analysis.Core.UpperCaseFilter.html">UpperCaseFilter</a>
</p>
<p><strong>NOTE:</strong> In Unicode, this transformation may lose information when the
upper case character represents more than one lower case character. Use this filter
when you Require uppercase tokens. Use the <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilter.html">LowerCaseFilter</a> for
general search matching
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.UpperCaseFilterFactory.html">UpperCaseFilterFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.UpperCaseFilter.html">UpperCaseFilter</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_uppercase&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/>
&lt;filter class=&quot;solr.UpperCaseFilterFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
<p><strong>NOTE:</strong> In Unicode, this transformation may lose information when the
upper case character represents more than one lower case character. Use this filter
when you require uppercase tokens. Use the <a class="xref" href="Lucene.Net.Analysis.Core.LowerCaseFilterFactory.html">LowerCaseFilterFactory</a> for
general search matching
</p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceAnalyzer.html">WhitespaceAnalyzer</a></h4>
<section><p>An <span class="xref">Lucene.Net.Analysis.Analyzer</span> that uses <a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a>.
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span> compatibility
when creating <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html">CharTokenizer</a>:
<ul><li>As of 3.1, <a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a> uses an int based API to normalize and
detect token codepoints. See <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_IsTokenChar_System_Int32_">IsTokenChar(Int32)</a> and
<a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_Normalize_System_Int32_">Normalize(Int32)</a> for details.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a></h4>
<section><p>A <a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a> is a tokenizer that divides text at whitespace.
Adjacent sequences of non-Whitespace characters form tokens.
<p>
You must specify the required <span class="xref">Lucene.Net.Util.LuceneVersion</span> compatibility when creating
<a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a>:
<ul><li>As of 3.1, <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html">CharTokenizer</a> uses an int based API to normalize and
detect token characters. See <a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_IsTokenChar_System_Int32_">IsTokenChar(Int32)</a> and
<a class="xref" href="Lucene.Net.Analysis.Util.CharTokenizer.html#Lucene_Net_Analysis_Util_CharTokenizer_Normalize_System_Int32_">Normalize(Int32)</a> for details.</li></ul>
</p></p>
</section>
<h4><a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizerFactory.html">WhitespaceTokenizerFactory</a></h4>
<section><p>Factory for <a class="xref" href="Lucene.Net.Analysis.Core.WhitespaceTokenizer.html">WhitespaceTokenizer</a>. </p>
<pre><code>&lt;fieldType name=&quot;text_ws&quot; class=&quot;solr.TextField&quot; positionIncrementGap=&quot;100&quot;>
&lt;analyzer>
&lt;tokenizer class=&quot;solr.WhitespaceTokenizerFactory&quot;/>
&lt;/analyzer>
&lt;/fieldType></code></pre>
</section>
</article>
</div>
<div class="hidden-sm col-md-2" role="complementary">
<div class="sideaffix">
<div class="contribution">
<ul class="nav">
<li>
<a href="https://github.com/apache/lucenenet/blob/docs/4.8.0-beta00010/src/Lucene.Net.Analysis.Common/Analysis/Core/package.md/#L2" class="contribution-link">Improve this Doc</a>
</li>
</ul>
</div>
<nav class="bs-docs-sidebar hidden-print hidden-xs hidden-sm affix" id="affix">
<!-- <p><a class="back-to-top" href="#top">Back to top</a><p> -->
</nav>
</div>
</div>
</div>
</div>
<footer>
<div class="grad-bottom"></div>
<div class="footer">
<div class="container">
<span class="pull-right">
<a href="#top">Back to top</a>
</span>
Copyright © 2020 Licensed to the Apache Software Foundation (ASF)
</div>
</div>
</footer>
</div>
<script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.js"></script>
<script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.js"></script>
<script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.js"></script>
</body>
</html>