| <!DOCTYPE html> |
| <!--[if IE]><![endif]--> |
| <html> |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> |
| <title>Class TokenStream |
| | Apache Lucene.NET 4.8.0-beta00010 Documentation </title> |
| <meta name="viewport" content="width=device-width"> |
| <meta name="title" content="Class TokenStream |
| | Apache Lucene.NET 4.8.0-beta00010 Documentation "> |
| <meta name="generator" content="docfx 2.56.0.0"> |
| |
| <link rel="shortcut icon" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/favicon.ico"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.css"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.css"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.css"> |
| <meta property="docfx:navrel" content="toc.html"> |
| <meta property="docfx:tocrel" content="core/toc.html"> |
| |
| <meta property="docfx:rel" content="https://lucenenet.apache.org/docs/4.8.0-beta00009/"> |
| |
| </head> |
| <body data-spy="scroll" data-target="#affix" data-offset="120"> |
| <div id="wrapper"> |
| <header> |
| |
| <nav id="autocollapse" class="navbar ng-scope" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| |
| <a class="navbar-brand" href="/"> |
| <img id="logo" class="svg" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/lucene-net-color.png" alt=""> |
| </a> |
| </div> |
| <div class="collapse navbar-collapse" id="navbar"> |
| <form class="navbar-form navbar-right" role="search" id="search"> |
| <div class="form-group"> |
| <input type="text" class="form-control" id="search-query" placeholder="Search" autocomplete="off"> |
| </div> |
| </form> |
| </div> |
| </div> |
| </nav> |
| |
| <div class="subnav navbar navbar-default"> |
| <div class="container hide-when-search"> |
| <ul class="level0 breadcrumb"> |
| <li> |
| <a href="https://lucenenet.apache.org/docs/4.8.0-beta00009/">API</a> |
| <span id="breadcrumb"> |
| <ul class="breadcrumb"> |
| <li></li> |
| </ul> |
| </span> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </header> |
| <div class="container body-content"> |
| |
| <div id="search-results"> |
| <div class="search-list"></div> |
| <div class="sr-items"> |
| <p><i class="glyphicon glyphicon-refresh index-loading"></i></p> |
| </div> |
| <ul id="pagination"></ul> |
| </div> |
| </div> |
| <div role="main" class="container body-content hide-when-search"> |
| |
| <div class="sidenav hide-when-search"> |
| <a class="btn toc-toggle collapse" data-toggle="collapse" href="#sidetoggle" aria-expanded="false" aria-controls="sidetoggle">Show / Hide Table of Contents</a> |
| <div class="sidetoggle collapse" id="sidetoggle"> |
| <div id="sidetoc"></div> |
| </div> |
| </div> |
| <div class="article row grid-right"> |
| <div class="col-md-10"> |
| <article class="content wrap" id="_content" data-uid="Lucene.Net.Analysis.TokenStream"> |
| |
| |
| <h1 id="Lucene_Net_Analysis_TokenStream" data-uid="Lucene.Net.Analysis.TokenStream" class="text-break">Class TokenStream |
| </h1> |
| <div class="markdown level0 summary"><p>A <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> enumerates the sequence of tokens, either from |
| <a class="xref" href="Lucene.Net.Documents.Field.html">Field</a>s of a <a class="xref" href="Lucene.Net.Documents.Document.html">Document</a> or from query text. |
| <p> |
| this is an abstract class; concrete subclasses are: |
| <ul><li><a class="xref" href="Lucene.Net.Analysis.Tokenizer.html">Tokenizer</a>, a <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> whose input is a <span class="xref">System.IO.TextReader</span>; and</li><li><a class="xref" href="Lucene.Net.Analysis.TokenFilter.html">TokenFilter</a>, a <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> whose input is another |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>.</li></ul> |
| A new <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> API has been introduced with Lucene 2.9. this API |
| has moved from being <a class="xref" href="Lucene.Net.Analysis.Token.html">Token</a>-based to <a class="xref" href="Lucene.Net.Util.IAttribute.html">IAttribute</a>-based. While |
| <a class="xref" href="Lucene.Net.Analysis.Token.html">Token</a> still exists in 2.9 as a convenience class, the preferred way |
| to store the information of a <a class="xref" href="Lucene.Net.Analysis.Token.html">Token</a> is to use <span class="xref">System.Attribute</span>s. |
| <p> |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> now extends <a class="xref" href="Lucene.Net.Util.AttributeSource.html">AttributeSource</a>, which provides |
| access to all of the token <a class="xref" href="Lucene.Net.Util.IAttribute.html">IAttribute</a>s for the <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>. |
| Note that only one instance per <span class="xref">System.Attribute</span> is created and reused |
| for every token. This approach reduces object creation and allows local |
| caching of references to the <span class="xref">System.Attribute</span>s. See |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a> for further details. |
| <p> |
| <strong>The workflow of the new <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> API is as follows:</strong> |
| <ol><li>Instantiation of <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>/<a class="xref" href="Lucene.Net.Analysis.TokenFilter.html">TokenFilter</a>s which add/get |
| attributes to/from the <a class="xref" href="Lucene.Net.Util.AttributeSource.html">AttributeSource</a>.</li><li>The consumer calls <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_Reset">Reset()</a>.</li><li>The consumer retrieves attributes from the stream and stores local |
| references to all attributes it wants to access.</li><li>The consumer calls <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a> until it returns false |
| consuming the attributes after each call.</li><li>The consumer calls <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_End">End()</a> so that any end-of-stream operations |
| can be performed.</li><li>The consumer calls <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_Dispose">Dispose()</a> to release any resource when finished |
| using the <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>.</li></ol> |
| To make sure that filters and consumers know which attributes are available, |
| the attributes must be added during instantiation. Filters and consumers are |
| not required to check for availability of attributes in |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a>. |
| <p> |
| You can find some example code for the new API in the analysis |
| documentation. |
| <p> |
| Sometimes it is desirable to capture a current state of a <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>, |
| e.g., for buffering purposes (see <a class="xref" href="Lucene.Net.Analysis.CachingTokenFilter.html">CachingTokenFilter</a>, |
| TeeSinkTokenFilter). For this usecase |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_CaptureState">CaptureState()</a> and <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_RestoreState_Lucene_Net_Util_AttributeSource_State_">RestoreState(AttributeSource.State)</a> |
| can be used. |
| <p>The <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a>-API in Lucene is based on the decorator pattern. |
| Therefore all non-abstract subclasses must be sealed or have at least a sealed |
| implementation of <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a>! This is checked when assertions are enabled.</p> |
| </div> |
| <div class="markdown level0 conceptual"></div> |
| <div class="inheritance"> |
| <h5>Inheritance</h5> |
| <div class="level0"><span class="xref">System.Object</span></div> |
| <div class="level1"><a class="xref" href="Lucene.Net.Util.AttributeSource.html">AttributeSource</a></div> |
| <div class="level2"><span class="xref">TokenStream</span></div> |
| <div class="level3"><a class="xref" href="Lucene.Net.Analysis.NumericTokenStream.html">NumericTokenStream</a></div> |
| <div class="level3"><a class="xref" href="Lucene.Net.Analysis.TokenFilter.html">TokenFilter</a></div> |
| <div class="level3"><a class="xref" href="Lucene.Net.Analysis.Tokenizer.html">Tokenizer</a></div> |
| </div> |
| <div classs="implements"> |
| <h5>Implements</h5> |
| <div><span class="xref">System.IDisposable</span></div> |
| </div> |
| <div class="inheritedMembers"> |
| <h5>Inherited Members</h5> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetAttributeFactory">AttributeSource.GetAttributeFactory()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetAttributeClassesEnumerator">AttributeSource.GetAttributeClassesEnumerator()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetAttributeImplsEnumerator">AttributeSource.GetAttributeImplsEnumerator()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_AddAttributeImpl_Lucene_Net_Util_Attribute_">AttributeSource.AddAttributeImpl(Attribute)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_AddAttribute__1">AttributeSource.AddAttribute<T>()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_HasAttributes">AttributeSource.HasAttributes</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_HasAttribute__1">AttributeSource.HasAttribute<T>()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetAttribute__1">AttributeSource.GetAttribute<T>()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_ClearAttributes">AttributeSource.ClearAttributes()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_CaptureState">AttributeSource.CaptureState()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_RestoreState_Lucene_Net_Util_AttributeSource_State_">AttributeSource.RestoreState(AttributeSource.State)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetHashCode">AttributeSource.GetHashCode()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_Equals_System_Object_">AttributeSource.Equals(Object)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_ReflectAsString_System_Boolean_">AttributeSource.ReflectAsString(Boolean)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_ReflectWith_Lucene_Net_Util_IAttributeReflector_">AttributeSource.ReflectWith(IAttributeReflector)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_CloneAttributes">AttributeSource.CloneAttributes()</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_CopyTo_Lucene_Net_Util_AttributeSource_">AttributeSource.CopyTo(AttributeSource)</a> |
| </div> |
| <div> |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_ToString">AttributeSource.ToString()</a> |
| </div> |
| <div> |
| <span class="xref">System.Object.Equals(System.Object, System.Object)</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.GetType()</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.MemberwiseClone()</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.ReferenceEquals(System.Object, System.Object)</span> |
| </div> |
| </div> |
| <h6><strong>Namespace</strong>: <a class="xref" href="Lucene.Net.Analysis.html">Lucene.Net.Analysis</a></h6> |
| <h6><strong>Assembly</strong>: Lucene.Net.dll</h6> |
| <h5 id="Lucene_Net_Analysis_TokenStream_syntax">Syntax</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public abstract class TokenStream : AttributeSource, IDisposable</code></pre> |
| </div> |
| <h3 id="constructors">Constructors |
| </h3> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream__ctor.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.%23ctor%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L84">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream__ctor_" data-uid="Lucene.Net.Analysis.TokenStream.#ctor*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream__ctor" data-uid="Lucene.Net.Analysis.TokenStream.#ctor">TokenStream()</h4> |
| <div class="markdown level1 summary"><p>A <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> using the default attribute factory.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">protected TokenStream()</code></pre> |
| </div> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream__ctor_Lucene_Net_Util_AttributeSource_.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.%23ctor(Lucene.Net.Util.AttributeSource)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L93">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream__ctor_" data-uid="Lucene.Net.Analysis.TokenStream.#ctor*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream__ctor_Lucene_Net_Util_AttributeSource_" data-uid="Lucene.Net.Analysis.TokenStream.#ctor(Lucene.Net.Util.AttributeSource)">TokenStream(AttributeSource)</h4> |
| <div class="markdown level1 summary"><p>A <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> that uses the same attributes as the supplied one.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">protected TokenStream(AttributeSource input)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><a class="xref" href="Lucene.Net.Util.AttributeSource.html">AttributeSource</a></td> |
| <td><span class="parametername">input</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream__ctor_Lucene_Net_Util_AttributeSource_AttributeFactory_.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.%23ctor(Lucene.Net.Util.AttributeSource.AttributeFactory)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L104">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream__ctor_" data-uid="Lucene.Net.Analysis.TokenStream.#ctor*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream__ctor_Lucene_Net_Util_AttributeSource_AttributeFactory_" data-uid="Lucene.Net.Analysis.TokenStream.#ctor(Lucene.Net.Util.AttributeSource.AttributeFactory)">TokenStream(AttributeSource.AttributeFactory)</h4> |
| <div class="markdown level1 summary"><p>A <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> using the supplied <a class="xref" href="Lucene.Net.Util.AttributeSource.AttributeFactory.html">AttributeSource.AttributeFactory</a> |
| for creating new <a class="xref" href="Lucene.Net.Util.IAttribute.html">IAttribute</a> instances.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">protected TokenStream(AttributeSource.AttributeFactory factory)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><a class="xref" href="Lucene.Net.Util.AttributeSource.AttributeFactory.html">AttributeSource.AttributeFactory</a></td> |
| <td><span class="parametername">factory</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h3 id="methods">Methods |
| </h3> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream_Dispose.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.Dispose%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L181">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream_Dispose_" data-uid="Lucene.Net.Analysis.TokenStream.Dispose*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream_Dispose" data-uid="Lucene.Net.Analysis.TokenStream.Dispose">Dispose()</h4> |
| <div class="markdown level1 summary"></div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public void Dispose()</code></pre> |
| </div> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream_Dispose_System_Boolean_.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.Dispose(System.Boolean)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L194">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream_Dispose_" data-uid="Lucene.Net.Analysis.TokenStream.Dispose*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream_Dispose_System_Boolean_" data-uid="Lucene.Net.Analysis.TokenStream.Dispose(System.Boolean)">Dispose(Boolean)</h4> |
| <div class="markdown level1 summary"><p>Releases resources associated with this stream. |
| <p> |
| If you override this method, always call <code>base.Dispose(disposing)</code>, otherwise |
| some internal state will not be correctly reset (e.g., <a class="xref" href="Lucene.Net.Analysis.Tokenizer.html">Tokenizer</a> will |
| throw <span class="xref">System.InvalidOperationException</span> on reuse).</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">protected virtual void Dispose(bool disposing)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.Boolean</span></td> |
| <td><span class="parametername">disposing</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream_End.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.End%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L154">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream_End_" data-uid="Lucene.Net.Analysis.TokenStream.End*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream_End" data-uid="Lucene.Net.Analysis.TokenStream.End">End()</h4> |
| <div class="markdown level1 summary"><p>This method is called by the consumer after the last token has been |
| consumed, after <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a> returned <code>false</code> |
| (using the new <a class="xref" href="Lucene.Net.Analysis.TokenStream.html">TokenStream</a> API). Streams implementing the old API |
| should upgrade to use this feature. |
| <p> |
| This method can be used to perform any end-of-stream operations, such as |
| setting the final offset of a stream. The final offset of a stream might |
| differ from the offset of the last token eg in case one or more whitespaces |
| followed after the last token, but a WhitespaceTokenizer was used. |
| <p> |
| Additionally any skipped positions (such as those removed by a stopfilter) |
| can be applied to the position increment, or any adjustment of other |
| attributes where the end-of-stream value may be important. |
| <p> |
| If you override this method, always call <code>base.End();</code>.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public virtual void End()</code></pre> |
| </div> |
| <h5 class="exceptions">Exceptions</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Condition</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.IO.IOException</span></td> |
| <td><p>If an I/O error occurs </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream_IncrementToken.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.IncrementToken%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L134">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream_IncrementToken_" data-uid="Lucene.Net.Analysis.TokenStream.IncrementToken*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream_IncrementToken" data-uid="Lucene.Net.Analysis.TokenStream.IncrementToken">IncrementToken()</h4> |
| <div class="markdown level1 summary"><p>Consumers (i.e., <a class="xref" href="Lucene.Net.Index.IndexWriter.html">IndexWriter</a>) use this method to advance the stream to |
| the next token. Implementing classes must implement this method and update |
| the appropriate <a class="xref" href="Lucene.Net.Util.IAttribute.html">IAttribute</a>s with the attributes of the next |
| token. |
| <p> |
| The producer must make no assumptions about the attributes after the method |
| has been returned: the caller may arbitrarily change it. If the producer |
| needs to preserve the state for subsequent calls, it can use |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_CaptureState">CaptureState()</a> to create a copy of the current attribute state. |
| <p> |
| this method is called for every token of a document, so an efficient |
| implementation is crucial for good performance. To avoid calls to |
| <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_AddAttribute__1">AddAttribute<T>()</a> and <a class="xref" href="Lucene.Net.Util.AttributeSource.html#Lucene_Net_Util_AttributeSource_GetAttribute__1">GetAttribute<T>()</a>, |
| references to all <a class="xref" href="Lucene.Net.Util.IAttribute.html">IAttribute</a>s that this stream uses should be |
| retrieved during instantiation. |
| <p> |
| To ensure that filters and consumers know which attributes are available, |
| the attributes must be added during instantiation. Filters and consumers |
| are not required to check for availability of attributes in |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a>.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public abstract bool IncrementToken()</code></pre> |
| </div> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.Boolean</span></td> |
| <td><p>false for end of stream; true otherwise </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream_Reset.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream.Reset%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L176">View Source</a> |
| </span> |
| <a id="Lucene_Net_Analysis_TokenStream_Reset_" data-uid="Lucene.Net.Analysis.TokenStream.Reset*"></a> |
| <h4 id="Lucene_Net_Analysis_TokenStream_Reset" data-uid="Lucene.Net.Analysis.TokenStream.Reset">Reset()</h4> |
| <div class="markdown level1 summary"><p>This method is called by a consumer before it begins consumption using |
| <a class="xref" href="Lucene.Net.Analysis.TokenStream.html#Lucene_Net_Analysis_TokenStream_IncrementToken">IncrementToken()</a>. |
| <p> |
| Resets this stream to a clean state. Stateful implementations must implement |
| this method so that they can be reused, just as if they had been created fresh. |
| <p> |
| If you override this method, always call <code>base.Reset()</code>, otherwise |
| some internal state will not be correctly reset (e.g., <a class="xref" href="Lucene.Net.Analysis.Tokenizer.html">Tokenizer</a> will |
| throw <span class="xref">System.InvalidOperationException</span> on further usage).</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public virtual void Reset()</code></pre> |
| </div> |
| <h3 id="implements">Implements</h3> |
| <div> |
| <span class="xref">System.IDisposable</span> |
| </div> |
| </article> |
| </div> |
| |
| <div class="hidden-sm col-md-2" role="complementary"> |
| <div class="sideaffix"> |
| <div class="contribution"> |
| <ul class="nav"> |
| <li> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Analysis_TokenStream.md&value=---%0Auid%3A%20Lucene.Net.Analysis.TokenStream%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A" class="contribution-link">Improve this Doc</a> |
| </li> |
| <li> |
| <a href="https://github.com/apache/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net/Analysis/TokenStream.cs/#L79" class="contribution-link">View Source</a> |
| </li> |
| </ul> |
| </div> |
| <nav class="bs-docs-sidebar hidden-print hidden-xs hidden-sm affix" id="affix"> |
| <!-- <p><a class="back-to-top" href="#top">Back to top</a><p> --> |
| </nav> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| <footer> |
| <div class="grad-bottom"></div> |
| <div class="footer"> |
| <div class="container"> |
| <span class="pull-right"> |
| <a href="#top">Back to top</a> |
| </span> |
| Copyright © 2020 Licensed to the Apache Software Foundation (ASF) |
| |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.js"></script> |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.js"></script> |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.js"></script> |
| </body> |
| </html> |
