| <!DOCTYPE html> |
| <!--[if IE]><![endif]--> |
| <html> |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> |
| <title>Class TokenSources |
| | Apache Lucene.NET 4.8.0-beta00010 Documentation </title> |
| <meta name="viewport" content="width=device-width"> |
| <meta name="title" content="Class TokenSources |
| | Apache Lucene.NET 4.8.0-beta00010 Documentation "> |
| <meta name="generator" content="docfx 2.56.0.0"> |
| |
| <link rel="shortcut icon" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/favicon.ico"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.css"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.css"> |
| <link rel="stylesheet" href="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.css"> |
| <meta property="docfx:navrel" content="toc.html"> |
| <meta property="docfx:tocrel" content="highlighter/toc.html"> |
| |
| <meta property="docfx:rel" content="https://lucenenet.apache.org/docs/4.8.0-beta00009/"> |
| |
| </head> |
| <body data-spy="scroll" data-target="#affix" data-offset="120"> |
| <div id="wrapper"> |
| <header> |
| |
| <nav id="autocollapse" class="navbar ng-scope" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| |
| <a class="navbar-brand" href="/"> |
| <img id="logo" class="svg" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/logo/lucene-net-color.png" alt=""> |
| </a> |
| </div> |
| <div class="collapse navbar-collapse" id="navbar"> |
| <form class="navbar-form navbar-right" role="search" id="search"> |
| <div class="form-group"> |
| <input type="text" class="form-control" id="search-query" placeholder="Search" autocomplete="off"> |
| </div> |
| </form> |
| </div> |
| </div> |
| </nav> |
| |
| <div class="subnav navbar navbar-default"> |
| <div class="container hide-when-search"> |
| <ul class="level0 breadcrumb"> |
| <li> |
| <a href="https://lucenenet.apache.org/docs/4.8.0-beta00009/">API</a> |
| <span id="breadcrumb"> |
| <ul class="breadcrumb"> |
| <li></li> |
| </ul> |
| </span> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </header> |
| <div class="container body-content"> |
| |
| <div id="search-results"> |
| <div class="search-list"></div> |
| <div class="sr-items"> |
| <p><i class="glyphicon glyphicon-refresh index-loading"></i></p> |
| </div> |
| <ul id="pagination"></ul> |
| </div> |
| </div> |
| <div role="main" class="container body-content hide-when-search"> |
| |
| <div class="sidenav hide-when-search"> |
| <a class="btn toc-toggle collapse" data-toggle="collapse" href="#sidetoggle" aria-expanded="false" aria-controls="sidetoggle">Show / Hide Table of Contents</a> |
| <div class="sidetoggle collapse" id="sidetoggle"> |
| <div id="sidetoc"></div> |
| </div> |
| </div> |
| <div class="article row grid-right"> |
| <div class="col-md-10"> |
| <article class="content wrap" id="_content" data-uid="Lucene.Net.Search.Highlight.TokenSources"> |
| |
| |
| <h1 id="Lucene_Net_Search_Highlight_TokenSources" data-uid="Lucene.Net.Search.Highlight.TokenSources" class="text-break">Class TokenSources |
| </h1> |
| <div class="markdown level0 summary"><p>Hides implementation issues associated with obtaining a <span class="xref">Lucene.Net.Analysis.TokenStream</span> for use with |
| the <a class="xref" href="Lucene.Net.Search.Highlight.Highlighter.html">Highlighter</a> - can obtain from |
| term vectors with offsets and positions or from an Analyzer re-parsing the stored content. |
| see TokenStreamFromTermVector</p> |
| </div> |
| <div class="markdown level0 conceptual"></div> |
| <div class="inheritance"> |
| <h5>Inheritance</h5> |
| <div class="level0"><span class="xref">System.Object</span></div> |
| <div class="level1"><span class="xref">TokenSources</span></div> |
| </div> |
| <div class="inheritedMembers"> |
| <h5>Inherited Members</h5> |
| <div> |
| <span class="xref">System.Object.Equals(System.Object)</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.Equals(System.Object, System.Object)</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.GetHashCode()</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.GetType()</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.MemberwiseClone()</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.ReferenceEquals(System.Object, System.Object)</span> |
| </div> |
| <div> |
| <span class="xref">System.Object.ToString()</span> |
| </div> |
| </div> |
| <h6><strong>Namespace</strong>: <a class="xref" href="Lucene.Net.Search.Highlight.html">Lucene.Net.Search.Highlight</a></h6> |
| <h6><strong>Assembly</strong>: Lucene.Net.Highlighter.dll</h6> |
| <h5 id="Lucene_Net_Search_Highlight_TokenSources_syntax">Syntax</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public class TokenSources</code></pre> |
| </div> |
| <h3 id="methods">Methods |
| </h3> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Analysis_Analyzer_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream(Lucene.Net.Index.IndexReader%2CSystem.Int32%2CSystem.String%2CLucene.Net.Analysis.Analyzer)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L129">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Analysis_Analyzer_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream(Lucene.Net.Index.IndexReader,System.Int32,System.String,Lucene.Net.Analysis.Analyzer)">GetAnyTokenStream(IndexReader, Int32, String, Analyzer)</h4> |
| <div class="markdown level1 summary"><p>A convenience method that tries a number of approaches to getting a token stream. |
| The cost of finding there are no termVectors in the index is minimal (1000 invocations still |
| registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetAnyTokenStream(IndexReader reader, int docId, string field, Analyzer analyzer)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.IndexReader</span></td> |
| <td><span class="parametername">reader</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.Int32</span></td> |
| <td><span class="parametername">docId</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.Analyzer</span></td> |
| <td><span class="parametername">analyzer</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td><p>null if field not stored correctly</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="exceptions">Exceptions</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Condition</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.IO.IOException</span></td> |
| <td><p>If there is a low-level I/O error</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Documents_Document_Lucene_Net_Analysis_Analyzer_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream(Lucene.Net.Index.IndexReader%2CSystem.Int32%2CSystem.String%2CLucene.Net.Documents.Document%2CLucene.Net.Analysis.Analyzer)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L105">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetAnyTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Documents_Document_Lucene_Net_Analysis_Analyzer_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetAnyTokenStream(Lucene.Net.Index.IndexReader,System.Int32,System.String,Lucene.Net.Documents.Document,Lucene.Net.Analysis.Analyzer)">GetAnyTokenStream(IndexReader, Int32, String, Document, Analyzer)</h4> |
| <div class="markdown level1 summary"><p>A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to |
| using the passed in <span class="xref">Lucene.Net.Documents.Document</span> to retrieve the <span class="xref">Lucene.Net.Analysis.TokenStream</span>. This is useful when |
| you already have the document, but would prefer to use the vector first.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetAnyTokenStream(IndexReader reader, int docId, string field, Document doc, Analyzer analyzer)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.IndexReader</span></td> |
| <td><span class="parametername">reader</span></td> |
| <td><p>The <span class="xref">Lucene.Net.Index.IndexReader</span> to use to try and get the vector from</p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.Int32</span></td> |
| <td><span class="parametername">docId</span></td> |
| <td><p>The docId to retrieve.</p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td><p>The field to retrieve on the document</p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Documents.Document</span></td> |
| <td><span class="parametername">doc</span></td> |
| <td><p>The document to fall back on</p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.Analyzer</span></td> |
| <td><span class="parametername">analyzer</span></td> |
| <td><p>The analyzer to use for creating the TokenStream if the vector doesn't exist</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td><p>The <span class="xref">Lucene.Net.Analysis.TokenStream</span> for the <a class="xref" href="http://localhost:8080/api/core/Lucene.Net.Index.IIndexableField.html">IIndexableField</a> on the <span class="xref">Lucene.Net.Documents.Document</span></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="exceptions">Exceptions</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Condition</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.IO.IOException</span></td> |
| <td><p>if there was an error loading</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Documents_Document_System_String_Lucene_Net_Analysis_Analyzer_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Documents.Document%2CSystem.String%2CLucene.Net.Analysis.Analyzer)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L325">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Documents_Document_System_String_Lucene_Net_Analysis_Analyzer_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Documents.Document,System.String,Lucene.Net.Analysis.Analyzer)">GetTokenStream(Document, String, Analyzer)</h4> |
| <div class="markdown level1 summary"></div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStream(Document doc, string field, Analyzer analyzer)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Documents.Document</span></td> |
| <td><span class="parametername">doc</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.Analyzer</span></td> |
| <td><span class="parametername">analyzer</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Analysis_Analyzer_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.IndexReader%2CSystem.Int32%2CSystem.String%2CLucene.Net.Analysis.Analyzer)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L318">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_IndexReader_System_Int32_System_String_Lucene_Net_Analysis_Analyzer_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.IndexReader,System.Int32,System.String,Lucene.Net.Analysis.Analyzer)">GetTokenStream(IndexReader, Int32, String, Analyzer)</h4> |
| <div class="markdown level1 summary"></div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStream(IndexReader reader, int docId, string field, Analyzer analyzer)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.IndexReader</span></td> |
| <td><span class="parametername">reader</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.Int32</span></td> |
| <td><span class="parametername">docId</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.Analyzer</span></td> |
| <td><span class="parametername">analyzer</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_Terms_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.Terms)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L145">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_Terms_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.Terms)">GetTokenStream(Terms)</h4> |
| <div class="markdown level1 summary"></div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStream(Terms vector)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.Terms</span></td> |
| <td><span class="parametername">vector</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_Terms_System_Boolean_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.Terms%2CSystem.Boolean)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L193">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_Terms_System_Boolean_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(Lucene.Net.Index.Terms,System.Boolean)">GetTokenStream(Terms, Boolean)</h4> |
| <div class="markdown level1 summary"><p>Low level api. Returns a token stream generated from a <span class="xref">Lucene.Net.Index.Terms</span>. This |
| can be used to feed the highlighter with a pre-parsed token |
| stream. The <span class="xref">Lucene.Net.Index.Terms</span> must have offsets available. |
| <p> |
| In my tests the speeds to recreate 1000 token streams using this method are: |
| <ul><li> |
| with TermVector offset only data stored - 420 milliseconds |
| </li><li> |
| with TermVector offset AND position data stored - 271 milliseconds |
| (nb timings for TermVector with position data are based on a tokenizer with contiguous |
| positions - no overlaps or gaps) |
| </li><li> |
| The cost of not using TermPositionVector to store |
| pre-parsed content and using an analyzer to re-parse the original content:<p> |
| <ul> |
| <li>reanalyzing the original content - 980 milliseconds |
| </li></ul></li> |
| </ul> |
| <p>The re-analyze timings will typically vary depending on - |
| <ol><li> |
| The complexity of the analyzer code (timings above were using a |
| stemmer/lowercaser/stopword combo) |
| </li><li> |
| The number of other fields (Lucene reads ALL fields off the disk |
| when accessing just one document field - can cost dear!) |
| </li><li> |
| Use of compression on field storage - could be faster due to compression (less disk IO) |
| or slower (more CPU burn) depending on the content. |
| </li></ol></p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStream(Terms tpv, bool tokenPositionsGuaranteedContiguous)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.Terms</span></td> |
| <td><span class="parametername">tpv</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.Boolean</span></td> |
| <td><span class="parametername">tokenPositionsGuaranteedContiguous</span></td> |
| <td><p>true if the token position numbers have no overlaps or gaps. If looking |
| to eek out the last drops of performance, set to true. If in doubt, set to false.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="exceptions">Exceptions</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Condition</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.ArgumentException</span></td> |
| <td><p>if no offsets are available</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_System_String_System_String_Lucene_Net_Analysis_Analyzer_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(System.String%2CSystem.String%2CLucene.Net.Analysis.Analyzer)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L338">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_System_String_System_String_Lucene_Net_Analysis_Analyzer_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStream(System.String,System.String,Lucene.Net.Analysis.Analyzer)">GetTokenStream(String, String, Analyzer)</h4> |
| <div class="markdown level1 summary"></div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStream(string field, string contents, Analyzer analyzer)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">contents</span></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.Analyzer</span></td> |
| <td><span class="parametername">analyzer</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td></td> |
| </tr> |
| </tbody> |
| </table> |
| <span class="small pull-right mobile-hide"> |
| <span class="divider">|</span> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources_GetTokenStreamWithOffsets_Lucene_Net_Index_IndexReader_System_Int32_System_String_.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources.GetTokenStreamWithOffsets(Lucene.Net.Index.IndexReader%2CSystem.Int32%2CSystem.String)%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A">Improve this Doc</a> |
| </span> |
| <span class="small pull-right mobile-hide"> |
| <a href="https://github.com/NightOwl888/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L298">View Source</a> |
| </span> |
| <a id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStreamWithOffsets_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStreamWithOffsets*"></a> |
| <h4 id="Lucene_Net_Search_Highlight_TokenSources_GetTokenStreamWithOffsets_Lucene_Net_Index_IndexReader_System_Int32_System_String_" data-uid="Lucene.Net.Search.Highlight.TokenSources.GetTokenStreamWithOffsets(Lucene.Net.Index.IndexReader,System.Int32,System.String)">GetTokenStreamWithOffsets(IndexReader, Int32, String)</h4> |
| <div class="markdown level1 summary"><p>Returns a <span class="xref">Lucene.Net.Analysis.TokenStream</span> with positions and offsets constructed from |
| field termvectors. If the field has no termvectors or offsets |
| are not included in the termvector, return null. See |
| <a class="xref" href="Lucene.Net.Search.Highlight.TokenSources.html#Lucene_Net_Search_Highlight_TokenSources_GetTokenStream_Lucene_Net_Index_Terms_">GetTokenStream(Terms)</a> |
| for an explanation of what happens when positions aren't present.</p> |
| </div> |
| <div class="markdown level1 conceptual"></div> |
| <h5 class="decalaration">Declaration</h5> |
| <div class="codewrapper"> |
| <pre><code class="lang-csharp hljs">public static TokenStream GetTokenStreamWithOffsets(IndexReader reader, int docId, string field)</code></pre> |
| </div> |
| <h5 class="parameters">Parameters</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Name</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Index.IndexReader</span></td> |
| <td><span class="parametername">reader</span></td> |
| <td><p>the <span class="xref">Lucene.Net.Index.IndexReader</span> to retrieve term vectors from</p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.Int32</span></td> |
| <td><span class="parametername">docId</span></td> |
| <td><p>the document to retrieve term vectors for </p> |
| </td> |
| </tr> |
| <tr> |
| <td><span class="xref">System.String</span></td> |
| <td><span class="parametername">field</span></td> |
| <td><p>the field to retrieve term vectors for</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="returns">Returns</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">Lucene.Net.Analysis.TokenStream</span></td> |
| <td><p>a <span class="xref">Lucene.Net.Analysis.TokenStream</span>, or null if offsets are not available</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h5 class="exceptions">Exceptions</h5> |
| <table class="table table-bordered table-striped table-condensed"> |
| <thead> |
| <tr> |
| <th>Type</th> |
| <th>Condition</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="xref">System.IO.IOException</span></td> |
| <td><p>If there is a low-level I/O error</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </article> |
| </div> |
| |
| <div class="hidden-sm col-md-2" role="complementary"> |
| <div class="sideaffix"> |
| <div class="contribution"> |
| <ul class="nav"> |
| <li> |
| <a href="https://github.com/apache/lucenenet/new/docs/4.8.0-beta00010/websites/apidocs/apiSpec/new?filename=Lucene_Net_Search_Highlight_TokenSources.md&value=---%0Auid%3A%20Lucene.Net.Search.Highlight.TokenSources%0Asummary%3A%20'*You%20can%20override%20summary%20for%20the%20API%20here%20using%20*MARKDOWN*%20syntax'%0A---%0A%0A*Please%20type%20below%20more%20information%20about%20this%20API%3A*%0A%0A" class="contribution-link">Improve this Doc</a> |
| </li> |
| <li> |
| <a href="https://github.com/apache/lucenenet/blob/release/Lucene.Net_4_8_0_beta00010/src/Lucene.Net.Highlighter/Highlight/TokenSources.cs/#L35" class="contribution-link">View Source</a> |
| </li> |
| </ul> |
| </div> |
| <nav class="bs-docs-sidebar hidden-print hidden-xs hidden-sm affix" id="affix"> |
| <!-- <p><a class="back-to-top" href="#top">Back to top</a><p> --> |
| </nav> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| <footer> |
| <div class="grad-bottom"></div> |
| <div class="footer"> |
| <div class="container"> |
| <span class="pull-right"> |
| <a href="#top">Back to top</a> |
| </span> |
| Copyright © 2020 Licensed to the Apache Software Foundation (ASF) |
| |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.vendor.js"></script> |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/docfx.js"></script> |
| <script type="text/javascript" src="https://lucenenet.apache.org/docs/4.8.0-beta00009/styles/main.js"></script> |
| </body> |
| </html> |
