= Highlighting
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

Highlighting in Solr allows fragments of documents that match the user's query to be included with the query response.

The fragments are included in a special section of the query response (the `highlighting` section), and the client uses the formatting clues also included to determine how to present the snippets to users. Fragments are a portion of a document field that contains matches from the query and are sometimes also referred to as "snippets" or "passages".

Highlighting is extremely configurable, perhaps more than any other part of Solr. There are many parameters each for fragment sizing, formatting, ordering, backup/alternate behavior, and more options that are hard to categorize. Nonetheless, highlighting is very simple to use.

== Usage

=== Common Highlighter Parameters
You only need to set the `hl` and often `hl.fl` parameters to get results. The following table documents these and some other supported parameters. Note that many highlighting parameters support per-field overrides, such as: `f._title_txt_.hl.snippets`

`hl`::
Use this parameter to enable or disable highlighting. The default is `false`. If you want to use highlighting, you must set this to `true`.

`hl.method`::
The highlighting implementation to use. Acceptable values are: `unified`, `original`, `fastVector`. The default is `original`.
+
See the <<Choosing a Highlighter>> section below for more details on the differences between the available highlighters.

`hl.fl`::
Specifies a list of fields to highlight, either comma- or space-delimited.  These must be "stored".
A wildcard of `\*` (asterisk) can be used to match field globs, such as `text_*` or even `\*` to highlight on all fields where highlighting is possible.
When using `*`, consider adding `hl.requireFieldMatch=true`.
+
Note that the field(s) listed here ought to have compatible text-analysis (defined in the schema) with field(s) referenced in the query to be highlighted.
It may be necessary to modify `hl.q` and `hl.qparser` and/or modify the text analysis.
The following example uses the <<local-parameters-in-queries.adoc#local-parameters-in-queries,local-params>> syntax and <<the-extended-dismax-query-parser.adoc#the-extended-dismax-query-parser,the edismax parser>> to highlight fields in `hl.fl`:
`&hl.fl=field1 field2&hl.q={!edismax qf=$hl.fl v=$q}&hl.qparser=lucene&hl.requireFieldMatch=true` (along with other applicable parameters, of course).
+
The default is the value of the `df` parameter which in turn has no default.

`hl.q`::
A query to use for highlighting.
This parameter allows you to highlight different terms or fields than those being used to search for documents.
When setting this, you might also need to set `hl.qparser`.
+
The default is the value of the `q` parameter (already parsed).

`hl.qparser`::
The <<query-syntax-and-parsing.adoc#query-syntax-and-parsing,query parser>> to use for the `hl.q` query.  It only applies when `hl.q` is set.
+
The default is the value of the `defType` parameter which in turn defaults to `lucene`.

`hl.requireFieldMatch`::
By default, `false`, all query terms will be highlighted for each field to be highlighted (`hl.fl`) no matter what fields the parsed query refer to. If set to `true`, only query terms aligning with the field being highlighted will in turn be highlighted.
+
If the query references fields different from the field being highlighted and they have different text analysis, the query may not highlight query terms it should have and vice versa. The analysis used is that of the field being highlighted (`hl.fl`), not the query fields.

`hl.usePhraseHighlighter`::
If set to `true`, the default, Solr will highlight phrase queries (and other advanced position-sensitive queries) accurately – as phrases. If `false`, the parts of the phrase will be highlighted everywhere instead of only when it forms the given phrase.

`hl.highlightMultiTerm`::
If set to `true`, the default, Solr will highlight wildcard queries (and other `MultiTermQuery` subclasses). If `false`, they won't be highlighted at all.

`hl.snippets`::
Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. The default is `1`.

`hl.fragsize`::
Specifies the approximate size, in characters, of fragments to consider for highlighting. The default is `100`. Using `0` indicates that no fragmenting should be considered and the whole field value should be used.

`hl.tag.pre`::
(`hl.simple.pre` for the Original Highlighter) Specifies the “tag” to use before a highlighted term. This can be any string, but is most often an HTML or XML tag.
+
The default is `<em>`.

`hl.tag.post`::
(`hl.simple.post` for the Original Highlighter) Specifies the “tag” to use after a highlighted term. This can be any string, but is most often an HTML or XML tag.
+
The default is `</em>`.

`hl.encoder`::
If blank, the default, then the stored text will be returned without any escaping/encoding performed by the highlighter. If set to `html` then special HTML/XML characters will be encoded (e.g., `&` becomes `\&amp;`). The pre/post snippet characters are never encoded.

`hl.maxAnalyzedChars`::
The character limit to look for highlights, after which no highlighting will be done. This is mostly only a performance concern for an _analysis_ based offset source since it's the slowest. See <<Schema Options and Performance Considerations>>.
+
The default is `51200` characters.

There are more parameters supported as well depending on the highlighter (via `hl.method`) chosen.

=== Highlighting in the Query Response

In the response to a query, Solr includes highlighting data in a section separate from the documents. It is up to a client to determine how to process this response and display the highlights to users.

Using the example documents included with Solr, we can see how this might work:

In response to a query such as:

[source,text]
http://localhost:8983/solr/gettingstarted/select?hl=on&q=apple&hl.fl=manu&fl=id,name,manu,cat

we get a response such as this (truncated slightly for space):

[source,json]
----
{
  "response": {
    "numFound": 1,
    "start": 0,
    "docs": [{
      "id": "MA147LL/A",
      "name": "Apple 60 GB iPod with Video Playback Black",
      "manu": "Apple Computer Inc.",
      "cat": [
        "electronics",
        "music"
      ]
    }]
  },
  "highlighting": {
    "MA147LL/A": {
      "manu": [
        "<em>Apple</em> Computer Inc."
      ]
    }
  }
}
----

Note the two sections `docs` and `highlighting`. The `docs` section contains the fields of the document requested with the `fl` parameter of the query (only "id", "name", "manu", and "cat").

The `highlighting` section includes the ID of each document, and the field that contains the highlighted portion. In this example, we used the `hl.fl` parameter to say we wanted query terms highlighted in the "manu" field. When there is a match to the query term in that field, it will be included for each document ID in the list.

== Choosing a Highlighter

Solr provides a `HighlightComponent` (a <<requesthandlers-and-searchcomponents-in-solrconfig.adoc#search-components,`SearchComponent`>>) and it's in the default list of components for search handlers. It offers a somewhat unified API over multiple actual highlighting implementations (or simply "highlighters") that do the business of highlighting.

There are many parameters supported by more than one highlighter, and sometimes the implementation details and semantics will be a bit different, so don't expect identical results when switching highlighters. You should use the `hl.method` parameter to choose a highlighter but it's also possible to explicitly configure an implementation by class name in `solrconfig.xml`.

There are four highlighters available that can be chosen at runtime with the `hl.method` parameter, in order of general recommendation:


<<The Unified Highlighter,Unified Highlighter>>:: (`hl.method=unified`)
+
The Unified Highlighter is the newest highlighter (as of Solr 6.4), which stands out as the most performant and accurate of the options.
It can handle typical requirements and others possibly via plugins/extension.
We recommend that you try this highlighter even though it isn't the default (yet).
+
The UH highlights a query very _accurately_ and thus is true to what the underlying Lucene query actually matches.
Other highlighters highlight terms more liberally (over-highlight).
For esoteric/custom queries, this highlighter has a greater likelihood of supporting it than the others.
+
A strong benefit to this highlighter is that you can opt to configure Solr to put more information in the underlying index to speed up highlighting of large documents; multiple configurations are supported, even on a per-field basis.
There is little or no such flexibility of offset sources for the other highlighters.
More on this below.
+
There are some reasons not to choose this highlighter:
Passage scoring does not consider boosts in the query.
Some users want more/better passage breaking flexibility.
The "alternate" fallback options are more primitive.

<<The Original Highlighter,Original Highlighter>>:: (`hl.method=original`, the default)
+
The Original Highlighter, sometimes called the "Standard Highlighter" or "Default Highlighter", is Lucene's original highlighter – a venerable option with a high degree of customization options.
Its query accuracy is good enough for most needs, although it's not quite as good/perfect as the Unified Highlighter.
+
The Original Highlighter will normally analyze stored text on the fly in order to highlight. It will use full term vectors if available.
If the text isn't "stored" but is in doc values (`docValues="true"`), this highlighter can work with it.
+
Where this highlighter falls short is performance; it's often twice as slow as the Unified Highlighter. And despite being the most customizable, it doesn't have a BreakIterator based fragmenter (all the others do), which could pose a challenge for some languages.


<<The FastVector Highlighter,FastVector Highlighter>>:: (`hl.method=fastVector`)
+
The FastVector Highlighter _requires_ full term vector options (`termVectors`, `termPositions`, and `termOffsets`) on the field, and is optimized with that in mind. It is nearly as configurable as the Original Highlighter with some variability.
+
This highlighter notably supports multi-colored highlighting such that different query words can be denoted in the fragment with different marking, usually expressed as an HTML tag with a unique color.
+
This highlighter's query-representation is less advanced than the Original or Unified Highlighters: for example it will not work well with the `surround` parser, and there are multiple reported bugs pertaining to queries with stop-words.

Both the FastVector and Original Highlighters can be used in conjunction in a search request to highlight some fields with one and some the other. In contrast, the Unified Highlighter can only be chosen exclusively.


The Unified Highlighter is exclusively configured via search parameters. In contrast, some settings for the Original and FastVector Highlighters are set in `solrconfig.xml`. There's a robust example of the latter in the "```techproducts```" configset.

In addition to further information below, more information can be found in the {solr-javadocs}/solr-core/org/apache/solr/highlight/package-summary.html[Solr javadocs].

=== Schema Options and Performance Considerations

Fundamental to the internals of highlighting are detecting the _offsets_ of the individual words that match the query. Some of the highlighters can run the stored text through the analysis chain defined in the schema, some can look them up from _postings_, and some can look them up from _term vectors._ These choices have different trade-offs:

* *Analysis*: Supported by the Unified and Original Highlighters. If you don't go out of your way to configure the other options below, the highlighter will analyze the stored text on the fly (during highlighting) to calculate offsets.
+
The benefit of this approach is that your index won't grow larger with any extra data that isn't strictly necessary for highlighting.
+
The down side is that highlighting speed is roughly linear with the amount of text to process, with a large factor being the complexity of your analysis chain.
+
For "short" text, this is a good choice. Or maybe it's not short but you're prioritizing a smaller index and indexing speed over highlighting performance.
* *Postings*: Supported by the Unified Highlighter. Set `storeOffsetsWithPositions` to `true`. This adds a moderate amount of extra data to the index but it speeds up highlighting tremendously, especially compared to analysis with longer text fields.
+
However, wildcard queries will fall back to analysis unless "light" term vectors are added.

** *with Term Vectors (light)*: Supported only by the Unified Highlighter. To enable this mode set `termVectors` to `true` but no other term vector related options on the field being highlighted.
+
This adds even more data to the index than just `storeOffsetsWithPositions` but not as much as enabling all the extra term vector options. Term Vectors are only accessed by the highlighter when a wildcard query is used and will prevent a fall back to analysis of the stored text.
+
This is definitely the fastest option for highlighting wildcard queries on large text fields.
* *Term Vectors (full)*: Supported by the Unified, FastVector, and Original Highlighters. Set `termVectors`, `termPositions`, and `termOffsets` to `true`, and potentially `termPayloads` for advanced use cases.
+
This adds substantial weight to the index – similar in size to the compressed stored text. If you are using the Unified Highlighter then this is not a recommended configuration since it's slower and heavier than postings with light term vectors. However, this could make sense if full term vectors are already needed for another use-case.

== The Unified Highlighter

The Unified Highlighter supports these following additional parameters to the ones listed earlier:

`hl.offsetSource`::
By default, the Unified Highlighter will usually pick the right offset source (see above). However it may be ambiguous such as during a migration from one offset source to another that hasn't completed.
+
The offset source can be explicitly configured to one of: `ANALYSIS`, `POSTINGS`, `POSTINGS_WITH_TERM_VECTORS`, or `TERM_VECTORS`.

`hl.fragAlignRatio`::
This parameter influences where the first match (i.e. highlighted text) in a passage is positioned.
The default value of `0.5` means to align the match to the middle.
A value of `0.0` means to align the match to the left, while `1.0` to align it to the right.
This setting is a best-effort hint, as there are a variety of factors.
When there's lots of text to be highlighted, lowering this number can help performance a lot.

`hl.fragsizeIsMinimum`::
When `true` (the default), the `hl.fragsize` parameter is treated as a (soft) minimum fragment size;
provided there is enough text, the fragment is at least this size.
When `false`, it's an optimal target -- the highlighter will _on average_ produce highlights of this length.
A `false` setting is slower, particularly when there's lots of text and `hl.bs.type=SENTENCE`.

`hl.tag.ellipsis`::
By default, each snippet is returned as a separate value (as is done with the other highlighters). Set this parameter to instead return one string with this text as the delimiter. _Note: this is likely to be removed in the future._

`hl.defaultSummary`::
If `true`, use the leading portion of the text as a snippet if a proper highlighted snippet can't otherwise be generated. The default is `false`.

`hl.score.k1`::
Specifies BM25 term frequency normalization parameter 'k1'. For example, it can be set to `0` to rank passages solely based on the number of query terms that match. The default is `1.2`.

`hl.score.b`::
Specifies BM25 length normalization parameter 'b'. For example, it can be set to "0" to ignore the length of passages entirely when ranking. The default is `0.75`.

`hl.score.pivot`::
Specifies BM25 average passage length in characters. The default is `87`.

`hl.bs.language`::
Specifies the breakiterator language for dividing the document into passages.

`hl.bs.country`::
Specifies the breakiterator country for dividing the document into passages.

`hl.bs.variant`::
Specifies the breakiterator variant for dividing the document into passages.

`hl.bs.type`::
Specifies the breakiterator type for dividing the document into passages. Can be `SEPARATOR`, `SENTENCE`, `WORD`*, `CHARACTER`, `LINE`, or `WHOLE`. `SEPARATOR` is special value that splits text on a user-provided character in `hl.bs.separator`.
+
The default is `SENTENCE`.

`hl.bs.separator`::
Indicates which character to break the text on. Use only if you have defined `hl.bs.type=SEPARATOR`.
+
This is useful when the text has already been manipulated in advance to have a special delineation character at desired highlight passage boundaries. This character will still appear in the text as the last character of a passage.

`hl.weightMatches`::
Tells the UH to use Lucene's new "Weight Matches" API instead of doing SpanQuery conversion.
This is the most accurate highlighting mode reflecting the query.
Furthermore, phrases will be highlighted as a whole instead of word by word.
+
The default is `true`.
However if either `hl.usePhraseHighlighter` or `hl.multiTermQuery` are set to false, then this setting is effectively false no matter what you set it to.

== The Original Highlighter

The Original Highlighter supports these following additional parameters to the ones listed earlier:

`hl.mergeContiguous`::
Instructs Solr to collapse contiguous fragments into a single fragment. A value of `true` indicates contiguous fragments will be collapsed into single fragment. The default value, `false`, is also the backward-compatible setting.

`hl.maxMultiValuedToExamine`::
Specifies the maximum number of entries in a multi-valued field to examine before stopping. This can potentially return zero results if the limit is reached before any matches are found.
+
If used with the `maxMultiValuedToMatch`, whichever limit is reached first will determine when to stop looking.
+
The default is `Integer.MAX_VALUE`.

`hl.maxMultiValuedToMatch`::
Specifies the maximum number of matches in a multi-valued field that are found before stopping.
+
If `hl.maxMultiValuedToExamine` is also defined, whichever limit is reached first will determine when to stop looking.
+
The default is `Integer.MAX_VALUE`.

`hl.alternateField`::
Specifies a field to be used as a backup default summary if Solr cannot generate a snippet (i.e., because no terms match).

`hl.maxAlternateFieldLength`::
Specifies the maximum number of characters of the field to return. Any value less than or equal to `0` means the field's length is unlimited (the default behavior).
+
This parameter is only used in conjunction with the `hl.alternateField` parameter.

`hl.highlightAlternate`::
If set to `true`, the default, and `hl.alternateFieldName` is active, Solr will show the entire alternate field, with highlighting of occurrences. If `hl.maxAlternateFieldLength=N` is used, Solr returns max `N` characters surrounding the best matching fragment.
+
If set to `false`, or if there is no match in the alternate field either, the alternate field will be shown without highlighting.

`hl.formatter`::
Selects a formatter for the highlighted output. Currently the only legal value is `simple`, which surrounds a highlighted term with a customizable pre- and post-text snippet.

`hl.simple.pre`, `hl.simple.post`::
Specifies the text that should appear before (`hl.simple.pre`) and after (`hl.simple.post`) a highlighted term, when using the `simple` formatter. The default is `<em>` and `</em>`.

`hl.fragmenter`::
Specifies a text snippet generator for highlighted text. The standard (default) fragmenter is `gap`, which creates fixed-sized fragments with gaps for multi-valued fields.
+
Another option is `regex`, which tries to create fragments that resemble a specified regular expression.

`hl.regex.slop`::
When using the regex fragmenter (`hl.fragmenter=regex`), this parameter defines the factor by which the fragmenter can stray from the ideal fragment size (given by `hl.fragsize`) to accommodate a regular expression.
+
For instance, a slop of `0.2` with `hl.fragsize=100` should yield fragments between 80 and 120 characters in length. It is usually good to provide a slightly smaller `hl.fragsize` value when using the regex fragmenter.
+
The default is `0.6`.

`hl.regex.pattern`::
Specifies the regular expression for fragmenting. This could be used to extract sentences.

`hl.regex.maxAnalyzedChars`::
Instructs Solr to analyze only this many characters from a field when using the regex fragmenter (after which, the fragmenter produces fixed-sized fragments). The default is `10000`.
+
Note, applying a complicated regex to a huge field is computationally expensive.

`hl.preserveMulti`::
If `true`, multi-valued fields will return all values in the order they were saved in the index. If `false`, the default, only values that match the highlight request will be returned.

`hl.payloads`::
When `hl.usePhraseHighlighter` is `true` and the indexed field has payloads but not term vectors (generally quite rare), the index's payloads will be read into the highlighter's memory index along with the postings.
+
If this may happen and you know you don't need them for highlighting (i.e., your queries don't filter by payload) then you can save a little memory by setting this to false.

The Original Highlighter has a plugin architecture that enables new functionality to be registered in `solrconfig.xml`. The "```techproducts```" configset shows most of these settings explicitly. You can use it as a guide to provide your own components to include a `SolrFormatter`, `SolrEncoder`, and `SolrFragmenter.`

== The FastVector Highlighter

The FastVector Highlighter (FVH) can be used in conjunction with the Original Highlighter if not all fields should be highlighted with the FVH. In such a mode, set `hl.method=original` and `f.yourTermVecField.hl.method=fastVector` for all fields that should use the FVH. One annoyance to keep in mind is that the Original Highlighter uses `hl.simple.pre` whereas the FVH (and other highlighters) use `hl.tag.pre`.

In addition to the initial listed parameters, the following parameters documented for the Original Highlighter above are also supported by the FVH:

* `hl.alternateField`
* `hl.maxAlternateFieldLength`
* `hl.highlightAlternate`

And here are additional parameters supported by the FVH:

`hl.fragListBuilder`::
The snippet fragmenting algorithm. The `weighted` fragListBuilder uses IDF-weights to order fragments. This fragListBuilder is the default.
+
Other options are `single`, which returns the entire field contents as one snippet, or `simple`. You can select a fragListBuilder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".

`hl.fragmentsBuilder`::
The fragments builder is responsible for formatting the fragments, which uses `<em>` and `</em>` markup by default (if `hl.tag.pre` and `hl.tag.post` are not defined).
+
Another pre-configured choice is `colored`, which is an example of how to use the fragments builder to insert HTML into the snippets for colored highlights if you choose. You can also implement your own if you'd like. You can select a fragments builder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".

`hl.boundaryScanner`::
See <<Using Boundary Scanners with the FastVector Highlighter>> below.

`hl.bs.*`::
See <<Using Boundary Scanners with the FastVector Highlighter>> below.

`hl.phraseLimit`::
The maximum number of phrases to analyze when searching for the highest-scoring phrase. The default is `5000`.

`hl.multiValuedSeparatorChar`::
Text to use to separate one value from the next for a multi-valued field. The default is " " (a space).

=== Using Boundary Scanners with the FastVector Highlighter

The FastVector Highlighter will occasionally truncate highlighted words. To prevent this, implement a boundary scanner in `solrconfig.xml`, then use the `hl.boundaryScanner` parameter to specify the boundary scanner for highlighting.

Solr supports two boundary scanners: `breakIterator` and `simple`.

==== The breakIterator Boundary Scanner

The `breakIterator` boundary scanner offers excellent performance right out of the box by taking locale and boundary type into account. In most cases you will want to use the `breakIterator` boundary scanner. To implement the `breakIterator` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the type, language, and country values as appropriate to your application:

[source,xml]
----
<boundaryScanner name="breakIterator" class="solr.highlight.BreakIteratorBoundaryScanner">
   <lst name="defaults">
     <str name="hl.bs.type">WORD</str>
     <str name="hl.bs.language">en</str>
     <str name="hl.bs.country">US</str>
   </lst>
</boundaryScanner>
----

Possible values for the `hl.bs.type` parameter are WORD, LINE, SENTENCE, and CHARACTER.

==== The simple Boundary Scanner

The `simple` boundary scanner scans term boundaries for a specified maximum character value (`hl.bs.maxScan`) and for common delimiters such as punctuation marks (`hl.bs.chars`). To implement the `simple` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the values as appropriate to your application:

[source,xml]
----
<boundaryScanner name="simple" class="solr.highlight.SimpleBoundaryScanner" default="true">
   <lst name="defaults">
     <str name="hl.bs.maxScan">10</str>
     <str name="hl.bs.chars">.,!?\t\n</str>
   </lst>
</boundaryScanner>
----