| = Learning To Rank |
| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| With the *Learning To Rank* (or *LTR* for short) module you can configure and run machine learned ranking models in Solr. |
| |
| The module also supports feature logging inside Solr. |
| The only thing you need to do outside Solr is train your own ranking model. |
| |
| == Learning to Rank Concepts |
| |
| === Re-Ranking |
| |
| Re-Ranking allows you to run a simple query for matching documents and then re-rank the top N documents using the scores from a different, more complex query. |
| This page describes the use of *LTR* complex queries, information on other rank queries included in the Solr distribution can be found in xref:query-re-ranking.adoc[]. |
| |
| === Learning To Rank Models |
| |
| In information retrieval systems, https://en.wikipedia.org/wiki/Learning_to_rank[Learning to Rank] is used to re-rank the top N retrieved documents using trained machine learning models. |
| The hope is that such sophisticated models can make more nuanced ranking decisions than standard ranking functions like https://en.wikipedia.org/wiki/Tf%E2%80%93idf[TF-IDF] or https://en.wikipedia.org/wiki/Okapi_BM25[BM25]. |
| |
| ==== Ranking Model |
| |
| A ranking model computes the scores used to rerank documents. |
| Irrespective of any particular algorithm or implementation, a ranking model's computation can use three types of inputs: |
| |
| * parameters that represent the scoring algorithm |
| * features that represent the document being scored |
| * features that represent the query for which the document is being scored |
| |
| ==== Interleaving |
| |
| Interleaving is an approach to Online Search Quality evaluation that allows to compare two models interleaving their results in the final ranked list returned to the user. |
| |
| * currently only the Team Draft Interleaving algorithm is supported (and its implementation assumes all results are from the same shard) |
| |
| |
| ==== Feature |
| |
| A feature is a value, a number, that represents some quantity or quality of the document being scored or of the query for which documents are being scored. |
| For example documents often have a 'recency' quality and 'number of past purchases' might be a quantity that is passed to Solr as part of the search query. |
| |
| ==== Normalizer |
| |
| Some ranking models expect features on a particular scale. |
| A normalizer can be used to translate arbitrary feature values into normalized values e.g., on a 0..1 or 0..100 scale. |
| |
| === Training Models |
| |
| ==== Feature Engineering |
| |
| The LTR module includes several feature classes as well as support for custom features. |
| Each feature class's javadocs contain an example to illustrate use of that class. |
| The process of https://en.wikipedia.org/wiki/Feature_engineering[feature engineering] itself is then entirely up to your domain expertise and creativity. |
| |
| [cols=",,,",options="header",] |
| |=== |
| |Feature |Class |Example parameters |<<External Feature Information>> |
| |field length |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/FieldLengthFeature.html[FieldLengthFeature] |`{"field":"title"}` |not (yet) supported |
| |field value |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/FieldValueFeature.html[FieldValueFeature] |`{"field":"hits"}` |not (yet) supported |
| |original score |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/OriginalScoreFeature.html[OriginalScoreFeature] |`{}` |not applicable |
| |solr query |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/SolrFeature.html[SolrFeature] |`{"q":"{!func}` `recip(ms(NOW,last_modified)` `,3.16e-11,1,1)"}` |supported |
| |solr filter query |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/SolrFeature.html[SolrFeature] |`{"fq":["{!terms f=category}book"]}` |supported |
| |solr query + filter query |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/SolrFeature.html[SolrFeature] |`{"q":"{!func}` `recip(ms(NOW,last_modified),` `3.16e-11,1,1)",` `"fq":["{!terms f=category}book"]}` |supported |
| |value |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/ValueFeature.html[ValueFeature] |`{"value":"$\{userFromMobile}","required":true}` |supported |
| |(custom) |(custom class extending {solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/Feature.html[Feature]) | | |
| |=== |
| |
| [cols=",,",options="header",] |
| |=== |
| |Normalizer |Class |Example parameters |
| |Identity |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/norm/IdentityNormalizer.html[IdentityNormalizer] |`{}` |
| |MinMax |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/norm/MinMaxNormalizer.html[MinMaxNormalizer] |`{"min":"0", "max":"50" }` |
| |Standard |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/norm/StandardNormalizer.html[StandardNormalizer] |`{"avg":"42","std":"6"}` |
| |(custom) |(custom class extending {solr-javadocs}/modules/ltr/org/apache/solr/ltr/norm/Normalizer.html[Normalizer]) | |
| |=== |
| |
| ==== Feature Logging |
| |
| The ltr module includes a xref:document-transformers.adoc[`[features\]` transformer] to support the calculation and return of feature values for https://en.wikipedia.org/wiki/Feature_extraction[feature extraction] purposes including and especially when you do not yet have an actual reranking model. |
| |
| ==== Feature Selection and Model Training |
| |
| Feature selection and model training take place offline and outside Solr. |
| The ltr module supports two generalized forms of models as well as custom models. |
| Each model class's javadocs contain an example to illustrate configuration of that class. |
| In the form of JSON files your trained model or models (e.g., different models for different customer geographies) can then be directly uploaded into Solr using provided REST APIs. |
| |
| [cols=",,",options="header",] |
| |=== |
| |General form |Class |Specific examples |
| |Linear |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/LinearModel.html[LinearModel] |RankSVM, Pranking |
| |Multiple Additive Trees |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/MultipleAdditiveTreesModel.html[MultipleAdditiveTreesModel] |LambdaMART, Gradient Boosted Regression Trees (GBRT) |
| |Neural Network |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/NeuralNetworkModel.html[NeuralNetworkModel] |RankNet |
| |(wrapper) |{solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/DefaultWrapperModel.html[DefaultWrapperModel] |(not applicable) |
| |(custom) |(custom class extending {solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/AdapterModel.html[AdapterModel]) |(not applicable) |
| |(custom) |(custom class extending {solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/LTRScoringModel.html[LTRScoringModel]) |(not applicable) |
| |=== |
| |
| == Module |
| |
| This is provided via the `ltr` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use. |
| |
| == Installation of LTR |
| |
| The ltr module requires the `modules/ltr/lib/solr-ltr-*.jar` JARs. |
| |
| == LTR Configuration |
| |
| Learning-To-Rank is a module and therefore its plugins must be configured in `solrconfig.xml`. |
| |
| === Minimum Requirements |
| |
| * Include the required module JARs. |
| Note that by default paths are relative to the Solr core, so they may need adjustments to your configuration, or an explicit specification of the `$solr.install.dir`. |
| + |
| [source,xml] |
| ---- |
| <lib dir="${solr.install.dir:../../../..}/modules/ltr/lib/" regex=".*\.jar" /> |
| ---- |
| |
| * Declaration of the `ltr` query parser. |
| + |
| [source,xml] |
| ---- |
| <queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin"/> |
| ---- |
| |
| * Configuration of the feature values cache. |
| + |
| [source,xml] |
| ---- |
| <cache name="QUERY_DOC_FV" |
| class="solr.search.CaffeineCache" |
| size="4096" |
| initialSize="2048" |
| autowarmCount="4096" |
| regenerator="solr.search.NoOpRegenerator" /> |
| ---- |
| |
| * Declaration of the `[features]` transformer. |
| + |
| [source,xml] |
| ---- |
| <transformer name="features" class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory"> |
| <str name="fvCacheName">QUERY_DOC_FV</str> |
| </transformer> |
| ---- |
| |
| * Declaration of the `[interleaving]` transformer. |
| + |
| [source,xml] |
| ---- |
| <transformer name="interleaving" class="org.apache.solr.ltr.response.transform.LTRInterleavingTransformerFactory"/> |
| ---- |
| |
| == LTR Lifecycle |
| |
| === Feature Stores |
| |
| It is recommended that you organise all your features into stores which are akin to namespaces: |
| |
| * Features within a store must be named uniquely. |
| * Across stores identical or similar features can share the same name. |
| * If no store name is specified then the default `\_DEFAULT_` feature store will be used. |
| |
| To discover the names of all your feature stores: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/feature-store |
| |
| To inspect the content of the `commonFeatureStore` feature store: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/feature-store/commonFeatureStore |
| |
| === Models |
| |
| * A model uses features from exactly one feature store. |
| * If no store is specified then the default `\_DEFAULT_` feature store will be used. |
| * A model need not use all the features defined in a feature store. |
| * Multiple models can use the same feature store. |
| |
| To log features for `currentFeatureStore` 's features: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,score,[features store=currentFeatureStore] |
| |
| To log features for `nextFeatureStore` features whilst reranking with `currentModel` based on `currentFeatureStore`: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=currentModel reRankDocs=100}&fl=id,score,[features store=nextFeatureStore] |
| |
| To view all models: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/model-store |
| |
| To delete the `currentModel` model: |
| |
| [source,bash] |
| ---- |
| curl -XDELETE 'http://localhost:8983/solr/techproducts/schema/model-store/currentModel' |
| ---- |
| |
| IMPORTANT: A feature store may be deleted only when there are no models using it. |
| |
| To delete the `currentFeatureStore` feature store: |
| |
| [source,bash] |
| ---- |
| curl -XDELETE 'http://localhost:8983/solr/techproducts/schema/feature-store/currentFeatureStore' |
| ---- |
| |
| ==== Using Large Models |
| |
| With SolrCloud, large models may fail to upload due to the limitation of ZooKeeper's buffer. |
| In this case, `DefaultWrapperModel` may help you to separate the model definition from uploaded file. |
| |
| Assuming that you consider to use a large model placed at `/path/to/models/myModel.json` through `DefaultWrapperModel`. |
| |
| [source,json] |
| ---- |
| { |
| "store" : "largeModelsFeatureStore", |
| "name" : "myModel", |
| "class" : "...", |
| "features" : [ |
| "..." |
| ], |
| "params" : { |
| "...": "..." |
| } |
| } |
| ---- |
| |
| First, add the directory to Solr's resource paths with a xref:configuration-guide:libs.adoc#lib-directives-in-solrconfig[`<lib/>` directive]: |
| |
| [source,xml] |
| ---- |
| <lib dir="/path/to" regex="models" /> |
| ---- |
| |
| Then, configure `DefaultWrapperModel` to wrap `myModel.json`: |
| |
| [source,json] |
| ---- |
| { |
| "store" : "largeModelsFeatureStore", |
| "name" : "myWrapperModel", |
| "class" : "org.apache.solr.ltr.model.DefaultWrapperModel", |
| "params" : { |
| "resource" : "myModel.json" |
| } |
| } |
| ---- |
| |
| `myModel.json` will be loaded during the initialization and be able to use by specifying `model=myWrapperModel`. |
| |
| NOTE: No `"features"` are configured in `myWrapperModel` because the features of the wrapped model (`myModel`) will be used; also note that the `"store"` configured for the wrapper model must match that of the wrapped model i.e., in this example the feature store called `largeModelsFeatureStore` is used. |
| |
| CAUTION: `<lib dir="/path/to/models" regex=".*\.json" />` doesn't work as expected in this case, because `SolrResourceLoader` considers given resources as JAR if `<lib />` indicates files. |
| |
| As an alternative to the above-described `DefaultWrapperModel`, it is possible to xref:deployment-guide:zookeeper-ensemble.adoc#increasing-the-file-size-limit[increase ZooKeeper's file size limit]. |
| |
| === Applying Changes |
| |
| The feature store and the model store are both xref:configuration-guide:managed-resources.adoc[]. |
| Changes made to managed resources are not applied to the active Solr components until the Solr collection (or Solr core in single server mode) is reloaded. |
| |
| == Quick Start with LTR |
| |
| The `"techproducts"` example included with Solr is pre-configured to load the plugins required for learning-to-rank from the `ltr` xref:configuration-guide:solr-modules.adoc[Solr Module], but they are disabled by default. |
| |
| To enable the plugins, please specify the `solr.ltr.enabled` JVM System Property when running the `techproducts` example: |
| |
| [source,bash] |
| ---- |
| bin/solr start -e techproducts -Dsolr.ltr.enabled=true |
| ---- |
| |
| === Uploading Features |
| |
| To upload features in a `/path/myFeatures.json` file, please run: |
| |
| [source,bash] |
| ---- |
| curl -XPUT 'http://localhost:8983/solr/techproducts/schema/feature-store' --data-binary "@/path/myFeatures.json" -H 'Content-type:application/json' |
| ---- |
| |
| To view the features you just uploaded please open the following URL in a browser: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/feature-store/_DEFAULT_ |
| |
| .Example: /path/myFeatures.json |
| [source,json] |
| ---- |
| [ |
| { |
| "name" : "documentRecency", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "q" : "{!func}recip( ms(NOW,last_modified), 3.16e-11, 1, 1)" |
| } |
| }, |
| { |
| "name" : "isBook", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "fq": ["{!terms f=cat}book"] |
| } |
| }, |
| { |
| "name" : "originalScore", |
| "class" : "org.apache.solr.ltr.feature.OriginalScoreFeature", |
| "params" : {} |
| } |
| ] |
| ---- |
| |
| === Logging Features |
| |
| To log features as part of a query, add `[features]` to the `fl` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,score,[features] |
| |
| The output will include feature values as a comma-separated list, resembling the output shown here: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features]"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.959392,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.959392, |
| "[features]":"documentRecency=0.020893794,isBook=0.0,originalScore=1.959392"}, |
| { |
| "id":"UTF8TEST", |
| "score":1.5513437, |
| "[features]":"documentRecency=0.020893794,isBook=0.0,originalScore=1.5513437"}] |
| }} |
| ---- |
| |
| ==== Feature Logging Parameters |
| |
| The feature logger transformer accepts the parameters described below. |
| Examples on how to use them can be found in the <<ltr-examples>> section below. |
| |
| `store`:: |
| + |
| [%autowidth,frame=none] |
| |=== |
| |No Re-Ranking |Optional |Default: `\_DEFAULT_` |
| |Re-Ranking |Optional |Default: model feature store |
| |=== |
| + |
| This parameter specifies the feature store to use for logging features. |
| + |
| In a reranking query, the default feature store used is the model feature store (e.g. `[features]`). |
| |
| `logAll`:: |
| + |
| [%autowidth,frame=none] |
| |=== |
| |No Re-Ranking |Default: `true` |
| |=== |
| + |
| |
| + |
| [%autowidth,frame=none] |
| |=== |
| |Re-Ranking |Logger and Model have same feature store|Default: `false` |
| |Re-Ranking |Logger and Model have different feature store |Default: `true` |
| |=== |
| + |
| This parameter specifies the features to log. |
| + |
| If set to `true` all the features from the feature store are printed. |
| |
| If set to `false` only the features used by the model are printed. |
| |
| CAUTION: When no re-ranking query is passed, only `logAll` = 'true' is supported. Passing `false` will cause a Solr exception. |
| |
| CAUTION: In a logging scenario where a re-ranking query is passed, if the logger `store` is different from the model `store`, only `logAll` = 'true' is supported. Passing `false` will cause a Solr exception. |
| |
| `format`:: |
| + |
| [%autowidth,frame=none] |
| |=== |
| |Optional |Default: `dense` |
| |=== |
| + |
| This parameter specifies the format to use for logging features. The supported values are: `dense` and `sparse`. |
| + |
| You can change the default behavior to be sparse, putting `<str name="defaultFormat">sparse</str>` in the {solr-javadocs}/modules/ltr/org/apache/solr/ltr/response/transform/LTRFeatureLoggerTransformerFactory.html[feature logger transformer] declaration in `solrconfig.xml` as follows: |
| |
| [source,xml] |
| ---- |
| <transformer name="features" class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory"> |
| <str name="fvCacheName">QUERY_DOC_FV</str> |
| <str name="defaultFormat">sparse</str> |
| <str name="csvKeyValueDelimiter">:</str> |
| <str name="csvFeatureSeparator"> </str> |
| </transformer> |
| ---- |
| |
| === Uploading a Model |
| |
| To upload the model in a `/path/myModel.json` file, please run: |
| |
| [source,bash] |
| ---- |
| curl -XPUT 'http://localhost:8983/solr/techproducts/schema/model-store' --data-binary "@/path/myModel.json" -H 'Content-type:application/json' |
| ---- |
| |
| To view the model you just uploaded please open the following URL in a browser: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/model-store |
| |
| .Example: /path/myModel.json |
| [source,json] |
| ---- |
| { |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "name" : "myModel", |
| "features" : [ |
| { "name" : "documentRecency" }, |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "weights" : { |
| "documentRecency" : 1.0, |
| "isBook" : 0.1, |
| "originalScore" : 0.5 |
| } |
| } |
| } |
| ---- |
| |
| === Running a Rerank Query |
| |
| To rerank the results of a query, add the `rq` parameter to your search, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModel reRankDocs=100}&fl=id,score |
| |
| The addition of the `rq` parameter will not change the output of the search. |
| |
| To obtain the feature values computed during reranking, add `[features]` to the `fl` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModel reRankDocs=100}&fl=id,score,[features] |
| |
| The output will include feature values as a comma-separated list, resembling the output shown here: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features]", |
| "rq":"{!ltr model=myModel reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"documentRecency=0.020893792,isBook=0.0,originalScore=1.959392"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"documentRecency=0.020893792,isBook=0.0,originalScore=1.5513437"}] |
| }} |
| ---- |
| |
| === Running a Rerank Query and Query Limits |
| |
| Apache Solr allows to define Query Limits to interrupt particularly expensive queries (xref:query-guide:common-query-parameters.adoc#timeallowed-parameter[Time Allowed], xref:query-guide:common-query-parameters.adoc#cpuallowed-parameter[Cpu Allowed]). |
| |
| If a query limit is exceeded while reranking, the rescoring is aborted and fully reverted. |
| |
| The original ranked list is returned and the response marked with the responseHeader 'partialResults'. |
| The details of what limit was exceeded is returned in the responseHeader 'partialResultsDetails'. |
| |
| See xref:query-guide:common-query-parameters.adoc#partialresults-parameter[Partial Results Parameter] for more details on how to handle partial results. |
| |
| === Running a Rerank Query Interleaving Two Models |
| |
| To rerank the results of a query, interleaving two models (myModelA, myModelB) add the `rq` parameter to your search, passing two models in input, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModelA model=myModelB reRankDocs=100}&fl=id,score |
| |
| To obtain the model that interleaving picked for a search result, computed during reranking, add `[interleaving]` to the `fl` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModelA model=myModelB reRankDocs=100}&fl=id,score,[interleaving] |
| |
| The output will include the model picked for each search result, resembling the output shown here: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[interleaving]", |
| "rq":"{!ltr model=myModelA model=myModelB reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[interleaving]":"myModelB"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[interleaving]":"myModelA"}] |
| }} |
| ---- |
| |
| === Running a Rerank Query Interleaving a Model with the Original Ranking |
| When approaching Search Quality Evaluation with interleaving it may be useful to compare a model with the original ranking. |
| To rerank the results of a query, interleaving a model with the original ranking, add the `rq` parameter to your search, passing the special inbuilt `_OriginalRanking_` model identifier as one model and your comparison model as the other model, for example: |
| |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=_OriginalRanking_ model=myModel reRankDocs=100}&fl=id,score |
| |
| The addition of the `rq` parameter will not change the output of the search. |
| |
| To obtain the model that interleaving picked for a search result, computed during reranking, add `[interleaving]` to the `fl` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=_OriginalRanking_ model=myModel reRankDocs=100}&fl=id,score,[interleaving] |
| |
| The output will include the model picked for each search result, resembling the output shown here: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features]", |
| "rq":"{!ltr model=_OriginalRanking_ model=myModel reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[interleaving]":"_OriginalRanking_"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[interleaving]":"myModel"}] |
| }} |
| ---- |
| |
| === Running a Rerank Query with Interleaving Passing a Specific Algorithm |
| To rerank the results of a query, interleaving two models using a specific algorithm, add the `interleavingAlgorithm` local parameter to the ltr query parser, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModelA model=myModelB reRankDocs=100 interleavingAlgorithm=TeamDraft}&fl=id,score |
| |
| Currently, the only (and default) algorithm supported is 'TeamDraft'. |
| |
| === External Feature Information |
| |
| The {solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/ValueFeature.html[ValueFeature] and {solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/SolrFeature.html[SolrFeature] classes support the use of external feature information, `efi` for short. |
| |
| ==== Uploading Features |
| |
| To upload features in a `/path/myEfiFeatures.json` file, please run: |
| |
| [source,bash] |
| ---- |
| curl -XPUT 'http://localhost:8983/solr/techproducts/schema/feature-store' --data-binary "@/path/myEfiFeatures.json" -H 'Content-type:application/json' |
| ---- |
| |
| To view the features you just uploaded please open the following URL in a browser: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/feature-store/myEfiFeatureStore |
| |
| .Example: /path/myEfiFeatures.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store" : "myEfiFeatureStore", |
| "name" : "isPreferredManufacturer", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { "fq" : [ "{!field f=manu}${preferredManufacturer}" ] } |
| }, |
| { |
| "store" : "myEfiFeatureStore", |
| "name" : "userAnswerValue", |
| "class" : "org.apache.solr.ltr.feature.ValueFeature", |
| "params" : { "value" : "${answer:42}" } |
| }, |
| { |
| "store" : "myEfiFeatureStore", |
| "name" : "userFromMobileValue", |
| "class" : "org.apache.solr.ltr.feature.ValueFeature", |
| "params" : { "value" : "${fromMobile}", "required" : true } |
| }, |
| { |
| "store" : "myEfiFeatureStore", |
| "name" : "userTextCat", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { "q" : "{!field f=cat}${text}" } |
| } |
| ] |
| ---- |
| |
| ==== Logging Features |
| |
| To log `myEfiFeatureStore` features as part of a query, add `efi.*` parameters to the `[features]` part of the `fl` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,cat,manu,score,[features store=myEfiFeatureStore efi.text=test efi.preferredManufacturer=Apache efi.fromMobile=1] |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,cat,manu,score,[features store=myEfiFeatureStore efi.text=test efi.preferredManufacturer=Apache efi.fromMobile=0 efi.answer=13] |
| |
| ==== Uploading a Model |
| |
| To upload the model in a `/path/myEfiModel.json` file, please run: |
| |
| [source,bash] |
| ---- |
| curl -XPUT 'http://localhost:8983/solr/techproducts/schema/model-store' --data-binary "@/path/myEfiModel.json" -H 'Content-type:application/json' |
| ---- |
| |
| To view the model you just uploaded please open the following URL in a browser: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/schema/model-store |
| |
| .Example: /path/myEfiModel.json |
| [source,json] |
| ---- |
| { |
| "store" : "myEfiFeatureStore", |
| "name" : "myEfiModel", |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "features" : [ |
| { "name" : "isPreferredManufacturer" }, |
| { "name" : "userAnswerValue" }, |
| { "name" : "userFromMobileValue" }, |
| { "name" : "userTextCat" } |
| ], |
| "params" : { |
| "weights" : { |
| "isPreferredManufacturer" : 0.2, |
| "userAnswerValue" : 1.0, |
| "userFromMobileValue" : 1.0, |
| "userTextCat" : 0.1 |
| } |
| } |
| } |
| ---- |
| |
| ==== Running a Rerank Query |
| |
| To obtain the feature values computed during reranking, add `[features]` to the `fl` parameter and `efi.*` parameters to the `rq` parameter, for example: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myEfiModel efi.text=test efi.preferredManufacturer=Apache efi.fromMobile=1}&fl=id,cat,manu,score,[features] |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myEfiModel efi.text=test efi.preferredManufacturer=Apache efi.fromMobile=0 efi.answer=13}&fl=id,cat,manu,score,[features] |
| |
| Notice the absence of `efi.*` parameters in the `[features]` part of the `fl` parameter. |
| |
| ==== Logging Features While Reranking |
| |
| To log features for `myEfiFeatureStore` features while still reranking with `myModel`: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModel}&fl=id,cat,manu,score,[features store=myEfiFeatureStore efi.text=test efi.preferredManufacturer=Apache efi.fromMobile=1] |
| |
| Notice the absence of `efi.\*` parameters in the `rq` parameter (because `myModel` does not use `efi` feature) and the presence of `efi.*` parameters in the `[features]` part of the `fl` parameter (because `myEfiFeatureStore` contains `efi` features). |
| |
| === Training Example |
| |
| Example training data and a demo `train_and_upload_demo_model.py` script can be found in the `solr/modules/ltr/example` folder in the https://gitbox.apache.org/repos/asf?p=solr.git;a=tree;f=solr/modules/ltr/example[Apache Solr Git repository] (mirrored on https://github.com/apache/solr/tree/releases/solr/{solr-full-version}/solr/modules/ltr/example[github.com]). |
| This example folder is not shipped in the Solr binary release. |
| |
| === Advanced Options |
| |
| ==== LTRThreadModule |
| |
| A thread module can be configured for the query parser and/or the transformer to parallelize the creation of feature weights. |
| For details, please refer to the {solr-javadocs}/modules/ltr/org/apache/solr/ltr/LTRThreadModule.html[LTRThreadModule] javadocs. |
| |
| ==== Models handling features' null values |
| This feature is available only for {solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/MultipleAdditiveTreesModel.html[MultipleAdditiveTreesModel]. |
| |
| In some scenarios a null value for a feature has a different meaning than a zero value. There are models that are trained to distinguish the two (e.g. https://xgboost.readthedocs.io/en/stable/faq.html#how-to-deal-with-missing-values), in Solr an additional `missing` branch parameter has been introduced to support this feature. |
| |
| This defines the branch to follow when the corresponding feature value is null. With the default configuration a null and a zero value have the same meaning. |
| |
| To handle null values, the `myFeatures.json` file needs to be modified. A `defaultValue` parameter with a `NaN` value needs to be added to each feature that can assume a null value. |
| |
| .Example: /path/myFeatures.json |
| [source,json] |
| ---- |
| [ |
| { |
| "name": "matchedTitle", |
| "class": "org.apache.solr.ltr.feature.SolrFeature", |
| "params": { |
| "q": "{!terms f=title}${user_query}" |
| } |
| }, |
| { |
| "name": "productReviewScore", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "product_review_score", |
| "defaultValue": "NaN" |
| } |
| } |
| ] |
| ---- |
| |
| Also, the model configuration needs two additional parameter: |
| |
| * `isNullSameAsZero` needs to be defined in the model `params` and set to `false`; |
| |
| * the `missing` parameter needs to be added to each branch where the corresponding feature supports null values. This can assume one value between `left` and `right`. |
| |
| .Example: /path/myModel.json |
| [source,json] |
| ---- |
| { |
| "class":"org.apache.solr.ltr.model.MultipleAdditiveTreesModel", |
| "name":"multipleadditivetreesmodel", |
| "features":[ |
| { "name": "matchedTitle"}, |
| { "name": "productReviewScore"} |
| ], |
| "params":{ |
| "isNullSameAsZero": "false", |
| "trees": [ |
| { |
| "weight" : "1f", |
| "root": { |
| "feature": "matchedTitle", |
| "threshold": "0.5f", |
| "left" : { |
| "value" : "-100" |
| }, |
| "right": { |
| "feature" : "productReviewScore", |
| "threshold": "0f", |
| "missing": "left", |
| "left" : { |
| "value" : "50" |
| }, |
| "right" : { |
| "value" : "65" |
| } |
| } |
| } |
| } |
| ] |
| } |
| } |
| |
| ---- |
| |
| When `isNullSameAsZero` is `false` for your model, the feature vector changes. |
| |
| * dense format: all features values are shown, also the default values which can be zero or null values. |
| * sparse format: only non default values are shown. |
| |
| e.g. |
| |
| given the features defined before; if their values are `matchedTitle=0` and `productReviewScore=0`, the sparse format will return `productReviewScore:0` (0 is the default value of `matchedTitle=0` and therefore it is not returned, 0 is not the default value of `productReviewScore=0` and therefore it is returned). |
| |
| ==== Implementation and Contributions |
| |
| How does Solr Learning-To-Rank work under the hood?:: |
| Please refer to the `ltr` {solr-javadocs}/modules/ltr/org/apache/solr/ltr/package-summary.html[javadocs] for an implementation overview. |
| |
| How could I write additional models and/or features?:: |
| Contributions for further models, features, normalizers and interleaving algorithms are welcome. |
| Related links: |
| + |
| * {solr-javadocs}/modules/ltr/org/apache/solr/ltr/model/LTRScoringModel.html[LTRScoringModel javadocs] |
| * {solr-javadocs}/modules/ltr/org/apache/solr/ltr/feature/Feature.html[Feature javadocs] |
| * {solr-javadocs}/modules/ltr/org/apache/solr/ltr/norm/Normalizer.html[Normalizer javadocs] |
| * {solr-javadocs}/modules/ltr/org/apache/solr/ltr/interleaving/Interleaving.html[Interleaving javadocs] |
| * https://github.com/apache/solr/blob/main/CONTRIBUTING.md[Contributing to Solr] |
| * https://github.com/apache/lucene/blob/main/CONTRIBUTING.md[Contributing to Lucene] |
| |
| == LTR Examples |
| |
| === One Feature Store, Multiple Ranking Models |
| |
| * `leftModel` and `rightModel` both use features from `commonFeatureStore` and the only different between the two models is the weights attached to each feature. |
| * Conventions used: |
| ** `commonFeatureStore.json` file contains features for the `commonFeatureStore` feature store |
| ** `leftModel.json` file contains model named `leftModel` |
| ** `rightModel.json` file contains model named `rightModel` |
| ** The model's features and weights are sorted alphabetically by name, this makes it easy to see what the commonalities and differences between the two models are. |
| ** The stores features are sorted alphabetically by name, this makes it easy to lookup features used in the models |
| |
| .Example: /path/commonFeatureStore.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store" : "commonFeatureStore", |
| "name" : "documentRecency", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "q" : "{!func}recip( ms(NOW,last_modified), 3.16e-11, 1, 1)" |
| } |
| }, |
| { |
| "store" : "commonFeatureStore", |
| "name" : "isBook", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "fq": [ "{!terms f=category}book" ] |
| } |
| }, |
| { |
| "store" : "commonFeatureStore", |
| "name" : "originalScore", |
| "class" : "org.apache.solr.ltr.feature.OriginalScoreFeature", |
| "params" : {} |
| } |
| ] |
| ---- |
| |
| .Example: /path/leftModel.json |
| [source,json] |
| ---- |
| { |
| "store" : "commonFeatureStore", |
| "name" : "leftModel", |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "features" : [ |
| { "name" : "documentRecency" }, |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "weights" : { |
| "documentRecency" : 0.1, |
| "isBook" : 1.0, |
| "originalScore" : 0.5 |
| } |
| } |
| } |
| ---- |
| |
| .Example: /path/rightModel.json |
| [source,json] |
| ---- |
| { |
| "store" : "commonFeatureStore", |
| "name" : "rightModel", |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "features" : [ |
| { "name" : "documentRecency" }, |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "weights" : { |
| "documentRecency" : 1.0, |
| "isBook" : 0.1, |
| "originalScore" : 0.5 |
| } |
| } |
| } |
| ---- |
| |
| === Model Evolution |
| |
| * `linearModel201701` uses features from `featureStore201701` |
| * `treesModel201702` uses features from `featureStore201702` |
| * `linearModel201701` and `treesModel201702` and their feature stores can co-exist whilst both are needed. |
| * When `linearModel201701` has been deleted then `featureStore201701` can also be deleted. |
| * Conventions used: |
| ** `<store>.json` file contains features for the `<store>` feature store |
| ** `<model>.json` file contains model name `<model>` |
| ** a 'generation' id (e.g., `YYYYMM` year-month) is part of the feature store and model names |
| ** The model's features and weights are sorted alphabetically by name, this makes it easy to see what the commonalities and differences between the two models are. |
| ** The stores features are sorted alphabetically by name, this makes it easy to see what the commonalities and differences between the two feature stores are. |
| |
| .Example: /path/featureStore201701.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store" : "featureStore201701", |
| "name" : "documentRecency", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "q" : "{!func}recip( ms(NOW,last_modified), 3.16e-11, 1, 1)" |
| } |
| }, |
| { |
| "store" : "featureStore201701", |
| "name" : "isBook", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "fq": [ "{!terms f=category}book" ] |
| } |
| }, |
| { |
| "store" : "featureStore201701", |
| "name" : "originalScore", |
| "class" : "org.apache.solr.ltr.feature.OriginalScoreFeature", |
| "params" : {} |
| } |
| ] |
| ---- |
| |
| .Example: /path/linearModel201701.json |
| [source,json] |
| ---- |
| { |
| "store" : "featureStore201701", |
| "name" : "linearModel201701", |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "features" : [ |
| { "name" : "documentRecency" }, |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "weights" : { |
| "documentRecency" : 0.1, |
| "isBook" : 1.0, |
| "originalScore" : 0.5 |
| } |
| } |
| } |
| ---- |
| |
| .Example: /path/featureStore201702.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store" : "featureStore201702", |
| "name" : "isBook", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "fq": [ "{!terms f=category}book" ] |
| } |
| }, |
| { |
| "store" : "featureStore201702", |
| "name" : "originalScore", |
| "class" : "org.apache.solr.ltr.feature.OriginalScoreFeature", |
| "params" : {} |
| } |
| ] |
| ---- |
| |
| .Example: /path/treesModel201702.json |
| [source,json] |
| ---- |
| { |
| "store" : "featureStore201702", |
| "name" : "treesModel201702", |
| "class" : "org.apache.solr.ltr.model.MultipleAdditiveTreesModel", |
| "features" : [ |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "trees" : [ |
| { |
| "weight" : "1", |
| "root" : { |
| "feature" : "isBook", |
| "threshold" : "0.5", |
| "left" : { "value" : "-100" }, |
| "right" : { |
| "feature" : "originalScore", |
| "threshold" : "10.0", |
| "left" : { "value" : "50" }, |
| "right" : { "value" : "75" } |
| } |
| } |
| }, |
| { |
| "weight" : "2", |
| "root" : { |
| "value" : "-10" |
| } |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Features Logging |
| |
| ==== logAll parameter |
| |
| Suppose to have a complete feature store like: |
| |
| .Example: /path/completeFeaturesStore.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store" : "completeFeaturesStore", |
| "name" : "documentRecency", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "q" : "{!func}recip( ms(NOW,last_modified), 3.16e-11, 1, 1)" |
| } |
| }, |
| { |
| "store" : "completeFeaturesStore", |
| "name" : "isBook", |
| "class" : "org.apache.solr.ltr.feature.SolrFeature", |
| "params" : { |
| "fq": ["{!terms f=cat}book"] |
| } |
| }, |
| { |
| "store" : "completeFeaturesStore", |
| "name" : "originalScore", |
| "class" : "org.apache.solr.ltr.feature.OriginalScoreFeature", |
| "params" : {} |
| } |
| ] |
| ---- |
| |
| And suppose to have a simple linear model that uses just two of the `completeFeaturesStore` 's features: |
| |
| .Example: /path/linearModel.json |
| [source,json] |
| ---- |
| { |
| "store" : "completeFeaturesStore", |
| "name" : "linearModel", |
| "class" : "org.apache.solr.ltr.model.LinearModel", |
| "features" : [ |
| { "name" : "isBook" }, |
| { "name" : "originalScore" } |
| ], |
| "params" : { |
| "weights" : { |
| "isBook" : 1.0, |
| "originalScore" : 0.5 |
| } |
| } |
| } |
| ---- |
| |
| Making a logging + re-ranking query without defining the `store` and `logAll` parameters will print only the model features (default: `store=model store` and `logAll=false`). |
| |
| The query: |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=linearModel reRankDocs=100}&fl=id,score,[features] |
| |
| The output: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features]", |
| "rq":"{!ltr model=linearModel reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"isBook=0.0,originalScore=1.959392"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"isBook=0.0,originalScore=1.5513437"}] |
| }} |
| ---- |
| |
| Making a logging + re-ranking query without defining the `store` parameter and setting `logAll = true` will print all the features from the model store. |
| |
| The query: |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=linearModel reRankDocs=100}&fl=id,score,[features logAll=true] |
| |
| The output: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features logAll=true]", |
| "rq":"{!ltr model=linearModel reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"documentRecency=0.020893792,isBook=0.0,originalScore=1.959392"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"documentRecency=0.020893792,isBook=0.0,originalScore=1.5513437"}] |
| }} |
| ---- |
| |
| Suppose to have a different feature store like: |
| |
| .Example: /path/differentFeaturesStore.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store": "differentFeaturesStore", |
| "name": "valueFeature1", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "field1" |
| } |
| }, |
| { |
| "store": "differentFeaturesStore", |
| "name": "valueFeature2", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "field2" |
| } |
| } |
| ] |
| ---- |
| |
| Making a logging + re-ranking query defining a `store` parameter different from the model store without defining the `logAll` parameter will print all the features from the selected feature store (default: `logAll=true`). |
| |
| The query: |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=linearModel reRankDocs=100}&fl=id,score,[features store=differentFeaturesStore] |
| |
| The output: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features store=differentFeaturesStore]", |
| "rq":"{!ltr model=linearModel reRankDocs=100}"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"valueFeature1=0.1,valueFeature2=2.0"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"valueFeature1=1.3,valueFeature2=4.0"}] |
| }} |
| ---- |
| |
| ==== format parameter |
| |
| Suppose to have a feature store like: |
| |
| .Example: /path/myFeaturesStore.json |
| [source,json] |
| ---- |
| [ |
| { |
| "store": "myFeaturesStore", |
| "name": "featureA", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "field1" |
| } |
| }, |
| { |
| "store": "myFeaturesStore", |
| "name": "featureB", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "field2" |
| } |
| }, |
| { |
| "store": "myFeaturesStore", |
| "name": "featureC", |
| "class": "org.apache.solr.ltr.feature.FieldValueFeature", |
| "params": { |
| "field": "field3" |
| } |
| } |
| ] |
| ---- |
| |
| To return dense CSV values such as: `featureA=0.1,featureB=0.2,featureC=0.0`, pass the `format=dense` parameter to the feature logger transformer: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,score,[features store=myFeaturesStore format=dense] |
| |
| The output: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features store=myFeaturesStore format=dense]"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"featureA=0.1,featureB=0.2,featureC=0.0"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"featureA=1.3,featureB=0.0,featureC=2.1"}] |
| }} |
| ---- |
| |
| To return sparse CSV values such as: `featureA=0.1,featureB=0.2`, pass the `format=sparse` parameter to the feature logger transformer: |
| |
| [source,text] |
| http://localhost:8983/solr/techproducts/query?q=test&fl=id,score,[features store=myFeaturesStore format=sparse] |
| |
| The output: |
| |
| [source,json] |
| ---- |
| { |
| "responseHeader":{ |
| "status":0, |
| "QTime":0, |
| "params":{ |
| "q":"test", |
| "fl":"id,score,[features store=myFeaturesStore format=sparse]"}}, |
| "response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[ |
| { |
| "id":"GB18030TEST", |
| "score":1.0005897, |
| "[features]":"featureA=0.1,featureB=0.2"}, |
| { |
| "id":"UTF8TEST", |
| "score":0.79656565, |
| "[features]":"featureA=1.3,featureC=2.1"}] |
| }} |
| ---- |
| |
| == Additional LTR Resources |
| |
| * "Learning to Rank in Solr" presentation at Lucene/Solr Revolution 2015 in Austin: |
| ** Slides: http://www.slideshare.net/lucidworks/learning-to-rank-in-solr-presented-by-michael-nilsson-diego-ceccarelli-bloomberg-lp |
| ** Video: https://www.youtube.com/watch?v=M7BKwJoh96s |
| |
| * The importance of Online Testing in Learning To Rank: |
| ** Blog: https://sease.io/2020/04/the-importance-of-online-testing-in-learning-to-rank-part-1.html |
| ** Blog: https://sease.io/2020/05/online-testing-for-learning-to-rank-interleaving.html |