solr/solr-ref-guide/src/solr-upgrade-notes.adoc

= Solr Upgrade Notes
:page-children: major-changes-in-solr-8, major-changes-in-solr-7, major-changes-from-solr-5-to-solr-6
:page-toclevels: 3
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

The following notes describe changes to Solr in recent releases that you should be aware of before upgrading.

These notes highlight the biggest changes that may impact the largest number of
implementations. It is not a comprehensive list of all changes to Solr in any release.

When planning your Solr upgrade, consider the customizations you have made to
your system and review the {solr-javadocs}/changes/Changes.html[`CHANGES.txt`]
file found in your Solr package. That file includes all the changes and updates
that may effect your existing implementation.

Detailed steps for upgrading a Solr cluster are in the section <<upgrading-a-solr-cluster.adoc#,Upgrading a Solr Cluster>>.

== Upgrading to 8.x Releases

If you are upgrading from 7.x, see the section <<Upgrading from 7.x Releases>> below.

=== Solr 8.10

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote810[8.10 Release Notes^]
for an overview of the main new features of Solr 8.10.

When upgrading to 8.10.x users should be aware of the following major changes from 8.9.

*Schema Designer UI*

A new screen has been added to the Admin UI that allows you to interactively design a Solr schema using your documents.

The designer screen provides a safe environment for you to:

* Upload or paste sample documents to identify fields.
* Get a "first" guess at what Solr thinks the field types in the fields should be.
* Edit fields, field types, dynamic fields, and supporting files.
* See how a field's analysis will impact your text.
* Test how schema changes will impact query-time behavior.
* Save your changes to a configset to use with a new collection.

See the section <<schema-designer.adoc#,Schema Designer>> for full details.

*Backups in S3*

Following the redesign of backups in Solr 8.8 that allowed storage of incremental backups in Google Cloud environments, Solr 8.10 provides support for storing backups in Amazon S3 buckets.

See the section <<making-and-restoring-backups.adoc#s3backuprepository,S3BackupRepository>> for how to configure.

*Security Admin UI*

Solr's Admin UI also got a new screen to support management of users, roles, and permissions.

The new UI works when authentication and/or authorization has been enabled with `bin/solr auth` or by manually installing a `security.json` file.
Before this, it provides a warning that your Solr instance is unsecured.

See the section <<security-ui.adoc#,Security UI>> for details.

*Solr SQL Improvements*

A number of improvements have been made in Solr's SQL functionality:

* Support added for `LIKE`, `IS NOT NULL`, `IS NULL`, and wildcards (for simplistic `LIKE` functionality).
* Two new aggregation functions, `COUNT(DISTINCT field)` and `APPROX_COUNT_DISTINCT(field)`, have been added.
* Queries using an `ORDER BY` clause can support `OFFSET` and `FETCH` operations.
* Multi-valued fields can now be returned.
* User permissions have been simplified so access to query endpoints `/sql`, `/select`, and `/export` is sufficient for full access for all SQL queries.

*shards.preference*

A new option for the `shards.preference` parameter allows selection of nodes based on whether or not the replica is a leader.
Now adding `shards.preference=replica.leader:false` will limit queries only to replicas which are not currently their shard's leader.

See the section <<distributed-requests.adoc#shards-preference-parameter,shards.preference Parameter>> for details and examples.

*Metrics & Prometheus Exporter*

A new `expr` option in the Metrics API allows for more advanced filtering of metrics based on regular expressions.
See the section <<metrics-reporting.adoc#metrics-api,Metrics API>> for examples.

The Prometheus Exporter's default `solr-exporter.config` has been improved to use the new `expr` option in the Metrics API to get a smaller set of metrics.
The default metrics exported still include most metrics, but the configuration will be easier to trim as needed.
This should help provide performance improvements in busy clusters being monitored by Prometheus.

*ZooKeeper Credentials*

ZooKeeper credentials can now be stored in a file whose location is defined with a system property instead of being passed in plain-text.
See <<zookeeper-access-control.adoc#out-of-the-box-credential-implementations,Out of the Box Credential Implementations>> for how to set this up.

=== Solr 8.9

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote89[8.9 Release Notes^]
for an overview of the main new features of Solr 8.9.

When upgrading to 8.9.x users should be aware of the following major changes from 8.8.

*Backup and Restore*

Solr 8.9 introduces extensive changes to Solr's backup and restore support.

A new backup format has been introduced in Solr 8.9 which replaces the previous snapshot-based backup.
This new format enables ‘incremental’ backups.
Repeated backups to a given location will take advantage of the data stored by their predecessors and will only operate on files that have changed since the previous backup.
This is supported by default, simply by storing each backup file in the same location.

The old and new formats are not compatible, although backups in the old format, a full snapshot of all files, can still be used to restore to Solr for the time-being.
The old format is officially deprecated, and support for it is likely to be removed in Solr 9.0.

For the time-being the old format can be created by defining a parameter `incremental=false`.
Again, though, this support is likely to be removed in Solr 9.0.

More documentation on backups is available at <<making-and-restoring-backups.adoc#,Making and Restoring Backups>>.

New Collections API commands for backups:

* LISTBACKUP: Lists information about each backup stored at the specified repository location.
See <<collection-management.adoc#listbackup,List Backups>> for more details.
* DELETEBACKUP: Deletes specified backups from the repository. See <<collection-management.adoc#deletebackup,Delete Backups>> for more details.

A new option for backup repository is also available in 8.9, which is to use Google Cloud Storage (GCS).
This is a contrib (located in `contrib/gcs-repository`).
See <<making-and-restoring-backups.adoc#gcsbackuprepository,GCSBackupRepository>> for configuration details.
The Solr community is working to add support for S3 buckets in the near future.

*Nested Docs*

Child Doc Transformer's `childFilter` parameter no longer applies query syntax
escaping because it's inconsistent with the rest of Solr and was limiting.  This refers to `[child childFilter='field:value']`.
There was no escaping here prior to 8.0 either.

*Collapse and Expand*

* BlockCollapse: If documents have been (or could be) indexed in a way where documents with the same collapse key have been indexed contiguously in the index, a new "block collapse" provides a significant speed improvement over traditional collapse.
+
See <<collapse-and-expand-results.adoc#block-collapsing,Block Collapsing>> for details.

* Expand Null Groups: A new parameter `expand.nullGroup` allows an expanded group to be returned containing document with no value in the expanded field. See <<collapse-and-expand-results.adoc#expand-component,Expand Component>> for details.

*In-Place Updates*

A new request parameter `update.partial.requireInPlace=true` allows telling Solr to "fail fast" if all of the necessary conditions are not satisfied to allow an in-place update to succeed.
See also <<updating-parts-of-documents.adoc#in-place-updates,In-Place Updates>>.

*Metrics History*

The Metrics History feature, which allowed long-term storage and aggregation of Solr's metrics, has been deprecated and will be removed in 9.0.

*Embedded Solr Server*

When using EmbeddedSolrServer, it will no longer close CoreContainer instances that were passed to it.

=== Solr 8.8

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote88[8.8 Release Notes^]
for an overview of the main new features of Solr 8.8.

When upgrading to 8.8.x users should be aware of the following major changes from 8.7.

*Nested Documents*

* When doing atomic/partial updates to a child document:
** Supply the `\_root_` field (the ID of the root document) so that Solr understands you are manipulating a child document and not a root document.
In its absence, Solr looks at the `\_route_` parameter but this may change in the future because it's not an ideal substitute.
If neither are present, Solr assumes you are updating a root document.
If this assumption is false, Solr will do a cheap check that usually detects the problem and will
throw an exception to alert you of the need to specify the Root ID.
This backwards incompatible change was done to increase performance and robustness.
** This feature no longer requires `stored=true` or `docValues=true` on the `\_root_` field.
You might have it for other purposes though (e.g., for `uniqueBlock(...)`).
** This feature no longer requires the `\_nest_path_` field, although you probably ought to
continue to define it as it's useful for other things.

*Removed Contribs*

* The search results clustering contrib (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results. The contrib will be re-introduced in Solr 9.0.

*Learning to Rank*

* Interleaving support has been added to Learning to Rank (LTR).
Currently only the Team Draft Interleaving algorithm is supported.
For examples using this feature, see the section <<learning-to-rank.adoc#running-a-rerank-query-interleaving-two-models,Running a Rerank Query Interleaving Two Models>>.

*Metrics*

* Two metrics have been added for SolrCloud's Overseer:
** `solr_metrics_overseer_stateUpdateQueueSize`
** `solr_metrics_overseer_collectionWorkQueueSize`

*Prometheus Exporter*

* The `./bin` scripts included with the Prometheus Exporter contrib now allow use of custom java options with environment variables.
See the section <<monitoring-solr-with-prometheus-and-grafana.adoc#environment-variable-options,Environment Variable Options>> for more details.
* The default Grafana dashboards now include panels for query performance monitoring.
The default Prometheus Exporter configuration includes metrics like queries-per-second (QPS) and 95th percentiles (P95) to populate the new panels.
* The default Prometheus Exporter configuration also includes the two new metrics mentioned in the Metrics above.

*Solr Home*

* The internal logic for identifying 'Solr Home' (`solr.solr.home`) has been refactored to make testing less error prone.
Plugin developers using `SolrPaths.locateSolrHome()` or 'new `SolrResourceLoader`' should check deprecation warnings as existing some existing functionality will be removed in 9.0.
https://issues.apache.org/jira/browse/SOLR-14934[SOLR-14934] has more technical details about this change for those concerned.

base_url removed from stored state*

As of Solr 8.8.0, the `base_url` property was removed from the stored state for replicas (SOLR-12182). If you're able to upgrade SolrJ to 8.8.x
for all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state
in ZooKeeper with future versions of Solr. However, if you are not able to upgrade SolrJ to 8.8.x for all client applications,
then leave the default `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in ZooKeeper.

You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr. After upgrading all nodes in your cluster
to 8.8.0, collections should fully recover. Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.

=== Solr 8.7

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote87[8.7 Release Notes^]
for an overview of the main new features of Solr 8.7.

When upgrading to 8.7.x users should be aware of the following major changes from 8.6.

*Autoscaling*

* If upgrading from **8.6.0**, please see the <<Solr 8.6.1,8.6.1 Upgrade notes>> below for information on performance degradations introduced in 8.6.0 that require some intervention to resolve.
If you are already on 8.6.1 or higher, you can ignore these instructions.

*Deprecations*

* The autoscaling framework is now formally deprecated and will be removed in Solr 9.0.
The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released. Deprecations include: autoscaling policy, triggers, `withCollection` support, simulation framework, autoscaling suggestions tab in the UI, `autoAddReplicas` and `UTILIZENODE` command.

* Similarly, rule-based replica placement strategy has been deprecated and will be replaced
  in Solr 9.0 by APIs for replica placement and cluster events, with plugin-based implementations.

* Support for detecting spinning disks will be removed in Lucene 9.0 (LUCENE-9576). Corresponding
`spins` metrics in Solr are deprecated and will be removed in Solr 9.0.

*Legacy Scaling (non-SolrCloud) Terminology Updated*

* Solr has replaced the terms "master" and "slave" in the codebase and all documentation with "leader" and "follower".
+
This impacts the functionality described in the section <<legacy-scaling-and-distribution.adoc#,Legacy Scaling and Distribution>>.
This functionality has only changed in terms of parameter names changed, and we do not expect any back-compatibility issues on upgrade to 8.7 or even 9.0 later.
+
However, users should update their `solrconfig.xml` files after completing the upgrade on all nodes of a cluster.
Comparing your configuration to the updated configuration examples in <<index-replication.adoc#,Index Replication>> will show examples of what needs to change, but here are the main changes:
+
. On the replication leader, in the definition of the `/replication` request handler:
.. Replace "master" with "leader".
.. Replace "slave" with "follower" if the former term is used in the name of any follower `solrconfig.xml` file definitions.
This file can be named anything, so you can change it to whatever you'd like to call it if you'd like.
.. Replace "slave" with "follower" if the former term is used in a replication repeater configuration.
. On the replication follower, in the definition of the `/replication` request handler:
.. Replace "masterUrl" with "leaderUrl".
.. Replace "slave" with "follower" if the former term is used in a repeater configuration.

*JSON Facets*

* Performance enhancements for the `relatedness()` statistics function are included with 8.7.
These yield the highest benefits with high-cardinality fields and can be disabled if working with lower cardinality fields with a new `sweep_collection` parameter.
See the section <<json-facet-api.adoc#relatedness-options,relatedness() Options>> for details.

*solr.in.sh / solr.in.cmd*

* Solr has relied on the `SOLR_STOP_WAIT` parameter defined in `solr.in.sh` or `solr.in.cmd` to determine how long to wait on _startup_. A new parameter `SOLR_START_WAIT` allows defining how long Solr should wait for start up to complete.
+
If the time set by this parameter is exceeded, Solr will exit the startup process and return the last few lines of the `solr.log` file to the terminal.
+
By default, this parameter is set to the same value as `SOLR_STOP_WAIT`.

* The default ZooKeeper client timeout (`ZK_CLIENT_TIMEOUT`) is now 30 seconds (`30000` milliseconds) instead of 15.

*Configsets*

* It's now possible to overwrite an existing configset when uploading changes by supplying the `overwrite=true` parameter to the <<configsets-api.adoc#configsets-upload,Configset API>>.
+
A related parameter is `cleanup=true`, which allows deleting any files from the old configset that are left behind after the overwrite.
+
The default for both of these parameters is `false`.

* When deleting a collection that has an automatically created configset (i.e., the configset was copied from the `_default` collection when the collection was created), the configset will also be deleted if it is not in use by any other collection.

*Logging*

* A request ID (`rid`) is now logged for all distributed search requests (in SolrCloud) which can be used to correlate query events across the system.
A parameter `disableRequestId=true` can be added to disable this if desired.

=== Solr 8.6.1

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote861[8.6.1 Release Notes^]
for an overview of the fixes included in Solr 8.6.1.

When upgrading to 8.6.1 users should be aware of the following major changes from 8.6.0.

*Autoscaling*

* As mentioned in the 8.6 upgrade notes, a default autoscaling policy was provided starting in 8.6.0.
This default autoscaling policy resulted in increasingly slow collection creation calls in large clusters (50+ collections).
+
In 8.6.1 the default autoscaling policy has been removed, and clusters will not use autoscaling unless a policy has explicitly been created.
If your cluster is running 8.6.0 and *not using an explicit autoscaling policy*, upgrade to 8.6.1 and remove the default cluster policy and preferences via the following command.
+
Replace `localhost:8983` with your Solr endpoint.
+
[source,text]
curl -X POST -H 'Content-type:application/json'  -d '{set-cluster-policy : [], set-cluster-preferences : []}' http://localhost:8983/api/cluster/autoscaling
+
For more details on autoscaling policies, see the section <<solrcloud-autoscaling-overview.adoc#cluster-policy,Cluster Policy>>.
+
This information is only relevant for users upgrading from 8.6.0. If upgrading from an earlier version to 8.6.1+, this warning can be ignored.

=== Solr 8.6

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote86[8.6 Release Notes^]
for an overview of the main new features of Solr 8.6.

When upgrading to 8.6.x users should be aware of the following major changes from 8.5.

*Support for Block-Max WAND*

Lucene added support for Block-Max WAND in 8.0, and 8.6 makes this available for Solr also.

This can provide significant performance enhancements by not calculating the score for results which are not likely to appear in the top set of results.

It is enabled when using a new query parameter `minExactCount`.
This parameter tells Solr to accurately count the number of hits accurately until at least this value.
Once this value is reached, Solr can skip over documents that don't have a score high enough to be in the top set of documents, which has the potential for greatly speeding up searches.

It's important to note that when using this parameter, the hit count of searches may not be accurate.
It is guaranteed that the hit count is accurate up to the value of `minExactCount`, but any returned hit count higher than that may be an approximation.

A new boolean attribute `numFoundExact` is included in all responses to indicate if the hit count in the response is expected to be exact or not.

More information about this new feature is available in the section <<common-query-parameters.adoc#minexactcount-parameter,minExactCount Parameter>>.

*Autoscaling*

* **NOTE: The default autoscaling policy has been removed as of 8.6.1**
+
Solr now includes a default autoscaling policy.
This policy can be overridden with your custom rules or by specifying an empty policy to replace the default.
+
For details of the default policy, see the section <<solrcloud-autoscaling-overview.adoc#cluster-policy,Cluster Policy>>.

* The <<solrcloud-autoscaling-trigger-actions.adoc#compute-plan-action,ComputePlan action>> now supports a collection selector to identify collections based on collection properties to determine which collections should be operated on.

*Security*

* Prior to Solr 8.6 Solr APIs which take a file system location, such as core creation, backup, restore, and others, did not validate the path and Solr would allow any absolute or relative path.
Starting in 8.6 only paths that are relative to `SOLR_HOME`, `SOLR_DATA_HOME` and `coreRootDir` are allowed by default.
+
If you need to create a core or store a backup outside the default paths, you will need to tell Solr which paths to allow.
A new element in `solr.xml` called `allowPaths` takes a comma-separated list of allowed paths.
+
When using the `solr.xml` file that ships with 8.6, you can configure the list of paths to allow through the system property `solr.allowPaths`.
Please see `bin/solr.in.sh` or `bin\solr.in.cmd` for example usage.
Using the value `*` will allow any path as in earlier versions.
+
For more on this, see the section <<format-of-solr-xml.adoc#the-solr-element,Solr.xml Parameters>>.
+
Windows SMB shares on the UNC format, such as `\\myhost\myshare\mypath` are now always disallowed.
Please use drive letter mounts instead, i.e., `S:\mypath`.

* A new authorization plugin `ExternalRoleRuleBasedAuthorizationPlugin` is now available.
This plugin allows an authentication plugin (such as JWT) to supply a user's roles instead of maintaining a user-to-role mapping inside Solr.

* When authentication is enabled, the Admin UI Dashboard (main screen) now includes a panel that shows the authentication and authorization plugins in use, the logged in username, and the roles assigned to this user.
A new link will also appear in the left-hand navigation to allow a user to log out.

*Streaming Expressions*

* The `/export` handler now supports streaming expressions to allow limiting the output of the export to only matching documents.
+
For more information about how to use this, see the section <<exporting-result-sets.adoc#specifying-the-local-streaming-expression,Specifying the Local Streaming Expression>>.

* The `stats`, `facet`, and `timeseries` expressions now support percentiles and standard deviation aggregations.

*Highlighting*

For the Unified Highlighter: The setting `hl.fragsizeIsMinimum` now defaults to `false` because `true` was found to be a significant performance regression when highlighting lots of text.
This will yield longer highlights on average compared to Solr 8.5 but relatively unchanged compared to previous releases.
Furthermore, if your application highlights lots of text, you may want to experiment with lowering `hl.fragAlignRatio` to trade ideal fragment alignment for better performance.

*Deprecations*

A primary focus of the community is improving Solr's stability and supportability.
With the addition of the package manager system in 8.4, we now have the ability to move some features into plugins maintained by third-parties with the expertise to properly develop and support them.
Our goal is to make running Solr easier and less prone to outages and other headaches.
In this spirit, the following features have been deprecated and are slated to be removed in Solr 9.0.

* Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0.
This feature is unlikely to be replaced by an identical plugin.
However, the community is working on figuring out a replacement feature for disaster recovery and failover.

* The Data Import Handler (DIH) is deprecated and is scheduled to be removed in 9.0.
Work to replace DIH with a community-supported plugin is underway and may be available soon.

* Support to store indexes and backups in HDFS is deprecated and is scheduled to be removed in 9.0.
A community-supported version of this may be available as a plugin in the future.
For more details, please see https://issues.apache.org/jira/browse/SOLR-14021[SOLR-14021^].

Users interested in maintaining a feature as a plugin are encouraged to join the https://solr.apache.org/community.html#mailing-lists-chat[developer mailing list^] to find out more about how to help.

=== Solr 8.5

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote85[8.5 Release Notes^]
for an overview of the main new features of Solr 8.5.

When upgrading to 8.5.x users should be aware of the following major changes from 8.4.

__Note: an index incompatibility warning was retroactively added below to 8.4 for users choosing a non-default postings format (e.g., "FST50").__

*Considerations for a SolrCloud Upgrade*

Solr 8.5 introduces a change in the format used for the elements in the Overseer queues and maps (see https://issues.apache.org/jira/browse/SOLR-14095[SOLR-14095] for technical discussion of the change). This queue is used internally by the Overseer to reliably handle
operations, to communicate operation results between the Overseer and the coordinator node, and by the REQUESTSTATUS API for displaying information about async Collection operations.

This change won’t require you to change any client-side code you should see no differences on the client side.
However, it does require some care when upgrading an existing SolrCloud cluster depending on your upgrade strategy.

If you are upgrading Solr with an atomic restart strategy:

* If you don’t use async or REQUESTSTATUS operations, you should be able to restart and not see any issues.
* If you do use Collection API operations:
. Pause Collection API operations.
. Cleanup queues (See the section <<collections-api.adoc#deletestatus,DELETESTATUS>> for examples)
if you use async operations.
. Upgrade and restart the nodes.
. Resume all normal operations.

If you are upgrading Solr with a rolling restart strategy:

* If you don’t use Collection API operations, you should be able to do a rolling restart and not see
any issues.
* If you do use Collection API operations, but you can pause their use during the restart the easiest
way is to:
. Pause Collection API operations.
. Upgrade and restart all nodes.
. Cleanup queues (See the section <<collections-api.adoc#deletestatus,DELETESTATUS>> for examples)
if you use async operations.
. Resume all normal operations.

If you use Collection API operations and can’t pause them during the upgrade:

. Start 8.5 nodes with the system property: `-Dsolr.useUnsafeOverseerResponse=deserialization`. Ensure the
Overseer node is upgraded last.
. Once all nodes are in 8.5 and once you don’t need to read old status anymore, restart again removing the
system property.

If you prefer to keep the old (but insecure) serialization strategy, you can start your nodes using the system
property: `-Dsolr.useUnsafeOverseerResponse=true`. Keep in mind that this will be removed in future version of Solr.

*Security Manager*

Solr now has the ability to run with a Java security manager enabled. To enable this, set the property `SOLR_SECURITY_MANAGER_ENABLED=true` in `solr.in.sh` or `solr.in.cmd`. Note that if you are using HDFS to store indexes, you cannot enable the security manager.

In Solr 9.0, this will be the default.

See also the section <<securing-solr.adoc#enable-security-manager,Enable Security Manager>>.

*Block/Allow Specific IPs*

Solr has two new parameters to allow you to restrict access to Solr using IP addresses. Use `SOLR_IP_WHITELIST` to configure a whitelist, and `SOLR_IP_BLACKLIST` to configure a blacklist. These properties are defined in `solr.in.sh` or `solr.in.cmd`.

See also the section <<securing-solr.adoc#enable-ip-access-control,Enable IP Access Control>>.

*BlockJoin Facet Deprecation*

The BlockJoinFacetComponent is marked for deprecation and will be removed in 9.0.
Users are encouraged to migrate to `uniqueBlock()` in JSON Facet API.
More information about this is available in the section <<json-faceting-domain-changes.adoc#block-join-domain-changes,Block Join Domain Changes>>.

*Caching with the Boolean Query Parser*

By default, the <<other-parsers.adoc#boolean-query-parser,Boolean Query Parser>> caches queries in Solr's filterCache. It's now possible to disable this with the local param `cache=false`.

*Indexing Log Files*

Solr now includes a command line tool, `bin/postlogs` which will index Solr's log files into a collection.
This provides an easy way to use Solr or visualization tools (such as Zeppelin) to troubleshoot problems with the system.

This tool is not yet officially documented in the Reference Guide, but draft documentation is available in a branch and can be accessed https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc[via GitHub^].

*Highlighting*

Solr's Unified Highlighter now has two parameters to help control passage sizing, `hl.fragAlignRatio` and `hl.fragsizeIsMinimum`.
See the section <<highlighting.adoc#the-unified-highlighter,The Unified Highlighter>> for details about these new parameters.
Regardless of the settings, the passages may be sized differently than before.
_Warning: These default settings were found to be a significant performance regression for apps that highlight lots of text with the default sentence break iterator.
See the 8.6 upgrade notes for advise you can apply in 8.5._

*Shared Library System Parameter*

Solr's `solr.xml` file has long had support for a `sharedLib` parameter, which allows you to define a common location for .jar files that may need to be in the path for all cores.

This property can now be defined in `solr.in.sh` or `solr.in.cmd` as a system property (`-Dsolr.sharedLib=/path/to/lib`) added to `SOLR_OPTS` (see `solr.in.sh` or `solr.in.cmd` for details).

=== Solr 8.4

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote84[8.4 Release Notes^]
for an overview of the main new features of Solr 8.4.

When upgrading to 8.4.x users should be aware of the following major changes from 8.3.

*Reminder:*  If you set the `postingsFormat` or `docValuesFormat` in the schema in order to use a non-default option, you risk preventing yourself from upgrading your Lucene/Solr software at future versions.
Multiple non-default postings formats changed in 8.4, thus rendering the index data from a previous index.
This includes "FST50" which was recommended by the Solr TaggerHandler for performance reasons.
There is now improved documentation to navigate this trade-off choice.

*Package Management System*

Version 8.4 introduces a package management system to Solr. The goals of the
system are to allow hot (live) deployment of plugins, provide packaging
guidelines for plugins, and standardize Solr's approach by following familiar
concepts used in other package management systems.

The system is designed to eventually replace use of the `<lib ../>` directive,
the Blob Store, and other methods of deploying plugins and custom components
to Solr.

The system is currently considered experimental, so use with caution. It must
be enabled with a system parameter passed at start up before it can be used.
For details, please see the section <<package-manager.adoc#,Package Management>>.

With this feature Solr's <<adding-custom-plugins-in-solrcloud-mode.adoc#,Blob Store>>
functionality is now deprecated and will likely be removed in 9.0.

*Security*

The follow mix of changes were all made with the intention of making Solr more secure out of the box.

* The `solrconfig.xml` file in Solr's `_default` configset has been trimmed of
 the following previously pre-configured items:
+
** All `<lib .../>` directives. This means that Solr Cell (aka Tika), Learning
to Rank, Clustering (with Carrot2), language identification, and Velocity (for
the `/browse` sample search interface) are no longer enabled out of the box.
** The `/browse`, `/tvrh`, and `/update/extract` request handlers.
** The Term Vector Component.
** The XSLT and Velocity response writers.
+
All of these items can be added to your Solr implementation by manually editing
`solrconfig.xml` to add them back in, or use the <<config-api.adoc#,Config API>>.
+
The `sample_techproducts_configs` and the examples found in `./example` are unchanged.

* Configsets that have been uploaded with an unsecured Configset API (i.e., when authentication is not enabled) are considered "Untrusted Configsets".
+
In order to bolster Solr's out-of-the-box security, these untrusted configsets
are no longer allowed to use the `<lib .../>` directive to implement contribs
or custom Jars.
+
When upgrading to 8.4, if you are using untrusted configsets that contain `<lib ../>`
directives, their corresponding collections will not load (they will cease to
work). You have a few options in this case:

** You can secure your Solr instance with <<authentication-and-authorization-plugins.adoc#,authentication>>
and re-upload the configset (using the `bin/solr zk upconfig ...`
<<solr-control-script-reference.adoc#,Solr CLI>> command);
** You can put your custom Jars in Solr's classpath instead of `lib` directories;
** You can try the new package management system to manage your custom Jars.
+
See the section <<configsets-api.adoc#configsets-upload,Upload a Configset>>
 for more details about trusted vs. untrusted configsets.

* Our default Jetty configuration has been updated to now set a
Content-Security-Policy (CSP) by default. See `./server/etc/jetty.xml` for
details about how it is configured.
+
As a result of this change, any custom HTML served by Solr's HTTP server that contains inline Javascript will no longer execute in modern browsers. The options for you are:

** Change your JavaScript code to not run inline any longer;
** Edit `jetty.xml` to remove CSP (creating weaker security protection);
** Remove/alter the headers with a reverse proxy.

* Solr's Blob Store and runtime libs functionality are now deprecated and are planned to be removed from Solr in version 9.0. It has been replaced with the new package management system.

* The Velocity response writer is also now deprecated and is planned to be removed from Solr in version 9.0.

*Using Collapse with Group Disallowed*

Using the <<collapse-and-expand-results.adoc#,CollapsingQueryParser>>
with <<result-grouping.adoc#,Result Grouping>> has never been
supported as it causes inconsistent behavior and NullPointerException errors.
We have now explicitly disallowed this combination to prevent these errors.
If you are using these together, you will need to modify your queries.

*SolrJ*

* SolrJ now supports the `shards.preference` parameter for single-shard
scenarios to ensure multi-shard and single-shard request routing works in the same way.
+
See <<using-solrj.adoc#cloud-request-routing,Cloud Request Routing>> and
<<distributed-requests.adoc#shards-preference-parameter,shards.preference Parameter>> for details.

* `QueryResponse.getExplainMap()` type has changed from `Map<String, String>` to `Map<String, Object>` in order to support structured explanations.
+
This change is expected to be mostly back-compatible. Compiled third-party
components will work the same due to type erasure, but source code changes may
be required.

* Replica routing code has been moved to SolrJ, making those classes available
to clients if necessary.

*Streaming Expressions*

* A new DBSCAN clustering streaming evaluator has been added.

* The `precision` stream evaluator can now operate on matrices.

* The `random` streaming expression can now create the x-axis.

*JSON Facets*

* Two new aggregations have been added: `missing` and `countvals`.

* Several aggregations now support multi-valued fields: `min`, `max`, `avg`, `sum`, `sumsq`, `stddev`, `variance`, and `percentile`.

*Caches*

* After the addition of `CaffeineCache` in 8.3, legacy SolrCache
implementations are deprecated and likely to be removed in 9.0.
+
Users are encouraged to transition their cache configurations to use
`org.apache.solr.search.CaffeineCache` as soon as feasible.

=== Solr 8.3

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote83[8.3 Release Notes^] for an overview of the main new features of Solr 8.3.

When upgrading to 8.3.x users should be aware of the following major changes from 8.2.

*JWT Authentication*

JWT Authentication now supports multiple identity providers.
To allow this, the parameter `jwkUrl` has been deprecated and replaced with `jwksUrl`.
Implementations using `jwkUrl` will continue to work as normal, but users
 should plan to transition their configurations to use `jwksUrl` instead as
 soon as feasible.

*Caches*

* Solr has a new cache implementation, `CaffeineCache`, which is now recommended over other caches. This cache is expected to generally provide most users lower memory footprint, higher hit ratio, and better multi-threaded performance.
+
Since caching has a direct impact on the performance of your Solr
 implementation, before switching to any new cache implementation in
 production, take care to test for your environment and traffic patterns so
 you fully understand the ramifications of the change.

* A new parameter, `maxIdleTime`, allows automatic eviction of cache items that have not been used in the defined amount of time. This allows the cache to release some memory and should aid those who want or need to fine-tune their caches.

See the section <<query-settings-in-solrconfig.adoc#,Query Settings in SolrConfig>> for more details about these and other cache options and parameters.

=== Solr 8.2

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote82[8.2 Release Notes^] for an overview of the main new features of Solr 8.2.

When upgrading to 8.2.x, users should be aware of the following major changes from v8.1.

*ZooKeeper 3.5.5*

Solr 8.2 updates the version of ZooKeeper included with Solr to v3.5.5.

It is recommended that external ensembles set up to work with Solr also be updated to ZooKeeper 3.5.5.

This ZooKeeper release includes many new security features.
In order for Solr's Admin UI to work with 3.5.5, the `zoo.cfg` file must allow access to ZooKeeper's "four-letter commands".
At a minimum, `ruok`, `conf`, and `mntr` must be enabled, but other commands can optionally be enabled if you choose.
See the section <<setting-up-an-external-zookeeper-ensemble.adoc#configuration-for-a-zookeeper-ensemble,Configuration for a ZooKeeper Ensemble>> for details.

[WARNING]
Until 8.3, https://issues.apache.org/jira/browse/SOLR-13672[SOLR-13672] causes the ZK Status screen in the Admin UI to not be able to report status. This only impacts the UI, ZooKeeper still operates correctly.

*Routed Aliases*

* Routed aliases now use collection properties to identify collections that belong to the alias; prior to 8.2, these aliases used core properties.
+
This is backward-compatible and aliases created with prior versions will
 continue to work. However, new collections will no longer add the
 `routedAliasName` property to the `core.properties` file so any external code
 depending on this location will need to be updated.

// TODO: aliases.adoc still says this is per-core?

* Time-routed aliases now include a `TRA` infix in the collection name, in the pattern `<alias>_TRA_<timestamp>`. +
Collections created with older versions will continue to work.

*Distributed Tracing Support*

This release adds support for tracing requests in Solr. Please review the section <<solr-tracing.adoc#,Distributed Solr Tracing>> for details on how to configure this feature.

=== Solr 8.1

See the https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote810[8.1 Release Notes^] for an overview of the main new features of Solr 8.1.

When upgrading to 8.1.x, users should be aware of the following major changes from v8.0.

*Global maxBooleanClauses Parameter*

* The behavior of the `maxBooleanClauses` parameter has changed to reduce the risk of exponential query expansion when dealing with pathological query strings.
+
A default upper limit of 1024 clauses is now enforced at the node level. This was the default prior to 7.0, and it can be overridden with a new global parameter in `solr.xml`. This limit will be enforced for all queries whether explicitly defined by the user (or client), or created by Solr and Lucene internals.
+
An identical parameter is available in `solrconfig.xml` for limiting the size of queries explicitly defined by the user (or client), but this per-collection limit will still be restricted by the global limit set in `solr.xml`.
+
If your use case demands that you a lot of OR or AND clauses in your queries, upon upgrade to 8.1 you may need to adjust the global `maxBooleanClauses` parameter since between 7.0 and 8.1 the limit was effectively unbounded.
+
For more information about the new parameter, see the section <<format-of-solr-xml.adoc#global-maxbooleanclauses,Format of solr.xml: maxBooleanClauses>>.

*Security*

* JSON Web Tokens (JWT) are now supported for authentication. These allow Solr to assert a user is already authenticated via an external identity provider, such as an OpenID Connect-enabled IdP. For more information, see the section <<jwt-authentication-plugin.adoc#,JWT Authentication Plugin>>.

* A new security plugin for audit logging has been added. A default class `SolrLogAuditLoggerPlugin` is available and configurable in `security.json`. The base class is also extendable for adding custom audit plugins if needed. See the section <<audit-logging.adoc#,Audit Logging>> for more information.

*Collections API*

* The output of the REQUESTSTATUS command in the Collections API will now include internal asynchronous requests (if any) in the "success" or "failed" keys.

* The CREATE command will now return the appropriate status code (4xx, 5xx, etc.) when the command has failed. Previously, it always returned `0`, even in failure.

* The MODIFYCOLLECTION command now accepts an attribute to set a collection as read-only. This can be used to block a collection from receiving any updates while still allowing queries to be served. See the section <<collection-management.adoc#modifycollection,MODIFYCOLLECTION>> for details on how to use it.

* A new command RENAME allows renaming a collection by setting up a one-to-one alias using the new name. For more information, see the section <<collection-management.adoc#rename,RENAME>>.

* A new command REINDEXCOLLECTION allows indexing existing stored fields from a source collection into a new collection. For more information, please see the section <<collection-management.adoc#reindexcollection,REINDEXCOLLECTION>>.

*Logging*

* The default Log4j2 logging mode has been changed from synchronous to asynchronous. This will improve logging throughput and reduce system contention at the cost of a _slight_ chance that some logging messages may be missed in the event of abnormal Solr termination.
+
If even this slight risk is unacceptable, the Log4j configuration file found in `server/resources/log4j2.xml` has the synchronous logging configuration in a commented section and can be edited to re-enable synchronous logging.

*Metrics*

* The SolrGangliaReporter has been removed from Solr. The metrics library used by Solr, Dropwizard Metrics, was updated to version 4, and Ganglia support was removed from it due to a dependency on the LGPL license.

*Browse UI (Velocity)*

* Velocity and Velocity Tools were both upgraded as part of this release. Velocity upgraded from 1.7 to 2.0. Please see https://velocity.apache.org/engine/2.0/upgrading.html about upgrading. Velocity Tools upgraded from 2.0 to 3.0. For more details, please see https://velocity.apache.org/tools/3.0/upgrading.html for details about the upgrade.

*Default Garbage Collector (GC)*

* Solr's default GC has been changed from CMS to G1. If you prefer to use CMS or any other GC method, you can modify the `GC_TUNE` section of `solr.in.sh` (*nix) or `solr.in.cmd` (Windows).


== Upgrading from 7.x Releases

The upgrade from 7.x to Solr 8.0 introduces several major changes that you should be aware of before upgrading.
These changes are described in the section <<major-changes-in-solr-8.adoc#,Major Changes in Solr 8>>. It's strongly recommended that you do a thorough review of that section before starting your upgrade.

[NOTE]
If you run in SolrCloud mode, you must be on Solr version 7.3 or higher in order to upgrade to 8.x.

== Upgrading from Pre-7.x Versions

Users upgrading from versions of Solr prior to 7.x are strongly encouraged to consult {solr-javadocs}/changes/Changes.html[`CHANGES.txt`] for the details of _all_ changes since the version they are upgrading from.

The upgrade from Solr 6.x to Solr 7.0 introduced several *major* changes that you should be aware of before upgrading. Please do a thorough review of the section <<major-changes-in-solr-7.adoc#,Major Changes in Solr 7>> before starting your upgrade.

A summary of the significant changes between Solr 5.x and Solr 6.0 is in the section <<major-changes-from-solr-5-to-solr-6.adoc#,Major Changes from Solr 5 to Solr 6>>.