| = Taking Solr to Production |
| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| This section provides guidance on how to setup Solr to run in production on *nix platforms, such as Ubuntu. Specifically, we’ll walk through the process of setting up to run a single Solr instance on a Linux host and then provide tips on how to support multiple Solr nodes running on the same host. |
| |
| == Service Installation Script |
| |
| Solr includes a service installation script (`bin/install_solr_service.sh`) to help you install Solr as a service on Linux. Currently, the script only supports CentOS, Debian, Red Hat, SUSE and Ubuntu Linux distributions. Before running the script, you need to determine a few parameters about your setup. Specifically, you need to decide where to install Solr and which system user should be the owner of the Solr files and process. |
| |
| === Planning Your Directory Structure |
| |
| We recommend separating your live Solr files, such as logs and index files, from the files included in the Solr distribution bundle, as that makes it easier to upgrade Solr and is considered a good practice to follow as a system administrator. |
| |
| ==== Solr Installation Directory |
| |
| By default, the service installation script will extract the distribution archive into `/opt`. You can change this location using the `-i` option when running the installation script. The script will also create a symbolic link to the versioned directory of Solr. For instance, if you run the installation script for Solr {solr-docs-version}.0, then the following directory structure will be used: |
| |
| [source,plain,subs="attributes"] |
| ---- |
| /opt/solr-{solr-docs-version}.0 |
| /opt/solr -> /opt/solr-{solr-docs-version}.0 |
| ---- |
| |
| Using a symbolic link insulates any scripts from being dependent on the specific Solr version. If, down the road, you need to upgrade to a later version of Solr, you can just update the symbolic link to point to the upgraded version of Solr. We’ll use `/opt/solr` to refer to the Solr installation directory in the remaining sections of this page. |
| |
| ==== Separate Directory for Writable Files |
| |
| You should also separate writable Solr files into a different directory; by default, the installation script uses `/var/solr`, but you can override this location using the `-d` option. With this approach, the files in `/opt/solr` will remain untouched and all files that change while Solr is running will live under `/var/solr`. |
| |
| === Create the Solr User |
| |
| Running Solr as `root` is not recommended for security reasons, and the <<solr-control-script-reference.adoc#,control script>> start command will refuse to do so. Consequently, you should determine the username of a system user that will own all of the Solr files and the running Solr process. By default, the installation script will create the *solr* user, but you can override this setting using the -u option. If your organization has specific requirements for creating new user accounts, then you should create the user before running the script. The installation script will make the Solr user the owner of the `/opt/solr` and `/var/solr` directories. |
| |
| You are now ready to run the installation script. |
| |
| === Run the Solr Installation Script |
| |
| To run the script, you'll need to download the latest Solr distribution archive and then do the following: |
| |
| [source,bash,subs="attributes"] |
| ---- |
| tar xzf solr-{solr-docs-version}.0.tgz solr-{solr-docs-version}.0/bin/install_solr_service.sh --strip-components=2 |
| ---- |
| |
| The previous command extracts the `install_solr_service.sh` script from the archive into the current directory. If installing on Red Hat, please make sure *lsof* is installed before running the Solr installation script (`sudo yum install lsof`). The installation script must be run as root: |
| |
| [source,bash,subs="attributes"] |
| ---- |
| sudo bash ./install_solr_service.sh solr-{solr-docs-version}.0.tgz |
| ---- |
| |
| By default, the script extracts the distribution archive into `/opt`, configures Solr to write files into `/var/solr`, and runs Solr as the `solr` user. Consequently, the following command produces the same result as the previous command: |
| |
| [source,bash,subs="attributes"] |
| ---- |
| sudo bash ./install_solr_service.sh solr-{solr-docs-version}.0.tgz -i /opt -d /var/solr -u solr -s solr -p 8983 |
| ---- |
| |
| You can customize the service name, installation directories, port, and owner using options passed to the installation script. To see available options, simply do: |
| |
| [source,bash] |
| ---- |
| sudo bash ./install_solr_service.sh -help |
| ---- |
| |
| Once the script completes, Solr will be installed as a service and running in the background on your server (on port 8983). To verify, you can do: |
| |
| [source,bash] |
| ---- |
| sudo service solr status |
| ---- |
| |
| If you do not want to start the service immediately, pass the `-n` option. You can then start the service manually later, e.g., after completing the configuration setup. |
| |
| We'll cover some additional configuration settings you can make to fine-tune your Solr setup in a moment. Before moving on, let's take a closer look at the steps performed by the installation script. This gives you a better overview and will help you understand important details about your Solr installation when reading other pages in this guide; such as when a page refers to Solr home, you'll know exactly where that is on your system. |
| |
| ==== Solr Home Directory |
| |
| The Solr home directory (not to be confused with the Solr installation directory) is where Solr manages core directories with index files. By default, the installation script uses `/var/solr/data`. If the `-d` option is used on the install script, then this will change to the `data` subdirectory in the location given to the -d option. Take a moment to inspect the contents of the Solr home directory on your system. If you do not <<using-zookeeper-to-manage-configuration-files.adoc#,store `solr.xml` in ZooKeeper>>, the home directory must contain a `solr.xml` file. When Solr starts up, the Solr Control Script passes the location of the home directory using the `-Dsolr.solr.home=...` system property. |
| |
| ==== Environment Overrides Include File |
| |
| The service installation script creates an environment specific include file that overrides defaults used by the `bin/solr` script. The main advantage of using an include file is that it provides a single location where all of your environment-specific overrides are defined. Take a moment to inspect the contents of the `/etc/default/solr.in.sh` file, which is the default path setup by the installation script. If you used the `-s` option on the install script to change the name of the service, then the first part of the filename will be different. For a service named `solr-demo`, the file will be named `/etc/default/solr-demo.in.sh`. There are many settings that you can override using this file. However, at a minimum, this script needs to define the `SOLR_PID_DIR` and `SOLR_HOME` variables, such as: |
| |
| [source,bash] |
| ---- |
| SOLR_PID_DIR=/var/solr |
| SOLR_HOME=/var/solr/data |
| ---- |
| |
| The `SOLR_PID_DIR` variable sets the directory where the <<solr-control-script-reference.adoc#,control script>> will write out a file containing the Solr server’s process ID. |
| |
| ==== Log Settings |
| |
| Solr uses Apache Log4J for logging. The installation script copies `/opt/solr/server/resources/log4j2.xml` to `/var/solr/log4j2.xml`. Take a moment to verify that the Solr include file is configured to send logs to the correct location by checking the following settings in `/etc/default/solr.in.sh`: |
| |
| [source,bash] |
| ---- |
| LOG4J_PROPS=/var/solr/log4j2.xml |
| SOLR_LOGS_DIR=/var/solr/logs |
| ---- |
| |
| For more information about Log4J configuration, please see: <<configuring-logging.adoc#,Configuring Logging>> |
| |
| ==== init.d Script |
| |
| When running a service like Solr on Linux, it’s common to setup an init.d script so that system administrators can control Solr using the service tool, such as: `service solr start`. The installation script creates a very basic init.d script to help you get started. Take a moment to inspect the `/etc/init.d/solr` file, which is the default script name setup by the installation script. If you used the `-s` option on the install script to change the name of the service, then the filename will be different. Notice that the following variables are setup for your environment based on the parameters passed to the installation script: |
| |
| [source,bash] |
| ---- |
| SOLR_INSTALL_DIR=/opt/solr |
| SOLR_ENV=/etc/default/solr.in.sh |
| RUNAS=solr |
| ---- |
| |
| The `SOLR_INSTALL_DIR` and `SOLR_ENV` variables should be self-explanatory. The `RUNAS` variable sets the owner of the Solr process, such as `solr`; if you don’t set this value, the script will run Solr as **root**, which is not recommended for production. You can use the `/etc/init.d/solr` script to start Solr by doing the following as root: |
| |
| [source,bash] |
| ---- |
| service solr start |
| ---- |
| |
| The `/etc/init.d/solr` script also supports the **stop**, **restart**, and *status* commands. Please keep in mind that the init script that ships with Solr is very basic and is intended to show you how to setup Solr as a service. However, it’s also common to use more advanced tools like *supervisord* or *upstart* to control Solr as a service on Linux. While showing how to integrate Solr with tools like supervisord is beyond the scope of this guide, the `init.d/solr` script should provide enough guidance to help you get started. Also, the installation script sets the Solr service to start automatically when the host machine initializes. |
| |
| === Progress Check |
| |
| In the next section, we cover some additional environment settings to help you fine-tune your production setup. However, before we move on, let's review what we've achieved thus far. Specifically, you should be able to control Solr using `/etc/init.d/solr`. Please verify the following commands work with your setup: |
| |
| [source,bash] |
| ---- |
| sudo service solr restart |
| sudo service solr status |
| ---- |
| |
| The status command should give some basic information about the running Solr node that looks similar to: |
| |
| [source,text] |
| ---- |
| Solr process PID running on port 8983 |
| { |
| "version":"5.0.0 - ubuntu - 2014-12-17 19:36:58", |
| "startTime":"2014-12-19T19:25:46.853Z", |
| "uptime":"0 days, 0 hours, 0 minutes, 8 seconds", |
| "memory":"85.4 MB (%17.4) of 490.7 MB"} |
| ---- |
| |
| If the `status` command is not successful, look for error messages in `/var/solr/logs/solr.log`. |
| |
| == Fine-Tune Your Production Setup |
| |
| === Dynamic Defaults for ConcurrentMergeScheduler |
| |
| The Merge Scheduler is configured in `solrconfig.xml` and defaults to `ConcurrentMergeScheduler`. This scheduler uses multiple threads to merge Lucene segments in the background. |
| |
| By default, the `ConcurrentMergeScheduler` auto-detects whether the underlying disk drive is rotational or a SSD and sets defaults for `maxThreadCount` and `maxMergeCount` accordingly. If the disk drive is determined to be rotational then the `maxThreadCount` is set to 1 and `maxMergeCount` is set to 6. Otherwise, `maxThreadCount` is set to 4 or half the number of processors available to the JVM whichever is greater and `maxMergeCount` is set to `maxThreadCount+5`. |
| |
| This auto-detection works only on Linux and even then it is not guaranteed to be correct. On all other platforms, the disk is assumed to be rotational. Therefore, if the auto-detection fails or is incorrect then indexing performance can suffer badly due to the wrong defaults. |
| |
| The auto-detected value is exposed by the <<metrics-reporting.adoc#metrics-api, Metrics API>> with the key `solr.node:CONTAINER.fs.coreRoot.spins`. A value of `true` denotes that the disk is detected to be a rotational or spinning disk. |
| |
| It is safer to explicitly set values for `maxThreadCount` and `maxMergeCount` in the <<indexconfig-in-solrconfig.adoc#mergescheduler, IndexConfig section of SolrConfig.xml>> so that values appropriate to your hardware are used. |
| |
| Alternatively, the boolean system property `lucene.cms.override_spins` can be set in the `SOLR_OPTS` variable in the include file to override the auto-detected value. Similarly, the system property `lucene.cms.override_core_count` can be set to the number of CPU cores to override the auto-detected processor count. |
| |
| === Memory and GC Settings |
| |
| By default, the `bin/solr` script sets the maximum Java heap size to 512M (-Xmx512m), which is fine for getting started with Solr. For production, you’ll want to increase the maximum heap size based on the memory requirements of your search application; values between 8 and 16 gigabytes are not uncommon for production servers. When you need to change the memory settings for your Solr server, use the `SOLR_HEAP` variable in the include file, such as: |
| |
| [source,bash] |
| ---- |
| SOLR_HEAP="8g" |
| ---- |
| |
| [NOTE] |
| ==== |
| Do not allocate a very large Java Heap unless you know you need it. See <<jvm-settings.adoc#choosing-memory-heap-settings,Choosing Memory Heap Settings>> for details. |
| ==== |
| |
| Also, the <<solr-control-script-reference.adoc#,Solr Control Script>> comes with a set of pre-configured Garbage First Garbage Collection settings that have shown to work well with Solr for a number of different workloads. |
| However, these settings may not work well for your specific use of Solr. Consequently, you may need to change the GC settings, which should also be done with the `GC_TUNE` variable in the `/etc/default/solr.in.sh` include file. For more information about garbage collection settings refer to following articles: |
| 1. https://cwiki.apache.org/confluence/display/solr/ShawnHeisey |
| 2. https://www.oracle.com/technetwork/articles/java/g1gc-1984535.html |
| You can also refer to <<jvm-settings.adoc#,JVM Settings>> for tuning your memory and garbage collection settings. |
| |
| ==== Out-of-Memory Shutdown Hook |
| |
| The `bin/solr` script registers the `bin/oom_solr.sh` script to be called by the JVM if an OutOfMemoryError occurs. The `oom_solr.sh` script will issue a `kill -9` to the Solr process that experiences the `OutOfMemoryError`. This behavior is recommended when running in SolrCloud mode so that ZooKeeper is immediately notified that a node has experienced a non-recoverable error. Take a moment to inspect the contents of the `/opt/solr/bin/oom_solr.sh` script so that you are familiar with the actions the script will perform if it is invoked by the JVM. |
| |
| === Going to Production with SolrCloud |
| |
| To run Solr in SolrCloud mode, you need to set the `ZK_HOST` variable in the include file to point to your ZooKeeper ensemble. Running the embedded ZooKeeper is not supported in production environments. For instance, if you have a ZooKeeper ensemble hosted on the following three hosts on the default client port 2181 (zk1, zk2, and zk3), then you would set: |
| |
| [source,bash] |
| ---- |
| ZK_HOST=zk1,zk2,zk3 |
| ---- |
| |
| When the `ZK_HOST` variable is set, Solr will launch in "cloud" mode. |
| |
| ==== ZooKeeper chroot |
| |
| If you're using a ZooKeeper instance that is shared by other systems, it's recommended to isolate the SolrCloud znode tree using ZooKeeper's chroot support. For instance, to ensure all znodes created by SolrCloud are stored under `/solr`, you can put `/solr` on the end of your `ZK_HOST` connection string, such as: |
| |
| [source,bash] |
| ---- |
| ZK_HOST=zk1,zk2,zk3/solr |
| ---- |
| |
| Before using a chroot for the first time, you need to create the root path (znode) in ZooKeeper by using the <<solr-control-script-reference.adoc#,Solr Control Script>>. We can use the mkroot command for that: |
| |
| [source,bash] |
| ---- |
| bin/solr zk mkroot /solr -z <ZK_node>:<ZK_PORT> |
| ---- |
| |
| [NOTE] |
| ==== |
| If you also want to bootstrap ZooKeeper with existing `solr_home`, you can instead use the `zkcli.sh` / `zkcli.bat` `bootstrap` command, which will also create the chroot path if it does not exist. See <<command-line-utilities.adoc#,Command Line Utilities>> for more info. |
| ==== |
| |
| === Solr Hostname |
| |
| Use the `SOLR_HOST` variable in the include file to set the hostname of the Solr server. |
| |
| [source,bash] |
| ---- |
| SOLR_HOST=solr1.example.com |
| ---- |
| |
| Setting the hostname of the Solr server is recommended, especially when running in SolrCloud mode, as this determines the address of the node when it registers with ZooKeeper. |
| |
| === Environment Banner in Admin UI |
| |
| To guard against accidentally doing changes to the wrong cluster, you may configure a visual indication in the Admin UI of whether you currently work with a production environment or not. To do this, edit your `solr.in.sh` or `solr.in.cmd` file with a `-Dsolr.environment=prod` setting, or set the cluster property named `environment`. To specify label and/or color, use a comma-delimited format as below. The `+` character can be used instead of space to avoid quoting. Colors may be valid CSS colors or numeric, e.g., `#ff0000` for bright red. Examples of valid environment configs: |
| |
| * `prod` |
| * `test,label=Functional+test` |
| * `dev,label=MyDev,color=blue` |
| * `dev,color=blue` |
| |
| === Override Settings in solrconfig.xml |
| |
| Solr allows configuration properties to be overridden using Java system properties passed at startup using the `-Dproperty=value` syntax. For instance, in `solrconfig.xml`, the default auto soft commit settings are set to: |
| |
| [source,xml] |
| ---- |
| <autoSoftCommit> |
| <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime> |
| </autoSoftCommit> |
| ---- |
| |
| In general, whenever you see a property in a Solr configuration file that uses the `${solr.PROPERTY:DEFAULT_VALUE}` syntax, then you know it can be overridden using a Java system property. For instance, to set the maxTime for soft-commits to be 10 seconds, then you can start Solr with `-Dsolr.autoSoftCommit.maxTime=10000`, such as: |
| |
| [source,bash] |
| ---- |
| bin/solr start -Dsolr.autoSoftCommit.maxTime=10000 |
| ---- |
| |
| The `bin/solr` script simply passes options starting with `-D` on to the JVM during startup. For running in production, we recommend setting these properties in the `SOLR_OPTS` variable defined in the include file. Keeping with our soft-commit example, in `/etc/default/solr.in.sh`, you would do: |
| |
| [source,bash] |
| ---- |
| SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=10000" |
| ---- |
| |
| === Ulimit Settings (*nix Operating Systems) |
| |
| There are several settings that should be monitored and set as high as possible, "unlimited" by preference. On most "*nix" operating systems, you can see the current values by typing the following at a command prompt. |
| |
| [source,bash] |
| ---- |
| ulimit -a |
| ---- |
| |
| These four settings in particular are important to have set very high, unlimited if possible. |
| |
| * max processes (`ulimit -u`): 65,000 is the recommended _minimum_. |
| * file handles (`ulimit -n`): 65,000 is the recommended _minimum_. All the files used by all replicas have their file handles open at once so this can grow quite large. |
| * virtual memory (`ulimit -v`): Set to unlimited. This is used to by MMapping the indexes. |
| * max memory size (`ulimit -m`): Also used by MMap, set to unlimited. |
| * If your system supports it, `sysctl vm.max_map_count`, should be set to unlimited as well. |
| |
| We strongly recommend that these settings be permanently raised. The exact process to permanently raise them will vary per operating system. Some systems require editing configuration files and restarting your server. Consult your system administrators for guidance in your particular environment. |
| |
| [WARNING] |
| ==== |
| Check these limits every time you upgrade your kernel or operating system. These operations can reset these to their defaults. |
| ==== |
| |
| [WARNING] |
| ==== |
| If these limits are exceeded, the problems reported by Solr vary depending on the specific operation responsible for exceeding the limit. Errors such as "too many open files", "connection error", and "max processes exceeded" have been reported, as well as SolrCloud recovery failures. |
| ==== |
| |
| === Avoid Swapping (*nix Operating Systems) |
| |
| When running a Java application like Lucene/Solr, having the OS swap memory to disk is a very bad situation. We usually prefer a hard crash so other healthy Solr nodes can take over, instead of letting a Solr node swap, causing terrible performance, timeouts and an unstable system. So our recommendation is to disable swap on the host altogether or reduce the "swappiness". These instructions are valid for Linux environments. Also note that when running Solr in a Docker container, these changes must be applied to the *host*. |
| |
| ==== Disabling Swap |
| |
| To disable swap on a Linux system temporarily, you can run the `swapoff` command: |
| |
| [source,bash] |
| ---- |
| sudo swapoff -a |
| ---- |
| |
| If you want to make this setting permanent, first make sure you have more than enough physical RAM on your host and then consult the documentation for your Linux system for the correct procedure to disable swap. |
| |
| ==== Reduce Swappiness |
| |
| An alternative option is to reduce the "swappiness" of your system. A Linux system will by default be quite aggressive in swapping out memory to disk, by having a high default "swappiness" value. By reducing this to a very low value, it will have almost the same effect as using `swapoff`, but Linux will still be allowed to swap in case of emergency. To check the current swappiness setting, run: |
| |
| [source,bash] |
| ---- |
| cat /proc/sys/vm/swappiness |
| ---- |
| |
| Next, to change the setting permanently, open `/etc/sysctl.conf` as the root user. Then, change or add this line to the file: |
| |
| [source,bash] |
| ---- |
| vm.swappiness = 1 |
| ---- |
| |
| Alternatively, you can change the setting temporarily by running this comamnd: |
| |
| [source,bash] |
| ---- |
| echo 1 > /proc/sys/vm/swappiness |
| ---- |
| |
| == Running Multiple Solr Nodes per Host |
| |
| The `bin/solr` script is capable of running multiple instances on one machine, but for a *typical* installation, this is not a recommended setup. Extra CPU and memory resources are required for each additional instance. A single instance is easily capable of handling multiple indexes. |
| |
| .When to ignore the recommendation |
| [NOTE] |
| ==== |
| For every recommendation, there are exceptions. For the recommendation above, that exception is mostly applicable when discussing extreme scalability. The best reason for running multiple Solr nodes on one host is decreasing the need for extremely large heaps. |
| |
| When the Java heap gets very large, it can result in extremely long garbage collection pauses, even with the GC tuning that the startup script provides by default. The exact point at which the heap is considered "very large" will vary depending on how Solr is used. This means that there is no hard number that can be given as a threshold, but if your heap is reaching the neighborhood of 16 to 32 gigabytes, it might be time to consider splitting nodes. Ideally this would mean more machines, but budget constraints might make that impossible. |
| |
| There is another issue once the heap reaches 32GB. Below 32GB, Java is able to use compressed pointers, but above that point, larger pointers are required, which uses more memory and slows down the JVM. |
| |
| Because of the potential garbage collection issues and the particular issues that happen at 32GB, if a single instance would require a 64GB heap, performance is likely to improve greatly if the machine is set up with two nodes that each have a 31GB heap. |
| ==== |
| |
| If your use case requires multiple instances, at a minimum you will need unique Solr home directories for each node you want to run; ideally, each home should be on a different physical disk so that multiple Solr nodes don’t have to compete with each other when accessing files on disk. Having different Solr home directories implies that you’ll need a different include file for each node. Moreover, if using the `/etc/init.d/solr` script to control Solr as a service, then you’ll need a separate script for each node. The easiest approach is to use the service installation script to add multiple services on the same host, such as: |
| |
| [source,bash,subs="attributes"] |
| ---- |
| sudo bash ./install_solr_service.sh solr-{solr-docs-version}.0.tgz -s solr2 -p 8984 |
| ---- |
| |
| The command shown above will add a service named `solr2` running on port 8984 using `/var/solr2` for writable (aka "live") files; the second server will still be owned and run by the `solr` user and will use the Solr distribution files in `/opt`. After installing the solr2 service, verify it works correctly by doing: |
| |
| [source,bash] |
| ---- |
| sudo service solr2 restart |
| sudo service solr2 status |
| ---- |
