| .. Licensed under the Apache License, Version 2.0 (the "License"); you may not |
| .. use this file except in compliance with the License. You may obtain a copy of |
| .. the License at |
| .. |
| .. http://www.apache.org/licenses/LICENSE-2.0 |
| .. |
| .. Unless required by applicable law or agreed to in writing, software |
| .. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT |
| .. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the |
| .. License for the specific language governing permissions and limitations under |
| .. the License. |
| |
| .. _setup/cluster: |
| |
| ============== |
| Cluster Set Up |
| ============== |
| |
| This section describes everything you need to know to prepare, install, and |
| set up your first CouchDB 2.x cluster. |
| |
| Ports and Firewalls |
| =================== |
| |
| CouchDB uses the following ports: |
| |
| +-------------+----------+-----------------------+----------------------+ |
| | Port Number | Protocol | Recommended binding | Usage | |
| +=============+==========+=======================+======================+ |
| | 5984 | tcp | As desired, by | Standard clustered | |
| | | | default ``localhost`` | port for all HTTP | |
| | | | | API requests | |
| +-------------+----------+-----------------------+----------------------+ |
| | 5986 | tcp | ``localhost`` or | Administrative tasks | |
| | | | private network | such as node and | |
| | | | **ONLY** | shard management | |
| +-------------+----------+-----------------------+----------------------+ |
| | 4369 | tcp | All interfaces | Erlang port mapper | |
| | | | by default | daemon (epmd) | |
| +-------------+----------+-----------------------+----------------------+ |
| | Random | tcp | Automatic | Communication with | |
| | above 1024 | | | other CouchDB nodes | |
| | (see below) | | | in the cluster | |
| +-------------+----------+-----------------------+----------------------+ |
| |
| CouchDB in clustered mode uses the port ``5984``, just as in a standalone |
| configuration, but it also uses ``5986`` for node-local APIs. These APIs are |
| administrative tools only, such as node and shard management. Do not use |
| port ``5986`` for any other reason. The port is slated to be deprecated in a |
| future CouchDB release. |
| |
| .. warning:: |
| **Never expose the node-local port to the public Internet.** |
| |
| By default, CouchDB only exposes port ``5986`` **only** on localhost. |
| If you have a secondary network connection on nodes for management purposes |
| only, it is acceptable to expose the port on that network as well. |
| |
| CouchDB uses Erlang-native clustering functionality to achieve a clustered |
| installation. Erlang uses TCP port ``4369`` (EPMD) to find other nodes, so all |
| servers must be able to speak to each other on this port. In an Erlang cluster, |
| all nodes are connected to all other nodes, in a mesh network configuration. |
| |
| .. warning:: |
| If you expose the port ``4369`` to the Internet or any other untrusted |
| network, then the only thing protecting you is the Erlang |
| `cookie`_. |
| |
| .. _cookie: http://erlang.org/doc/reference_manual/distributed.html |
| |
| Every Erlang application running on that machine (such as CouchDB) then uses |
| automatically assigned ports for communciation with other nodes. Yes, this |
| means random ports. This will obviously not work with a firewall, but it is |
| possible to force an Erlang application to use a specific port range. |
| |
| This documentation will use the range TCP ``9100-9200``, but this range is |
| unnecessarily broad. If you only have a single Erlang application running on a |
| machine, the range can be limited to a single port: ``9100-9100``, since the |
| ports epmd assign are for *inbound connections* only. Three CouchDB nodes |
| running on a single machine, as in a development cluster scenario, would need |
| three ports in this range. |
| |
| Configure and Test the Communication with Erlang |
| ================================================ |
| |
| Make CouchDB use correct IP|FQDN and the open ports |
| ---------------------------------------------------- |
| |
| In file ``etc/vm.args`` change the line ``-name couchdb@127.0.0.1`` to |
| ``-name couchdb@<reachable-ip-address|fully-qualified-domain-name>`` which defines |
| the name of the node. Each node must have an identifier that allows remote |
| systems to talk to it. The node name is of the form |
| ``<name>@<reachable-ip-address|fully-qualified-domain-name>``. |
| |
| The name portion can be couchdb on all nodes, unless you are running more than |
| 1 CouchDB node on the same server with the same IP address or domain name. In |
| that case, we recommend names of ``couchdb1``, ``couchdb2``, etc. |
| |
| The second portion of the node name must be an identifier by which other nodes |
| can access this node -- either the node's fully qualified domain name (FQDN) or |
| the node's IP address. The FQDN is preferred so that you can renumber the node's |
| IP address without disruption to the cluster. (This is common in cloud-hosted |
| environments.) |
| |
| .. warning:: |
| |
| Tricks with ``/etc/hosts`` and ``libresolv`` don't work with Erlang. |
| Either properly set up DNS and use fully-qualified domain names, or |
| use IP addresses. DNS and FQDNs are preferred. |
| |
| Open ``etc/vm.args``, on all nodes, and add ``-kernel inet_dist_listen_min 9100`` |
| and ``-kernel inet_dist_listen_max 9200`` like below: |
| |
| .. code-block:: erlang |
| |
| -name ... |
| -setcookie ... |
| ... |
| -kernel inet_dist_listen_min 9100 |
| -kernel inet_dist_listen_max 9200 |
| |
| Again, a small range is fine, down to a single port (set both to ``9100``) if you |
| only ever run a single CouchDB node on each machine. |
| |
| Confirming connectivity between nodes |
| ------------------------------------- |
| |
| For this test, you need 2 servers with working hostnames. Let us call them |
| server1.test.com and server2.test.com. They reside at ``192.168.0.1`` and |
| ``192.168.0.2``, respectively. |
| |
| On server1.test.com: |
| |
| .. code-block:: bash |
| |
| erl -name bus@192.168.0.1 -setcookie 'brumbrum' -kernel inet_dist_listen_min 9100 -kernel inet_dist_listen_max 9200 |
| |
| Then on server2.test.com: |
| |
| .. code-block:: bash |
| |
| erl -name car@192.168.0.2 -setcookie 'brumbrum' -kernel inet_dist_listen_min 9100 -kernel inet_dist_listen_max 9200 |
| |
| An explanation to the commands: |
| * ``erl`` the Erlang shell. |
| * ``-name bus@192.168.0.1`` the name of the Erlang node and its IP address or FQDN. |
| * ``-setcookie 'brumbrum'`` the "password" used when nodes connect to each |
| other. |
| * ``-kernel inet_dist_listen_min 9100`` the lowest port in the range. |
| * ``-kernel inet_dist_listen_max 9200`` the highest port in the range. |
| |
| This gives us 2 Erlang shells. shell1 on server1, shell2 on server2. |
| Time to connect them. Enter the following, being sure to end the line with a |
| period (``.``): |
| |
| In shell1: |
| |
| .. code-block:: erlang |
| |
| net_kernel:connect_node(car@192.168.0.2). |
| |
| This will connect to the node called ``car`` on the server called |
| ``192.168.0.2``. |
| |
| If that returns true, then you have an Erlang cluster, and the firewalls are |
| open. This means that 2 CouchDB nodes on these two servers will be able to |
| communicate with each other successfully. If you get false or nothing at all, |
| then you have a problem with the firewall, DNS, or your settings. Try again. |
| |
| If you're concerned about firewall issues, or having trouble connecting all |
| nodes of your cluster later on, repeat the above test between all pairs of |
| servers to confirm connectivity and system configuration is correct. |
| |
| .. _cluster/setup/prepare: |
| |
| Preparing CouchDB nodes to be joined into a cluster |
| =================================================== |
| |
| Before you can add nodes to form a cluster, you must have them listening on an |
| IP address accessible from the other nodes in the cluster. You should also ensure |
| that a few critical settings are identical across all nodes before joining them. |
| |
| The settings we recommend you set now, before joining the nodes into a cluster, |
| are: |
| |
| 1. ``etc/vm.args`` settings as described in the |
| :ref:`previous two sections<setup/cluster>` |
| 2. At least one :ref:`server administrator<config/admins>` |
| user (and password) |
| 3. Bind the node's clustered interface (port ``5984``) to a reachable IP address |
| 4. A consistent :config:option:`UUID <couchdb/uuid>`. The UUID is used in identifying |
| the cluster when replicating. If this value is not consistent across all nodes |
| in the cluster, replications may be forced to rewind the changes feed to zero, |
| leading to excessive memory, CPU and network use. |
| 5. A consistent :config:option:`httpd secret <couch_httpd_auth/secret>`. The secret |
| is used in calculating and evaluating cookie and proxy authentication, and should |
| be set consistently to avoid unnecessary repeated session cookie requests. |
| |
| As of CouchDB 3.0, steps 4 and 5 above are automatically performed for you when |
| using the setup API endpoints described below. |
| |
| If you use a configuration management tool, such as Chef, Ansible, Puppet, etc., |
| then you can place these settings in a ``.ini`` file and distribute them to all |
| nodes ahead of time. Be sure to pre-encrypt the password (cutting and pasting |
| from a test instance is easiest) if you use this route to avoid CouchDB rewriting |
| the file. |
| |
| If you do not use configuration management, or are just experimenting with |
| CouchDB for the first time, use these commands *once per server* to perform |
| steps 2-4 above. Be sure to change the ``password`` to something secure, and |
| again, use the same password on all nodes. You may have to run these commands |
| locally on each node; if so, replace ``<server-IP|FQDN>`` below with ``127.0.0.1``. |
| |
| .. code-block:: bash |
| |
| # First, get two UUIDs to use later on. Be sure to use the SAME UUIDs on all nodes. |
| curl http://<server-IP|FQDN>:5984/_uuids?count=2 |
| |
| # CouchDB will respond with something like: |
| # {"uuids":["60c9e8234dfba3e2fdab04bf92001142","60c9e8234dfba3e2fdab04bf92001cc2"]} |
| # Copy the provided UUIDs into your clipboard or a text editor for later use. |
| # Use the first UUID as the cluster UUID. |
| # Use the second UUID as the cluster shared http secret. |
| |
| # Create the admin user and password: |
| curl -X PUT http://<server-IP|FQDN>:5984/_node/_local/_config/admins/admin -d '"password"' |
| |
| # Now, bind the clustered interface to all IP addresses availble on this machine |
| curl -X PUT http://<server-IP|FQDN>:5984/_node/_local/_config/chttpd/bind_address -d '"0.0.0.0"' |
| |
| # If not using the setup wizard / API endpoint, the following 2 steps are required: |
| # Set the UUID of the node to the first UUID you previously obtained: |
| curl -X PUT http://<server-IP|FQDN>:5984/_node/_local/_config/couchdb/uuid -d '"FIRST-UUID-GOES-HERE"' |
| |
| # Finally, set the shared http secret for cookie creation to the second UUID: |
| curl -X PUT http://<server-IP|FQDN>:5984/_node/_local/_config/couch_httpd_auth/secret -d '"SECOND-UUID-GOES-HERE"' |
| |
| .. _cluster/setup/wizard: |
| |
| The Cluster Setup Wizard |
| ======================== |
| |
| CouchDB 2.x comes with a convenient Cluster Setup Wizard as part of the Fauxton |
| web administration interface. For first-time cluster setup, and for |
| experimentation, this is your best option. |
| |
| It is **strongly recommended** that the minimum number of nodes in a cluster is |
| 3. For more explanation, see the :ref:`Cluster Theory <cluster/theory>` section |
| of this documentation. |
| |
| After installation and initial start-up of all nodes in your cluster, ensuring |
| all nodes are reachable, and the pre-configuration steps listed above, visit |
| Fauxton at ``http://<server1>:5984/_utils#setup``. You will be asked to set up |
| CouchDB as a single-node instance or set up a cluster. |
| |
| When you click "Setup Cluster" you are asked for admin credentials again, and |
| then to add nodes by IP address. To get more nodes, go through the same install |
| procedure on other machines. Be sure to specify the total number of nodes you |
| expect to add to the cluster before adding nodes. |
| |
| Now enter each node's IP address or FQDN in the setup wizard, ensuring you also |
| enter the previously set server admin username and password. |
| |
| Once you have added all nodes, click "Setup" and Fauxton will finish the |
| cluster configuration for you. |
| |
| To check that all nodes have been joined correctly, visit |
| ``http://<server-IP|FQDN>:5984/_membership`` on each node. The returned list |
| should show all of the nodes in your cluster: |
| |
| .. code-block:: javascript |
| |
| { |
| "all_nodes": [ |
| "couchdb@server1.test.com", |
| "couchdb@server2.test.com", |
| "couchdb@server3.test.com" |
| ], |
| "cluster_nodes": [ |
| "couchdb@server1.test.com", |
| "couchdb@server2.test.com", |
| "couchdb@server3.test.com" |
| ] |
| } |
| |
| The ``all_nodes`` section is the list of *expected* nodes; the ``cluster_nodes`` |
| section is the list of *actually connected* nodes. Be sure the two lists match. |
| |
| Now your cluster is ready and available! You can send requests to any one of |
| the nodes, and all three will respond as if you are working with a single |
| CouchDB cluster. |
| |
| For a proper production setup, you'd now set up an HTTP reverse proxy in front |
| of the cluster, for load balancing and SSL termination. We recommend |
| `HAProxy`_, but others can be used. Sample configurations are available in the |
| :ref:`best-practices` section. |
| |
| .. _cluster/setup/api: |
| |
| The Cluster Setup API |
| ===================== |
| |
| If you would prefer to manually configure your CouchDB cluster, CouchDB exposes |
| the ``_cluster_setup`` endpoint for that purpose. After installation and |
| initial setup/config, we can set up the cluster. On each node we need to run |
| the following command to set up the node: |
| |
| .. code-block:: bash |
| |
| curl -X POST -H "Content-Type: application/json" http://admin:password@127.0.0.1:5984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "node_count":"3"}' |
| |
| After that we can join all the nodes together. Choose one node as the "setup |
| coordination node" to run all these commands on. This "setup coordination |
| node" only manages the setup and requires all other nodes to be able to see it |
| and vice versa. *It has no special purpose beyond the setup process; CouchDB |
| does not have the concept of a "master" node in a cluster.* |
| |
| Setup will not work with unavailable nodes. All nodes must be online and properly |
| preconfigured before the cluster setup process can begin. |
| |
| To join a node to the cluster, run these commands for each node you want to add: |
| |
| .. code-block:: bash |
| |
| curl -X POST -H "Content-Type: application/json" http://admin:password@<setup-coordination-node>:5984/_cluster_setup -d '{"action": "enable_cluster", "bind_address":"0.0.0.0", "username": "admin", "password":"password", "port": 5984, "node_count": "3", "remote_node": "<remote-node-ip>", "remote_current_user": "<remote-node-username>", "remote_current_password": "<remote-node-password>" }' |
| curl -X POST -H "Content-Type: application/json" http://admin:password@<setup-coordination-node>:5984/_cluster_setup -d '{"action": "add_node", "host":"<remote-node-ip>", "port": <remote-node-port>, "username": "admin", "password":"password"}' |
| |
| This will join the two nodes together. Keep running the above commands for each |
| node you want to add to the cluster. Once this is done run the following |
| command to complete the cluster setup and add the system databases: |
| |
| .. code-block:: bash |
| |
| curl -X POST -H "Content-Type: application/json" http://admin:password@<setup-coordination-node>:5984/_cluster_setup -d '{"action": "finish_cluster"}' |
| |
| Verify install: |
| |
| .. code-block:: bash |
| |
| curl http://admin:password@<setup-coordination-node>:5984/_cluster_setup |
| |
| Response: |
| |
| .. code-block:: bash |
| |
| {"state":"cluster_finished"} |
| |
| Verify all cluster nodes are connected: |
| |
| .. code-block:: bash |
| |
| curl http://admin:password@<setup-coordination-node>:5984/_membership |
| |
| Response: |
| |
| .. code-block:: bash |
| |
| { |
| "all_nodes": [ |
| "couchdb@couch1.test.com", |
| "couchdb@couch2.test.com", |
| "couchdb@couch3.test.com", |
| ], |
| "cluster_nodes": [ |
| "couchdb@couch1.test.com", |
| "couchdb@couch2.test.com", |
| "couchdb@couch3.test.com", |
| ] |
| } |
| |
| Ensure the ``all_nodes`` and ``cluster_nodes`` lists match. |
| |
| You CouchDB cluster is now set up. |
| |
| .. _HAProxy: http://haproxy.org/ |
| .. _example configuration for HAProxy: https://github.com/apache/couchdb/blob/master/rel/haproxy.cfg |