| <!-- |
| |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| --> |
| |
| # 1. Purpose |
| |
| This document describes how to install and start IoTDB Cluster (1.0.0). |
| |
| # 2. Prerequisites |
| |
| 1. JDK>=1.8. |
| 2. Max open file 65535. |
| 3. Disable the swap memory. |
| 4. Ensure that data/confignode directory has been cleared when starting ConfigNode for the first time, |
| and data/datanode directory has been cleared when starting DataNode for the first time |
| 5. Turn off the firewall of the server if the entire cluster is in a trusted environment. |
| 6. By default, IoTDB Cluster will use ports 10710, 10720 for the ConfigNode and |
| 6667, 10730, 10740, 10750 and 10760 for the DataNode. |
| Please make sure those ports are not occupied, or you will modify the ports in configuration files. |
| |
| # 3. Get the Installation Package |
| |
| You can either download the binary release files (see Chap 3.1) or compile with source code (see Chap 3.2). |
| |
| ## 3.1 Download the binary distribution |
| |
| 1. Open our website [Download Page](https://iotdb.apache.org/Download/). |
| 2. Download the binary distribution. |
| 3. Decompress to get the apache-iotdb-1.0.0-all-bin directory. |
| |
| ## 3.2 Compile with source code |
| |
| ### 3.2.1 Download the source code |
| |
| **Git** |
| ``` |
| git clone https://github.com/apache/iotdb.git |
| git checkout v1.0.0 |
| ``` |
| |
| **Website** |
| 1. Open our website [Download Page](https://iotdb.apache.org/Download/). |
| 2. Download the source code. |
| 3. Decompress to get the apache-iotdb-1.0.0 directory. |
| |
| ### 3.2.2 Compile source code |
| |
| Under the source root folder: |
| ``` |
| mvn clean package -pl distribution -am -DskipTests |
| ``` |
| |
| Then you will get the binary distribution under |
| **distribution/target/apache-iotdb-1.0.0-SNAPSHOT-all-bin/apache-iotdb-1.0.0-SNAPSHOT-all-bin**. |
| |
| # 4. Binary Distribution Content |
| |
| | **Folder** | **Description** | |
| |-------------------------|---------------------------------------------------------------------------------------------------| |
| | conf | Configuration files folder, contains configuration files of ConfigNode, DataNode, JMX and logback | |
| | data | Data files folder, contains data files of ConfigNode and DataNode | |
| | lib | Jar files folder | |
| | licenses | Licenses files folder | |
| | logs | Logs files folder, contains logs files of ConfigNode and DataNode | |
| | sbin | Shell files folder, contains start/stop/remove shell of ConfigNode and DataNode, cli shell | |
| | tools | System tools | |
| |
| # 5. Cluster Installation and Configuration |
| |
| ## 5.1 Cluster Installation |
| |
| `apache-iotdb-1.0.0-SNAPSHOT-all-bin` contains both the ConfigNode and the DataNode. |
| Please deploy the files to all servers of your target cluster. |
| A best practice is deploying the files into the same directory in all servers. |
| |
| If you want to try the cluster mode on one server, please read |
| [Cluster Quick Start](https://iotdb.apache.org/UserGuide/Master/QuickStart/ClusterQuickStart.html). |
| |
| ## 5.2 Cluster Configuration |
| |
| We need to modify the configurations on each server. |
| Therefore, login each server and switch the working directory to `apache-iotdb-1.0.0-SNAPSHOT-all-bin`. |
| The configuration files are stored in the `./conf` directory. |
| |
| For all ConfigNode servers, we need to modify the common configuration (see Chap 5.2.1) |
| and ConfigNode configuration (see Chap 5.2.2). |
| |
| For all DataNode servers, we need to modify the common configuration (see Chap 5.2.1) |
| and DataNode configuration (see Chap 5.2.3). |
| |
| ### 5.2.1 Common configuration |
| |
| Open the common configuration file ./conf/iotdb-common.properties, |
| and set the following parameters base on the |
| [Deployment Recommendation](https://iotdb.apache.org/UserGuide/Master/Cluster/Deployment-Recommendation.html): |
| |
| | **Configuration** | **Description** | **Default** | |
| |--------------------------------------------|--------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| |
| | cluster\_name | Cluster name for which the Node to join in | defaultCluster | |
| | config\_node\_consensus\_protocol\_class | Consensus protocol of ConfigNode | org.apache.iotdb.consensus.ratis.RatisConsensus | |
| | schema\_replication\_factor | Schema replication factor, no more than DataNode number | 1 | |
| | schema\_region\_consensus\_protocol\_class | Consensus protocol of schema replicas | org.apache.iotdb.consensus.ratis.RatisConsensus | |
| | data\_replication\_factor | Data replication factor, no more than DataNode number | 1 | |
| | data\_region\_consensus\_protocol\_class | Consensus protocol of data replicas. Note that RatisConsensus currently does not support multiple data directories | org.apache.iotdb.consensus.iot.IoTConsensus | |
| |
| **Notice: The preceding configuration parameters cannot be changed after the cluster is started. Ensure that the common configurations of all Nodes are the same. Otherwise, the Nodes cannot be started.** |
| |
| ### 5.2.2 ConfigNode configuration |
| |
| Open the ConfigNode configuration file ./conf/iotdb-confignode.properties, |
| and set the following parameters based on the IP address and available port of the server or VM: |
| |
| | **Configuration** | **Description** | **Default** | **Usage** | |
| |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | cn\_internal\_address | Internal rpc service address of ConfigNode | 127.0.0.1 | Set to the IPV4 address or domain name of the server | |
| | cn\_internal\_port | Internal rpc service port of ConfigNode | 10710 | Set to any unoccupied port | |
| | cn\_consensus\_port | ConfigNode replication consensus protocol communication port | 10720 | Set to any unoccupied port | |
| | cn\_target\_config\_node\_list | ConfigNode address to which the node is connected when it is registered to the cluster. Note that Only one ConfigNode can be configured. | 127.0.0.1:10710 | For Seed-ConfigNode, set to its own cn\_internal\_address:cn\_internal\_port; For other ConfigNodes, set to other one running ConfigNode's cn\_internal\_address:cn\_internal\_port | |
| |
| **Notice: The preceding configuration parameters cannot be changed after the node is started. Ensure that all ports are not occupied. Otherwise, the Node cannot be started.** |
| |
| ### 5.2.3 DataNode configuration |
| |
| Open the DataNode configuration file ./conf/iotdb-datanode.properties, |
| and set the following parameters based on the IP address and available port of the server or VM: |
| |
| | **Configuration** | **Description** | **Default** | **Usage** | |
| |-------------------------------------|--------------------------------------------------------|-----------------|---------------------------------------------------------------------------------------------------------------------------------------| |
| | dn\_rpc\_address | Client RPC Service address | 127.0.0.1 | Set to the IPV4 address or domain name of the server | |
| | dn\_rpc\_port | Client RPC Service port | 6667 | Set to any unoccupied port | |
| | dn\_internal\_address | Control flow address of DataNode inside cluster | 127.0.0.1 | Set to the IPV4 address or domain name of the server | |
| | dn\_internal\_port | Control flow port of DataNode inside cluster | 10730 | Set to any unoccupied port | |
| | dn\_mpp\_data\_exchange\_port | Data flow port of DataNode inside cluster | 10740 | Set to any unoccupied port | |
| | dn\_data\_region\_consensus\_port | Data replicas communication port for consensus | 10750 | Set to any unoccupied port | |
| | dn\_schema\_region\_consensus\_port | Schema replicas communication port for consensus | 10760 | Set to any unoccupied port | |
| | dn\_target\_config\_node\_list | Running ConfigNode of the Cluster | 127.0.0.1:10710 | Set to any running ConfigNode's cn\_internal\_address:cn\_internal\_port. You can set multiple values, separate them with commas(",") | |
| |
| **Notice: The preceding configuration parameters cannot be changed after the node is started. Ensure that all ports are not occupied. Otherwise, the Node cannot be started.** |
| |
| # 6. Cluster Operation |
| |
| ## 6.1 Starting the cluster |
| |
| This section describes how to start a cluster that includes several ConfigNodes and DataNodes. |
| The cluster can provide services only by starting at least one ConfigNode |
| and no less than the number of data/schema_replication_factor DataNodes. |
| |
| The total process are three steps: |
| |
| * Start the Seed-ConfigNode |
| * Add ConfigNode (Optional) |
| * Add DataNode |
| |
| ### 6.1.1 Start the Seed-ConfigNode |
| |
| **The first Node started in the cluster must be ConfigNode. The first started ConfigNode must follow the tutorial in this section.** |
| |
| The first ConfigNode to start is the Seed-ConfigNode, which marks the creation of the new cluster. |
| Before start the Seed-ConfigNode, please open the common configuration file ./conf/iotdb-common.properties and check the following parameters: |
| |
| | **Configuration** | **Check** | |
| |--------------------------------------------|-------------------------------------------------| |
| | cluster\_name | Is set to the expected name | |
| | config\_node\_consensus\_protocol\_class | Is set to the expected consensus protocol | |
| | schema\_replication\_factor | Is set to the expected schema replication count | |
| | schema\_region\_consensus\_protocol\_class | Is set to the expected consensus protocol | |
| | data\_replication\_factor | Is set to the expected data replication count | |
| | data\_region\_consensus\_protocol\_class | Is set to the expected consensus protocol | |
| |
| **Notice:** Please set these parameters carefully based on the [Deployment Recommendation](https://iotdb.apache.org/UserGuide/Master/Cluster/Deployment-Recommendation.html). |
| These parameters are not modifiable after the Node first startup. |
| |
| Then open its configuration file ./conf/iotdb-confignode.properties and check the following parameters: |
| |
| | **Configuration** | **Check** | |
| |--------------------------------|-----------------------------------------------------------------------------------------------------| |
| | cn\_internal\_address | Is set to the IPV4 address or domain name of the server | |
| | cn\_internal\_port | The port isn't occupied | |
| | cn\_consensus\_port | The port isn't occupied | |
| | cn\_target\_config\_node\_list | Is set to its own internal communication address, which is cn\_internal\_address:cn\_internal\_port | |
| |
| After checking, you can run the startup script on the server: |
| |
| ``` |
| # Linux foreground |
| bash ./sbin/start-confignode.sh |
| |
| # Linux background |
| nohup bash ./sbin/start-confignode.sh >/dev/null 2>&1 & |
| |
| # Windows |
| .\sbin\start-confignode.bat |
| ``` |
| |
| For more details about other configuration parameters of ConfigNode, see the |
| [ConfigNode Configurations](https://iotdb.apache.org/UserGuide/Master/Reference/ConfigNode-Config-Manual.html). |
| |
| ### 6.1.2 Add more ConfigNodes (Optional) |
| |
| **The ConfigNode who isn't the first one started must follow the tutorial in this section.** |
| |
| You can add more ConfigNodes to the cluster to ensure high availability of ConfigNodes. |
| A common configuration is to add extra two ConfigNodes to make the cluster has three ConfigNodes. |
| |
| Ensure that all configuration parameters in the ./conf/iotdb-common.properites are the same as those in the Seed-ConfigNode; |
| otherwise, it may fail to start or generate runtime errors. |
| Therefore, please check the following parameters in common configuration file: |
| |
| | **Configuration** | **Check** | |
| |--------------------------------------------|----------------------------------------| |
| | cluster\_name | Is consistent with the Seed-ConfigNode | |
| | config\_node\_consensus\_protocol\_class | Is consistent with the Seed-ConfigNode | |
| | schema\_replication\_factor | Is consistent with the Seed-ConfigNode | |
| | schema\_region\_consensus\_protocol\_class | Is consistent with the Seed-ConfigNode | |
| | data\_replication\_factor | Is consistent with the Seed-ConfigNode | |
| | data\_region\_consensus\_protocol\_class | Is consistent with the Seed-ConfigNode | |
| |
| Then, please open its configuration file ./conf/iotdb-confignode.properties and check the following parameters: |
| |
| | **Configuration** | **Check** | |
| |--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | cn\_internal\_address | Is set to the IPV4 address or domain name of the server | |
| | cn\_internal\_port | The port isn't occupied | |
| | cn\_consensus\_port | The port isn't occupied | |
| | cn\_target\_config\_node\_list | Is set to the internal communication address of an other running ConfigNode. The internal communication address of the seed ConfigNode is recommended. | |
| |
| After checking, you can run the startup script on the server: |
| |
| ``` |
| # Linux foreground |
| bash ./sbin/start-confignode.sh |
| |
| # Linux background |
| nohup bash ./sbin/start-confignode.sh >/dev/null 2>&1 & |
| |
| # Windows |
| .\sbin\start-confignode.bat |
| ``` |
| |
| For more details about other configuration parameters of ConfigNode, see the |
| [ConfigNode Configurations](https://iotdb.apache.org/UserGuide/Master/Reference/ConfigNode-Config-Manual.html). |
| |
| ### 6.1.3 Start DataNode |
| |
| **Before adding DataNodes, ensure that there exists at least one ConfigNode is running in the cluster.** |
| |
| You can add any number of DataNodes to the cluster. |
| Before adding a new DataNode, |
| |
| please open its common configuration file ./conf/iotdb-common.properties and check the following parameters: |
| |
| | **Configuration** | **Check** | |
| |--------------------------------------------|----------------------------------------| |
| | cluster\_name | Is consistent with the Seed-ConfigNode | |
| |
| Then open its configuration file ./conf/iotdb-datanode.properties and check the following parameters: |
| |
| | **Configuration** | **Check** | |
| |-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | dn\_rpc\_address | Is set to the IPV4 address or domain name of the server | |
| | dn\_rpc\_port | The port isn't occupied | |
| | dn\_internal\_address | Is set to the IPV4 address or domain name of the server | |
| | dn\_internal\_port | The port isn't occupied | |
| | dn\_mpp\_data\_exchange\_port | The port isn't occupied | |
| | dn\_data\_region\_consensus\_port | The port isn't occupied | |
| | dn\_schema\_region\_consensus\_port | The port isn't occupied | |
| | dn\_target\_config\_node\_list | Is set to the internal communication address of other running ConfigNodes. The internal communication address of the seed ConfigNode is recommended. | |
| |
| After checking, you can run the startup script on the server: |
| |
| ``` |
| # Linux foreground |
| bash ./sbin/start-datanode.sh |
| |
| # Linux background |
| nohup bash ./sbin/start-datanode.sh >/dev/null 2>&1 & |
| |
| # Windows |
| .\sbin\start-datanode.bat |
| ``` |
| |
| For more details about other configuration parameters of DataNode, see the |
| [DataNode Configurations](https://iotdb.apache.org/UserGuide/Master/Reference/DataNode-Config-Manual.html). |
| |
| **Notice: The cluster can provide services only if the number of its DataNodes is no less than the number of replicas(max{schema\_replication\_factor, data\_replication\_factor}).** |
| |
| ## 6.2 Start Cli |
| |
| If the cluster is in local environment, you can directly run the Cli startup script in the ./sbin directory: |
| |
| ``` |
| # Linux |
| ./sbin/start-cli.sh |
| |
| # Windows |
| .\sbin\start-cli.bat |
| ``` |
| |
| If you want to use the Cli to connect to a cluster in the production environment, |
| Please read the [Cli manual](https://iotdb.apache.org/UserGuide/Master/QuickStart/Command-Line-Interface.html). |
| |
| ## 6.3 Verify Cluster |
| |
| Use a 3C3D(3 ConfigNodes and 3 DataNodes) as an example. |
| Assumed that the IP addresses of the 3 ConfigNodes are 192.168.1.10, 192.168.1.11 and 192.168.1.12, and the default ports 10710 and 10720 are used. |
| Assumed that the IP addresses of the 3 DataNodes are 192.168.1.20, 192.168.1.21 and 192.168.1.22, and the default ports 6667, 10730, 10740, 10750 and 10760 are used. |
| |
| After starting the cluster successfully according to chapter 6.1, you can run the `show cluster details` command on the Cli, and you will see the following results: |
| |
| ``` |
| IoTDB> show cluster details |
| +------+----------+-------+---------------+------------+-------------------+------------+-------+-------+-------------------+-----------------+ |
| |NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort| RpcAddress|RpcPort|MppPort|SchemaConsensusPort|DataConsensusPort| |
| +------+----------+-------+---------------+------------+-------------------+------------+-------+-------+-------------------+-----------------+ |
| | 0|ConfigNode|Running| 192.168.1.10| 10710| 10720| | | | | | |
| | 2|ConfigNode|Running| 192.168.1.11| 10710| 10720| | | | | | |
| | 3|ConfigNode|Running| 192.168.1.12| 10710| 10720| | | | | | |
| | 1| DataNode|Running| 192.168.1.20| 10730| |192.168.1.20| 6667| 10740| 10750| 10760| |
| | 4| DataNode|Running| 192.168.1.21| 10730| |192.168.1.21| 6667| 10740| 10750| 10760| |
| | 5| DataNode|Running| 192.168.1.22| 10730| |192.168.1.22| 6667| 10740| 10750| 10760| |
| +------+----------+-------+---------------+------------+-------------------+------------+-------+-------+-------------------+-----------------+ |
| Total line number = 6 |
| It costs 0.012s |
| ``` |
| |
| If the status of all Nodes is **Running**, the cluster deployment is successful. |
| Otherwise, read the run logs of the Node that fails to start and |
| check the corresponding configuration parameters. |
| |
| ## 6.4 Stop IoTDB |
| |
| This section describes how to manually shut down the ConfigNode or DataNode process of the IoTDB. |
| |
| ### 6.4.1 Stop ConfigNode by script |
| |
| Run the stop ConfigNode script: |
| |
| ``` |
| # Linux |
| ./sbin/stop-confignode.sh |
| |
| # Windows |
| .\sbin\stop-confignode.bat |
| ``` |
| |
| ### 6.4.2 Stop DataNode by script |
| |
| Run the stop DataNode script: |
| |
| ``` |
| # Linux |
| ./sbin/stop-datanode.sh |
| |
| # Windows |
| .\sbin\stop-datanode.bat |
| ``` |
| |
| ### 6.4.3 Kill Node process |
| |
| Get the process number of the Node: |
| |
| ``` |
| jps |
| |
| # or |
| |
| ps aux | grep iotdb |
| ``` |
| |
| Kill the process： |
| |
| ``` |
| kill -9 <pid> |
| ``` |
| |
| **Notice Some ports require root access, in which case use sudo** |
| |
| ## 6.5 Shrink the Cluster |
| |
| This section describes how to remove ConfigNode or DataNode from the cluster. |
| |
| ### 6.5.1 Remove ConfigNode |
| |
| Before removing a ConfigNode, ensure that there is at least one active ConfigNode in the cluster after the removal. |
| Run the remove-confignode script on an active ConfigNode: |
| |
| ``` |
| # Linux |
| # Remove the ConfigNode with confignode_id |
| ./sbin/remove-confignode.sh <confignode_id> |
| |
| # Remove the ConfigNode with address:port |
| ./sbin/remove-confignode.sh <cn_internal_address>:<cn_internal_port> |
| |
| |
| # Windows |
| # Remove the ConfigNode with confignode_id |
| .\sbin\remove-confignode.bat <confignode_id> |
| |
| # Remove the ConfigNode with address:port |
| .\sbin\remove-confignode.bat <cn_internal_address>:<cn_internal_portcn_internal_port> |
| ``` |
| |
| ### 6.5.2 Remove DataNode |
| |
| Before removing a DataNode, ensure that the cluster has at least the number of data/schema replicas DataNodes. |
| Run the remove-datanode script on an active DataNode: |
| |
| ``` |
| # Linux |
| # Remove the DataNode with datanode_id |
| ./sbin/remove-datanode.sh <datanode_id> |
| |
| # Remove the DataNode with rpc address:port |
| ./sbin/remove-datanode.sh <dn_rpc_address>:<dn_rpc_port> |
| |
| |
| # Windows |
| # Remove the DataNode with datanode_id |
| .\sbin\remove-datanode.bat <datanode_id> |
| |
| # Remove the DataNode with rpc address:port |
| .\sbin\remove-datanode.bat <dn_rpc_address>:<dn_rpc_port> |
| ``` |
| |
| # 7. FAQ |
| |
| See [FAQ](https://iotdb.apache.org/UserGuide/Master/FAQ/FAQ-for-cluster-setup.html) |
