This article is the setup process of IoTDB Cluster (1.0.0).
JDK>=1.8.
Max open file 65535.
Disable the swap memory.
Download the binary distribution from website Download Page.
Download the source code:
git clone https://github.com/apache/iotdb.git
The default branch is master, you should checkout to the release tag:
git checkout v1.0.0
Under the source root folder:
mvn clean package -pl distribution -am -DskipTests
Then you will get the binary distribution under distribution/target, in which the all-bin contains ConfigNode and DataNode, and DataNode contains the Cli.
| Folder | Description |
|---|---|
| conf | Configuration files folder, contains configuration files of ConfigNode and DataNode |
| data | Data files folder, contains data files of ConfigNode and DataNode |
| lib | Jar files folder |
| licenses | Licenses files folder |
| logs | Logs files folder, contains logs files of ConfigNode and DataNode |
| sbin | Shell files folder, contains start/stop/remove shell of ConfigNode and DataNode, cli shell |
| tools | System tools |
Users could start a cluster which contains multiple ConfigNode and DataNode. A cluster need at least one ConfigNode and no less than the number of data/schema_replication_factor DataNodes.
The total process are three steps:
Please set the important parameters in conf/iotdb-confignode.properties and conf/iotdb-common.properties:
iotdb-confignode.properties:
| Configuration | Description |
|---|---|
| cn_internal_address | Internal rpc service address of ConfigNode |
| cn_internal_port | Internal rpc service port of ConfigNode |
| cn_consensus_port | ConfigNode replication consensus protocol communication port |
| cn_target_config_node_list | Target ConfigNode address, if the current ConfigNode is the first one, then set its own address:port |
iotdb-common.properties:
| Configuration | Description |
|---|---|
| data_replication_factor | Data replication factor, no more than DataNode number |
| data_region_consensus_protocol_class | Consensus protocol of data replicas. Note that RatisConsensus currently does not support multiple data directories |
| schema_replication_factor | Schema replication factor, no more than DataNode number |
| schema_region_consensus_protocol_class | Consensus protocol of schema replicas |
Start on Linux:
# Foreground bash ./sbin/start-confignode.sh # Background nohup bash ./sbin/start-confignode.sh >/dev/null 2>&1 &
Start on Windows:
sbin\start-confignode.bat
More details ConfigNode Configurations.
This will add the replication factor of ConfigNode, except for the ports that couldn't conflict with, make sure other configurations are the same with existing ConfigNode in Cluster, and set parameter cn_target_config_nodes_list as an active ConfigNode in Cluster.
The adding ConfigNode also use the start-confignode.sh/bat.
You could add any number of DataNode.
Please set the important parameters in iotdb-datanode.properties:
| Configuration | Description |
|---|---|
| dn_rpc_address | Client RPC Service address |
| dn_rpc_port | Client RPC Service port |
| dn_internal_address | Control flow address of DataNode inside cluster |
| dn_internal_port | Control flow port of DataNode inside cluster |
| dn_mpp_data_exchange_port | Data flow port of DataNode inside cluster |
| dn_data_region_consensus_port | Data replicas communication port for consensus |
| dn_schema_region_consensus_port | Schema replicas communication port for consensus |
| dn_target_config_node_list | Running ConfigNode of the Cluster |
Start on Linux:
# Foreground bash ./sbin/start-datanode.sh # Background nohup bash ./sbin/start-datanode.sh >/dev/null 2>&1 &
Start on Windows:
sbin\start-datanode.bat
More details are in DataNode Configurations.
When you meet problem, and want to stop IoTDB ConfigNode and DataNode directly, our shells can help you do this.
In Windows:
sbin\stop-datanode.bat
sbin\stop-confignode.bat
In Linux:
bash sbin/stop-datanode.sh
bash sbin/stop-confignode.sh
Be careful not to miss the “sudo” label, because some port info‘s acquisition may require root authority. If you can’t sudo, just use “jps” or “ps aux | grep iotdb” to get the process's id, then use “kill -9 ” to stop the process.
If you just want to setup your IoTDB locally, You can quickly init 1C1D (i.e. 1 Confignode and 1 Datanode) environment by our shells.
This will work well if you don't change our default settings.
Start on Windows:
sbin\start-standalone.bat
Start on Linux:
bash sbin/start-standalone.sh
Besides, with our shell, you can also directly kill these processes.
Stop on Windows:
sbin\stop-standalone.bat
Stop on Linux:
bash sbin/stop-standalone.sh
Note: On Linux, the 1C1D processes both launches in the background, and you can see the logs for details.
The stop-standalone.sh may not work well without sudo, since IoTDB's port numbers may be invisible without permission. If stop-standalone.sh meets some error, you can use “jps” or “ps aux | grep iotdb” to obtain the process ids, and use “sudo kill -9 ” to manually stop the processes.
Cli shell is in sbin folder.
Start on Linux:
./datanode/sbin/start-cli.sh
Start on Windows:
datanode\sbin\start-cli.bat
Execute the remove-confignode shell on an active ConfigNode, and make sure that there is at least one active ConfigNode in Cluster after removing.
Remove on Linux:
# Remove the ConfigNode with confignode_id ./confignode/sbin/remove-confignode.sh <confignode_id> # Remove the ConfigNode with address:port ./confignode/sbin/remove-confignode.sh <internal_address>:<internal_port>
Remove on Windows:
# Remove the ConfigNode with confignode_id confignode\sbin\remove-confignode.bat <confignode_id> # Remove the ConfigNode with address:port confignode\sbin\remove-confignode.bat <internal_address>:<internal_port>
Execute the remove-datanode shell on an active DataNode, and make sure that the number of active DataNodes are no less than the number of data/schema_replication_factor in Cluster after removing.
Remove on Linux:
# Remove the DataNode with datanode_id bash ./datanode/sbin/remove-datanode.sh <datanode_id> # Remove the DataNode with rpc address:port bash ./datanode/sbin/remove-datanode.sh <rpc_address>:<rpc_port>
Remove on Windows:
# Remove the DataNode with datanode_id datanode\sbin\remove-datanode.bat <datanode_id> # Remove the DataNode with rpc address:port datanode\sbin\remove-datanode.bat <rpc_address>:<rpc_port>
This section uses a local environment as an example to illustrate how to start, expand, and shrink a IoTDB Cluster.
Unzip the apache-iotdb-1.0.0-all-bin.zip file to cluster0 folder.
Start the Cluster version with one ConfigNode and one DataNode(1C1D), and the default number of replicas is one.
./cluster0/sbin/start-confignode.sh ./cluster0/sbin/start-datanode.sh
./cluster0/sbin/start-cli.sh
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 127.0.0.1| 22277| | 1| DataNode|Running| 127.0.0.1| 9003| +------+----------+-------+---------------+------------+ Total line number = 2 It costs 0.160s
Unzip the apache-iotdb-1.0.0-all-bin.zip file to cluster1 and cluster2 folder.
For folder cluster1:
| configuration item | value |
|---|---|
| cn_internal_address | 127.0.0.1 |
| cn_internal_port | 22279 |
| cn_consensus_port | 22280 |
| cn_target_config_node_list | 127.0.0.1:22277 |
| configuration item | value |
|---|---|
| dn_rpc_address | 127.0.0.1 |
| dn_rpc_port | 6668 |
| dn_internal_address | 127.0.0.1 |
| dn_internal_port | 9004 |
| dn_mpp_data_exchange_port | 8778 |
| dn_data_region_consensus_port | 40011 |
| dn_schema_region_consensus_port | 50011 |
| dn_target_config_node_list | 127.0.0.1:22277 |
For folder cluster2:
| configuration item | value |
|---|---|
| cn_internal_address | 127.0.0.1 |
| cn_internal_port | 22281 |
| cn_consensus_port | 22282 |
| cn_target_config_node_list | 127.0.0.1:22277 |
| configuration item | value |
|---|---|
| dn_rpc_address | 127.0.0.1 |
| dn_rpc_port | 6669 |
| dn_internal_address | 127.0.0.1 |
| dn_internal_port | 9005 |
| dn_mpp_data_exchange_port | 8779 |
| dn_data_region_consensus_port | 40012 |
| dn_schema_region_consensus_port | 50012 |
| dn_target_config_node_list | 127.0.0.1:22277 |
Expanding the Cluster to three ConfigNode and three DataNode(3C3D). The following commands can be executed in arbitrary order.
./cluster1/sbin/start-confignode.sh ./cluster1/sbin/start-datanode.sh ./cluster2/sbin/start-confignode.sh ./cluster2/sbin/start-datanode.sh
Execute the show cluster command, then the result is shown below:
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 127.0.0.1| 22277| | 2|ConfigNode|Running| 127.0.0.1| 22279| | 3|ConfigNode|Running| 127.0.0.1| 22281| | 1| DataNode|Running| 127.0.0.1| 9003| | 4| DataNode|Running| 127.0.0.1| 9004| | 5| DataNode|Running| 127.0.0.1| 9005| +------+----------+-------+---------------+------------+ Total line number = 6 It costs 0.012s
./cluster0/sbin/remove-confignode.sh 127.0.0.1:22279
./cluster0/sbin/remove-datanode.sh 127.0.0.1:6668
Execute the show cluster command, then the result is shown below:
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 127.0.0.1| 22277| | 3|ConfigNode|Running| 127.0.0.1| 22281| | 1| DataNode|Running| 127.0.0.1| 9003| | 5| DataNode|Running| 127.0.0.1| 9005| +------+----------+-------+---------------+------------+ Total line number = 4 It costs 0.007s