Cluster Quick Start

The purpose of this article is to show how to start, expand, and shrink an IoTDB cluster in an easy way.

See also: FAQ

1. Installation and deployment

As an example, we'd like to start an IoTDB cluster with 3 ConfigNodes and 3 DataNodes(3C3D) with minimum modifications. Thus,

  • the cluster name is defaultCluster
  • data/schema replica is 1
  • the max heap size of ConfigNode take the 1/4 of the computer
  • the max heap size of DataNode take the 1/4 of the computer

Suppose there are 3 computers(3 nodes we called here) with Linux OS and JDK installed(detail see Prerequisites) and IoTDB working directory is /data/iotdb. IP address and configurations is like below:

Node IP192.168.132.10192.168.132.11192.168.132.12
serviceConfigNodeConfigNodeConfigNode
serviceDataNodeDataNodeDataNode

Port:

ServiceConfigNodeDataNode
port10710, 107206667, 10730, 10740, 10750, 10760

illustration：

  • We could use IP address or hostname/domain to set up an IoTDB cluster, then we would take IP address. If using hostname/domain, /etc/hosts must be set well.
  • JVM memory configuration: MAX_HEAP_SIZE in confignode-env.sh and datanode-env.sh, equal to or greater than 1G is recommended. It's enough for ConfigNode taking 1~2G. The memory taking of DataNode is decided by the inputing and querying data.

1.1 download

In every computer, Download the IoTDB install package and extract it to working directory of /data/iotdb. Then get the directory tree:

/data/iotdb/
├── conf    # configuration files
├── lib     # jar library
├── sbin    # start/stop shell etc.
└── tools   # other tools

1.2. configuration

Configuration files are in /data/iotdb/conf. Modify the specified configuration file according to the table below:

ConfigurationConfiguration OptionIP:192.168.132.10IP:192.168.132.11IP:192.168.132.12
iotdb-confignode.propertiescn_internal_address192.168.132.10192.168.132.11192.168.132.12
cn_target_config_node_list192.168.132.10:10710192.168.132.10:10710192.168.132.10:10710
iotdb-datanode.propertiesdn_rpc_address192.168.132.10192.168.132.11192.168.132.12
dn_internal_address192.168.132.10192.168.132.11192.168.132.12
dn_target_config_node_list192.168.132.10:10710192.168.132.10:10710192.168.132.10:10710

Notice: It's recommended that the configurations of iotdb-common.properties and the heap size of JVM in all nodes are the same.

1.3. start IoTDB cluster

Before starting the IoTDB cluster, make sure the configurations are correct and there is no any data in the working directory.

1.3.1. start the first node

That is cn_target_config_node_list in above configuration table. Execute these commands below in node of 192.168.132.10.

cd /data/iotdb
# start ConfigNode and DataNode services
sbin/start-standalone.sh
    
# check DataNode logs to see whether starting successfully or not
tail -f logs/log_datanode_all.log
# expecting statements like below
# 2023-07-21 20:26:01,881 [main] INFO  o.a.i.db.service.DataNode:192 - Congratulation, IoTDB DataNode is set up successfully. Now, enjoy yourself!

If there is no such logs mentioned abolve or there are some Exceptions in log files, it's failed. Then please check log_confignode_all.log and log_datanode_all.log in directory of /data/iotdb/logs.

Notice：

  • Make sure the first node, especially the first ConfigNode that cn_target_config_node_list specified, starting successfully, and then start the other services.
  • If starting failed，it's necessary to do cleanup before starting again.
  • How to start service ConfigNode or DataNode alone:
# start ConfigNode alone in daemon
sbin/start-confignode.sh -d
# start DataNode alone in daemon
sbin/start-datanode.sh -d

1.3.2. start service ConfigNode and DataNode in other nodes

Execute commands below in both 192.168.132.11 and 192.168.132.12:

cd /data/iotdb
# start service ConfigNode and DataNode
sbin/start-standalone.sh

If starting failed, it's necessary to do cleanup in all nodes, and then doging all again from starting the first node.

1.3.3. check the cluster status

If everything goes well, the cluster will start successfully. Then, we can start the Cli for verification.

/data/iotdb/sbin/start-cli.sh -h 192.168.132.10
IoTDB>show cluster;
# example result:
+------+----------+-------+---------------+------------+-------+---------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|Version|BuildInfo|
+------+----------+-------+---------------+------------+-------+---------+
|     0|ConfigNode|Running| 192.168.132.10|       10710|1.x.x  |  xxxxxxx|
|     1|  DataNode|Running| 192.168.132.10|       10730|1.x.x  |  xxxxxxx|
|     2|ConfigNode|Running| 192.168.132.11|       10710|1.x.x  |  xxxxxxx|
|     3|  DataNode|Running| 192.168.132.11|       10730|1.x.x  |  xxxxxxx|
|     4|ConfigNode|Running| 192.168.132.12|       10710|1.x.x  |  xxxxxxx|
|     5|  DataNode|Running| 192.168.132.12|       10730|1.x.x  |  xxxxxxx|
+------+----------+-------+---------------+------------+-------+---------+

illustration: The IP address of start-cli.sh -h could be any IP address of DataNode service.

【reference】Cleanup

Execute commands in every node:

  1. End processes of ConfigNode and DataNode
# 1. Stop services ConfigNode and DataNode
sbin/stop-standalone.sh

# 2. Check whether there are IoTDB processes left or not
jps
# 或者
ps -ef|gerp iotdb

# 3. If there is any IoTDB process left, kill it
kill -9 <pid>
# If there is only 1 IoTDB instance, execue command below to remove all IoTDB process
ps -ef|grep iotdb|grep -v grep|tr -s '  ' ' ' |cut -d ' ' -f2|xargs kill -9
  1. Remove directories of data and logs
cd /data/iotdb
rm -rf data logs

illustration: It‘s necessary to remove directory of data but it’s not necessary to remove directory of logs, only for convenience.

2. Expand

Expand means add services of ConfigNode or DataNode into an existing IoTDB cluster.

It's the same as starting the other nodes mentioned above. That is downloading IoTDB install package, extracting, configurating and then starting. The new node here is 192.168.132.13. Notice

  • It's must be cleaned up, in other words doing cleanup in it.
  • cluster_name of iotdb-common.properties must be the same to the cluster.
  • cn_target_config_node_list and dn_target_config_node_list must be the same to the cluster.
  • The old data wouldn't be moved to the new node but the new data would be.

2.1. configuration

Modify the specified configuration file according to the table below:

ConfigurationConfiguration OptionIP:192.168.132.13
iotdb-confignode.propertiescn_internal_address192.168.132.13
cn_target_config_node_list192.168.132.10:10710
iotdb-datanode.propertiesdn_rpc_address192.168.132.13
dn_internal_address192.168.132.13
dn_target_config_node_list192.168.132.10:10710

2.2. expand

Execute commands below in new node of 192.168.132.13:

cd /data/iotdb
# start service ConfigNode and DataNode
sbin/start-standalone.sh

2.3. check the result

Execute show cluster through Cli and the result like below:

/data/iotdb/sbin/start-cli.sh -h 192.168.132.10
IoTDB>show cluster;
# example result：
+------+----------+-------+---------------+------------+-------+---------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|Version|BuildInfo|
+------+----------+-------+---------------+------------+-------+---------+
|     0|ConfigNode|Running| 192.168.132.10|       10710|1.x.x  |  xxxxxxx|
|     1|  DataNode|Running| 192.168.132.10|       10730|1.x.x  |  xxxxxxx|
|     2|ConfigNode|Running| 192.168.132.11|       10710|1.x.x  |  xxxxxxx|
|     3|  DataNode|Running| 192.168.132.11|       10730|1.x.x  |  xxxxxxx|
|     4|ConfigNode|Running| 192.168.132.12|       10710|1.x.x  |  xxxxxxx|
|     5|  DataNode|Running| 192.168.132.12|       10730|1.x.x  |  xxxxxxx|
|     6|ConfigNode|Running| 192.168.132.13|       10710|1.x.x  |  xxxxxxx|
|     7|  DataNode|Running| 192.168.132.13|       10730|1.x.x  |  xxxxxxx|
+------+----------+-------+---------------+------------+-------+---------+

3. Remove service

Shrink means removing a service from the IoTDB cluster. Notice:

  • Shrink could be done in any node within the cluster
  • Any service could be shrinked within cluster. But the DataNode service of the cluster must greater than the data replica of iotdb-common.properties.
  • Be patient to wait for the end of shrinking, and then read the guide in logs carefully.

3.1 shrink service ConfigNode

cd /data/iotdb
# way 1: shrink with ip:port
sbin/remove-confignode.sh 192.168.132.13:10710

# way 2: shrink with NodeID of `show cluster`
sbin/remove-confignode.sh 6

3.2 shrink service DataNode

cd /data/iotdb
# way 1: shrink with ip:port
sbin/remove-datanode.sh 192.168.132.13:6667

# way 2: shrink with NodeID of `show cluster`
sbin/remove-confignode.sh 7

3.3 check the result

Execute show cluster through Cli, the result is like below:

+------+----------+-------+---------------+------------+-------+---------+
|NodeID|  NodeType| Status|InternalAddress|InternalPort|Version|BuildInfo|
+------+----------+-------+---------------+------------+-------+---------+
|     0|ConfigNode|Running| 192.168.132.10|       10710|1.x.x  |  xxxxxxx|
|     1|  DataNode|Running| 192.168.132.10|       10730|1.x.x  |  xxxxxxx|
|     2|ConfigNode|Running| 192.168.132.11|       10710|1.x.x  |  xxxxxxx|
|     3|  DataNode|Running| 192.168.132.11|       10730|1.x.x  |  xxxxxxx|
|     4|ConfigNode|Running| 192.168.132.12|       10710|1.x.x  |  xxxxxxx|
|     5|  DataNode|Running| 192.168.132.12|       10730|1.x.x  |  xxxxxxx|
+------+----------+-------+---------------+------------+-------+---------+