This article is the setup process of IoTDB Cluster (0.14.0-preview1).
JDK>=1.8.
Max open file 65535.
Disable the swap memory.
Download the binary distribution from website Download Page.
Download the source code:
git clone https://github.com/apache/iotdb.git
The default branch is master, you should checkout to the release tag:
git checkout v0.14.0-preview1
Under the source root folder:
mvn clean package -pl distribution -am -DskipTests
Then you will get the binary distribution under distribution/target, in which the all-bin contains ConfigNode and DataNode, and DataNode contains the Cli.
| Folder | Description |
|---|---|
| confignode | Contains start/stop/remove shell, configurations, logs, data of ConfigNode |
| datanode | Contains start/stop/remove shell, configurations, logs, data of DataNode, cli shell |
| grafana-metrics-example | Grafana metric page module |
| lib | Jar files folder |
| tools | System tools |
Users could start a cluster which contains multiple ConfigNode and DataNode. A cluster need at least one ConfigNode and no less than the number of data/schema_replication_factor DataNodes.
The total process are three steps:
Please set the important parameters in iotdb-confignode.properties:
| Configuration | Description |
|---|---|
| internal_address | Internal rpc service address of ConfigNode |
| internal_port | Internal rpc service address of ConfigNode |
| consensus_port | ConfigNode replication consensus protocol communication port |
| target_config_nodes | Target ConfigNode address, if the current is the first ConfigNode, then set its address:port |
| data_replication_factor | Data replication factor, no more than DataNode number |
| data_region_consensus_protocol_class | Consensus protocol of data replicas |
| schema_replication_factor | Schema replication factor, no more than DataNode number |
| schema_region_consensus_protocol_class | Consensus protocol of schema replicas |
Start on Linux:
# Foreground ./confignode/sbin/start-confignode.sh # Background nohup ./confignode/sbin/start-confignode.sh >/dev/null 2>&1 &
Start on Windows:
confignode\sbin\start-confignode.bat
More details ConfigNode Configurations.
This will add the replication factor of ConfigNode, except for the port couldn't conflict, make sure other configurations are the same with existing ConfigNode in Cluster.
The adding ConfigNode also use the start-confignode.sh/bat.
You could add any number of DataNode.
Please set the important parameters in iotdb-datanode.properties.
| Configuration | Description |
|---|---|
| rpc_address | Client RPC Service address |
| rpc_port | Client RPC Service port |
| internal_address | Control flow address of DataNode inside cluster |
| internal_port | Control flow port of DataNode inside cluster |
| mpp_data_exchange_port | Data flow port of DataNode inside cluster |
| data_region_consensus_port | Data replicas communication port for consensus |
| schema_region_consensus_port | Schema replicas communication port for consensus |
| target_config_nodes | Running ConfigNode of the Cluster |
Start on Linux:
# Foreground ./datanode/sbin/start-datanode.sh # Background nohup ./datanode/sbin/start-datanode.sh >/dev/null 2>&1 &
Start on Windows:
datanode\sbin\start-datanode.bat
More details DataNode Configurations.
Cli is in datanode/sbin folder.
Start on Linux:
./datanode/sbin/start-cli.sh
Start on Windows:
datanode\sbin\start-cli.bat
Execute the remove-confignode shell on an active ConfigNode.
Remove on Linux:
./confignode/sbin/remove-confignode.sh <confignode_id> ./confignode/sbin/remove-confignode.sh <internal_address>:<internal_port>
Remove on Windows:
confignode\sbin\remove-confignode.bat <confignode_id> confignode\sbin\remove-confignode.bat <internal_address>:<internal_port>
Execute the remove-datanode shell on an active DataNode.
Remove on Linux:
./datanode/sbin/remove-datanode.sh <datanode_id> ./datanode/sbin/remove-datanode.sh <rpc_address>:<rpc_port>
Remove on Windows:
datanode\sbin\remove-datanode.bat <datanode_id> datanode\sbin\remove-datanode.bat <rpc_address>:<rpc_port>
This section uses a local environment as an example to illustrate how to start, expand, and shrink a IoTDB Cluster.
Unzip the apache-iotdb-0.14.0-preview1-all-bin.zip file to cluster0 folder.
Starting the Cluster version with one ConfigNode and one DataNode(1C1D), the default number of replica is one.
./cluster0/confignode/sbin/start-confignode.sh ./cluster0/datanode/sbin/start-datanode.sh
./cluster0/datanode/sbin/start-cli.sh
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 0.0.0.0| 22277| | 1| DataNode|Running| 127.0.0.1| 9003| +------+----------+-------+---------------+------------+ Total line number = 2 It costs 0.160s
Unzip the apache-iotdb-0.14.0-preview1-all-bin.zip file to cluster1 and cluster2 folder.
For folder cluster1:
| configuration item | value |
|---|---|
| internal_address | 0.0.0.0 |
| internal_port | 22279 |
| consensus_port | 22280 |
| target_config_nodes | 0.0.0.0:22277 |
| configuration item | value |
|---|---|
| rpc_address | 0.0.0.0 |
| rpc_port | 6668 |
| internal_address | 127.0.0.1 |
| internal_port | 9004 |
| mpp_data_exchange_port | 8778 |
| data_region_consensus_port | 40011 |
| schema_region_consensus_port | 50011 |
| target_config_nodes | 127.0.0.1:22277 |
For folder cluster1:
| configuration item | value |
|---|---|
| internal_address | 0.0.0.0 |
| internal_port | 22281 |
| consensus_port | 22282 |
| target_config_nodes | 0.0.0.0:22277 |
| configuration item | value |
|---|---|
| rpc_address | 0.0.0.0 |
| rpc_port | 6669 |
| internal_address | 127.0.0.1 |
| internal_port | 9005 |
| mpp_data_exchange_port | 8779 |
| data_region_consensus_port | 40012 |
| schema_region_consensus_port | 50012 |
| target_config_nodes | 127.0.0.1:22277 |
Expanding the Cluster to three ConfigNode and three DataNode(3C3D). The following commands can be executed in no particular order.
./cluster1/confignode/sbin/start-confignode.sh ./cluster1/datanode/sbin/start-datanode.sh ./cluster2/confignode/sbin/start-confignode.sh ./cluster2/datanode/sbin/start-datanode.sh
Execute the show cluster command, the result is shown below:
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 0.0.0.0| 22277| | 2|ConfigNode|Running| 0.0.0.0| 22279| | 3|ConfigNode|Running| 0.0.0.0| 22281| | 1| DataNode|Running| 127.0.0.1| 9003| | 4| DataNode|Running| 127.0.0.1| 9004| | 5| DataNode|Running| 127.0.0.1| 9005| +------+----------+-------+---------------+------------+ Total line number = 6 It costs 0.012s
./cluster0/confignode/sbin/remove-confignode.sh 127.0.0.1:22279
./cluster0/datanode/sbin/remove-datanode.sh 127.0.0.1:6668
Execute the show cluster command, the result is shown below:
IoTDB> show cluster +------+----------+-------+---------------+------------+ |NodeID| NodeType| Status|InternalAddress|InternalPort| +------+----------+-------+---------------+------------+ | 0|ConfigNode|Running| 0.0.0.0| 22277| | 3|ConfigNode|Running| 0.0.0.0| 22281| | 1| DataNode|Running| 127.0.0.1| 9003| | 5| DataNode|Running| 127.0.0.1| 9005| +------+----------+-------+---------------+------------+ Total line number = 4 It costs 0.007s