This document focuses on how to manage Doris metadata in a real production environment. It includes the proposed deployment of FE nodes, some commonly used operational methods, and common error resolution methods.
For the time being, read the Doris metadata design document to understand how Doris metadata works.
FeMetaVersion.java file in the FE code), it is usually impossible to roll back to the old version after upgrading to the new version. Therefore, before upgrading FE, be sure to test metadata compatibility according to the operations in the Upgrade Document.Let's assume that the path of meta_dir specified in fe.conf is path/to/palo-meta. In a normal Doris cluster, the directory structure of metadata should be as follows:
/path/to/palo-meta/
|-- bdb/
| |-- 00000000.jdb
| |-- je.config.csv
| |-- je.info.0
| |-- je.info.0.lck
| |-- je.lck
| `-- je.stat.csv
`-- image/
|-- ROLE
|-- VERSION
`-- image.xxxx
bdb
We use bdbje as a distributed kV system to store metadata journal. This BDB directory is equivalent to the “data directory” of bdbje.
The .jdb suffix is the data file of bdbje. These data files will increase with the increasing number of metadata journals. When Doris regularly completes the image, the old log is deleted. So normally, the total size of these data files varies from several MB to several GB (depending on how Doris is used, such as import frequency). When the total size of the data file is larger than 10GB, you may need to wonder whether the image failed or the historical journals that failed to distribute the image could not be deleted.
je.info.0 is the running log of bdbje. The time in this log is UTC + 0 time zone. We may fix this in a later version. From this log, you can also see how some bdbje works.
image directory
The image directory is used to store metadata mirrors generated regularly by Doris. Usually, you will see a image.xxxxx mirror file. Where xxxxx is a number. This number indicates that the image contains all metadata journal before xxxx. And the generation time of this file (viewed through ls -al) is usually the generation time of the mirror.
You may also see a image.ckpt file. This is a metadata mirror being generated. The du -sh command should show that the file size is increasing, indicating that the mirror content is being written to the file. When the mirror is written, it automatically renames itself to a new image.xxxxx and replaces the old image file.
Only FE with a Master role will actively generate image files on a regular basis. After each generation, FE is pushed to other non-Master roles. When it is confirmed that all other FEs have received this image, Master FE deletes the metadata journal in bdbje. Therefore, if image generation fails or image push fails to other FEs, data in bdbje will accumulate.
ROLE file records the type of FE (FOLLOWER or OBSERVER), which is a text file.
VERSION file records the cluster ID of the Doris cluster and the token used to access authentication between nodes, which is also a text file.
ROLE file and VERSION file may only exist at the same time, or they may not exist at the same time (e.g. at the first startup).
Single node FE is the most basic deployment mode. A complete Doris cluster requires at least one FE node. When there is only one FE node, the type of the node is Follower and the role is Master.
First start-up
Suppose the path of meta_dir specified in fe.conf is path/to/palo-meta.
Ensure that path/to/palo-meta already exists, that the permissions are correct and that the directory is empty.
Start directly through sh bin/start_fe.sh.
After booting, you should be able to see the following log in fe.log:
The above logs are not necessarily strictly in this order, but they are basically similar.
The first start-up of a single-node FE usually does not encounter problems. If you haven‘t seen the above logs, generally speaking, you haven’t followed the document steps carefully, please read the relevant wiki carefully.
Restart
Stopped FE nodes can be restarted by using sh bin/start_fe.sh.
After restarting, you should be able to see the following log in fe.log:
Palo FE starting...
finished to get cluster id: xxxx, role: FOLLOWER and node name: xxxx
If no image has been generated before reboot, you will see:
image does not exist: /path/to/palo-meta/image/image.0
If an image is generated before the restart, you will see:
start load image from /path/to/palo-meta/image/image.xxx. is ckpt: false
finished load image in xxx ms
transfer from INIT to UNKNOWN
replayed journal id is xxxx, replay to journal id is yyyy
transfer from UNKNOWN to MASTER
finish replay in xxx msec
master finish replay journal, can write now.
begin to generate new image: image.xxxx
start save image to /path/to/palo-meta/image/image.ckpt. is ckpt: true
finished save image /path/to/palo-meta/image/image.ckpt in xxx ms. checksum is xxxx
push image.xxx to other nodes. totally xx nodes, push successed xx nodes
QE service start
thrift server started
The above logs are not necessarily strictly in this order, but they are basically similar.
Common problems
For the deployment of single-node FE, start-stop usually does not encounter any problems. If you have any questions, please refer to the relevant Wiki and check your operation steps carefully.
Adding FE processes is described in detail in the Elastic Expansion Documents and will not be repeated. Here are some points for attention, as well as common problems.
Notes
--helper parameter is added to point to Master FE. There is no need to add --helper when restarting. (If --helper is specified, FE will directly ask the helper node for its role. If not, FE will try to obtain information from ROLE and VERSION files in the palo-meta/image/ directory.meta_dir of the FE is created, has correct permissions and is empty.ALTER SYSTEM ADD FOLLOWER/OBSERVER statement adds FE to metadata in a sequence that is not required. If a new FE is started first and no statement is executed, the current node is not added to the group. Please add it first. in the new FE log. When the statement is executed, it enters the normal process.ALTER SYSTEM ADD FOLLOWER/OBSERVER stmt.Common problems
this need is DETACHED
When you first start a FE to be added, if the data in palo-meta/bdb on Master FE is large, you may see the words this node is DETACHED. in the FE log to be added. At this point, bdbje is copying data, and you can see that the bdb/ directory of FE to be added is growing. This process usually takes several minutes (depending on the amount of data in bdbje). Later, there may be some bdbje-related error stack information in fe. log. If QE service start and thrift server start are displayed in the final log, the start is usually successful. You can try to connect this FE via mysql-client. If these words do not appear, it may be the problem of bdbje replication log timeout. At this point, restarting the FE directly will usually solve the problem.
Failure to add due to various reasons
If OBSERVER is added, because OBSERVER-type FE does not participate in the majority of metadata writing, it can theoretically start and stop at will. Therefore, for the case of adding OBSERVER failure. The process of OBSERVER FE can be killed directly. After clearing the metadata directory of OBSERVER, add the process again.
If FOLLOWER is added, because FOLLOWER is mostly written by participating metadata. So it is possible that FOLLOWER has joined the bdbje electoral team. If there are only two FOLLOWER nodes (including MASTER), then stopping one FE may cause another FE to quit because it cannot write most of the time. At this point, we should first delete the newly added FOLLOWER node from the metadata through the ALTER SYSTEM DROP FOLLOWER command, then kill the FOLLOWER process, empty the metadata and re-add the process.
The corresponding type of FE can be deleted by the ALTER SYSTEM DROP FOLLOWER/OBSERVER command. The following points should be noted:
For OBSERVER type FE, direct DROP is enough, without risk.
For FOLLOWER type FE. First, you should make sure that you start deleting an odd number of FOLLOWERs (three or more).
FE may fail to start bdbje and synchronize between FEs for some reasons. Phenomena include the inability to write metadata, the absence of MASTER, and so on. At this point, we need to manually restore the FE. The general principle of manual recovery of FE is to start a new MASTER through metadata in the current meta_dir, and then add other FEs one by one. Please follow the following steps strictly:
First, stop all FE processes and all business access. Make sure that during metadata recovery, external access will not lead to other unexpected problems.
Identify which FE node's metadata is up-to-date:
meta_dir directories first.meta_dir/image directory. The larger the number, the newer the metadata.The following operations are performed on the FE nodes selected in step 2.
role=OBSERVER in the meta_dir/image/ROLE file to role=FOLLOWER. (Recovery from the OBSERVER node will be more cumbersome, first follow the steps here, followed by a separate description)metadata_failure_recovery=true.sh bin/start_fe.sh to start the FEStart a single node FE. You should see the words transfer from XXXX to MASTER in fe.log.show frontends; command, you should see all the FEs you added before, and the current FE is master.metadata_failure_recovery=true configuration item in fe.conf, or set it to false, and restart the FE (Important).If you are recovering metadata from an OBSERVER node, after completing the above steps, you will find that the current FE role is OBSERVER, but
IsMasterappears astrue. This is because the “OBSERVER” seen here is recorded in Doris‘s metadata, but whether it is master or not, is recorded in bdbje’s metadata. Because we recovered from an OBSERVER node, there was inconsistency. Please take the following steps to fix this problem (we will fix it in a later version):
- First, all FE nodes except this “OBSERVER” are DROPed out.
- A new FOLLOWER FE is added through the
ADD FOLLOWERcommand, assuming that it is on hostA.- Start a new FE on hostA and join the cluster by
helper.- After successful startup, you should see two FEs through the
show frontends;statement, one is the previous OBSERVER, the other is the newly added FOLLOWER, and the OBSERVER is the master.- After confirming that the new FOLLOWER is working properly, the new FOLLOWER metadata is used to perform a failure recovery operation again.
- The purpose of the above steps is to manufacture a metadata of FOLLOWER node artificially, and then use this metadata to restart fault recovery. This avoids inconsistencies in recovering metadata from OBSERVER.
The meaning of
metadata_failure_recovery = trueis to empty the metadata ofbdbje. In this way, bdbje will not contact other FEs before, but start as a separate FE. This parameter needs to be set to true only when restoring startup. After recovery, it must be set to false. Otherwise, once restarted, the metadata of bdbje will be emptied again, which will make other FEs unable to work properly.
After the successful execution of step 3, we delete the previous FEs from the metadata by using the ALTER SYSTEM DROP FOLLOWER/OBSERVER command and add them again by adding new FEs.
If the above operation is normal, it will be restored.
If you need to change the existing FOLLOWER/OBSERVER type FE to OBSERVER/FOLLOWER type, please delete FE in the way described above, and then add the corresponding type FE.
If you need to migrate one FE from the current node to another, there are several scenarios.
FOLLOWER, or OBSERVER migration for non-MASTER nodes
After adding a new FOLLOWER / OBSERVER directly, delete the old FOLLOWER / OBSERVER.
Single-node MASTER migration
When there is only one FE, refer to the Failure Recovery section. Copy the palo-meta directory of FE to the new node and start the new MASTER in Step 3 of the Failure Recovery section
A set of FOLLOWER migrates from one set of nodes to another set of new nodes
Deploy FE on the new node and add the new node first by adding FOLLOWER. The old nodes can be dropped by DROP one by one. In the process of DROP-by-DROP, MASTER automatically selects the new FOLLOWER node.
FE currently has the following ports
edit_log_port
If this port needs to be replaced, it needs to be restored with reference to the operations in the Failure Recovery section. Because the port has been persisted into bdbje‘s own metadata (also recorded in Doris’s own metadata), it is necessary to clear bdbje's metadata by setting metadata_failure_recovery=true.
http_port
All FE http_ports must be consistent. So if you want to modify this port, all FEs need to be modified and restarted. Modifying this port will be more complex in the case of multiple FOLLOWER deployments (involving laying eggs and laying hens...), so this operation is not recommended. If necessary, follow the operation in the Failure Recovery section directly.
rpc_port
After modifying the configuration, restart FE directly. Master FE informs BE of the new port through heartbeat. Only this port of Master FE will be used. However, it is still recommended that all FE ports be consistent.
query_port
After modifying the configuration, restart FE directly. This only affects mysql's connection target.
In some extreme cases, the image file on the disk may be damaged, but the metadata in the memory is intact. At this point, we can dump the metadata from the memory and replace the image file on the disk to recover the metadata. the entire non-stop query service operation steps are as follows:
Stop all Load, Create, Alter operations.
Execute the following command to dump metadata from the Master FE memory: (hereafter called image_mem)
curl -u $root_user:$password http://$master_hostname:8030/dump
Replace the image file in the meta_dir/image directory on the OBSERVER FE node with the image_mem file, restart the OBSERVER FE node, and verify the integrity and correctness of the image_mem file. You can check whether the DB and Table metadata are normal on the FE Web page, whether there is an exception in fe.log, whether it is in a normal replayed jour.
Replace the image file in the meta_dir/image directory on the FOLLOWER FE node with the image_mem file in turn, restart the FOLLOWER FE node, and confirm that the metadata and query services are normal.
Replace the image file in the meta_dir/image directory on the Master FE node with the image_mem file, restart the Master FE node, and then confirm that the FE Master switch is normal and The Master FE node can generate a new image file through checkpoint.
Recover all Load, Create, Alter operations.
Note: If the Image file is large, the entire process can take a long time, so during this time, make sure Master FE does not generate a new image file via checkpoint. When the image.ckpt file in the meta_dir/image directory on the Master FE node is observed to be as large as the image.xxx file, the image.ckpt file can be deleted directly.
The metadata log of FE is stored in BDBJE in the form of Key-Value. In some abnormal situations, FE may not be started due to metadata errors. In this case, Doris provides a way to help users query the data stored in BDBJE to facilitate troubleshooting.
First, you need to add configuration in fe.conf: enable_bdbje_debug_mode=true, and then start FE through sh start_fe.sh --daemon.
At this time, FE will enter the debug mode, only start the http server and MySQL server, and open the BDBJE instance, but will not load any metadata and other subsequent startup processes.
This is, we can view the data stored in BDBJE by visiting the web page of FE, or after connecting to Doris through the MySQL client, through show proc /bdbje;.
mysql> show proc "/bdbje"; +----------+---------------+---------+ | DbNames | JournalNumber | Comment | +----------+---------------+---------+ | 110589 | 4273 | | | epochDB | 4 | | | metricDB | 430694 | | +----------+---------------+---------+
The first level directory will display all the database names in BDBJE and the number of entries in each database.
mysql> show proc "/bdbje/110589"; +-----------+ | JournalId | +-----------+ | 1 | | 2 | ... | 114858 | | 114859 | | 114860 | | 114861 | +-----------+ 4273 rows in set (0.06 sec)
Entering the second level, all the entry keys under the specified database will be listed.
mysql> show proc "/bdbje/110589/114861"; +-----------+--------------+---------------------------------------------+ | JournalId | OpType | Data | +-----------+--------------+---------------------------------------------+ | 114861 | OP_HEARTBEAT | org.apache.doris.persist.HbPackage@6583d5fb | +-----------+--------------+---------------------------------------------+ 1 row in set (0.05 sec)
The third level can display the value information of the specified key.
The deployment recommendation of FE is described in the Installation and Deployment Document. Here are some supplements.
If you don‘t know the operation logic of FE metadata very well, or you don’t have enough experience in the operation and maintenance of FE metadata, we strongly recommend that only one FOLLOWER-type FE be deployed as MASTER in practice, and the other FEs are OBSERVER, which can reduce many complex operation and maintenance problems. Don't worry too much about the failure of MASTER single point to write metadata. First, if you configure it properly, FE as a java process is very difficult to hang up. Secondly, if the MASTER disk is damaged (the probability is very low), we can also use the metadata on OBSERVER to recover manually through fault recovery.
The JVM of the FE process must ensure sufficient memory. We strongly recommend that FE's JVM memory should be at least 10GB and 32GB to 64GB. And deploy monitoring to monitor JVM memory usage. Because if OOM occurs in FE, metadata writing may fail, resulting in some failures that cannot recover!
FE nodes should have enough disk space to prevent the excessive metadata from causing insufficient disk space. At the same time, FE logs also take up more than a dozen gigabytes of disk space.
Output meta out of date. current time: xxx, synchronized time: xxx, has log: xxx, fe type: xxx in fe.log
This is usually because the FE cannot elect Master. For example, if three FOLLOWERs are configured, but only one FOLLOWER is started, this FOLLOWER will cause this problem. Usually, just start the remaining FOLLOWER. If the problem has not been solved after the start-up, manual recovery may be required in accordance with the way in the Failure Recovery section.
Clock delta: xxxx ms. between Feeder: xxxx and this Replica exceeds max permissible delta: xxxx ms.
Bdbje requires that clock errors between nodes should not exceed a certain threshold. If exceeded, the node will exit abnormally. The default threshold is 5000ms, which is controlled by FE parameter `max_bdbje_clock_delta_ms', and can be modified as appropriate. But we suggest using NTP and other clock synchronization methods to ensure the clock synchronization of Doris cluster hosts.
Mirror files in the image/ directory have not been updated for a long time
Master FE generates a mirror file by default for every 50,000 metadata journal. In a frequently used cluster, a new image file is usually generated every half to several days. If you find that the image file has not been updated for a long time (for example, more than a week), you can see the reasons in sequence as follows:
Search for memory is not enough to do checkpoint. Committed memroy XXXX Bytes, used memory XXXX Bytes. in the fe.log of Master FE. If found, it indicates that the current FE's JVM memory is insufficient for image generation (usually we need to reserve half of the FE memory for image generation). Then you need to add JVM memory and restart FE before you can observe. Each time Master FE restarts, a new image is generated directly. This restart method can also be used to actively generate new images. Note that if there are multiple FOLLOWER deployments, then when you restart the current Master FE, another FOLLOWER FE will become MASTER, and subsequent image generation will be the responsibility of the new Master. Therefore, you may need to modify the JVM memory configuration of all FOLLOWER FE.
Search for begin to generate new image: image.xxxx in the fe.log of Master FE. If it is found, then the image is generated. Check the subsequent log of this thread, and if checkpoint finished save image.xxxx appears, the image is written successfully. If Exception when generating new image file occurs, the generation fails and specific error messages need to be viewed.
The size of the bdb/ directory is very large, reaching several Gs or more.
The BDB directory will remain large for some time after eliminating the error that the new image cannot be generated. Maybe it's because Master FE failed to push image. You can search push image.XXXX to other nodes. totally XX nodes, push successed YY nodes in the fe. log of Master FE. If YY is smaller than xx, then some FEs are not pushed successfully. You can see the specific error Exception when pushing image file.url = xxx in the fe. log.
At the same time, you can add the configuration in the FE configuration file: edit_log_roll_num = xxxx. This parameter sets the number of metadata journals and makes an image once. The default is 50000. This number can be reduced appropriately to make images more frequent, thus speeding up the deletion of old journals.
FOLLOWER FE hangs up one after another
Because Doris‘s metadata adopts the majority writing strategy, that is, a metadata journal must be written to at least a number of FOLLOWER FEs (for example, three FOLLOWERs, two must be written successfully) before it can be considered successful. If the write fails, the FE process exits on its own initiative. So suppose there are three FOLLOWERs: A, B and C. C hangs up first, and then B hangs up, then A will hang up. So as described in the Best Practices section, if you don’t have extensive experience in metadata operations and maintenance, it's not recommended to deploy multiple FOLLOWERs.
fe.log 中出现 get exception when try to close previously opened bdb database. ignore it
If there is the word ignore it behind it, there is usually no need to deal with it. If you are interested, you can search for this error in BDBEnvironment.java, and see the annotations.
From show frontends; Look, the Join of a FE is listed as true, but actually the FE is abnormal.
Through show frontends; see the Join information. If the column is true, it only means that the FE has joined the cluster. It does not mean that it still exists normally in the cluster. If false, it means that the FE has never joined the cluster.
Configuration of FE master_sync_policy, replica_sync_policy, and txn_rollback_limit.
master_sync_policy is used to specify whether fsync (), replica_sync_policy is called when Leader FE writes metadata log, and replica_sync_policy is used to specify whether other Follower FE calls fsync () when FE HA deploys synchronous metadata. In earlier versions of Oris, these two parameters defaulted to WRITE_NO_SYNC, i.e., fsync () was not called. In the latest version of Oris, the default has been changed to SYNC, that is, fsync () is called. Calling fsync () significantly reduces the efficiency of metadata disk writing. In some environments, IOPS may drop to several hundred and the latency increases to 2-3ms (but it's still enough for Doris metadata manipulation). Therefore, we recommend the following configuration:
master_sync_policy is set to SYNC, which prevents the loss of metadata due to the downtime of the FE system.master_sync_policy and replica_sync_policy to WRITE_NO_SYNC, because we think that the probability of simultaneous outage of multiple systems is very low.If master_sync_policy is set to WRITE_NO_SYNC in a single Follower FE deployment, then a FE system outage may occur, resulting in loss of metadata. At this point, if other Observer FE attempts to restart, it may report an error:
Node xxx must rollback xx total commits(numPassedDurableCommits of which were durable) to the earliest point indicated by transaction xxxx in order to rejoin the replication group, but the transaction rollback limit of xxx prohibits this.
This means that some transactions that have been persisted need to be rolled back, but the number of entries exceeds the upper limit. Here our default upper limit is 100, which can be changed by setting txn_rollback_limit. This operation is only used to attempt to start FE normally, but lost metadata cannot be recovered.