commit | 802d2c257bd5aa98518e5dc389bad8ce14cd0bf1 | [log] [tgz] |
---|---|---|
author | Guohao Li <gehafearless@apache.org> | Wed Mar 13 11:32:31 2024 +0800 |
committer | GitHub <noreply@github.com> | Wed Mar 13 11:32:31 2024 +0800 |
tree | b2cdba07375cb30aea1762d5eff5c79f76ce85f9 | |
parent | c1788c3db37ca4d2ff2d55e3d3ca6b667575a95b [diff] |
feat(FQDN): replace rpc_address to host_prot for function & add host_port on thrift struct (#1658) https://github.com/apache/incubator-pegasus/issues/1659 1. Add a new host_port type field if rpc_address field exists in IDL structures. 2. Use both rpc_address and the optional host_port fields in C++ code. 3. Use host_port to communicate with remote nodes firstly. TODO: - Make sure everything works well after upgrading from old versions without any config changes - Make sure old version client is able to access the new version cluster - Make sure new version client is able to access the old version cluster NOTE: It's not able to upgrade Pegasus cluster and clients smoothly to the new version with FQDN enabled, it's needed to follow the steps if you want to enable the FQDN feature: 1. Stop the cluster and clients 2. Set the new configs (using hostnames instead of IPs) 4. Rebuild the metadata of the cluster (See https://pegasus.apache.org/zh/administration/meta-recovery) 5. Start the whole cluster and application clients NOTE: Do not support rolling restart servers in the cluster with FQDN related configs changed
Note: The master
branch may be in an unstable or even in a broken state during development. Please use GitHub Releases instead of the master
branch in order to get stable binaries.
Apache Pegasus is a distributed key-value storage system which is designed to be:
Pegasus targets to fill the gap between Redis and HBase. As the former is in-memory, low latency, but does not provide a strong-consistency guarantee. And unlike the latter, Pegasus is entirely written in C++ and its write-path relies merely on the local filesystem.
Apart from the performance requirements, we also need a storage system to ensure multiple-level data safety and support fast data migration between data centers, automatic load balancing, and online partition split.
Persistence of data: Each write is replicated three-way to different ReplicaServers before responding to the client. Using PacificA protocol, Pegasus has the ability for strong consistent replication and membership changes.
Automatic load balancing over ReplicaServers: Load balancing is a builtin function of MetaServer, which manages the distribution of replicas. When the cluster is in an inbalance state, the administrator can invoke a simple rebalance command that automatically schedules the replica migration.
Cold Backup: Pegasus supports an extensible backup and restore mechanism to ensure data safety. The location of snapshot could be a distributed filesystem like HDFS or local filesystem. The snapshot storing in the filesystem can be further used for analysis based on pegasus-spark.
Eventually-consistent intra-datacenter replication: This is a feature we called duplication. It allows a change made in the local cluster accesible after a short time period by the remote cluster. It help achieving higher availability of your service and gaining better performance by accessing only local cluster.
See our documentation on the Pegasus Website.
Pegasus has support for several languages:
Send emails to the Apache Pegasus developer mailing list: dev@pegasus.apache.org
. This is the place where topics around development, community, and problems are officially discussed. Please remember to subscribe to the mailing list via dev-subscribe@pegasus.apache.org
.
GitHub Issues: submit an issue when you have any idea to improve Pegasus, and when you encountered some bugs or problems.
Test tools:
Data import/export tools:
Copyright 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0