| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Duplication</title> |
| <link rel="stylesheet" href="/assets/css/app.css"> |
| <link rel="shortcut icon" href="/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Duplication | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.3" /> |
| <meta property="og:title" content="Duplication" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content=" In Pegasus, cross-datacenter data synchronization is also known as ‘hot’ backup or duplication, abbreviated as “dup”. The main purpose of this feature is to ensure data center-level availability. When the business needs to ensure that services and data can tolerate data center failures, this feature can be considered." /> |
| <meta property="og:description" content=" In Pegasus, cross-datacenter data synchronization is also known as ‘hot’ backup or duplication, abbreviated as “dup”. The main purpose of this feature is to ensure data center-level availability. When the business needs to ensure that services and data can tolerate data center failures, this feature can be considered." /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2024-04-22T06:39:52+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Duplication" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2024-04-22T06:39:52+00:00","datePublished":"2024-04-22T06:39:52+00:00","description":" In Pegasus, cross-datacenter data synchronization is also known as ‘hot’ backup or duplication, abbreviated as “dup”. The main purpose of this feature is to ensure data center-level availability. When the business needs to ensure that services and data can tolerate data center failures, this feature can be considered.","headline":"Duplication","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/duplication"},"url":"/administration/duplication"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| |
| <body> |
| <div class="dashboard is-full-height"> |
| <!-- left panel --> |
| <div class="dashboard-panel is-medium is-hidden-mobile pl-0"> |
| <div class="dashboard-panel-header has-text-centered"> |
| <a href="/"> |
| <img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;"> |
| </a> |
| |
| </div> |
| <div class="dashboard-panel-main is-scrollable pl-6"> |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label">The Pegasus documentation</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/downloads" |
| class=""> |
| Downloads |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Building Pegasus</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/build/compile-by-docker" |
| class=""> |
| Compile by docker (recommended) |
| </a> |
| </li> |
| |
| <li> |
| <a href="/docs/build/compile-from-source" |
| class=""> |
| Compile from source |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Client Libs</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/clients/java-client" |
| class=""> |
| Java Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/cpp-client" |
| class=""> |
| C++ Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class=""> |
| Golang Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/python-client" |
| class=""> |
| Python Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/node-client" |
| class=""> |
| NodeJS Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/scala-client" |
| class=""> |
| Scala Client |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Tools</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/tools/shell" |
| class=""> |
| Pegasus Shell |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class=""> |
| Admin CLI |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/pegic" |
| class=""> |
| Pegasus data access CLI |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">API</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/api/ttl" |
| class=""> |
| TTL(Time To Live) |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/single-atomic" |
| class=""> |
| Single-Atomic Operations |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/redis" |
| class=""> |
| Redis Adaption |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/geo" |
| class=""> |
| GEO Support |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/http" |
| class=""> |
| HTTP API |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Admin</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/administration/deployment" |
| class=""> |
| Deployment |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/config" |
| class=""> |
| Configurations |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/rebalance" |
| class=""> |
| Rebalance |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/monitoring" |
| class=""> |
| Monitoring |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/rolling-update" |
| class=""> |
| Rolling Restart and Upgrade |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/scale-in-out" |
| class=""> |
| Scale-in and Scale-out |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/resource-management" |
| class=""> |
| Resource Management |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/cold-backup" |
| class=""> |
| Cold Backup |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/meta-recovery" |
| class=""> |
| Metadata Recovery |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/replica-recovery" |
| class=""> |
| Replica Data Recovery |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/zk-migration" |
| class=""> |
| Zookeeper Migration |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-migration" |
| class=""> |
| Table Migration |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-soft-delete" |
| class=""> |
| Table Soft-Delete |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-env" |
| class=""> |
| Table Environment Variables |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/remote-commands" |
| class=""> |
| Remote Command |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/partition-split" |
| class=""> |
| Partition-Split |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/duplication" |
| class="is-active"> |
| Duplication |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/compression" |
| class=""> |
| Data Compression |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/throttling" |
| class=""> |
| Throttling |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/experiences" |
| class=""> |
| Experiences |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/manual-compact" |
| class=""> |
| Manual Compact |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/usage-scenario" |
| class=""> |
| Usage Scenario |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/bad-disk" |
| class=""> |
| Bad Disk Repair |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/whitelist" |
| class=""> |
| Replica Server Whitelist |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/backup-request" |
| class=""> |
| Backup Request |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/hotspot-detection" |
| class=""> |
| Hotspot Detection |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| </div> |
| </div> |
| |
| <!-- main section --> |
| <div class="dashboard-main is-scrollable"> |
| <nav class="navbar is-hidden-desktop"> |
| <div class="navbar-brand"> |
| <a href="/" class="navbar-item"> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus-square.png"> |
| </a> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-light is-outlined is-inverted" href="/zh/administration/duplication"><strong>中</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| The Pegasus documentation |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/downloads" |
| class="navbar-item "> |
| Downloads |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Building Pegasus |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/build/compile-by-docker" |
| class="navbar-item "> |
| Compile by docker (recommended) |
| </a> |
| |
| <a href="/docs/build/compile-from-source" |
| class="navbar-item "> |
| Compile from source |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Client Libs |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/clients/java-client" |
| class="navbar-item "> |
| Java Client |
| </a> |
| |
| <a href="/clients/cpp-client" |
| class="navbar-item "> |
| C++ Client |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class="navbar-item "> |
| Golang Client |
| </a> |
| |
| <a href="/clients/python-client" |
| class="navbar-item "> |
| Python Client |
| </a> |
| |
| <a href="/clients/node-client" |
| class="navbar-item "> |
| NodeJS Client |
| </a> |
| |
| <a href="/clients/scala-client" |
| class="navbar-item "> |
| Scala Client |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Tools |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/tools/shell" |
| class="navbar-item "> |
| Pegasus Shell |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class="navbar-item "> |
| Admin CLI |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/pegic" |
| class="navbar-item "> |
| Pegasus data access CLI |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| API |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/api/ttl" |
| class="navbar-item "> |
| TTL(Time To Live) |
| </a> |
| |
| <a href="/api/single-atomic" |
| class="navbar-item "> |
| Single-Atomic Operations |
| </a> |
| |
| <a href="/api/redis" |
| class="navbar-item "> |
| Redis Adaption |
| </a> |
| |
| <a href="/api/geo" |
| class="navbar-item "> |
| GEO Support |
| </a> |
| |
| <a href="/api/http" |
| class="navbar-item "> |
| HTTP API |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Admin |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/administration/deployment" |
| class="navbar-item "> |
| Deployment |
| </a> |
| |
| <a href="/administration/config" |
| class="navbar-item "> |
| Configurations |
| </a> |
| |
| <a href="/administration/rebalance" |
| class="navbar-item "> |
| Rebalance |
| </a> |
| |
| <a href="/administration/monitoring" |
| class="navbar-item "> |
| Monitoring |
| </a> |
| |
| <a href="/administration/rolling-update" |
| class="navbar-item "> |
| Rolling Restart and Upgrade |
| </a> |
| |
| <a href="/administration/scale-in-out" |
| class="navbar-item "> |
| Scale-in and Scale-out |
| </a> |
| |
| <a href="/administration/resource-management" |
| class="navbar-item "> |
| Resource Management |
| </a> |
| |
| <a href="/administration/cold-backup" |
| class="navbar-item "> |
| Cold Backup |
| </a> |
| |
| <a href="/administration/meta-recovery" |
| class="navbar-item "> |
| Metadata Recovery |
| </a> |
| |
| <a href="/administration/replica-recovery" |
| class="navbar-item "> |
| Replica Data Recovery |
| </a> |
| |
| <a href="/administration/zk-migration" |
| class="navbar-item "> |
| Zookeeper Migration |
| </a> |
| |
| <a href="/administration/table-migration" |
| class="navbar-item "> |
| Table Migration |
| </a> |
| |
| <a href="/administration/table-soft-delete" |
| class="navbar-item "> |
| Table Soft-Delete |
| </a> |
| |
| <a href="/administration/table-env" |
| class="navbar-item "> |
| Table Environment Variables |
| </a> |
| |
| <a href="/administration/remote-commands" |
| class="navbar-item "> |
| Remote Command |
| </a> |
| |
| <a href="/administration/partition-split" |
| class="navbar-item "> |
| Partition-Split |
| </a> |
| |
| <a href="/administration/duplication" |
| class="navbar-item is-active"> |
| Duplication |
| </a> |
| |
| <a href="/administration/compression" |
| class="navbar-item "> |
| Data Compression |
| </a> |
| |
| <a href="/administration/throttling" |
| class="navbar-item "> |
| Throttling |
| </a> |
| |
| <a href="/administration/experiences" |
| class="navbar-item "> |
| Experiences |
| </a> |
| |
| <a href="/administration/manual-compact" |
| class="navbar-item "> |
| Manual Compact |
| </a> |
| |
| <a href="/administration/usage-scenario" |
| class="navbar-item "> |
| Usage Scenario |
| </a> |
| |
| <a href="/administration/bad-disk" |
| class="navbar-item "> |
| Bad Disk Repair |
| </a> |
| |
| <a href="/administration/whitelist" |
| class="navbar-item "> |
| Replica Server Whitelist |
| </a> |
| |
| <a href="/administration/backup-request" |
| class="navbar-item "> |
| Backup Request |
| </a> |
| |
| <a href="/administration/hotspot-detection" |
| class="navbar-item "> |
| Hotspot Detection |
| </a> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <nav class="navbar is-hidden-mobile"> |
| <div class="navbar-start w-full"> |
| <div class="navbar-item pl-0 w-full"> |
| <!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input, |
| I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released. |
| Related issue: https://github.com/algolia/docsearch/issues/230--> |
| <div id="docsearch"></div> |
| </div> |
| </div> |
| <div class="navbar-end"> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-light is-outlined is-inverted" href="/zh/administration/duplication"><strong>中</strong></a> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section class="hero is-info lg:mr-3"> |
| <div class="hero-body"> |
| |
| <p class="title is-size-2 is-centered">Duplication</p> |
| </div> |
| </section> |
| <section class="section" style="padding-top: 2rem;"> |
| <div class="content"> |
| <p> In Pegasus, cross-datacenter data synchronization is also known as ‘hot’ backup or duplication, abbreviated as “dup”. The main purpose of this feature is to ensure data center-level availability. When the business needs to ensure that services and data can tolerate data center failures, this feature can be considered.</p> |
| |
| <p> In addition, when the Pegasus client is distributed across multiple data centers, it often encounters high latency issues when accessing the Pegasus service across data centers. In this case, we can deploy the Pegasus service and client in the same data center, and the client can only read and write the local data center’s service. Then, the duplication feature synchronizes the writes to all data centers. This approach can ensure that all data centers have complete data while avoiding the latency overhead of cross-datacenter access.</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> client client client |
| + + + |
| +---------v-------+ +--------v--------+ +------v-----------+ |
| | | | | | | |
| | pegasus-beijing <---> pegasus-tianjin <---> pegasus-shanghai | |
| | | | | | | |
| +----------^------+ +-----------------+ +---------^--------+ |
| | | |
| +------------------------------------------+ |
| </code></pre></div></div> |
| |
| <p> We can achieve a single-master configuration as well as a multi-master configuration across multiple data centers, which users can configure according to their needs. It is important to note that cross-datacenter synchronization is asynchronous data replication and not completely real-time. Unlike within a single data center, this feature does not provide consistency guarantees for read-after-write across data centers. Currently, in a healthy cross-datacenter network environment, data latency is about 10 seconds, meaning that data written in data center A will be written to data center B after approximately 10 seconds.</p> |
| |
| <h2 id="get-started">Get started</h2> |
| |
| <p>Suppose we have 2 pegasus clusters <em>bjsrv-account</em> (source cluster) and <em>tjsrv-account</em> (target cluster), located in two computer rooms in Beijing and Tianjin respectively. The table <code class="language-plaintext highlighter-rouge">my_source_app</code> stores extremely critical user account data. It needs to be available in dual clusters, so we implement duplication for it:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> ./run.sh shell -n bjsrv-account |
| |
| Type "help" for more information. |
| Type "Ctrl-D" or "Ctrl-C" to exit the shell. |
| |
| The cluster name is: bjsrv-account |
| The cluster meta list is: ****** |
| |
| (use meta list to connect cluster) |
| >./admin-cli -m ****** |
| |
| >>> ls |
| app_id status app_name |
| 12 AVAILABLE my_source_app |
| |
| >>>use my_source_app |
| >>> dup add -c my_target_cluster -p |
| successfully add duplication [dupid: 1669972761] |
| |
| >>> dup list |
| [ |
| { |
| "dupid": 1692240106, |
| "status": "DS_LOG", |
| "remote": "tjsrv-account", |
| "create_ts": 1692240106066, |
| "fail_mode": "FAIL_SLOW" |
| } |
| ] |
| </code></pre></div></div> |
| |
| <p> By using the “dup add” command, the pegasus table “my_source_app” in the “bjsrv-account” cluster will be replicated to the “tjsrv-account” cluster in near real-time. This means that every write operation in the Beijing data center will eventually be replicated to the Tianjin data center. Duplication uses log asynchronous replication to achieve cross-cluster synchronization, which can be compared to MySQL’s binlog replication and HBase replication.</p> |
| |
| <p> The duplication function is based on the granularity of tables, and you can implement duplication for only a part of the tables in the cluster. The table names of the two clusters for duplication need to be consistent, but the number of partitions does not need to be the same. For example, users can create tables as follows:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## bjsrv-account |
| >>> create my_source_app -p 128 |
| |
| ## tjsrv-account |
| >>> create my_source_app -p 32 |
| </code></pre></div></div> |
| |
| <h2 id="sample-begin-duplicat-on-a-online-pegasus-table">Sample: Begin duplicat on a online Pegasus Table</h2> |
| |
| <p> Sometimes, an online table may not have considered the need for cross-datacenter synchronization during its initial design, and it is only after serving for a period of time that it is decided to perform duplication. At this time, we need to copy all the existing data from the source cluster to the destination cluster. Because it is an online table, we require that during the copying process:</p> |
| |
| <ol> |
| <li>The service cannot be stopped</li> |
| <li>The incremental data written during the copying process cannot be lost</li> |
| </ol> |
| |
| <p>To meet this requirement, our operational approach is:</p> |
| |
| <ol> |
| <li>First, the source cluster retains all the incremental writes (i.e., WAL logs) from this moment on</li> |
| <li>Move the full-data snapshot (cold backup) of the source cluster to the specified path and wait for the backup cluster (destination cluster) to learn from this data</li> |
| <li>After the target cluster completes learning the existing rdb data, it informs the source cluster to enter the WAL log sending phase</li> |
| <li>After that, the source cluster starts duplication and copies the accumulated incremental writes, sending them to the remote destination cluster.</li> |
| </ol> |
| |
| <table> |
| <thead> |
| <tr> |
| <th>master cluster</th> |
| <th> </th> |
| <th>follower cluster</th> |
| <th> </th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>meta</td> |
| <td>primary</td> |
| <td>primary</td> |
| <td>meta</td> |
| </tr> |
| <tr> |
| <td> </td> |
| <td>0.A period RPC sending task will be found during replica server initialization,which is used to communicate between replica server and meta server</td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td>0. Recive the RPC send by replica server,summary duplication tasks and calculate progress and then reply replica server.</td> |
| <td> </td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td>1.Initiate a request to add a duplication task for the table with the command “add_duplication”, and add the related “dup_info”.</td> |
| <td> </td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td>2. Enter dup status <strong>DS_PREPARE</strong>，synchronization checkpoint</td> |
| <td>3. Get the new dup_info from meta，build a class named replica_duplicator，use trigger_manual_emergency_checkpoint generate checkpoint</td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td>4. Received replica server report that all checkpoints have been generated, and now starting to create the table “create_follower_app_for_duplication” for the standby cluster.</td> |
| <td>–> –>–> RPC_CM_CREATE_APP->–></td> |
| <td>–> –>–> –> With master pegasus table info–> –> –> –></td> |
| <td>5.Received RPC_CM_CREATE_APP request, starting to create the table “duplicate_checkpoint”.</td> |
| </tr> |
| <tr> |
| <td> </td> |
| <td><– <– <– <– <– <– <– <– <– <– <– <– <-</td> |
| <td><– <– <– <–Create Table success return ERR_OK <– <-</td> |
| <td>6.Using the checkpoint of the master pegasus table. Send a request to fetch the checkpoint. The ‘nfs’ underlying method “async_duplicate_checkpoint_from_master_replica” is called to copy target checkpoint.</td> |
| </tr> |
| <tr> |
| <td>7. Received ERR_OK response,and enter the dup status <strong>DS_APP</strong></td> |
| <td> </td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td>8. In the next communication round, check the created table in the DS_APP state. If there are no errors, enter the dup status <strong>DS_LOG</strong> “check_follower_app_if_create_completed”.</td> |
| <td> </td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td> </td> |
| <td>9.The replica server first learns that the status has switched to <strong>DS_LOG</strong>, and starts dup the part of the PLOG data using “start_dup_log”.</td> |
| <td> </td> |
| <td> </td> |
| </tr> |
| <tr> |
| <td> </td> |
| <td>10.”load” function replay the PLOG , and “ship” function package and send those mutation</td> |
| <td>11.As a server, receive the “ship” package, unpack it, and process it based on the specific RPC type. contains.pegasus_write_service::duplicate</td> |
| <td> </td> |
| </tr> |
| </tbody> |
| </table> |
| |
| <p>Below are the steps required to enable duplication for an online pegasus table:</p> |
| |
| <h2 id="step-1-set-the-cluster-duplication-parameters-">Step 1: Set the cluster duplication parameters .</h2> |
| |
| <p> The relevant parameters under the replication and duplication-group items on both the master cluster and backup clusters must be consistent. The master cluster refers to the sender of the synchronized data, while the backup cluster refers to the receiver.</p> |
| |
| <p>Example configuration for the master cluster:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[replication] |
| duplication_enabled = true |
| duplicate_log_batch_bytes = 4096 # 0 meaning no batch，usually we set it to 4096 |
| |
| [pegasus.clusters] |
| # the master cluster have to set the meta address of backup cluster |
| tjsrv-account = xxxxxxxxx |
| |
| # both clusters which have been join in duplication should register the cluster_id of each other |
| [[duplication-group]] |
| bjsrv-account = 1 |
| tjsrv-account = 2 |
| </code></pre></div></div> |
| |
| <p>Example configuration for the backup cluster:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[replication] |
| duplication_enabled = true |
| duplicate_log_batch_bytes = 4096 |
| |
| [pegasus.clusters] |
| # backup cluster can set nothing in this config project |
| |
| [[duplication-group]] |
| bjsrv-account = 1 |
| tjsrv-account = 2 |
| </code></pre></div></div> |
| |
| <h2 id="step-2-connect-to-the-domain-proxy-system-on-demand">Step 2: Connect to the domain proxy system on demand.</h2> |
| |
| <p> The main purpose of cross-datacenter duplication is to provide datacenter-level disaster recovery. In order to provide the ability to switch traffic across datacenters, business applications that require duplication must connect to the meta-proxy. The logic of the meta-proxy is that the client accesses the proxy, the proxy looks up the path of the corresponding table on ZooKeeper, obtains a real cluster meta address, and then accesses this meta. The path configuration of the meta-proxy on ZooKeeper is at the table level, so it is best to avoid having tables with the same name for different businesses within the same region. Of course, if the business side can modify the meta address by themselves, they do not need to connect to the domain proxy system.</p> |
| |
| <h2 id="step-3-enable-duplication">Step 3: Enable duplication.</h2> |
| |
| <p>Before enabling duplication, it is necessary to consider whether to synchronize all data of the table (full data synchronization) or only synchronize from the current moment (incremental synchronization).</p> |
| |
| <ol> |
| <li>If a full copy is performed, the checkpoint needs to be copied, and the backup cluster cannot have a table with the same name. This command creates a new table and enables the primary-backup task.</li> |
| <li>If incremental synchronization is performed, the checkpoint does not need to be copied (i.e., only incremental data is synchronized), and it is necessary to ensure that the backup cluster has already created a table with the same name.</li> |
| </ol> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># typing in pegasus admin-cli |
| # dup add -c {cluster_name} -p {-p meaing full data synchronization，without -p meaing incremental synchronization} |
| |
| >> use my_source_app |
| >> dup add -c tjsrv-account -p |
| successfully add duplication [dupid: 1669972761] |
| </code></pre></div></div> |
| |
| <h3 id="step-4-pauserestartremove-a-duplication">Step 4: Pause/Restart/Remove a duplication.</h3> |
| |
| <pre><code class="language-C#"># TIPS：only in DS_LOG status duplication can be paused |
| >> dup pause/start/remove -d {the id of current duplication，use dup list can check it} |
| </code></pre> |
| |
| <p><strong>FYI：After pause, the accumulated unsent logs continue to pile up. After remove, the unsent logs are directly cleared to zero.</strong></p> |
| |
| <h2 id="reliability-of-duplication">Reliability of duplication</h2> |
| |
| <h3 id="automatic-fault-handling">Automatic fault handling.</h3> |
| |
| <p>Duplication is an online service integrated in ReplicaServer, so we have high requirements for the reliability of this function. To deal with various failures that may occur during duplication, we provide several options for fault handling:</p> |
| |
| <ul> |
| <li>fail-slow: In this fault handling mode, duplication will retry indefinitely for any failure. Our operations personnel need to set alarms for some key monitoring items to be aware of the occurrence of failures. This is the default fault handling mode for Pegasus.</li> |
| <li>fail-skip: When a failure occurs, if multiple retries are unsuccessful, duplication will skip the current batch of data and replicate the next batch of data. This is suitable for business scenarios that can tolerate data loss. This option trades data loss for better availability.</li> |
| </ul> |
| |
| <p>Commond</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set_dup_fail_mode <app_name> <dupid> <slow|skip> |
| </code></pre></div></div> |
| |
| <h3 id="important-monitoring">Important Monitoring</h3> |
| |
| <p>In the operation and maintenance of duplication , we recommend observing several core monitoring indicators to continuously monitor the service situation:</p> |
| |
| <ul> |
| <li> |
| <p>collector<em>app.pegasus</em>app.stat.dup_failed_shipping_ops#<app_name>: The number of failed write replication RPCs. Failure often means that the remote cluster or cross-cluster network is unavailable.</app_name></p> |
| </li> |
| <li> |
| <p>replica<em>app.pegasus</em>dup.time_lag_ms@<app_name>: P99 data replication delay. That is, how long it takes for a write from the source cluster to reach the destination cluster.</app_name></p> |
| </li> |
| <li> |
| <p>replica<em>app.pegasus</em>dup.lagging_writes@<app_name>: The number of writes that have taken too long to reach the destination cluster. We can configure a threshold, and a replication that takes longer than this threshold will be recorded once:</app_name></p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[pegasus.server] |
| dup_lagging_write_threshold_ms = 10000 |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>replica<em>eon.replica_stub</em>dup.pending_mutations_count: The number of writes that are piled up in the source cluster and have not yet been replicated. If everything is normal, this monitoring item will remain stable at a certain value. When a fault occurs in a certain link of duplication, there will often be a large number of writes piled up, and this value will continue to rise.</p> |
| </li> |
| <li> |
| <p>replica<em>eon.replica_stub</em>dup.load_file_failed_count: The number of times the source cluster failed to read the log file. Reading the log file is a key link in duplication. If this link fails for some reason, it will cause duplication to be blocked.</p> |
| </li> |
| </ul> |
| |
| <h2 id="the-metadata-of-duplication">The metadata of duplication</h2> |
| |
| <p>The metadata of the duplication will be persisted on Zookeeper through MetaServer, and its storage path is as follows:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> |
| <cluster_root> <app_id> <dupid> |
| | | | |
| | | | |
| [zk: 127.0.0.1:22181(CONNECTED) 0] get /pegasus/bjsrv-account/0.0.x.x/apps/1/duplication/1537336970 |
| |
| {"remote":"tjsrv-account","status":"DS_START","create_timestamp_ms":1537336970483} |
| </code></pre></div></div> |
| |
| <h2 id="complete-configuration-parameters-item-list">Complete configuration parameters item list</h2> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[replication] |
| # default is true |
| duplication_enabled = true |
| |
| [pegasus.clusters] |
| # master cluster HAVE TO config it |
| tjsrv-account = 127.0.0.1:51601,127.0.0.1:51601 |
| |
| [pegasus.server] |
| dup_lagging_write_threshold_ms = 10000 |
| |
| [duplication-group] |
| bjsrv-account = 1 |
| tjsrv-account = 2 |
| </code></pre></div></div> |
| |
| <p> We add a prefix of <code class="language-plaintext highlighter-rouge">timestamp+cluster_id</code> before each piece of data. The timestamp is the time when the data is written to Pegasus, and the cluster_id is configured in the duplication-group. The cluster_id for bjsrv is 1, and the cluster_id for tjsrv is 2.</p> |
| |
| <p> The purpose of cluster_id is that in case of a write conflict, for example, if tjsrv and bjsrv both write to the key “user_1” at the same time, the system will first check the timestamps of the two writes and take the one with the larger timestamp as the final value.</p> |
| |
| <p> In the extremely rare case where the timestamps are the same, the one with the larger cluster_id will be taken as the final value. Using this mechanism, we can ensure that the final values of the two clusters are always the same.</p> |
| |
| <h2 id="complete-list-of-monitoring-items">Complete list of monitoring items</h2> |
| |
| <table> |
| <thead> |
| <tr> |
| <th>monitoring items</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.log_read_bytes_rate</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.log_read_mutations_rate</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.shipped_bytes_rate</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.confirmed_rate</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.pending_mutations_count</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.time_lag(ms)</code> (XiaoMi/rdsn#393)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica_stub*dup.load_file_failed_count</code> (XiaoMi/rdsn#425)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*eon.replica*dup.disabled_non_idempotent_write_count@<app_name></code> (XiaoMi/rdsn#411)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*app.pegasus*dup_shipped_ops@<gpid></code> (#399)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*app.pegasus*dup_failed_shipping_ops@<gpid></code> (#399)</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*app.pegasus*dup.time_lag_ms@<app_name></code> #526</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">replica*app.pegasus*dup.lagging_writes@<app_name></code> #526</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">collector*app.pegasus*app.stat.duplicate_qps#<app_name></code> #520</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">collector*app.pegasus*app.stat.dup_shipped_ops#<app_name></code> #520</td> |
| </tr> |
| <tr> |
| <td><code class="language-plaintext highlighter-rouge">collector*app.pegasus*app.stat.dup_failed_shipping_ops#<app_name></code> #520</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| <h2 id="complete-http-interface-list">Complete HTTP interface list</h2> |
| |
| <ul> |
| <li> |
| <p><code class="language-plaintext highlighter-rouge">http://0.0.0.0:34602/meta/app/duplication?name=temp</code></p> |
| </li> |
| <li> |
| <p><code class="language-plaintext highlighter-rouge">http://0.0.0.0:34801/replica/duplication?appid=2</code></p> |
| </li> |
| </ul> |
| |
| <h2 id="known-limitations">Known Limitations</h2> |
| |
| <ul> |
| <li>Duplication is not recommended to have two data copies written simultaneously in two different data centers. Based on our experience, it is usually acceptable to have only one data copy written at a time when dealing with two data centers.</li> |
| <li>Users can distribute their data evenly between the tjsrv and bjsrv data centers.Duplication can ensure that in the event of a data center outage,so that only a few seconds of data will be lost (assuming stable network connections between the data centers).</li> |
| </ul> |
| |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <!-- right panel --> |
| <div class="dashboard-panel is-small is-scrollable is-hidden-mobile"> |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| Table of contents |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#get-started">Get started</a></li> |
| <li><a href="#sample-begin-duplicat-on-a-online-pegasus-table">Sample: Begin duplicat on a online Pegasus Table</a></li> |
| <li><a href="#step-1-set-the-cluster-duplication-parameters-">Step 1: Set the cluster duplication parameters .</a></li> |
| <li><a href="#step-2-connect-to-the-domain-proxy-system-on-demand">Step 2: Connect to the domain proxy system on demand.</a></li> |
| <li><a href="#step-3-enable-duplication">Step 3: Enable duplication.</a> |
| <ul> |
| <li><a href="#step-4-pauserestartremove-a-duplication">Step 4: Pause/Restart/Remove a duplication.</a></li> |
| </ul> |
| </li> |
| <li><a href="#reliability-of-duplication">Reliability of duplication</a> |
| <ul> |
| <li><a href="#automatic-fault-handling">Automatic fault handling.</a></li> |
| <li><a href="#important-monitoring">Important Monitoring</a></li> |
| </ul> |
| </li> |
| <li><a href="#the-metadata-of-duplication">The metadata of duplication</a></li> |
| <li><a href="#complete-configuration-parameters-item-list">Complete configuration parameters item list</a></li> |
| <li><a href="#complete-list-of-monitoring-items">Complete list of monitoring items</a></li> |
| <li><a href="#complete-http-interface-list">Complete HTTP interface list</a></li> |
| <li><a href="#known-limitations">Known Limitations</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| <script> |
| docsearch({ |
| container: '#docsearch', |
| appId: 'QRN30RBW0S', |
| indexName: 'pegasus-apache', |
| apiKey: 'd3a3252fa344359766707a106c4ed88f', |
| debug: true |
| }); |
| </script> |
| |
| </body> |
| |
| </html> |
