| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Architecture</title> |
| <link rel="stylesheet" href="/assets/css/app.css"> |
| <link rel="shortcut icon" href="/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Architecture | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.3" /> |
| <meta property="og:title" content="Architecture" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="Overall architecture" /> |
| <meta property="og:description" content="Overall architecture" /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2024-04-22T13:02:52+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Architecture" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2024-04-22T13:02:52+00:00","datePublished":"2024-04-22T13:02:52+00:00","description":"Overall architecture","headline":"Architecture","mainEntityOfPage":{"@type":"WebPage","@id":"/overview/architecture"},"url":"/overview/architecture"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| <body> |
| |
| |
| |
| |
| <nav class="navbar is-primary"> |
| <div class="container"> |
| <!--container will be unwrapped when it's in docs--> |
| <div class="navbar-brand"> |
| <a href="/" class="navbar-item "> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus.svg"> |
| </a> |
| <div class="navbar-item"> |
| <a href="/docs" class="button is-primary is-outlined is-inverted"> |
| <span class="icon"><i class="fas fa-book"></i></span> |
| <span>Docs</span> |
| </a> |
| </div> |
| <div class="navbar-item is-hidden-desktop"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-primary is-outlined is-inverted" href="/zh/overview/architecture"><strong>中</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable "> |
| <a href="" |
| class="navbar-link "> |
| |
| <span class="icon" style="margin-right: .25em"> |
| <i class="fas fa-users"></i> |
| </span> |
| |
| <span> |
| ASF |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="https://www.apache.org/" |
| class="navbar-item "> |
| Foundation |
| </a> |
| |
| <a href="https://www.apache.org/licenses/" |
| class="navbar-item "> |
| License |
| </a> |
| |
| <a href="https://www.apache.org/events/current-event.html" |
| class="navbar-item "> |
| Events |
| </a> |
| |
| <a href="https://www.apache.org/foundation/sponsorship.html" |
| class="navbar-item "> |
| Sponsorship |
| </a> |
| |
| <a href="https://www.apache.org/security/" |
| class="navbar-item "> |
| Security |
| </a> |
| |
| <a href="https://privacy.apache.org/policies/privacy-policy-public.html" |
| class="navbar-item "> |
| Privacy |
| </a> |
| |
| <a href="https://www.apache.org/foundation/thanks.html" |
| class="navbar-item "> |
| Thanks |
| </a> |
| |
| </div> |
| </div> |
| |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable "> |
| <a href="/community" |
| class="navbar-link "> |
| |
| <span class="icon" style="margin-right: .25em"> |
| <i class="fas fa-user-plus"></i> |
| </span> |
| |
| <span> |
| Community |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/community/#contact-us" |
| class="navbar-item "> |
| Contact Us |
| </a> |
| |
| <a href="/community/#contribution" |
| class="navbar-item "> |
| Contribution |
| </a> |
| |
| <a href="https://cwiki.apache.org/confluence/display/PEGASUS/Coding+guides" |
| class="navbar-item "> |
| Coding Guides |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/issues?q=is%3Aissue+is%3Aopen+label%3Atype%2Fbug" |
| class="navbar-item "> |
| Bug Tracking |
| </a> |
| |
| <a href="https://cwiki.apache.org/confluence/display/INCUBATOR/PegasusProposal" |
| class="navbar-item "> |
| Apache Proposal |
| </a> |
| |
| </div> |
| </div> |
| |
| |
| |
| <a href="/blogs" |
| class="navbar-item "> |
| |
| <span class="icon" style="margin-right: .25em"> |
| <i class="fas fa-rss"></i> |
| </span> |
| |
| <span>Blog</span> |
| </a> |
| |
| |
| |
| <a href="/docs/downloads" |
| class="navbar-item "> |
| |
| <span class="icon" style="margin-right: .25em"> |
| <i class="fas fa-fire"></i> |
| </span> |
| |
| <span>Releases</span> |
| </a> |
| |
| |
| </div> |
| <div class="navbar-item is-hidden-mobile"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-primary is-outlined is-inverted" href="/zh/overview/architecture"><strong>中</strong></a> |
| |
| </div> |
| </div> |
| </div> |
| </nav> |
| |
| <section class="section"> |
| <div class="container"> |
| <div class="columns is-multiline"> |
| <div class="column is-one-fourth"> |
| |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label"></p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/overview" |
| class=""> |
| Overview |
| </a> |
| </li> |
| |
| <li> |
| <a href="/overview/background" |
| class=""> |
| Background |
| </a> |
| </li> |
| |
| <li> |
| <a href="/overview/architecture" |
| class="is-active"> |
| Architecture |
| </a> |
| </li> |
| |
| <li> |
| <a href="/overview/data-model" |
| class=""> |
| Data Model |
| </a> |
| </li> |
| |
| <li> |
| <a href="/overview/benchmark" |
| class=""> |
| Benchmark |
| </a> |
| </li> |
| |
| <li> |
| <a href="/docs/build/compile-by-docker" |
| class=""> |
| Installation |
| </a> |
| </li> |
| |
| <li> |
| <a href="/overview/onebox" |
| class=""> |
| Onebox |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| |
| </div> |
| <div class="column is-half"> |
| |
| |
| |
| <div class="content"> |
| <h1 id="Architecture">Architecture</h1> |
| <h2 id="overall-architecture">Overall architecture</h2> |
| |
| <p>The overall architecture of the Pegasus is shown in the following figure, which is divided into four parts:</p> |
| <ul> |
| <li>Client Lib: Encapsulates the mapping rules of user data to the storage server (Replica Server), as well as the communication details with the storage server, and provides a concise Key-Value access interfaces for users to use</li> |
| <li>Replica Server: Responsible for data storage and responding to data access requests from Client Lib</li> |
| <li>Meta Server: Responsible for the Replica Server survival detection, data sharding management, load balancing, etc. Usually, a deployment mode of one-leader and multiple-backups is adopted</li> |
| <li>Zookeeper: Responsible for selecting the leader between the Meta Servers to achieve high availability, and store various metadata of the cluster</li> |
| </ul> |
| |
| <p><img src="/assets/images/pegasus-architecture-components.png" alt="pegasus-architecture-components" class="img-responsive" /></p> |
| |
| <h2 id="replica-server">Replica Server</h2> |
| |
| <p>Replica Server is mainly responsible for data storage and access. Provide services using replica as the management unit:</p> |
| <ul> |
| <li>The serving replica can be in the role of either primary replica or secondary replica</li> |
| <li>Using RocksDB as the storage engine at the underlying level</li> |
| <li>Manage commit logs and implement data replication protocols to ensure data consistency</li> |
| </ul> |
| |
| <h2 id="meta-server">Meta Server</h2> |
| |
| <p>The Meta Servers usually adopted in the deployment mode of one-leader and multiple-backups, and all states are persisted to the Zookeeper and the leader is selected through the Zookeeper. When the leader fails, another backup immediately grabs the lock and restores its state from the Zookeeper, becoming the new leader. The functions that Meta Servers are responsible for include:</p> |
| <ul> |
| <li>Cluster initialization</li> |
| <li>Management of Replica Servers</li> |
| <li>Assignment, management, and load balancing scheduling of replicas</li> |
| <li>Creating and deleting tables</li> |
| <li>Respond to client requests and provide the latest routing table (partition configuration) to the client</li> |
| </ul> |
| |
| <h2 id="apache-zookeeper">Apache Zookeeper</h2> |
| |
| <p>In Pegasus, Zookeeper has two main functions:</p> |
| <ul> |
| <li>Cluster metadata storage</li> |
| <li>Meta Server leader selection</li> |
| </ul> |
| |
| <h2 id="client-lib">Client Lib</h2> |
| |
| <p>Client Lib provides users with data access interfaces, including features:</p> |
| <ul> |
| <li>Concise interfaces: Provides simple interfaces for users, encapsulating details such as data locating and fault tolerance internally</li> |
| <li>Simple configuration: Users only need to specify the Meta Server addresses list through the configuration file to access the Pegasus cluster</li> |
| <li>Try to interact directly with Replica Servers and minimize access frequency to Meta Servers to avoid single-point issues, and don’t rely on Zookeeper</li> |
| </ul> |
| |
| </div> |
| </div> |
| <div class="column is-one-fourth is-hidden-mobile" style="padding-left: 3rem"> |
| |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| Table of contents |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#Architecture">Architecture</a> |
| <ul> |
| <li><a href="#overall-architecture">Overall architecture</a></li> |
| <li><a href="#replica-server">Replica Server</a></li> |
| <li><a href="#meta-server">Meta Server</a></li> |
| <li><a href="#apache-zookeeper">Apache Zookeeper</a></li> |
| <li><a href="#client-lib">Client Lib</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| |
| </div> |
| </div> |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| </body> |
| </html> |
