| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Meta Recovery</title> |
| <link rel="stylesheet" href="/zh/assets/css/app.css"> |
| <link rel="shortcut icon" href="/zh/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/zh/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/zh/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Meta Recovery | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.3" /> |
| <meta property="og:title" content="Meta Recovery" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="功能目标 在Pegasus bootstrap的过程中,meta server需要先从zookeeper上拉取table的元信息以及所有replica的拓扑结构,再开始服务。" /> |
| <meta property="og:description" content="功能目标 在Pegasus bootstrap的过程中,meta server需要先从zookeeper上拉取table的元信息以及所有replica的拓扑结构,再开始服务。" /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2024-04-22T06:39:52+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Meta Recovery" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2024-04-22T06:39:52+00:00","datePublished":"2024-04-22T06:39:52+00:00","description":"功能目标 在Pegasus bootstrap的过程中,meta server需要先从zookeeper上拉取table的元信息以及所有replica的拓扑结构,再开始服务。","headline":"Meta Recovery","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/meta-recovery"},"url":"/administration/meta-recovery"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| |
| <body> |
| <div class="dashboard is-full-height"> |
| <!-- left panel --> |
| <div class="dashboard-panel is-medium is-hidden-mobile pl-0"> |
| <div class="dashboard-panel-header has-text-centered"> |
| <a href="/zh/"> |
| <img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;"> |
| </a> |
| |
| </div> |
| <div class="dashboard-panel-main is-scrollable pl-6"> |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label">Pegasus 产品文档</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/downloads" |
| class=""> |
| 下载 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">编译构建</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/build/compile-by-docker" |
| class=""> |
| 使用 Docker 完成编译(推荐) |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/docs/build/compile-from-source" |
| class=""> |
| 从源码编译 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">客户端库</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/clients/java-client" |
| class=""> |
| Java 客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/cpp-client" |
| class=""> |
| C++ 客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class=""> |
| Golang 客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/python-client" |
| class=""> |
| Python 客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/node-client" |
| class=""> |
| NodeJS 客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/scala-client" |
| class=""> |
| Scala 客户端 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">生态工具</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/tools/shell" |
| class=""> |
| Pegasus Shell 工具 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class=""> |
| 集群管理命令行 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/pegic" |
| class=""> |
| 数据访问命令行 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">用户接口</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/api/ttl" |
| class=""> |
| TTL |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/single-atomic" |
| class=""> |
| 单行原子操作 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/redis" |
| class=""> |
| Redis 适配 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/geo" |
| class=""> |
| GEO 支持 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/http" |
| class=""> |
| HTTP 接口 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">高效运维</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/administration/deployment" |
| class=""> |
| 集群部署 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/config" |
| class=""> |
| 配置说明 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rebalance" |
| class=""> |
| 负载均衡 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/monitoring" |
| class=""> |
| 可视化监控 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rolling-update" |
| class=""> |
| 集群重启和升级 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/scale-in-out" |
| class=""> |
| 集群扩容缩容 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/resource-management" |
| class=""> |
| 资源管理 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/cold-backup" |
| class=""> |
| 冷备份 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/meta-recovery" |
| class="is-active"> |
| 元数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/replica-recovery" |
| class=""> |
| Replica 数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/zk-migration" |
| class=""> |
| Zookeeper 迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-migration" |
| class=""> |
| Table 迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-soft-delete" |
| class=""> |
| Table 软删除 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-env" |
| class=""> |
| Table 环境变量 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/remote-commands" |
| class=""> |
| 远程命令 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/partition-split" |
| class=""> |
| Partition-Split |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/duplication" |
| class=""> |
| 跨机房同步 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/compression" |
| class=""> |
| 数据压缩 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/throttling" |
| class=""> |
| 流量控制 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/experiences" |
| class=""> |
| 运维经验 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/manual-compact" |
| class=""> |
| Manual Compact 功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/usage-scenario" |
| class=""> |
| Usage Scenario 功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/bad-disk" |
| class=""> |
| 坏盘检修 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/whitelist" |
| class=""> |
| Replica Server 白名单 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/backup-request" |
| class=""> |
| Backup Request |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/hotspot-detection" |
| class=""> |
| 热点检测 |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| </div> |
| </div> |
| |
| <!-- main section --> |
| <div class="dashboard-main is-scrollable"> |
| <nav class="navbar is-hidden-desktop"> |
| <div class="navbar-brand"> |
| <a href="/zh/" class="navbar-item"> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus-square.png"> |
| </a> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/meta-recovery"><strong>En</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Pegasus 产品文档 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/downloads" |
| class="navbar-item "> |
| 下载 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 编译构建 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/build/compile-by-docker" |
| class="navbar-item "> |
| 使用 Docker 完成编译(推荐) |
| </a> |
| |
| <a href="/zh/docs/build/compile-from-source" |
| class="navbar-item "> |
| 从源码编译 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 客户端库 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/clients/java-client" |
| class="navbar-item "> |
| Java 客户端 |
| </a> |
| |
| <a href="/zh/clients/cpp-client" |
| class="navbar-item "> |
| C++ 客户端 |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class="navbar-item "> |
| Golang 客户端 |
| </a> |
| |
| <a href="/zh/clients/python-client" |
| class="navbar-item "> |
| Python 客户端 |
| </a> |
| |
| <a href="/zh/clients/node-client" |
| class="navbar-item "> |
| NodeJS 客户端 |
| </a> |
| |
| <a href="/zh/clients/scala-client" |
| class="navbar-item "> |
| Scala 客户端 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 生态工具 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/tools/shell" |
| class="navbar-item "> |
| Pegasus Shell 工具 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class="navbar-item "> |
| 集群管理命令行 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/pegic" |
| class="navbar-item "> |
| 数据访问命令行 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 用户接口 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/api/ttl" |
| class="navbar-item "> |
| TTL |
| </a> |
| |
| <a href="/zh/api/single-atomic" |
| class="navbar-item "> |
| 单行原子操作 |
| </a> |
| |
| <a href="/zh/api/redis" |
| class="navbar-item "> |
| Redis 适配 |
| </a> |
| |
| <a href="/zh/api/geo" |
| class="navbar-item "> |
| GEO 支持 |
| </a> |
| |
| <a href="/zh/api/http" |
| class="navbar-item "> |
| HTTP 接口 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 高效运维 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/administration/deployment" |
| class="navbar-item "> |
| 集群部署 |
| </a> |
| |
| <a href="/zh/administration/config" |
| class="navbar-item "> |
| 配置说明 |
| </a> |
| |
| <a href="/zh/administration/rebalance" |
| class="navbar-item "> |
| 负载均衡 |
| </a> |
| |
| <a href="/zh/administration/monitoring" |
| class="navbar-item "> |
| 可视化监控 |
| </a> |
| |
| <a href="/zh/administration/rolling-update" |
| class="navbar-item "> |
| 集群重启和升级 |
| </a> |
| |
| <a href="/zh/administration/scale-in-out" |
| class="navbar-item "> |
| 集群扩容缩容 |
| </a> |
| |
| <a href="/zh/administration/resource-management" |
| class="navbar-item "> |
| 资源管理 |
| </a> |
| |
| <a href="/zh/administration/cold-backup" |
| class="navbar-item "> |
| 冷备份 |
| </a> |
| |
| <a href="/zh/administration/meta-recovery" |
| class="navbar-item is-active"> |
| 元数据恢复 |
| </a> |
| |
| <a href="/zh/administration/replica-recovery" |
| class="navbar-item "> |
| Replica 数据恢复 |
| </a> |
| |
| <a href="/zh/administration/zk-migration" |
| class="navbar-item "> |
| Zookeeper 迁移 |
| </a> |
| |
| <a href="/zh/administration/table-migration" |
| class="navbar-item "> |
| Table 迁移 |
| </a> |
| |
| <a href="/zh/administration/table-soft-delete" |
| class="navbar-item "> |
| Table 软删除 |
| </a> |
| |
| <a href="/zh/administration/table-env" |
| class="navbar-item "> |
| Table 环境变量 |
| </a> |
| |
| <a href="/zh/administration/remote-commands" |
| class="navbar-item "> |
| 远程命令 |
| </a> |
| |
| <a href="/zh/administration/partition-split" |
| class="navbar-item "> |
| Partition-Split |
| </a> |
| |
| <a href="/zh/administration/duplication" |
| class="navbar-item "> |
| 跨机房同步 |
| </a> |
| |
| <a href="/zh/administration/compression" |
| class="navbar-item "> |
| 数据压缩 |
| </a> |
| |
| <a href="/zh/administration/throttling" |
| class="navbar-item "> |
| 流量控制 |
| </a> |
| |
| <a href="/zh/administration/experiences" |
| class="navbar-item "> |
| 运维经验 |
| </a> |
| |
| <a href="/zh/administration/manual-compact" |
| class="navbar-item "> |
| Manual Compact 功能 |
| </a> |
| |
| <a href="/zh/administration/usage-scenario" |
| class="navbar-item "> |
| Usage Scenario 功能 |
| </a> |
| |
| <a href="/zh/administration/bad-disk" |
| class="navbar-item "> |
| 坏盘检修 |
| </a> |
| |
| <a href="/zh/administration/whitelist" |
| class="navbar-item "> |
| Replica Server 白名单 |
| </a> |
| |
| <a href="/zh/administration/backup-request" |
| class="navbar-item "> |
| Backup Request |
| </a> |
| |
| <a href="/zh/administration/hotspot-detection" |
| class="navbar-item "> |
| 热点检测 |
| </a> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <nav class="navbar is-hidden-mobile"> |
| <div class="navbar-start w-full"> |
| <div class="navbar-item pl-0 w-full"> |
| <!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input, |
| I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released. |
| Related issue: https://github.com/algolia/docsearch/issues/230--> |
| <div id="docsearch"></div> |
| </div> |
| </div> |
| <div class="navbar-end"> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/meta-recovery"><strong>En</strong></a> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section class="hero is-info lg:mr-3"> |
| <div class="hero-body"> |
| |
| <p class="title is-size-2 is-centered">元数据恢复</p> |
| </div> |
| </section> |
| <section class="section" style="padding-top: 2rem;"> |
| <div class="content"> |
| <h1 id="功能目标">功能目标</h1> |
| <p>在Pegasus bootstrap的过程中,meta server需要先从zookeeper上拉取table的元信息以及所有replica的拓扑结构,再开始服务。</p> |
| |
| <p>元数据恢复的目标就是:<strong>让Pegasus可以不依赖zookeeper的任何信息,完成系统的bootstrap</strong>。</p> |
| |
| <p>具体流程就是:用户只需提供集群有效的replica server的集合;meta server通过和这些replica server交互,尝试重建出table元信息和replica拓扑结构,并写入到新的zookeeper节点,完成bootstrap。</p> |
| |
| <p><strong>注意:元数据恢复功能只是zookeeper数据损坏或者丢失之后的补救措施,运维人员要尽力避免这种情况的发生。</strong></p> |
| |
| <h1 id="操作流程">操作流程</h1> |
| <h2 id="使用onebox集群演示">使用onebox集群演示</h2> |
| <ol> |
| <li> |
| <p>初始化onebox集群</p> |
| |
| <p>只启动一个meta server:</p> |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh clear_onebox |
| ./run.sh start_onebox <span class="nt">-m</span> 1 <span class="nt">-w</span> |
| </code></pre></div> </div> |
| |
| <p>此时通过shell的<code class="language-plaintext highlighter-rouge">cluster_info</code>命令,可以看到zookeeper的节点路径:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zookeeper_root : /pegasus/onebox/x.x.x.x |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>使用bench工具灌数据</p> |
| |
| <p>灌数据是为了测试集群元数据恢复前后数据的完整性:</p> |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh bench <span class="nt">--app_name</span> temp <span class="nt">-t</span> fillseq_pegasus <span class="nt">-n</span> 10000 |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>修改配置文件</p> |
| |
| <p>使用以下命令修改meta server的配置文件:</p> |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'s@/pegasus/onebox@/pegasus/onebox_recovery@'</span> onebox/meta1/config.ini |
| <span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'s@recover_from_replica_server = false@recover_from_replica_server = true@'</span> onebox/meta1/config.ini |
| </code></pre></div> </div> |
| |
| <p>以上命令将配置文件<code class="language-plaintext highlighter-rouge">onebox/meta1/config.ini</code>中的zookeeper路径进行修改,并设置为recovery模式:</p> |
| <ul> |
| <li><code class="language-plaintext highlighter-rouge">cluster_root = /pegasus/onebox/x.x.x.x</code> 改为 <code class="language-plaintext highlighter-rouge">cluster_root = /pegasus/onebox_recovery/x.x.x.x</code></li> |
| <li><code class="language-plaintext highlighter-rouge">distributed_lock_service_parameters = /pegasus/onebox/x.x.x.x</code> 改为 <code class="language-plaintext highlighter-rouge">distributed_lock_service_parameters = /pegasus/onebox_recovery/x.x.x.x</code></li> |
| <li><code class="language-plaintext highlighter-rouge">recover_from_replica_server = false</code> 改为 <code class="language-plaintext highlighter-rouge">recover_from_replica_server = true</code></li> |
| </ul> |
| </li> |
| <li> |
| <p>重启meta</p> |
| |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh stop_onebox_instance <span class="nt">-m</span> 1 |
| ./run.sh start_onebox_instance <span class="nt">-m</span> 1 |
| </code></pre></div> </div> |
| |
| <p>重启成功后,meta server会进入recovery模式,此时除了start_recovery请求,所有其他RPC请求都会返回ERR_UNDER_RECOVERY,譬如,使用shell的<code class="language-plaintext highlighter-rouge">ls</code>命令得到结果如下:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> ls |
| list apps failed, error=ERR_UNDER_RECOVERY |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>通过shell发送recover命令</p> |
| |
| <p>首先准备<code class="language-plaintext highlighter-rouge">recover_node_list</code>文件,用于指定有效的replica server节点,格式为每行一个节点,譬如:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># comment line |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| </code></pre></div> </div> |
| |
| <p>通过shell的<code class="language-plaintext highlighter-rouge">recover</code>命令,向meta server发送start_recovery请求:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list |
| Wait seconds: 100 |
| Skip bad nodes: false |
| Skip lost partitions: false |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| ============================= |
| Recover result: ERR_OK |
| </code></pre></div> </div> |
| |
| <p>当返回结果为ERR_OK时,表示恢复成功,可以通过shell的<code class="language-plaintext highlighter-rouge">ls</code>命令看到正常的表信息。</p> |
| |
| <p>通过shell的<code class="language-plaintext highlighter-rouge">cluster_info</code>命令,可以看到zookeeper节点路径已经改变:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zookeeper_root : /pegasus/onebox_recovery/x.x.x.x |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>检查数据完整性</p> |
| |
| <p>使用bench工具查询之前写入的数据是否完整存在:</p> |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh bench <span class="nt">--app_name</span> temp <span class="nt">-t</span> readrandom_pegasus <span class="nt">-n</span> 10000 |
| </code></pre></div> </div> |
| |
| <p>在最后的统计结果中能看到<code class="language-plaintext highlighter-rouge">(10000 of 10000 found)</code>,表示恢复后数据完整存在。</p> |
| </li> |
| <li> |
| <p>修改配置文件并重启meta</p> |
| |
| <p>恢复成功后,需要修改配置文件,重新改回非recovery模式:</p> |
| <ul> |
| <li><code class="language-plaintext highlighter-rouge">recover_from_replica_server = true</code> 改为 <code class="language-plaintext highlighter-rouge">recover_from_replica_server = false</code></li> |
| </ul> |
| |
| <p>重启meta server:</p> |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./run.sh stop_onebox_instance <span class="nt">-m</span> 1 |
| ./run.sh start_onebox_instance <span class="nt">-m</span> 1 |
| </code></pre></div> </div> |
| |
| <p>这样做是避免meta server发生重启时,再次进入recovery模式,使集群变得不可用。</p> |
| </li> |
| </ol> |
| |
| <h2 id="线上集群恢复">线上集群恢复</h2> |
| |
| <p>对线上集群进行元数据恢复时,请遵循上面的<code class="language-plaintext highlighter-rouge">3~7</code>步骤,需注意以下几点:</p> |
| <ul> |
| <li>在<code class="language-plaintext highlighter-rouge">recover_node_list</code>中指定有效的replica server节点时,请保证所有节点都是正常可用的。</li> |
| <li>恢复之前不要忘记将配置文件中<code class="language-plaintext highlighter-rouge">recover_from_replica_server</code>设置为true。</li> |
| <li>只能恢复到zookeeper的新节点或者空节点。</li> |
| <li>恢复完成后重新将配置文件中<code class="language-plaintext highlighter-rouge">recover_from_replica_server</code>设置为false。</li> |
| </ul> |
| |
| <h2 id="常见问题整理">常见问题整理</h2> |
| |
| <ul> |
| <li> |
| <p>恢复到zookeeper的非空节点</p> |
| |
| <p>此时MetaServer应当启动失败并coredump:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>F12:16:26.793 (1488341786793734532 26cc) meta.default0.0000269c00010001: /home/mi/git.xiaomi/Pegasus/pegasus/rdsn/src/dist/replication/meta_server/server_state.cpp:698:initialize_data_structure(): assertion expression: false |
| F12:16:26.793 (1488341786793754317 26cc) meta.default0.0000269c00010001: /home/mi/git.xiaomi/Pegasus/pegasus/rdsn/src/dist/replication/meta_server/server_state.cpp:698:initialize_data_structure(): find apps from remote storage, but [meta_server].recover_from_replica_server = true |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>忘记设置recover_from_replica_server为true</p> |
| |
| <p>meta server会进入正常启动逻辑,从zookeeper上获取的apps为空,但是在config sync过程中发现replica server上有无法识别的replica,出现元数据不一致的情况,会coredump:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>F12:22:21.228 (1488342141228270056 2764) meta.meta_state0.0102000000000001: /home/mi/git.xiaomi/Pegasus/pegasus/rdsn/src/dist/replication/meta_server/server_state.cpp:823:on_config_sync(): assertion expression: false |
| F12:22:21.228 (1488342141228314857 2764) meta.meta_state0.0102000000000001: /home/mi/git.xiaomi/Pegasus/pegasus/rdsn/src/dist/replication/meta_server/server_state.cpp:823:on_config_sync(): gpid(2.7) on node(10.235.114.240:34801) is not exist on meta server, administrator should check consistency of meta data |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>恢复时连不上replica server</p> |
| |
| <p>如果恢复时连不上replica server,recover命令就会执行失败:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list |
| Wait seconds: 100 |
| Skip bad nodes: false |
| Skip lost partitions: false |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| x.x.x.x:34804 |
| ============================= |
| Recover result: ERR_TRY_AGAIN |
| ============================= |
| ERROR: collect app and replica info from node(x.x.x.x:34804) failed with err(ERR_NETWORK_FAILURE), you can skip it by set skip_bad_nodes option |
| ============================= |
| </code></pre></div> </div> |
| |
| <p>可以通过指定<code class="language-plaintext highlighter-rouge">--skip_bad_nodes</code>参数,强制忽略有问题的节点。但是要注意,忽略问题节点可能造成部分partition的备份数不全,有丢数据风险。</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list --skip_bad_nodes |
| Wait seconds: 100 |
| Skip bad nodes: true |
| Skip lost partitions: false |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| ============================= |
| Recover result: ERR_OK |
| ============================= |
| WARNING: collect app and replica info from node(x.x.x.x:34804) failed with err(ERR_NETWORK_FAILURE), skip the bad node |
| WARNING: partition(1.0) only collects 2/3 of replicas, may lost data |
| WARNING: partition(1.1) only collects 2/3 of replicas, may lost data |
| WARNING: partition(1.3) only collects 2/3 of replicas, may lost data |
| WARNING: partition(1.5) only collects 2/3 of replicas, may lost data |
| WARNING: partition(1.7) only collects 2/3 of replicas, may lost data |
| ============================= |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>恢复时发现某个partition的备份数不全</p> |
| |
| <p>当遇到partition备份数不全的情况时,依然能正常恢复,但是会打印告警提示:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list |
| Wait seconds: 100 |
| Skip bad nodes: false |
| Skip lost partitions: false |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| ============================= |
| Recover result: ERR_OK |
| ============================= |
| WARNING: partition(1.0) only collects 1/3 of replicas, may lost data |
| ============================= |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>恢复时发现某个partition没有可用的replica</p> |
| |
| <p>当某个partition完全没有收集到可用的replica备份时,恢复会失败:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list |
| Wait seconds: 100 |
| Skip bad nodes: false |
| Skip lost partitions: false |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| ============================= |
| Recover result: ERR_TRY_AGAIN |
| ============================= |
| ERROR: partition(1.0) has no replica collected, you can force recover it by set skip_lost_partitions option |
| ============================= |
| </code></pre></div> </div> |
| |
| <p>可以通过指定<code class="language-plaintext highlighter-rouge">--skip_lost_partitions</code>参数,强制其继续执行恢复,此时partition(1.0)会初始化为空replica。<strong>此操作需慎重,需确定能够容忍数据丢失</strong>:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> recover -f recover_node_list --skip_lost_partitions |
| Wait seconds: 100 |
| Skip bad nodes: false |
| Skip lost partitions: true |
| Node list: |
| ============================= |
| x.x.x.x:34801 |
| x.x.x.x:34802 |
| x.x.x.x:34803 |
| ============================= |
| Recover result: ERR_OK |
| ============================= |
| WARNING: partition(1.0) has no replica collected, force recover the lost partition to empty |
| ============================= |
| </code></pre></div> </div> |
| </li> |
| <li> |
| <p>恢复软删除的表</p> |
| |
| <p>对于已经删除的表,由于有<a href="table-soft-delete">Table软删除</a>功能,只要没有超过保留时间,replica server上的replica数据就不会被清理,所以该表能被恢复,且被认为是一个正常的未删除的表,也就是说丢掉了删除信息,但是不会丢数据。</p> |
| |
| <p>由于表删除后可以创建新的同名表,所以在恢复过程中可能会发现多个表都使用相同表名,出现表名冲突。此时,id最大的那个表会使用原始表名,其他表的表名都改为<code class="language-plaintext highlighter-rouge">{name}-{id}</code>的形式。</p> |
| </li> |
| </ul> |
| |
| <h1 id="设计与实现">设计与实现</h1> |
| <p>元数据恢复功能的设计与实现:</p> |
| <ul> |
| <li>meta server提供一个配置项,用来标识“当从zookeeper上获取不到任何元数据信息时,是否进入元数据恢复的模式”。</li> |
| <li>shell端提供一个recovery的命令,来触发meta启动元数据恢复流程。</li> |
| <li>如果进入了元数据恢复流程,meta server会接收replica server的探活心跳,并只响应一个特殊的<code class="language-plaintext highlighter-rouge">start_recovery</code>RPC,而不响应任何其他类型的RPC。</li> |
| <li>用户需要指定一个replica server的集合;meta server只和该集合中节点进行通信,响应其<code class="language-plaintext highlighter-rouge">start_recovery</code>RPC,以进行信息收集,用于bootstrap;meta server和任何节点的交互失败都会导致恢复流程的失败。</li> |
| </ul> |
| |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <!-- right panel --> |
| <div class="dashboard-panel is-small is-scrollable is-hidden-mobile"> |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| 本页导航 |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#功能目标">功能目标</a></li> |
| <li><a href="#操作流程">操作流程</a> |
| <ul> |
| <li><a href="#使用onebox集群演示">使用onebox集群演示</a></li> |
| <li><a href="#线上集群恢复">线上集群恢复</a></li> |
| <li><a href="#常见问题整理">常见问题整理</a></li> |
| </ul> |
| </li> |
| <li><a href="#设计与实现">设计与实现</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| <script> |
| docsearch({ |
| container: '#docsearch', |
| appId: 'QRN30RBW0S', |
| indexName: 'pegasus-apache', |
| apiKey: 'd3a3252fa344359766707a106c4ed88f', |
| debug: true |
| }); |
| </script> |
| |
| </body> |
| |
| </html> |