| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Rolling Update</title> |
| <link rel="stylesheet" href="/zh/assets/css/app.css"> |
| <link rel="shortcut icon" href="/zh/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/zh/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/zh/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Rolling Update | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.2" /> |
| <meta property="og:title" content="Rolling Update" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="功能目标" /> |
| <meta property="og:description" content="功能目标" /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2023-11-23T14:51:44+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Rolling Update" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2023-11-23T14:51:44+00:00","datePublished":"2023-11-23T14:51:44+00:00","description":"功能目标","headline":"Rolling Update","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/rolling-update"},"url":"/administration/rolling-update"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| |
| <body> |
| <div class="dashboard is-full-height"> |
| <!-- left panel --> |
| <div class="dashboard-panel is-medium is-hidden-mobile pl-0"> |
| <div class="dashboard-panel-header has-text-centered"> |
| <a href="/zh/"> |
| <img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;"> |
| </a> |
| |
| </div> |
| <div class="dashboard-panel-main is-scrollable pl-6"> |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label">Pegasus产品文档</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/downloads" |
| class=""> |
| 下载 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">编译构建</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/build/compile-by-docker" |
| class=""> |
| 使用Docker完成编译（推荐） |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/docs/build/compile-from-source" |
| class=""> |
| 从源码编译 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">客户端库</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/clients/java-client" |
| class=""> |
| Java客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/cpp-client" |
| class=""> |
| C++客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class=""> |
| Golang客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/python2-client" |
| class=""> |
| Python2客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/python3-client" |
| class=""> |
| Python3客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/node-client" |
| class=""> |
| NodeJS客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/scala-client" |
| class=""> |
| Scala客户端 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">生态工具</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/tools/shell" |
| class=""> |
| Pegasus Shell 工具 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class=""> |
| 集群管理命令行 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/pegic" |
| class=""> |
| 数据访问命令行 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">用户接口</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/api/ttl" |
| class=""> |
| TTL |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/single-atomic" |
| class=""> |
| 单行原子操作 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/redis" |
| class=""> |
| Redis适配 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/geo" |
| class=""> |
| GEO支持 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/http" |
| class=""> |
| HTTP接口 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">高效运维</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/administration/deployment" |
| class=""> |
| 集群部署 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/config" |
| class=""> |
| 配置说明 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rebalance" |
| class=""> |
| 负载均衡 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/monitoring" |
| class=""> |
| 可视化监控 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rolling-update" |
| class="is-active"> |
| 集群升级 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/scale-in-out" |
| class=""> |
| 集群扩容缩容 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/resource-management" |
| class=""> |
| 资源管理 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/cold-backup" |
| class=""> |
| 冷备份 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/meta-recovery" |
| class=""> |
| 元数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/replica-recovery" |
| class=""> |
| Replica数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/zk-migration" |
| class=""> |
| Zookeeper迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-migration" |
| class=""> |
| Table迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-soft-delete" |
| class=""> |
| Table软删除 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-env" |
| class=""> |
| Table环境变量 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/remote-commands" |
| class=""> |
| 远程命令 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/partition-split" |
| class=""> |
| Partition-Split |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/duplication" |
| class=""> |
| 跨机房同步 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/compression" |
| class=""> |
| 数据压缩 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/throttling" |
| class=""> |
| 流量控制 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/experiences" |
| class=""> |
| 运维经验 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/manual-compact" |
| class=""> |
| Manual Compact功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/usage-scenario" |
| class=""> |
| Usage Scenario功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/bad-disk" |
| class=""> |
| 坏盘检修 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/whitelist" |
| class=""> |
| 白名单 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/backup-request" |
| class=""> |
| Backup Request |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/hotspot-detection" |
| class=""> |
| 热点检测 |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| </div> |
| </div> |
| |
| <!-- main section --> |
| <div class="dashboard-main is-scrollable"> |
| <nav class="navbar is-hidden-desktop"> |
| <div class="navbar-brand"> |
| <a href="/zh/" class="navbar-item"> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus-square.png"> |
| </a> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/rolling-update"><strong>En</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Pegasus产品文档 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/downloads" |
| class="navbar-item "> |
| 下载 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 编译构建 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/build/compile-by-docker" |
| class="navbar-item "> |
| 使用Docker完成编译（推荐） |
| </a> |
| |
| <a href="/zh/docs/build/compile-from-source" |
| class="navbar-item "> |
| 从源码编译 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 客户端库 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/clients/java-client" |
| class="navbar-item "> |
| Java客户端 |
| </a> |
| |
| <a href="/zh/clients/cpp-client" |
| class="navbar-item "> |
| C++客户端 |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class="navbar-item "> |
| Golang客户端 |
| </a> |
| |
| <a href="/zh/clients/python2-client" |
| class="navbar-item "> |
| Python2客户端 |
| </a> |
| |
| <a href="/zh/clients/python3-client" |
| class="navbar-item "> |
| Python3客户端 |
| </a> |
| |
| <a href="/zh/clients/node-client" |
| class="navbar-item "> |
| NodeJS客户端 |
| </a> |
| |
| <a href="/zh/clients/scala-client" |
| class="navbar-item "> |
| Scala客户端 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 生态工具 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/tools/shell" |
| class="navbar-item "> |
| Pegasus Shell 工具 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class="navbar-item "> |
| 集群管理命令行 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/pegic" |
| class="navbar-item "> |
| 数据访问命令行 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 用户接口 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/api/ttl" |
| class="navbar-item "> |
| TTL |
| </a> |
| |
| <a href="/zh/api/single-atomic" |
| class="navbar-item "> |
| 单行原子操作 |
| </a> |
| |
| <a href="/zh/api/redis" |
| class="navbar-item "> |
| Redis适配 |
| </a> |
| |
| <a href="/zh/api/geo" |
| class="navbar-item "> |
| GEO支持 |
| </a> |
| |
| <a href="/zh/api/http" |
| class="navbar-item "> |
| HTTP接口 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 高效运维 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/administration/deployment" |
| class="navbar-item "> |
| 集群部署 |
| </a> |
| |
| <a href="/zh/administration/config" |
| class="navbar-item "> |
| 配置说明 |
| </a> |
| |
| <a href="/zh/administration/rebalance" |
| class="navbar-item "> |
| 负载均衡 |
| </a> |
| |
| <a href="/zh/administration/monitoring" |
| class="navbar-item "> |
| 可视化监控 |
| </a> |
| |
| <a href="/zh/administration/rolling-update" |
| class="navbar-item is-active"> |
| 集群升级 |
| </a> |
| |
| <a href="/zh/administration/scale-in-out" |
| class="navbar-item "> |
| 集群扩容缩容 |
| </a> |
| |
| <a href="/zh/administration/resource-management" |
| class="navbar-item "> |
| 资源管理 |
| </a> |
| |
| <a href="/zh/administration/cold-backup" |
| class="navbar-item "> |
| 冷备份 |
| </a> |
| |
| <a href="/zh/administration/meta-recovery" |
| class="navbar-item "> |
| 元数据恢复 |
| </a> |
| |
| <a href="/zh/administration/replica-recovery" |
| class="navbar-item "> |
| Replica数据恢复 |
| </a> |
| |
| <a href="/zh/administration/zk-migration" |
| class="navbar-item "> |
| Zookeeper迁移 |
| </a> |
| |
| <a href="/zh/administration/table-migration" |
| class="navbar-item "> |
| Table迁移 |
| </a> |
| |
| <a href="/zh/administration/table-soft-delete" |
| class="navbar-item "> |
| Table软删除 |
| </a> |
| |
| <a href="/zh/administration/table-env" |
| class="navbar-item "> |
| Table环境变量 |
| </a> |
| |
| <a href="/zh/administration/remote-commands" |
| class="navbar-item "> |
| 远程命令 |
| </a> |
| |
| <a href="/zh/administration/partition-split" |
| class="navbar-item "> |
| Partition-Split |
| </a> |
| |
| <a href="/zh/administration/duplication" |
| class="navbar-item "> |
| 跨机房同步 |
| </a> |
| |
| <a href="/zh/administration/compression" |
| class="navbar-item "> |
| 数据压缩 |
| </a> |
| |
| <a href="/zh/administration/throttling" |
| class="navbar-item "> |
| 流量控制 |
| </a> |
| |
| <a href="/zh/administration/experiences" |
| class="navbar-item "> |
| 运维经验 |
| </a> |
| |
| <a href="/zh/administration/manual-compact" |
| class="navbar-item "> |
| Manual Compact功能 |
| </a> |
| |
| <a href="/zh/administration/usage-scenario" |
| class="navbar-item "> |
| Usage Scenario功能 |
| </a> |
| |
| <a href="/zh/administration/bad-disk" |
| class="navbar-item "> |
| 坏盘检修 |
| </a> |
| |
| <a href="/zh/administration/whitelist" |
| class="navbar-item "> |
| 白名单 |
| </a> |
| |
| <a href="/zh/administration/backup-request" |
| class="navbar-item "> |
| Backup Request |
| </a> |
| |
| <a href="/zh/administration/hotspot-detection" |
| class="navbar-item "> |
| 热点检测 |
| </a> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <nav class="navbar is-hidden-mobile"> |
| <div class="navbar-start w-full"> |
| <div class="navbar-item pl-0 w-full"> |
| <!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input, |
| I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released. |
| Related issue: https://github.com/algolia/docsearch/issues/230--> |
| <div id="docsearch"></div> |
| </div> |
| </div> |
| <div class="navbar-end"> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/rolling-update"><strong>En</strong></a> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section class="hero is-info lg:mr-3"> |
| <div class="hero-body"> |
| |
| <p class="title is-size-2 is-centered">集群升级</p> |
| </div> |
| </section> |
| <section class="section" style="padding-top: 2rem;"> |
| <div class="content"> |
| <h1 id="功能目标">功能目标</h1> |
| |
| <p>当需要升级server版本或者修改config配置时，都需要对集群进行升级。对于分布式集群来说，常用的升级方法就是<strong>滚动升级(Rolling-Update)</strong>，即不停止服务，对一台一台server逐个进行升级。</p> |
| |
| <p>集群升级的重要目标在于<strong>平稳</strong>，即不停服，并且对可用性的影响降至最低。为了达到这个目标，我们先看看在升级过程中哪些地方可能会影响可用性：</p> |
| <ul> |
| <li>replica server进程被kill后，该进程服务的replica无法提供服务： |
| <ul> |
| <li>对于primary replica：因为直接向客户端提供读写服务，所以进程kill后肯定会影响读写，需要等metaserver重新分派新的primary replica后才能恢复。meta server通过心跳感知replica server的存活状态，failure detection的时间延迟取决于配置参数<code class="language-plaintext highlighter-rouge">fd_grace_seconds</code>，通常配置为10秒，即最多需要经过<strong>10秒</strong>，meta server才能知道replica server挂了，然后重新分派新的primary replica。</li> |
| <li>对于secondary replica：由于不服务读，所以理论上对读无影响。但是会影响写，因为一致性协议要求一主两备都写成功，写操作才能提交。进程kill后，primary replica在执行写操作过程中会发现该secondary replica已失联，然后通知meta server将其踢掉，经过<code class="language-plaintext highlighter-rouge">reconfiguration</code>阶段后变成一主一备，继续提供写服务。在切换过程中尚未完成的写操作，即使有<code class="language-plaintext highlighter-rouge">reconciliation</code>阶段重新执行，但客户端那边大概率已经超时了，对可用性有一定影响。但是这个影响相对小些，因为<code class="language-plaintext highlighter-rouge">reconfiguration</code>的速度是比较快的，通常在<strong>1秒</strong>以内就能完成。</li> |
| </ul> |
| </li> |
| <li>升级meta server：升级meta server对可用度的影响几乎可以忽略不计，因为客户端会在本地缓存各partition的服务节点信息，通常情况下并不需要向meta server查询，因此meta server重启过程中的短暂失联对客户端基本没有影响。不过考虑到meta server需要与replica server维持心跳，所以要避免连续kill meta server进程，造成replica server心跳失联的风险。</li> |
| <li>升级collector：升级collector对可用度没有影响。但是可用度统计是在collector上进行的，所以可能会对统计数据有轻微影响。</li> |
| </ul> |
| |
| <p>因此，在集群升级过程要提高可用性，需要考虑如下几点：</p> |
| <ul> |
| <li>一次只能升级一个进程，且在该进程重启并完全恢复进入服务状态后，才能升级下一个进程。 |
| <ul> |
| <li>因为如果升级一个进程后，集群没有恢复到完全健康状态，有的partition还只有一主一备，这时再kill一个replica server的话，很可能进入只有一主的状态，无法提供写服务。</li> |
| <li>另外，等待集群所有partition都恢复三备份后再继续升级下一个进程，也能有效降低数据丢失的风险。</li> |
| </ul> |
| </li> |
| <li>尽量主动迁移replica，而不是被动迁移replica，避免failure detection的时间延迟影响可用度。 |
| <ul> |
| <li>被动迁移需要等待failure detection来感知节点失联，而主动迁移就是在kill掉replica server之前，先将这个进程服务的primary replica都迁移到其他节点上，这个<code class="language-plaintext highlighter-rouge">reconfiguration</code>过程是很快的，基本1秒以内完成。</li> |
| <li>更进一步，还可以在kill掉replica server之前，将这个进程服务的secondary replica手动降级，将<code class="language-plaintext highlighter-rouge">reconfiguration</code>过程由“写失败被动触发”变为“主动触发”，也能降低对可用度的影响。</li> |
| </ul> |
| </li> |
| <li>尽量减少进程重启时恢复过程的工作量，缩短进程重启时间。 |
| <ul> |
| <li>replica server在重启时需要replay log来恢复数据。如果直接kill掉，需要replay的数据量可能很大。但是如果在kill之前，先主动触发memtable的flush操作，让内存数据先落地，在重启时需要replay的数据量就会大大减少，重启时间会缩短很多，而整个集群升级所需的时间也能大大缩短。</li> |
| </ul> |
| </li> |
| <li>尽量减少不必要的节点间数据拷贝，避免因为增加CPU/网络/IO负载影响可用度。 |
| <ul> |
| <li>replica server挂掉后，部分partition进入一主一备的状态。如果meta server立即在其他replica server上补充备份，会带来大量的跨节点数据拷贝，增加CPU/网络/IO负载压力，影响集群稳定性。Pegasus解决这个问题的办法是，允许在一段时间内维持一主一备状态，给原来的replica server进行恢复的机会。如果长时间没有恢复，才会在新的replica server上补充备份。这样兼顾了数据的安全性和集群的稳定性。可以通过配置参数<code class="language-plaintext highlighter-rouge">replica_assign_delay_ms_for_dropouts</code>控制等待时间，默认为10分钟。</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h1 id="升级流程">升级流程</h1> |
| |
| <h2 id="高可用升级">高可用升级</h2> |
| <p>根据以上对高可用度的考虑，我们建议完善的升级流程如下：</p> |
| <ul> |
| <li>准备好新的Server程序包和配置文件</li> |
| <li>使用shell工具将集群的meta level设置为steady，关闭<a href="rebalance">负载均衡功能</a>，避免不必要的replica迁移 |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> set_meta_level steady |
| </code></pre></div> </div> |
| </li> |
| <li>升级replica server进程，采用逐个升级的策略。升级单个replica server： |
| <ul> |
| <li>通过shell向meta server发送<a href="remote-commands#meta-server">远程命令</a>，禁掉<code class="language-plaintext highlighter-rouge">add_secondary</code>操作： |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> remote_command -t meta-server meta.lb.add_secondary_max_count_for_one_node 0 |
| </code></pre></div> </div> |
| </li> |
| <li>通过migrate_node命令，将replica server上的primary replica都迁走： |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./run.sh migrate_node <span class="nt">-c</span> <span class="nv">$meta_list</span> <span class="nt">-n</span> <span class="nv">$node</span> <span class="nt">-t</span> run |
| </code></pre></div> </div> |
| <p>通过shell的<code class="language-plaintext highlighter-rouge">nodes -d</code>命令查看该节点的服务replica情况，等待primary replica的个数变为0；如果长时间不变为0，重新执行上面命令。</p> |
| </li> |
| <li>通过downgrade_node命令，将replica server上的secondary replica都降级为INACTIVE： |
| <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./run.sh downgrade_node <span class="nt">-c</span> <span class="nv">$meta_list</span> <span class="nt">-n</span> <span class="nv">$node</span> <span class="nt">-t</span> run |
| </code></pre></div> </div> |
| <p>通过shell的<code class="language-plaintext highlighter-rouge">nodes -d</code>命令查看该节点的服务replica情况，等待secondary replica的个数变为0；如果长时间不变为0，重新执行上面命令。</p> |
| </li> |
| <li>通过shell向replica server发送远程命令，将所有replica都关闭，以触发flush操作，将数据都落地： |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> remote_command -l $node replica.kill_partition |
| </code></pre></div> </div> |
| <p>等待大约1分钟，让数据完成落地。</p> |
| </li> |
| <li>通过shell向meta server发送<a href="remote-commands#meta-server">远程命令</a>，开启<code class="language-plaintext highlighter-rouge">add_secondary</code>操作： |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> remote_command -t meta-server meta.lb.add_secondary_max_count_for_one_node 100 |
| </code></pre></div> </div> |
| </li> |
| <li>替换程序包和配置文件</li> |
| <li>重启meta server进程</li> |
| <li>使用shell的<code class="language-plaintext highlighter-rouge">ls -d</code>命令查看集群状态，等待所有partition都完全恢复健康</li> |
| <li>继续升级下一个replica server</li> |
| </ul> |
| </li> |
| <li>升级meta server进程，采用逐个升级的策略。升级单个meta server： |
| <ul> |
| <li>kill掉meta server进程</li> |
| <li>替换程序包和配置文件</li> |
| <li>重启meta server进程</li> |
| <li>等待30秒以上，保证meta server与replica server心跳的连续性</li> |
| <li>继续升级下一个meta server</li> |
| </ul> |
| </li> |
| <li>升级collector进程： |
| <ul> |
| <li>kill掉collector进程</li> |
| <li>替换程序包和配置文件</li> |
| <li>重启collector进程</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h2 id="简化版升级">简化版升级</h2> |
| <p>如果对可用性要求没那么高，升级流程可简化如下：</p> |
| <ul> |
| <li>准备好新的Server程序包和配置文件</li> |
| <li>使用shell工具将集群的meta level设置为steady，关闭<a href="rebalance">负载均衡功能</a>，避免不必要的replica迁移 |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> set_meta_level steady |
| </code></pre></div> </div> |
| </li> |
| <li>升级replica server进程，采用逐个升级的策略。升级单个replica server： |
| <ul> |
| <li>kill掉replica server进程</li> |
| <li>替换程序包和配置文件</li> |
| <li>重启replica server进程</li> |
| <li>使用shell的<code class="language-plaintext highlighter-rouge">ls -d</code>命令查看集群状态，等待所有partition都完全恢复健康</li> |
| <li>继续升级下一个replica server</li> |
| </ul> |
| </li> |
| <li>升级meta server进程，采用逐个升级的策略。升级单个meta server： |
| <ul> |
| <li>kill掉meta server进程</li> |
| <li>替换程序包和配置文件</li> |
| <li>重启meta server进程</li> |
| <li>等待30秒以上，保证meta server与replica server心跳的连续性</li> |
| <li>继续升级下一个meta server</li> |
| </ul> |
| </li> |
| <li>升级collector进程： |
| <ul> |
| <li>kill掉collector进程</li> |
| <li>替换程序包和配置文件</li> |
| <li>重启collector进程</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h1 id="升级脚本">升级脚本</h1> |
| |
| <p>我们提供了集群升级脚本<a href="https://github.com/apache/incubator-pegasus/blob/master/scripts/pegasus_rolling_update.sh">scripts/pegasus_rolling_update.sh</a>。该脚本采用<a href="#高可用升级">高可用升级</a>流程，用于小米内部的集群升级。</p> |
| |
| <p>不过这个脚本并不能直接使用，因为其依赖minos部署工具来完成以下事情：</p> |
| <ul> |
| <li>获取集群的进程列表</li> |
| <li>自动部署更新程序包和配置文件，并重启进程</li> |
| </ul> |
| |
| <p>你可以修改该脚本，针对你们自己的部署系统，修改以上通过minos完成的部分，使其可以正常工作。如需帮助，请联系我们。</p> |
| |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <!-- right panel --> |
| <div class="dashboard-panel is-small is-scrollable is-hidden-mobile"> |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| 本页导航 |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#功能目标">功能目标</a></li> |
| <li><a href="#升级流程">升级流程</a> |
| <ul> |
| <li><a href="#高可用升级">高可用升级</a></li> |
| <li><a href="#简化版升级">简化版升级</a></li> |
| </ul> |
| </li> |
| <li><a href="#升级脚本">升级脚本</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| <script> |
| docsearch({ |
| container: '#docsearch', |
| appId: 'QRN30RBW0S', |
| indexName: 'pegasus-apache', |
| apiKey: 'd3a3252fa344359766707a106c4ed88f', |
| debug: true |
| }); |
| </script> |
| |
| </body> |
| |
| </html> |
