| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Usage Scenario</title> |
| <link rel="stylesheet" href="/zh/assets/css/app.css"> |
| <link rel="shortcut icon" href="/zh/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/zh/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/zh/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Usage Scenario | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.2" /> |
| <meta property="og:title" content="Usage Scenario" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="注：Usage Scenario功能从v1.8.1版本开始支持。" /> |
| <meta property="og:description" content="注：Usage Scenario功能从v1.8.1版本开始支持。" /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2023-11-23T14:57:08+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Usage Scenario" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2023-11-23T14:57:08+00:00","datePublished":"2023-11-23T14:57:08+00:00","description":"注：Usage Scenario功能从v1.8.1版本开始支持。","headline":"Usage Scenario","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/usage-scenario"},"url":"/administration/usage-scenario"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| |
| <body> |
| <div class="dashboard is-full-height"> |
| <!-- left panel --> |
| <div class="dashboard-panel is-medium is-hidden-mobile pl-0"> |
| <div class="dashboard-panel-header has-text-centered"> |
| <a href="/zh/"> |
| <img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;"> |
| </a> |
| |
| </div> |
| <div class="dashboard-panel-main is-scrollable pl-6"> |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label">Pegasus产品文档</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/downloads" |
| class=""> |
| 下载 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">编译构建</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/build/compile-by-docker" |
| class=""> |
| 使用Docker完成编译（推荐） |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/docs/build/compile-from-source" |
| class=""> |
| 从源码编译 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">客户端库</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/clients/java-client" |
| class=""> |
| Java客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/cpp-client" |
| class=""> |
| C++客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class=""> |
| Golang客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/python-client" |
| class=""> |
| Python客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/node-client" |
| class=""> |
| NodeJS客户端 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/clients/scala-client" |
| class=""> |
| Scala客户端 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">生态工具</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/docs/tools/shell" |
| class=""> |
| Pegasus Shell 工具 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class=""> |
| 集群管理命令行 |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/pegic" |
| class=""> |
| 数据访问命令行 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">用户接口</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/api/ttl" |
| class=""> |
| TTL |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/single-atomic" |
| class=""> |
| 单行原子操作 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/redis" |
| class=""> |
| Redis适配 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/geo" |
| class=""> |
| GEO支持 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/api/http" |
| class=""> |
| HTTP接口 |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">高效运维</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/zh/administration/deployment" |
| class=""> |
| 集群部署 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/config" |
| class=""> |
| 配置说明 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rebalance" |
| class=""> |
| 负载均衡 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/monitoring" |
| class=""> |
| 可视化监控 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/rolling-update" |
| class=""> |
| 集群升级 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/scale-in-out" |
| class=""> |
| 集群扩容缩容 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/resource-management" |
| class=""> |
| 资源管理 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/cold-backup" |
| class=""> |
| 冷备份 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/meta-recovery" |
| class=""> |
| 元数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/replica-recovery" |
| class=""> |
| Replica数据恢复 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/zk-migration" |
| class=""> |
| Zookeeper迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-migration" |
| class=""> |
| Table迁移 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-soft-delete" |
| class=""> |
| Table软删除 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/table-env" |
| class=""> |
| Table环境变量 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/remote-commands" |
| class=""> |
| 远程命令 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/partition-split" |
| class=""> |
| Partition-Split |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/duplication" |
| class=""> |
| 跨机房同步 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/compression" |
| class=""> |
| 数据压缩 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/throttling" |
| class=""> |
| 流量控制 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/experiences" |
| class=""> |
| 运维经验 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/manual-compact" |
| class=""> |
| Manual Compact功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/usage-scenario" |
| class="is-active"> |
| Usage Scenario功能 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/bad-disk" |
| class=""> |
| 坏盘检修 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/whitelist" |
| class=""> |
| 白名单 |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/backup-request" |
| class=""> |
| Backup Request |
| </a> |
| </li> |
| |
| <li> |
| <a href="/zh/administration/hotspot-detection" |
| class=""> |
| 热点检测 |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| </div> |
| </div> |
| |
| <!-- main section --> |
| <div class="dashboard-main is-scrollable"> |
| <nav class="navbar is-hidden-desktop"> |
| <div class="navbar-brand"> |
| <a href="/zh/" class="navbar-item"> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus-square.png"> |
| </a> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/usage-scenario"><strong>En</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Pegasus产品文档 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/downloads" |
| class="navbar-item "> |
| 下载 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 编译构建 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/build/compile-by-docker" |
| class="navbar-item "> |
| 使用Docker完成编译（推荐） |
| </a> |
| |
| <a href="/zh/docs/build/compile-from-source" |
| class="navbar-item "> |
| 从源码编译 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 客户端库 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/clients/java-client" |
| class="navbar-item "> |
| Java客户端 |
| </a> |
| |
| <a href="/zh/clients/cpp-client" |
| class="navbar-item "> |
| C++客户端 |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class="navbar-item "> |
| Golang客户端 |
| </a> |
| |
| <a href="/zh/clients/python-client" |
| class="navbar-item "> |
| Python客户端 |
| </a> |
| |
| <a href="/zh/clients/node-client" |
| class="navbar-item "> |
| NodeJS客户端 |
| </a> |
| |
| <a href="/zh/clients/scala-client" |
| class="navbar-item "> |
| Scala客户端 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 生态工具 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/docs/tools/shell" |
| class="navbar-item "> |
| Pegasus Shell 工具 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class="navbar-item "> |
| 集群管理命令行 |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/pegic" |
| class="navbar-item "> |
| 数据访问命令行 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 用户接口 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/api/ttl" |
| class="navbar-item "> |
| TTL |
| </a> |
| |
| <a href="/zh/api/single-atomic" |
| class="navbar-item "> |
| 单行原子操作 |
| </a> |
| |
| <a href="/zh/api/redis" |
| class="navbar-item "> |
| Redis适配 |
| </a> |
| |
| <a href="/zh/api/geo" |
| class="navbar-item "> |
| GEO支持 |
| </a> |
| |
| <a href="/zh/api/http" |
| class="navbar-item "> |
| HTTP接口 |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| 高效运维 |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/zh/administration/deployment" |
| class="navbar-item "> |
| 集群部署 |
| </a> |
| |
| <a href="/zh/administration/config" |
| class="navbar-item "> |
| 配置说明 |
| </a> |
| |
| <a href="/zh/administration/rebalance" |
| class="navbar-item "> |
| 负载均衡 |
| </a> |
| |
| <a href="/zh/administration/monitoring" |
| class="navbar-item "> |
| 可视化监控 |
| </a> |
| |
| <a href="/zh/administration/rolling-update" |
| class="navbar-item "> |
| 集群升级 |
| </a> |
| |
| <a href="/zh/administration/scale-in-out" |
| class="navbar-item "> |
| 集群扩容缩容 |
| </a> |
| |
| <a href="/zh/administration/resource-management" |
| class="navbar-item "> |
| 资源管理 |
| </a> |
| |
| <a href="/zh/administration/cold-backup" |
| class="navbar-item "> |
| 冷备份 |
| </a> |
| |
| <a href="/zh/administration/meta-recovery" |
| class="navbar-item "> |
| 元数据恢复 |
| </a> |
| |
| <a href="/zh/administration/replica-recovery" |
| class="navbar-item "> |
| Replica数据恢复 |
| </a> |
| |
| <a href="/zh/administration/zk-migration" |
| class="navbar-item "> |
| Zookeeper迁移 |
| </a> |
| |
| <a href="/zh/administration/table-migration" |
| class="navbar-item "> |
| Table迁移 |
| </a> |
| |
| <a href="/zh/administration/table-soft-delete" |
| class="navbar-item "> |
| Table软删除 |
| </a> |
| |
| <a href="/zh/administration/table-env" |
| class="navbar-item "> |
| Table环境变量 |
| </a> |
| |
| <a href="/zh/administration/remote-commands" |
| class="navbar-item "> |
| 远程命令 |
| </a> |
| |
| <a href="/zh/administration/partition-split" |
| class="navbar-item "> |
| Partition-Split |
| </a> |
| |
| <a href="/zh/administration/duplication" |
| class="navbar-item "> |
| 跨机房同步 |
| </a> |
| |
| <a href="/zh/administration/compression" |
| class="navbar-item "> |
| 数据压缩 |
| </a> |
| |
| <a href="/zh/administration/throttling" |
| class="navbar-item "> |
| 流量控制 |
| </a> |
| |
| <a href="/zh/administration/experiences" |
| class="navbar-item "> |
| 运维经验 |
| </a> |
| |
| <a href="/zh/administration/manual-compact" |
| class="navbar-item "> |
| Manual Compact功能 |
| </a> |
| |
| <a href="/zh/administration/usage-scenario" |
| class="navbar-item is-active"> |
| Usage Scenario功能 |
| </a> |
| |
| <a href="/zh/administration/bad-disk" |
| class="navbar-item "> |
| 坏盘检修 |
| </a> |
| |
| <a href="/zh/administration/whitelist" |
| class="navbar-item "> |
| 白名单 |
| </a> |
| |
| <a href="/zh/administration/backup-request" |
| class="navbar-item "> |
| Backup Request |
| </a> |
| |
| <a href="/zh/administration/hotspot-detection" |
| class="navbar-item "> |
| 热点检测 |
| </a> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <nav class="navbar is-hidden-mobile"> |
| <div class="navbar-start w-full"> |
| <div class="navbar-item pl-0 w-full"> |
| <!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input, |
| I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released. |
| Related issue: https://github.com/algolia/docsearch/issues/230--> |
| <div id="docsearch"></div> |
| </div> |
| </div> |
| <div class="navbar-end"> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <!--If you don't want a url to be relativized, you can add a space explicitly into the href to |
| prevents a url from being relativized by polyglot.--> |
| <a class="button is-light is-outlined is-inverted" href=" /administration/usage-scenario"><strong>En</strong></a> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section class="hero is-info lg:mr-3"> |
| <div class="hero-body"> |
| |
| <p class="title is-size-2 is-centered">Usage Scenario功能</p> |
| </div> |
| </section> |
| <section class="section" style="padding-top: 2rem;"> |
| <div class="content"> |
| <p>注：Usage Scenario功能从v1.8.1版本开始支持。</p> |
| |
| <h1 id="原理">原理</h1> |
| <p>Usage Scenario功能，是指对于Pegasus的表，可以指定其使用场景。针对不同的场景，通过优化底层RocksDB的配置，以获得更好的读写性能。</p> |
| |
| <p>我们知道，RocksDB的LSM-Tree设计具有明显的写放大效应。数据先是写入到memtable中，当memtable满了，就会flush到sstable中，并放在level-0层。当level-0层的文件个数达到预设的限制时，就会触发compaction操作，将level-0层的所有文件merge到level-1层。同理，当level-1层的数据量达到预设的限制时，也会触发level-1层的compaction，将挑选一些文件merge到level-2层。这样，数据会从低层往高层逐层上移。</p> |
| |
| <p>RocksDB是一个可配置性很强的引擎，上面的各种行为都可以通过配置参数来调节，并且有很多参数都是可以动态修改的。这里给出几个比较关键的配置参数：</p> |
| <ul> |
| <li>write_buffer_size：memtable的大小限制，配置得越小，写入同样数据量产生的sstable文件数就越多。</li> |
| <li>level0_file_num_compaction_trigger：level-0层的文件个数限制，当达到这个限制时，就会触发compaction。</li> |
| <li>level0_slowdown_writes_trigger：当level-0层的文件个数超过这个值的时候，就会触发slowdown-writes行为，通过主动提升写操作延迟，来降低写入速度。</li> |
| <li>level0_stop_writes_trigger：当level-0层的文件个数超过这个值的时候，就会触发stop-writes行为，拒绝写入新的数据。</li> |
| <li>max_bytes_for_level_base：level-1层的数据量限制，当达到这个限制时，就会挑选一些文件merge到level-2层。</li> |
| <li>max_bytes_for_level_multiplier：数据量随层数的增长因子，譬如如果设为10，表示level-2的数据量限制就是level-1的10倍，level-3的数据量限制也是level-2的10倍。</li> |
| </ul> |
| |
| <p>Pegasus在提供读写服务的时候，需要考虑这些因素：</p> |
| <ul> |
| <li>当写数据的速度较大时，memtable很快就会写满，就会不断flush memtable产生新的sstable文件；</li> |
| <li>新sstable文件的产生就会触发compaction，并从低层向高层逐层蔓延；</li> |
| <li>compaction需要耗费大量的CPU和IO，造成<strong>机器的CPU和IO负载居高不下</strong>，影响读写操作的性能；</li> |
| <li>如果compaction速度赶不上数据写入的速度，level-0的文件数就会越堆越多，最终达到<code class="language-plaintext highlighter-rouge">level0_slowdown_writes_trigger</code>的限制，<strong>使写操作的延迟陡增</strong>；甚至进一步达到<code class="language-plaintext highlighter-rouge">level0_stop_writes_trigger</code>的限制，<strong>使写操作失败</strong>，影响系统的稳定性和服务的可用性。</li> |
| <li><strong>读写需求很难同时得到满足</strong>，二者需要权衡。compaction进行得越快，level-0的文件数维持得越少，读的时候需要读取的文件个数就越少，读性能就越高；但是compaction越快，带来的写放大效应就越大，CPU和IO的负载就越重，也会影响读写性能。</li> |
| </ul> |
| |
| <p>所幸的是，RocksDB针对这个问题也给出了一些解决方案，譬如在<a href="https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ">RocksDB-FAQ</a>中给出的方案：</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: What's the fastest way to load data into RocksDB? |
| |
| A: A fast way to direct insert data to the DB: |
| 1. using single writer thread and insert in sorted order |
| 2. batch hundreds of keys into one write batch |
| 3. use vector memtable |
| 4. make sure options.max_background_flushes is at least 4 |
| 5. before inserting the data, disable automatic compaction, set options.level0_file_num_compaction_trigger, |
| options.level0_slowdown_writes_trigger and options.level0_stop_writes_trigger to very large. After inserting all the |
| data, issue a manual compaction. |
| |
| 3-5 will be automatically done if you call Options::PrepareForBulkLoad() to your option |
| </code></pre></div></div> |
| |
| <p>而我们的思路正是：通过针对不同业务场景，设置不同的RocksDB参数，调节RocksDB的行为，以提供更好的读写性能。具体来说：</p> |
| <ul> |
| <li>通过<a href="table-env">Table环境变量</a>设置<code class="language-plaintext highlighter-rouge">rocksdb.usage_scenario</code>来指定当前的业务场景。</li> |
| <li>Replica在检测到该环境变量发生变化时，就会根据业务场景，动态修改RocksDB的配置参数。具体设置了哪些参数，请参见<a href="https://github.com/apache/incubator-pegasus/blob/master/src/server/pegasus_server_impl.cpp">src/server/pegasus_server_impl.cpp</a>中的<code class="language-plaintext highlighter-rouge">set_usage_scenario()</code>方法。</li> |
| </ul> |
| |
| <h1 id="支持场景">支持场景</h1> |
| |
| <p>目前支持三种场景：</p> |
| <ul> |
| <li>normal：正常场景，读写兼顾。这也是表的默认场景，该场景不会对写进行特别的优化，适合大部分读多写少或者读写均衡的应用。</li> |
| <li>prefer_write：写较多的场景。主要是增大<code class="language-plaintext highlighter-rouge">write_buffer_size</code>以降低sstable的产生速度。</li> |
| <li>bulk_load：灌数据场景。应用上面RocksDB-FAQ中提到的优化，避免compaction过程。因为Bulk load模式停止compaction，所以写入的数据都会堆放在level-0层，对读不友好。因此，Bulk load模式通常与<a href="manual-compact">Manual Compact功能</a>配合使用，在数据加载完成后进行一次Manual Compact，以去除垃圾数据、提升读性能（参见后面的<a href="#应用示例">应用示例</a>）。另外，当不需要加载数据时，应当恢复为Normal模式。典型的灌数据流程： |
| <ul> |
| <li>设置表的Usage Scenario模式为bulk_load</li> |
| <li>灌数据：在bulk load模式下灌数据的QPS会更高，流量更稳定</li> |
| <li>执行Manual Compact：该过程消耗大量的CPU和IO，可能对集群读写性能有影响</li> |
| <li>恢复表的Usage Scenario模式为normal</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h1 id="如何设置">如何设置</h1> |
| <h2 id="通过shell设置">通过shell设置</h2> |
| <p>通过shell的<a href="/zh/overview/shell#set_app_envs">set_app_envs命令</a>来设置，譬如设置temp表为bulk_load模式：</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> use temp |
| >>> set_app_envs rocksdb.usage_scenario bulk_load |
| </code></pre></div></div> |
| |
| <p>Table环境变量不会立即生效，大约需要等几十秒后才能在所有replica上生效。</p> |
| |
| <h2 id="通过脚本设置">通过脚本设置</h2> |
| <p>我们提供了一个脚本工具<a href="https://github.com/apache/incubator-pegasus/blob/master/scripts/pegasus_set_usage_scenario.sh">scripts/pegasus_set_usage_scenario.sh</a>来方便地设置，用法：</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./scripts/pegasus_set_usage_scenario.sh |
| This tool is for set usage scenario of specified table(app). |
| USAGE: ./scripts/pegasus_set_usage_scenario.sh <cluster-meta-list> <app-name> <normal|prefer_write|bulk_load> |
| </code></pre></div></div> |
| |
| <p>该工具会调用shell命令设置Table环境变量，然后还会检测是否在所有replica上都已经生效，只有所有都生效了才算执行完成。</p> |
| |
| <h2 id="应用示例">应用示例</h2> |
| |
| <p>bulk_load模式通常用于灌数据，但是在灌数据过程中因为消耗大量的CPU和IO，对读性能会产生较大影响，造成读延迟陡增、超时率升高等。如果业务对读性能要求比较苛刻，可以考虑<strong>读写分离的双集群方案</strong>。</p> |
| |
| <p>假设两个集群分别为A和B，最初线上流量访问A集群，灌数据流程：</p> |
| <ul> |
| <li>第一步：设置B模式为bulk_load -> 灌数据至B -> Manual Compact B -> 设置B模式为normal -> 切线上流量至B</li> |
| <li>第二步：设置A模式为bulk_load -> 灌数据至A -> Manual Compact A -> 设置A模式为normal -> 切线上流量至A</li> |
| </ul> |
| |
| <p>关于如何Manual Compact，请参考<a href="manual-compact">Manual-Compact功能</a>。</p> |
| |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <!-- right panel --> |
| <div class="dashboard-panel is-small is-scrollable is-hidden-mobile"> |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| 本页导航 |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#原理">原理</a></li> |
| <li><a href="#支持场景">支持场景</a></li> |
| <li><a href="#如何设置">如何设置</a> |
| <ul> |
| <li><a href="#通过shell设置">通过shell设置</a></li> |
| <li><a href="#通过脚本设置">通过脚本设置</a></li> |
| <li><a href="#应用示例">应用示例</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| <script> |
| docsearch({ |
| container: '#docsearch', |
| appId: 'QRN30RBW0S', |
| indexName: 'pegasus-apache', |
| apiKey: 'd3a3252fa344359766707a106c4ed88f', |
| debug: true |
| }); |
| </script> |
| |
| </body> |
| |
| </html> |
