| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Pegasus | Resource Management</title> |
| <link rel="stylesheet" href="/assets/css/app.css"> |
| <link rel="shortcut icon" href="/assets/images/favicon.ico"> |
| <link rel="stylesheet" href="/assets/css/utilities.min.css"> |
| <link rel="stylesheet" href="/assets/css/docsearch.v3.css"> |
| <script src="/assets/js/jquery.min.js"></script> |
| <script src="/assets/js/all.min.js"></script> |
| <script src="/assets/js/docsearch.v3.js"></script> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Resource Management | Pegasus</title> |
| <meta name="generator" content="Jekyll v4.3.3" /> |
| <meta property="og:title" content="Resource Management" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="Background Introduction" /> |
| <meta property="og:description" content="Background Introduction" /> |
| <meta property="og:site_name" content="Pegasus" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2024-04-22T13:02:52+00:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Resource Management" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2024-04-22T13:02:52+00:00","datePublished":"2024-04-22T13:02:52+00:00","description":"Background Introduction","headline":"Resource Management","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/resource-management"},"url":"/administration/resource-management"}</script> |
| <!-- End Jekyll SEO tag --> |
| </head> |
| |
| |
| <body> |
| <div class="dashboard is-full-height"> |
| <!-- left panel --> |
| <div class="dashboard-panel is-medium is-hidden-mobile pl-0"> |
| <div class="dashboard-panel-header has-text-centered"> |
| <a href="/"> |
| <img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;"> |
| </a> |
| |
| </div> |
| <div class="dashboard-panel-main is-scrollable pl-6"> |
| |
| |
| <aside class="menu"> |
| |
| <p class="menu-label">The Pegasus documentation</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/downloads" |
| class=""> |
| Downloads |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Building Pegasus</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/build/compile-by-docker" |
| class=""> |
| Compile by docker (recommended) |
| </a> |
| </li> |
| |
| <li> |
| <a href="/docs/build/compile-from-source" |
| class=""> |
| Compile from source |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Client Libs</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/clients/java-client" |
| class=""> |
| Java Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/cpp-client" |
| class=""> |
| C++ Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class=""> |
| Golang Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/python-client" |
| class=""> |
| Python Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/node-client" |
| class=""> |
| NodeJS Client |
| </a> |
| </li> |
| |
| <li> |
| <a href="/clients/scala-client" |
| class=""> |
| Scala Client |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Tools</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/docs/tools/shell" |
| class=""> |
| Pegasus Shell |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class=""> |
| Admin CLI |
| </a> |
| </li> |
| |
| <li> |
| <a href="https://github.com/pegasus-kv/pegic" |
| class=""> |
| Pegasus data access CLI |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">API</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/api/ttl" |
| class=""> |
| TTL(Time To Live) |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/single-atomic" |
| class=""> |
| Single-Atomic Operations |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/redis" |
| class=""> |
| Redis Adaption |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/geo" |
| class=""> |
| GEO Support |
| </a> |
| </li> |
| |
| <li> |
| <a href="/api/http" |
| class=""> |
| HTTP API |
| </a> |
| </li> |
| |
| </ul> |
| |
| <p class="menu-label">Admin</p> |
| <ul class="menu-list"> |
| |
| <li> |
| <a href="/administration/deployment" |
| class=""> |
| Deployment |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/config" |
| class=""> |
| Configurations |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/rebalance" |
| class=""> |
| Rebalance |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/monitoring" |
| class=""> |
| Monitoring |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/rolling-update" |
| class=""> |
| Rolling Restart and Upgrade |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/scale-in-out" |
| class=""> |
| Scale-in and Scale-out |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/resource-management" |
| class="is-active"> |
| Resource Management |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/cold-backup" |
| class=""> |
| Cold Backup |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/meta-recovery" |
| class=""> |
| Metadata Recovery |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/replica-recovery" |
| class=""> |
| Replica Data Recovery |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/zk-migration" |
| class=""> |
| Zookeeper Migration |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-migration" |
| class=""> |
| Table Migration |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-soft-delete" |
| class=""> |
| Table Soft-Delete |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/table-env" |
| class=""> |
| Table Environment Variables |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/remote-commands" |
| class=""> |
| Remote Command |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/partition-split" |
| class=""> |
| Partition-Split |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/duplication" |
| class=""> |
| Duplication |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/compression" |
| class=""> |
| Data Compression |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/throttling" |
| class=""> |
| Throttling |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/experiences" |
| class=""> |
| Experiences |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/manual-compact" |
| class=""> |
| Manual Compact |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/usage-scenario" |
| class=""> |
| Usage Scenario |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/bad-disk" |
| class=""> |
| Bad Disk Repair |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/whitelist" |
| class=""> |
| Replica Server Whitelist |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/backup-request" |
| class=""> |
| Backup Request |
| </a> |
| </li> |
| |
| <li> |
| <a href="/administration/hotspot-detection" |
| class=""> |
| Hotspot Detection |
| </a> |
| </li> |
| |
| </ul> |
| |
| </aside> |
| </div> |
| </div> |
| |
| <!-- main section --> |
| <div class="dashboard-main is-scrollable"> |
| <nav class="navbar is-hidden-desktop"> |
| <div class="navbar-brand"> |
| <a href="/" class="navbar-item"> |
| <!-- Pegasus Icon --> |
| <img src="/assets/images/pegasus-square.png"> |
| </a> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-light is-outlined is-inverted" href="/zh/administration/resource-management"><strong>中</strong></a> |
| |
| </div> |
| <a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu"> |
| <!-- Appears in mobile mode only --> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| <span aria-hidden="true"></span> |
| </a> |
| </div> |
| <div class="navbar-menu" id="navMenu"> |
| <div class="navbar-end"> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| The Pegasus documentation |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/downloads" |
| class="navbar-item "> |
| Downloads |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Building Pegasus |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/build/compile-by-docker" |
| class="navbar-item "> |
| Compile by docker (recommended) |
| </a> |
| |
| <a href="/docs/build/compile-from-source" |
| class="navbar-item "> |
| Compile from source |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Client Libs |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/clients/java-client" |
| class="navbar-item "> |
| Java Client |
| </a> |
| |
| <a href="/clients/cpp-client" |
| class="navbar-item "> |
| C++ Client |
| </a> |
| |
| <a href="https://github.com/apache/incubator-pegasus/tree/master/go-client" |
| class="navbar-item "> |
| Golang Client |
| </a> |
| |
| <a href="/clients/python-client" |
| class="navbar-item "> |
| Python Client |
| </a> |
| |
| <a href="/clients/node-client" |
| class="navbar-item "> |
| NodeJS Client |
| </a> |
| |
| <a href="/clients/scala-client" |
| class="navbar-item "> |
| Scala Client |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Tools |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/docs/tools/shell" |
| class="navbar-item "> |
| Pegasus Shell |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/admin-cli" |
| class="navbar-item "> |
| Admin CLI |
| </a> |
| |
| <a href="https://github.com/pegasus-kv/pegic" |
| class="navbar-item "> |
| Pegasus data access CLI |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| API |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/api/ttl" |
| class="navbar-item "> |
| TTL(Time To Live) |
| </a> |
| |
| <a href="/api/single-atomic" |
| class="navbar-item "> |
| Single-Atomic Operations |
| </a> |
| |
| <a href="/api/redis" |
| class="navbar-item "> |
| Redis Adaption |
| </a> |
| |
| <a href="/api/geo" |
| class="navbar-item "> |
| GEO Support |
| </a> |
| |
| <a href="/api/http" |
| class="navbar-item "> |
| HTTP API |
| </a> |
| |
| </div> |
| </div> |
| |
| <!--dropdown--> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a href="" |
| class="navbar-link "> |
| <span> |
| Admin |
| </span> |
| </a> |
| <div class="navbar-dropdown"> |
| |
| <a href="/administration/deployment" |
| class="navbar-item "> |
| Deployment |
| </a> |
| |
| <a href="/administration/config" |
| class="navbar-item "> |
| Configurations |
| </a> |
| |
| <a href="/administration/rebalance" |
| class="navbar-item "> |
| Rebalance |
| </a> |
| |
| <a href="/administration/monitoring" |
| class="navbar-item "> |
| Monitoring |
| </a> |
| |
| <a href="/administration/rolling-update" |
| class="navbar-item "> |
| Rolling Restart and Upgrade |
| </a> |
| |
| <a href="/administration/scale-in-out" |
| class="navbar-item "> |
| Scale-in and Scale-out |
| </a> |
| |
| <a href="/administration/resource-management" |
| class="navbar-item is-active"> |
| Resource Management |
| </a> |
| |
| <a href="/administration/cold-backup" |
| class="navbar-item "> |
| Cold Backup |
| </a> |
| |
| <a href="/administration/meta-recovery" |
| class="navbar-item "> |
| Metadata Recovery |
| </a> |
| |
| <a href="/administration/replica-recovery" |
| class="navbar-item "> |
| Replica Data Recovery |
| </a> |
| |
| <a href="/administration/zk-migration" |
| class="navbar-item "> |
| Zookeeper Migration |
| </a> |
| |
| <a href="/administration/table-migration" |
| class="navbar-item "> |
| Table Migration |
| </a> |
| |
| <a href="/administration/table-soft-delete" |
| class="navbar-item "> |
| Table Soft-Delete |
| </a> |
| |
| <a href="/administration/table-env" |
| class="navbar-item "> |
| Table Environment Variables |
| </a> |
| |
| <a href="/administration/remote-commands" |
| class="navbar-item "> |
| Remote Command |
| </a> |
| |
| <a href="/administration/partition-split" |
| class="navbar-item "> |
| Partition-Split |
| </a> |
| |
| <a href="/administration/duplication" |
| class="navbar-item "> |
| Duplication |
| </a> |
| |
| <a href="/administration/compression" |
| class="navbar-item "> |
| Data Compression |
| </a> |
| |
| <a href="/administration/throttling" |
| class="navbar-item "> |
| Throttling |
| </a> |
| |
| <a href="/administration/experiences" |
| class="navbar-item "> |
| Experiences |
| </a> |
| |
| <a href="/administration/manual-compact" |
| class="navbar-item "> |
| Manual Compact |
| </a> |
| |
| <a href="/administration/usage-scenario" |
| class="navbar-item "> |
| Usage Scenario |
| </a> |
| |
| <a href="/administration/bad-disk" |
| class="navbar-item "> |
| Bad Disk Repair |
| </a> |
| |
| <a href="/administration/whitelist" |
| class="navbar-item "> |
| Replica Server Whitelist |
| </a> |
| |
| <a href="/administration/backup-request" |
| class="navbar-item "> |
| Backup Request |
| </a> |
| |
| <a href="/administration/hotspot-detection" |
| class="navbar-item "> |
| Hotspot Detection |
| </a> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <nav class="navbar is-hidden-mobile"> |
| <div class="navbar-start w-full"> |
| <div class="navbar-item pl-0 w-full"> |
| <!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input, |
| I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released. |
| Related issue: https://github.com/algolia/docsearch/issues/230--> |
| <div id="docsearch"></div> |
| </div> |
| </div> |
| <div class="navbar-end"> |
| <div class="navbar-item"> |
| |
| |
| <!--A simple language switch button that only supports zh and en.--> |
| <!--IF its language is zh, then switches to en.--> |
| |
| <a class="button is-light is-outlined is-inverted" href="/zh/administration/resource-management"><strong>中</strong></a> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section class="hero is-info lg:mr-3"> |
| <div class="hero-body"> |
| |
| <p class="title is-size-2 is-centered">Resource Management</p> |
| </div> |
| </section> |
| <section class="section" style="padding-top: 2rem;"> |
| <div class="content"> |
| <h1 id="background-introduction">Background Introduction</h1> |
| |
| <p>The main resources used by Pegasus include CPU, disk, memory, network, etc. The usage load of these system resources should not be too high, otherwise the Pegasus service may become unstable or even crash. It’s recommend:</p> |
| <ul> |
| <li>The storage usage of a single disk should not exceed 80%.</li> |
| <li>Memory usage should not exceed 80% of each node.</li> |
| <li>The number of network connections should not exceed the system’s limit, and it is recommended to limit the number of connections less than 50000.</li> |
| </ul> |
| |
| <p>By adjusting these configurations, the use of disk storage can be reduced:</p> |
| <ul> |
| <li>Set <code class="language-plaintext highlighter-rouge">max_replicas_in_group = 3</code>, refer to <a href="#replica-management">Replica management</a>.</li> |
| <li>Set <code class="language-plaintext highlighter-rouge">gc_disk_error_replica_interval_seconds = 3600</code> and <code class="language-plaintext highlighter-rouge">gc_disk_garbage_replica_interval_seconds = 3600</code>, refer to <a href="#garbage-directory-management">Garbage directory management</a>.</li> |
| <li>Set <code class="language-plaintext highlighter-rouge">checkpoint_reserve_min_count = 2</code> and <code class="language-plaintext highlighter-rouge">checkpoint_reserve_time_seconds = 1200</code>, refer to <a href="#rocksdb-checkpoints-management">RocksDB checkpoints management</a>.</li> |
| </ul> |
| |
| <h1 id="replica-management">Replica management</h1> |
| |
| <p>Pegasus recommends using 3 replicas (1 primary + 2 secondaries) and setting the <code class="language-plaintext highlighter-rouge">-r</code> parameter to 3 when creating tables.</p> |
| |
| <p>However, the actual number of replicas in the cluster may exceed 3, which is determined by the following configuration:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[meta_server] |
| max_replicas_in_group = 4 |
| </code></pre></div></div> |
| |
| <p>The meaning of this configuration: allow the maximum number of replicas (including primary and secondaries) of a partition, with a default value of 4 (indicating allowing the retention of 1 inactive replica). Although there are 3 active replicas (1 primary + 2 secondaries) being served, during the procedure of downtime recovery or load balancing, replicas may migrate from server A to server B. After the migration, the data on server A is actually no longer needed. However, with sufficient storage, the replica on server A can be retained. If the replica is re-migrated to server A in the future, these retained replicas may be reused to avoid re-transmission.</p> |
| |
| <p>In aim to reduce disk storage usage and delete useless replicas data in time, you can set <code class="language-plaintext highlighter-rouge">max_replicas_in_group = 3</code>, restart Meta Server to make the configuration take effect, and then set the <a href="rebalance">Load Rebalance</a> level to <code class="language-plaintext highlighter-rouge">lively</code>, allowing Meta Server to delete useless replicas data.</p> |
| |
| <h1 id="garbage-directory-management">Garbage directory management</h1> |
| |
| <p>If the replica directory in Replica Server is no longer needed or damaged, it becomes a garbage directory: unnecessary directory has a <code class="language-plaintext highlighter-rouge">.gar</code> suffix, and damaged directory has a <code class="language-plaintext highlighter-rouge">.err</code> suffix. These directories are not deleted immediately, as they may still have value in certain extreme situations (such as recall data through them in the event of a cluster crash).</p> |
| |
| <p>There are two configurations that determine the actual deletion time for these directories:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[replication] |
| gc_disk_error_replica_interval_seconds = 604800 |
| gc_disk_garbage_replica_interval_seconds = 86400 |
| </code></pre></div></div> |
| <p>For these two types of directories, the last modification time (before Pegasus 2.6 is the last modification time of the directory, starting from Pegasus 2.6 is the timestamp field in the directory name) of the directory will be checked, and deletion is executed when the gap between the last modification time and the current time exceeds the corresponding configuration.</p> |
| |
| <p>In aim to reduce disk storage usage by deleting these garbage directories in time, it can be achieved by reducing the values of these two configurations.</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[replication] |
| gc_disk_error_replica_interval_seconds = 3600 |
| gc_disk_garbage_replica_interval_seconds = 3600 |
| </code></pre></div></div> |
| <ul> |
| <li>If the Pegasus version is less than 1.11.3, it’s needed to restart the Replica Server for the configurations to take effect.</li> |
| <li>If the Pegasus version is between 1.11.3 and 2.1, these two configurations can be modified and take effect at runtime through the <code class="language-plaintext highlighter-rouge">useless-dir-reserve-seconds</code> command in <a href="remote-commands">Remote commands</a>, without restarting the Replica Server process. For example, modify these two configurations to 0 for emergency cleaning of the garbage directories: |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> remote_command -t replica-server useless-dir-reserve-seconds 0 |
| </code></pre></div> </div> |
| <p>After confirming that the cleaning is complete, restore to the configurations:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> remote_command -t replica-server useless-dir-reserve-seconds DEFAULT |
| </code></pre></div> </div> |
| </li> |
| <li>Starting from version 2.2, the configurations can be modified and take effect at runtime through the <a href="/api/http">HTTP API</a> without restarting the Replica Server process.</li> |
| </ul> |
| |
| <h1 id="rocksdb-checkpoints-management">RocksDB checkpoints management</h1> |
| |
| <p>The storage engine of Replica Server is RocksDB, it generates <a href="https://github.com/facebook/rocksdb/wiki/Checkpoints">checkpoint</a> regularly. The checkpoints are placed in the data directory of the replica, and are suffixed by the <code class="language-plaintext highlighter-rouge">last_durable_decree</code>.</p> |
| |
| <p>As shown in the figure below, the data directory of the replica contains the currently using <code class="language-plaintext highlighter-rouge">rdb</code> directory and several checkpoint directories: |
| <img src="/assets/images/checkpoint_dirs.png" alt="checkpoint_dirs.png" class="img-responsive" /></p> |
| |
| <p>When generating a checkpoint, the files in the checkpoint are generated through hard linking rather than copying. One of the sstable files may be held by the <code class="language-plaintext highlighter-rouge">rdb</code> or by one or more checkpoints. As long as any one of them holds, the data of that file exists on the disk, consuming storage space. The file can be deleted only when <code class="language-plaintext highlighter-rouge">rdb</code> and all checkpoints do not hold it.</p> |
| |
| <p>The RocksDB is continuously performing background compactions, so the sstable held by any checkpoint may no longer be held by <code class="language-plaintext highlighter-rouge">rdb</code> (call it expired). If the retention time of the checkpoints are too long, these expired sstables cannot be deleted in time, which consume extra disk storage space. Especially for tables with high write throughput, compaction occurs more frequently, and the lifecycle of a single sstable file is very short. If the number of checkpoints is kept relatively high, the storage space consumed may be several times larger than the current actual data size.</p> |
| |
| <p>The following configurations determine the strategy for deleting checkpoints:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[pegasus.server] |
| checkpoint_reserve_min_count = 2 |
| checkpoint_reserve_time_seconds = 1800 |
| </code></pre></div></div> |
| <ul> |
| <li>checkpoint_reserve_min_count: represents the minimum number of reserved checkpoints. Only when the number of checkpoints exceeds this limit, the oldest checkpoint may be deleted.</li> |
| <li>checkpoint_reserve_time_seconds: represents the minimum retention time of the checkpoint. Only when the generation time of the checkpoint exceeds this value from the current time can the oldest one be deleted.</li> |
| <li>The checkpoint will only be deleted when meet the 2 conditions simultaneously.</li> |
| </ul> |
| |
| <p>In aim to reduce disk storage usage by deleting the old checkpoint directories in time, you can lower these two configurations. For example:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[pegasus.server] |
| checkpoint_reserve_min_count = 1 |
| checkpoint_reserve_time_seconds = 1200 |
| </code></pre></div></div> |
| <p>Note: It is not recommended to set <code class="language-plaintext highlighter-rouge">checkpoint_reserve_time_seconds</code> too low. Considering the impact on learning, it should be larger than <code class="language-plaintext highlighter-rouge">replica_assign_delay_ms_for_dropouts</code> (default is 5 minutes).</p> |
| |
| <h2 id="set-table-level-configuration">Set table level configuration</h2> |
| |
| <p>Since Pegasus 1.11.3, it is supported to modify these two configurations at runtime through the <a href="table-env">Table environment variable</a> for a specified table, without restarting the Replica Server process. For example:</p> |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> use <table_name> |
| >>> set_app_envs rocksdb.checkpoint.reserve_min_count 1 |
| >>> set_app_envs rocksdb.checkpoint.reserve_time_seconds 600 |
| </code></pre></div></div> |
| |
| </div> |
| </section> |
| <footer class="footer"> |
| <div class="container"> |
| <div class="content is-small has-text-centered"> |
| <div style="margin-bottom: 20px;"> |
| <a href="http://incubator.apache.org"> |
| <img src="/assets/images/egg-logo.png" |
| width="15%" |
| alt="Apache Incubator"/> |
| </a> |
| </div> |
| Copyright © 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>. |
| Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version |
| 2.0</a>. |
| <br><br> |
| |
| Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF), |
| sponsored by the Apache Incubator. Incubation is required of all newly accepted projects |
| until a further review indicates that the infrastructure, communications, and decision making process |
| have stabilized in a manner consistent with other successful ASF projects. While incubation status is |
| not necessarily a reflection of the completeness or stability of the code, it does indicate that the |
| project has yet to be fully endorsed by the ASF. |
| |
| <br><br> |
| Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either |
| registered trademarks or trademarks of The Apache Software Foundation in the United States and other |
| countries. |
| </div> |
| </div> |
| </footer> |
| </div> |
| |
| <!-- right panel --> |
| <div class="dashboard-panel is-small is-scrollable is-hidden-mobile"> |
| <p class="menu-label"> |
| <span class="icon"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </span> |
| Table of contents |
| </p> |
| <ul class="menu-list"> |
| <li><a href="#background-introduction">Background Introduction</a></li> |
| <li><a href="#replica-management">Replica management</a></li> |
| <li><a href="#garbage-directory-management">Garbage directory management</a></li> |
| <li><a href="#rocksdb-checkpoints-management">RocksDB checkpoints management</a> |
| <ul> |
| <li><a href="#set-table-level-configuration">Set table level configuration</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| </div> |
| </div> |
| |
| <script src="/assets/js/app.js" type="text/javascript"></script> |
| <script> |
| docsearch({ |
| container: '#docsearch', |
| appId: 'QRN30RBW0S', |
| indexName: 'pegasus-apache', |
| apiKey: 'd3a3252fa344359766707a106c4ed88f', |
| debug: true |
| }); |
| </script> |
| |
| </body> |
| |
| </html> |
