Google Git
Sign in
apache / incubator-pegasus-website / refs/heads/asf-site / . / output / administration / table-env.html
blob: ab248dded0ba25810a58a9fe1f08cacbaa14e26b [file] [log] [blame]
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Pegasus | Table Env</title>
<link rel="stylesheet" href="/assets/css/app.css">
<link rel="shortcut icon" href="/assets/images/favicon.ico">
<link rel="stylesheet" href="/assets/css/utilities.min.css">
<link rel="stylesheet" href="/assets/css/docsearch.v3.css">
<script src="/assets/js/jquery.min.js"></script>
<script src="/assets/js/all.min.js"></script>
<script src="/assets/js/docsearch.v3.js"></script>
<!-- Begin Jekyll SEO tag v2.8.0 -->
<title>Table Env | Pegasus</title>
<meta name="generator" content="Jekyll v4.3.3" />
<meta property="og:title" content="Table Env" />
<meta property="og:locale" content="en_US" />
<meta name="description" content="Introduction" />
<meta property="og:description" content="Introduction" />
<meta property="og:site_name" content="Pegasus" />
<meta property="og:type" content="article" />
<meta property="article:published_time" content="2024-04-22T13:02:52+00:00" />
<meta name="twitter:card" content="summary" />
<meta property="twitter:title" content="Table Env" />
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"BlogPosting","dateModified":"2024-04-22T13:02:52+00:00","datePublished":"2024-04-22T13:02:52+00:00","description":"Introduction","headline":"Table Env","mainEntityOfPage":{"@type":"WebPage","@id":"/administration/table-env"},"url":"/administration/table-env"}</script>
<!-- End Jekyll SEO tag -->
</head>
<body>
<div class="dashboard is-full-height">
<!-- left panel -->
<div class="dashboard-panel is-medium is-hidden-mobile pl-0">
<div class="dashboard-panel-header has-text-centered">
<a href="/">
<img src="/assets/images/pegasus-logo-inv.png" style="width: 80%;">
</a>
</div>
<div class="dashboard-panel-main is-scrollable pl-6">
<aside class="menu">
<p class="menu-label">The Pegasus documentation</p>
<ul class="menu-list">
<li>
<a href="/docs/downloads"
class="">
Downloads
</a>
</li>
</ul>
<p class="menu-label">Building Pegasus</p>
<ul class="menu-list">
<li>
<a href="/docs/build/compile-by-docker"
class="">
Compile by docker (recommended)
</a>
</li>
<li>
<a href="/docs/build/compile-from-source"
class="">
Compile from source
</a>
</li>
</ul>
<p class="menu-label">Client Libs</p>
<ul class="menu-list">
<li>
<a href="/clients/java-client"
class="">
Java Client
</a>
</li>
<li>
<a href="/clients/cpp-client"
class="">
C++ Client
</a>
</li>
<li>
<a href="https://github.com/apache/incubator-pegasus/tree/master/go-client"
class="">
Golang Client
</a>
</li>
<li>
<a href="/clients/python-client"
class="">
Python Client
</a>
</li>
<li>
<a href="/clients/node-client"
class="">
NodeJS Client
</a>
</li>
<li>
<a href="/clients/scala-client"
class="">
Scala Client
</a>
</li>
</ul>
<p class="menu-label">Tools</p>
<ul class="menu-list">
<li>
<a href="/docs/tools/shell"
class="">
Pegasus Shell
</a>
</li>
<li>
<a href="https://github.com/pegasus-kv/admin-cli"
class="">
Admin CLI
</a>
</li>
<li>
<a href="https://github.com/pegasus-kv/pegic"
class="">
Pegasus data access CLI
</a>
</li>
</ul>
<p class="menu-label">API</p>
<ul class="menu-list">
<li>
<a href="/api/ttl"
class="">
TTL(Time To Live)
</a>
</li>
<li>
<a href="/api/single-atomic"
class="">
Single-Atomic Operations
</a>
</li>
<li>
<a href="/api/redis"
class="">
Redis Adaption
</a>
</li>
<li>
<a href="/api/geo"
class="">
GEO Support
</a>
</li>
<li>
<a href="/api/http"
class="">
HTTP API
</a>
</li>
</ul>
<p class="menu-label">Admin</p>
<ul class="menu-list">
<li>
<a href="/administration/deployment"
class="">
Deployment
</a>
</li>
<li>
<a href="/administration/config"
class="">
Configurations
</a>
</li>
<li>
<a href="/administration/rebalance"
class="">
Rebalance
</a>
</li>
<li>
<a href="/administration/monitoring"
class="">
Monitoring
</a>
</li>
<li>
<a href="/administration/rolling-update"
class="">
Rolling Restart and Upgrade
</a>
</li>
<li>
<a href="/administration/scale-in-out"
class="">
Scale-in and Scale-out
</a>
</li>
<li>
<a href="/administration/resource-management"
class="">
Resource Management
</a>
</li>
<li>
<a href="/administration/cold-backup"
class="">
Cold Backup
</a>
</li>
<li>
<a href="/administration/meta-recovery"
class="">
Metadata Recovery
</a>
</li>
<li>
<a href="/administration/replica-recovery"
class="">
Replica Data Recovery
</a>
</li>
<li>
<a href="/administration/zk-migration"
class="">
Zookeeper Migration
</a>
</li>
<li>
<a href="/administration/table-migration"
class="">
Table Migration
</a>
</li>
<li>
<a href="/administration/table-soft-delete"
class="">
Table Soft-Delete
</a>
</li>
<li>
<a href="/administration/table-env"
class="is-active">
Table Environment Variables
</a>
</li>
<li>
<a href="/administration/remote-commands"
class="">
Remote Command
</a>
</li>
<li>
<a href="/administration/partition-split"
class="">
Partition-Split
</a>
</li>
<li>
<a href="/administration/duplication"
class="">
Duplication
</a>
</li>
<li>
<a href="/administration/compression"
class="">
Data Compression
</a>
</li>
<li>
<a href="/administration/throttling"
class="">
Throttling
</a>
</li>
<li>
<a href="/administration/experiences"
class="">
Experiences
</a>
</li>
<li>
<a href="/administration/manual-compact"
class="">
Manual Compact
</a>
</li>
<li>
<a href="/administration/usage-scenario"
class="">
Usage Scenario
</a>
</li>
<li>
<a href="/administration/bad-disk"
class="">
Bad Disk Repair
</a>
</li>
<li>
<a href="/administration/whitelist"
class="">
Replica Server Whitelist
</a>
</li>
<li>
<a href="/administration/backup-request"
class="">
Backup Request
</a>
</li>
<li>
<a href="/administration/hotspot-detection"
class="">
Hotspot Detection
</a>
</li>
</ul>
</aside>
</div>
</div>
<!-- main section -->
<div class="dashboard-main is-scrollable">
<nav class="navbar is-hidden-desktop">
<div class="navbar-brand">
<a href="/" class="navbar-item">
<!-- Pegasus Icon -->
<img src="/assets/images/pegasus-square.png">
</a>
<div class="navbar-item">
<!--A simple language switch button that only supports zh and en.-->
<!--IF its language is zh, then switches to en.-->
<a class="button is-light is-outlined is-inverted" href="/zh/administration/table-env"><strong>中</strong></a>
</div>
<a role="button" class="navbar-burger burger" aria-label="menu" aria-expanded="false" data-target="navMenu">
<!-- Appears in mobile mode only -->
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
</a>
</div>
<div class="navbar-menu" id="navMenu">
<div class="navbar-end">
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
The Pegasus documentation
</span>
</a>
<div class="navbar-dropdown">
<a href="/docs/downloads"
class="navbar-item ">
Downloads
</a>
</div>
</div>
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
Building Pegasus
</span>
</a>
<div class="navbar-dropdown">
<a href="/docs/build/compile-by-docker"
class="navbar-item ">
Compile by docker (recommended)
</a>
<a href="/docs/build/compile-from-source"
class="navbar-item ">
Compile from source
</a>
</div>
</div>
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
Client Libs
</span>
</a>
<div class="navbar-dropdown">
<a href="/clients/java-client"
class="navbar-item ">
Java Client
</a>
<a href="/clients/cpp-client"
class="navbar-item ">
C++ Client
</a>
<a href="https://github.com/apache/incubator-pegasus/tree/master/go-client"
class="navbar-item ">
Golang Client
</a>
<a href="/clients/python-client"
class="navbar-item ">
Python Client
</a>
<a href="/clients/node-client"
class="navbar-item ">
NodeJS Client
</a>
<a href="/clients/scala-client"
class="navbar-item ">
Scala Client
</a>
</div>
</div>
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
Tools
</span>
</a>
<div class="navbar-dropdown">
<a href="/docs/tools/shell"
class="navbar-item ">
Pegasus Shell
</a>
<a href="https://github.com/pegasus-kv/admin-cli"
class="navbar-item ">
Admin CLI
</a>
<a href="https://github.com/pegasus-kv/pegic"
class="navbar-item ">
Pegasus data access CLI
</a>
</div>
</div>
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
API
</span>
</a>
<div class="navbar-dropdown">
<a href="/api/ttl"
class="navbar-item ">
TTL(Time To Live)
</a>
<a href="/api/single-atomic"
class="navbar-item ">
Single-Atomic Operations
</a>
<a href="/api/redis"
class="navbar-item ">
Redis Adaption
</a>
<a href="/api/geo"
class="navbar-item ">
GEO Support
</a>
<a href="/api/http"
class="navbar-item ">
HTTP API
</a>
</div>
</div>
<!--dropdown-->
<div class="navbar-item has-dropdown is-hoverable">
<a href=""
class="navbar-link ">
<span>
Admin
</span>
</a>
<div class="navbar-dropdown">
<a href="/administration/deployment"
class="navbar-item ">
Deployment
</a>
<a href="/administration/config"
class="navbar-item ">
Configurations
</a>
<a href="/administration/rebalance"
class="navbar-item ">
Rebalance
</a>
<a href="/administration/monitoring"
class="navbar-item ">
Monitoring
</a>
<a href="/administration/rolling-update"
class="navbar-item ">
Rolling Restart and Upgrade
</a>
<a href="/administration/scale-in-out"
class="navbar-item ">
Scale-in and Scale-out
</a>
<a href="/administration/resource-management"
class="navbar-item ">
Resource Management
</a>
<a href="/administration/cold-backup"
class="navbar-item ">
Cold Backup
</a>
<a href="/administration/meta-recovery"
class="navbar-item ">
Metadata Recovery
</a>
<a href="/administration/replica-recovery"
class="navbar-item ">
Replica Data Recovery
</a>
<a href="/administration/zk-migration"
class="navbar-item ">
Zookeeper Migration
</a>
<a href="/administration/table-migration"
class="navbar-item ">
Table Migration
</a>
<a href="/administration/table-soft-delete"
class="navbar-item ">
Table Soft-Delete
</a>
<a href="/administration/table-env"
class="navbar-item is-active">
Table Environment Variables
</a>
<a href="/administration/remote-commands"
class="navbar-item ">
Remote Command
</a>
<a href="/administration/partition-split"
class="navbar-item ">
Partition-Split
</a>
<a href="/administration/duplication"
class="navbar-item ">
Duplication
</a>
<a href="/administration/compression"
class="navbar-item ">
Data Compression
</a>
<a href="/administration/throttling"
class="navbar-item ">
Throttling
</a>
<a href="/administration/experiences"
class="navbar-item ">
Experiences
</a>
<a href="/administration/manual-compact"
class="navbar-item ">
Manual Compact
</a>
<a href="/administration/usage-scenario"
class="navbar-item ">
Usage Scenario
</a>
<a href="/administration/bad-disk"
class="navbar-item ">
Bad Disk Repair
</a>
<a href="/administration/whitelist"
class="navbar-item ">
Replica Server Whitelist
</a>
<a href="/administration/backup-request"
class="navbar-item ">
Backup Request
</a>
<a href="/administration/hotspot-detection"
class="navbar-item ">
Hotspot Detection
</a>
</div>
</div>
</div>
</div>
</nav>
<nav class="navbar is-hidden-mobile">
<div class="navbar-start w-full">
<div class="navbar-item pl-0 w-full">
<!--TODO(wutao): Given the limitation of docsearch that couldn't handle multiple input,
I make searchbox only shown in desktop. Fix this issue when docsearch.js v3 released.
Related issue: https://github.com/algolia/docsearch/issues/230-->
<div id="docsearch"></div>
</div>
</div>
<div class="navbar-end">
<div class="navbar-item">
<!--A simple language switch button that only supports zh and en.-->
<!--IF its language is zh, then switches to en.-->
<a class="button is-light is-outlined is-inverted" href="/zh/administration/table-env"><strong>中</strong></a>
</div>
</div>
</nav>
<section class="hero is-info lg:mr-3">
<div class="hero-body">
<p class="title is-size-2 is-centered">Table Environment Variables</p>
</div>
</section>
<section class="section" style="padding-top: 2rem;">
<div class="content">
<h1 id="introduction">Introduction</h1>
<p>In order to control some of the behaviors of the table, Pegasus provides the table environment variables, also known as <code class="language-plaintext highlighter-rouge">app envs</code>.</p>
<p>The table environment variables are stored in the metadata of the table in the form of a map (in the Thrift structure <code class="language-plaintext highlighter-rouge">app_info</code>) and persisted to the Apache Zookeeper.</p>
<p>View the table information through the shell tool’s <a href="/docs/tools/shell/#ls"><code class="language-plaintext highlighter-rouge">ls</code> command</a>, and the last column <code class="language-plaintext highlighter-rouge">envs_count</code> indicates the number of environment variables of the table. For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt;&gt;&gt; ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
2 AVAILABLE temp pegasus 8 3 true 2024-04-15_06:58:38 - - 1
</code></pre></div></div>
<p>If you want to view specific table environment variables, you need to use the <a href="#get_app_envs">get_app_envs</a> command.</p>
<p>Table environment variables have the following characteristics:</p>
<ul>
<li>Persist the metadata of the table onto Apache Zookeeper.</li>
<li>It can be dynamically modified through the command line, and once the modification is successful, it will be immediately updated to Zookeeper.</li>
<li>Synchronize the variables though the periodic synchronization message <code class="language-plaintext highlighter-rouge">config_sync</code> between the Meta Server and the Replica Server to take effect on each Replica Server.
<blockquote>
<p>Due to periodic synchronization, the updated environment variables may not take immediate effect on the Replica Server, but rather have a delay. This delay time depends on the value of the configuration ‘config_sync_interval_ms’, which defaults to 30 seconds.</p>
</blockquote>
</li>
<li>The key of environmental variables is usually separated by <code class="language-plaintext highlighter-rouge">.</code> for easy classification.</li>
</ul>
<p>The functions currently supported through table environment variables include:</p>
<ul>
<li><a href="manual-compact">Manual-Compact</a></li>
<li><a href="usage-scenario">Usage-Scenario</a></li>
</ul>
<h1 id="control-commands">Control commands</h1>
<p>Pegasus provides <a href="/docs/tools/shell/">Shell commands</a> for manipulating table environment variables.</p>
<blockquote>
<p>Before executing these commands, <code class="language-plaintext highlighter-rouge">use &lt;table_name&gt;</code> needs to be executed to select the table that needs to be operated on.</p>
</blockquote>
<h2 id="get_app_envs">get_app_envs</h2>
<p>Refer to: <a href="/docs/tools/shell/#get_app_envs"><code class="language-plaintext highlighter-rouge">get_app_envs</code> command</a></p>
<h2 id="set_app_envs">set_app_envs</h2>
<p>Refer to: <a href="/docs/tools/shell/#set_app_envs"><code class="language-plaintext highlighter-rouge">set_app_envs</code> command</a></p>
<h2 id="del_app_envs">del_app_envs</h2>
<p>Refer to: <a href="/docs/tools/shell/#del_app_envs"><code class="language-plaintext highlighter-rouge">del_app_envs</code> command</a></p>
<h2 id="clear_app_envs">clear_app_envs</h2>
<p>Refer to: <a href="/docs/tools/shell/#clear_app_envs"><code class="language-plaintext highlighter-rouge">clear_app_envs</code> command</a></p>
<h1 id="supported-environment-variables">Supported environment variables</h1>
<p>Starting from Pegasus 2.6, all supported table environment variables can be obtained through Meta Server <a href="/api/http">HTTP API</a> <code class="language-plaintext highlighter-rouge">/envs/list</code>.
For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl 127.0.0.1:34601/envs/list
{
"business.info": {
"limitation": "",
"sample": "",
"type": "string"
},
"default_ttl": {
"limitation": "&gt;= 0",
"sample": "86400",
"type": "unsigned int32"
},
"manual_compact.disabled": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"manual_compact.max_concurrent_running_count": {
"limitation": "&gt;= 0",
"sample": "8",
"type": "unsigned int32"
},
"manual_compact.once.bottommost_level_compaction": {
"limitation": "force | skip",
"sample": "skip",
"type": "string"
},
"manual_compact.once.target_level": {
"limitation": "-1 or &gt;= 1",
"sample": "6",
"type": "unsigned int64"
},
"manual_compact.once.trigger_time": {
"limitation": "&gt; 0, timestamp (in seconds) to trigger the once manual compaction",
"sample": "1700000000",
"type": "unsigned int64"
},
"manual_compact.periodic.bottommost_level_compaction": {
"limitation": "force | skip",
"sample": "skip",
"type": "string"
},
"manual_compact.periodic.target_level": {
"limitation": "-1 or &gt;= 1",
"sample": "6",
"type": "unsigned int64"
},
"manual_compact.periodic.trigger_time": {
"limitation": "",
"sample": "",
"type": "string"
},
"replica.backup_request_throttling": {
"limitation": "&lt;size[K|M]&gt;*&lt;delay|reject&gt;*&lt;milliseconds&gt;",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.deny_client_request": {
"limitation": "timeout*all | timeout*write | timeout*read | reconfig*all | reconfig*write | reconfig*read",
"sample": "timeout*all",
"type": "string"
},
"replica.read_throttling": {
"limitation": "&lt;size[K|M]&gt;*&lt;delay|reject&gt;*&lt;milliseconds&gt;",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.read_throttling_by_size": {
"limitation": "",
"sample": "20000*delay*100,20000*reject*100",
"type": "string"
},
"replica.rocksdb_block_cache_enabled": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"replica.rocksdb_iteration_threshold_time_ms": {
"limitation": "&gt;= 0",
"sample": "1000",
"type": "unsigned int64"
},
"replica.slow_query_threshold": {
"limitation": "&gt;= 20",
"sample": "1000",
"type": "unsigned int64"
},
"replica.split.validate_partition_hash": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"replica.write_throttling": {
"limitation": "&lt;size[K|M]&gt;*&lt;delay|reject&gt;*&lt;milliseconds&gt;",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica.write_throttling_by_size": {
"limitation": "&lt;size[K|M]&gt;*&lt;delay|reject&gt;*&lt;milliseconds&gt;",
"sample": "10000*delay*100,20000*reject*100",
"type": "string"
},
"replica_access_controller.allowed_users": {
"limitation": "",
"sample": "",
"type": "string"
},
"replica_access_controller.ranger_policies": {
"limitation": "",
"sample": "",
"type": "string"
},
"rocksdb.allow_ingest_behind": {
"limitation": "true | false",
"sample": "true",
"type": "bool"
},
"rocksdb.checkpoint.reserve_min_count": {
"limitation": "&gt; 0",
"sample": "2",
"type": "unsigned int32"
},
"rocksdb.checkpoint.reserve_time_seconds": {
"limitation": "&gt;= 0",
"sample": "3600",
"type": "unsigned int32"
},
"rocksdb.num_levels": {
"limitation": "In range [1, 10]",
"sample": "6",
"type": "unsigned int64"
},
"rocksdb.usage_scenario": {
"limitation": "bulk_load | normal | prefer_write",
"sample": "normal",
"type": "string"
},
"rocksdb.write_buffer_size": {
"limitation": "In range [16777216, 536870912]",
"sample": "16777216",
"type": "unsigned int64"
},
"user_specified_compaction": {
"limitation": "",
"sample": "",
"type": "string"
}
}
</code></pre></div></div>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="content is-small has-text-centered">
<div style="margin-bottom: 20px;">
<a href="http://incubator.apache.org">
<img src="/assets/images/egg-logo.png"
width="15%"
alt="Apache Incubator"/>
</a>
</div>
Copyright &copy; 2023 <a href="http://www.apache.org">The Apache Software Foundation</a>.
Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version
2.0</a>.
<br><br>
Apache Pegasus is an effort undergoing incubation at The Apache Software Foundation (ASF),
sponsored by the Apache Incubator. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects. While incubation status is
not necessarily a reflection of the completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.
<br><br>
Apache Pegasus, Pegasus, Apache, the Apache feather logo, and the Apache Pegasus project logo are either
registered trademarks or trademarks of The Apache Software Foundation in the United States and other
countries.
</div>
</div>
</footer>
</div>
<!-- right panel -->
<div class="dashboard-panel is-small is-scrollable is-hidden-mobile">
<p class="menu-label">
<span class="icon">
<i class="fa fa-bars" aria-hidden="true"></i>
</span>
Table of contents
</p>
<ul class="menu-list">
<li><a href="#introduction">Introduction</a></li>
<li><a href="#control-commands">Control commands</a>
<ul>
<li><a href="#get_app_envs">get_app_envs</a></li>
<li><a href="#set_app_envs">set_app_envs</a></li>
<li><a href="#del_app_envs">del_app_envs</a></li>
<li><a href="#clear_app_envs">clear_app_envs</a></li>
</ul>
</li>
<li><a href="#supported-environment-variables">Supported environment variables</a></li>
</ul>
</div>
</div>
<script src="/assets/js/app.js" type="text/javascript"></script>
<script>
docsearch({
container: '#docsearch',
appId: 'QRN30RBW0S',
indexName: 'pegasus-apache',
apiKey: 'd3a3252fa344359766707a106c4ed88f',
debug: true
});
</script>
</body>
</html>