Google Git
Sign in
apache / cassandra-website / 6d522089c5327a655b259124e598e26ab44d759a / . / src / doc / 4.1 / operating / compression.html
blob: 271a69b5269f44573bf9947f67ee45ee4af29b6a [file] [log] [blame]
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Compression &mdash; Apache Cassandra Documentation v4.1</title>
<script type="text/javascript" src="../_static/js/modernizr.min.js"></script>
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/language_data.js"></script>
<script async="async" type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/javascript" src="../_static/js/theme.js"></script>
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/extra.css" type="text/css" />
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Change Data Capture" href="cdc.html" />
<link rel="prev" title="Bloom Filters" href="bloom_filters.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home"> Apache Cassandra
</a>
<div class="version">
4.1
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../getting_started/index.html">Getting Started</a></li>
<li class="toctree-l1"><a class="reference internal" href="../new/index.html">New Features in Apache Cassandra 4.0</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/index.html">Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../cql/index.html">The Cassandra Query Language (CQL)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../data_modeling/index.html">Data Modeling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration/index.html">Configuring Cassandra</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Operating Cassandra</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="snitch.html">Snitch</a></li>
<li class="toctree-l2"><a class="reference internal" href="topo_changes.html">Adding, replacing, moving and removing nodes</a></li>
<li class="toctree-l2"><a class="reference internal" href="repair.html">Repair</a></li>
<li class="toctree-l2"><a class="reference internal" href="read_repair.html">Read repair</a></li>
<li class="toctree-l2"><a class="reference internal" href="hints.html">Hints</a></li>
<li class="toctree-l2"><a class="reference internal" href="compaction/index.html">Compaction</a></li>
<li class="toctree-l2"><a class="reference internal" href="bloom_filters.html">Bloom Filters</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Compression</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#configuring-compression">Configuring Compression</a></li>
<li class="toctree-l3"><a class="reference internal" href="#benefits-and-uses">Benefits and Uses</a></li>
<li class="toctree-l3"><a class="reference internal" href="#operational-impact">Operational Impact</a></li>
<li class="toctree-l3"><a class="reference internal" href="#advanced-use">Advanced Use</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="cdc.html">Change Data Capture</a></li>
<li class="toctree-l2"><a class="reference internal" href="backups.html">Backups</a></li>
<li class="toctree-l2"><a class="reference internal" href="bulk_loading.html">Bulk Loading</a></li>
<li class="toctree-l2"><a class="reference internal" href="metrics.html">Monitoring</a></li>
<li class="toctree-l2"><a class="reference internal" href="security.html">Security</a></li>
<li class="toctree-l2"><a class="reference internal" href="hardware.html">Hardware Choices</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../tools/index.html">Cassandra Tools</a></li>
<li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development/index.html">Contributing to Cassandra</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq/index.html">Frequently Asked Questions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins/index.html">Third-Party Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../bugs.html">Reporting Bugs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contactus.html">Contact us</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">Apache Cassandra</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html">Docs</a> &raquo;</li>
<li><a href="index.html">Operating Cassandra</a> &raquo;</li>
<li>Compression</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/operating/compression.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="compression">
<h1>Compression<a class="headerlink" href="#compression" title="Permalink to this headline"></a></h1>
<p>Cassandra offers operators the ability to configure compression on a per-table basis. Compression reduces the size of
data on disk by compressing the SSTable in user-configurable compression <code class="docutils literal notranslate"><span class="pre">chunk_length_in_kb</span></code>. As Cassandra SSTables
are immutable, the CPU cost of compressing is only necessary when the SSTable is written - subsequent updates
to data will land in different SSTables, so Cassandra will not need to decompress, overwrite, and recompress data when
UPDATE commands are issued. On reads, Cassandra will locate the relevant compressed chunks on disk, decompress the full
chunk, and then proceed with the remainder of the read path (merging data from disks and memtables, read repair, and so
on).</p>
<p>Compression algorithms typically trade off between the following three areas:</p>
<ul class="simple">
<li><p><strong>Compression speed</strong>: How fast does the compression algorithm compress data. This is critical in the flush and
compaction paths because data must be compressed before it is written to disk.</p></li>
<li><p><strong>Decompression speed</strong>: How fast does the compression algorithm de-compress data. This is critical in the read
and compaction paths as data must be read off disk in a full chunk and decompressed before it can be returned.</p></li>
<li><p><strong>Ratio</strong>: By what ratio is the uncompressed data reduced by. Cassandra typically measures this as the size of data
on disk relative to the uncompressed size. For example a ratio of <code class="docutils literal notranslate"><span class="pre">0.5</span></code> means that the data on disk is 50% the size
of the uncompressed data. Cassandra exposes this ratio per table as the <code class="docutils literal notranslate"><span class="pre">SSTable</span> <span class="pre">Compression</span> <span class="pre">Ratio</span></code> field of
<code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">tablestats</span></code>.</p></li>
</ul>
<p>Cassandra offers five compression algorithms by default that make different tradeoffs in these areas. While
benchmarking compression algorithms depends on many factors (algorithm parameters such as compression level,
the compressibility of the input data, underlying processor class, etc …), the following table should help you pick
a starting point based on your application’s requirements with an extremely rough grading of the different choices
by their performance in these areas (A is relatively good, F is relatively bad):</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 39%" />
<col style="width: 20%" />
<col style="width: 11%" />
<col style="width: 13%" />
<col style="width: 6%" />
<col style="width: 11%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Compression Algorithm</p></th>
<th class="head"><p>Cassandra Class</p></th>
<th class="head"><p>Compression</p></th>
<th class="head"><p>Decompression</p></th>
<th class="head"><p>Ratio</p></th>
<th class="head"><p>C* Version</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><a class="reference external" href="https://lz4.github.io/lz4/">LZ4</a></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">LZ4Compressor</span></code></p></td>
<td><p>A+</p></td>
<td><p>A+</p></td>
<td><p>C+</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">&gt;=1.2.2</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="https://lz4.github.io/lz4/">LZ4HC</a></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">LZ4Compressor</span></code></p></td>
<td><p>C+</p></td>
<td><p>A+</p></td>
<td><p>B+</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">&gt;=</span> <span class="pre">3.6</span></code></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://facebook.github.io/zstd/">Zstd</a></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">ZstdCompressor</span></code></p></td>
<td><p>A-</p></td>
<td><p>A-</p></td>
<td><p>A+</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">&gt;=</span> <span class="pre">4.0</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference external" href="http://google.github.io/snappy/">Snappy</a></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">SnappyCompressor</span></code></p></td>
<td><p>A-</p></td>
<td><p>A</p></td>
<td><p>C</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">&gt;=</span> <span class="pre">1.0</span></code></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference external" href="https://zlib.net">Deflate (zlib)</a></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">DeflateCompressor</span></code></p></td>
<td><p>C</p></td>
<td><p>C</p></td>
<td><p>A</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">&gt;=</span> <span class="pre">1.0</span></code></p></td>
</tr>
</tbody>
</table>
<p>Generally speaking for a performance critical (latency or throughput) application <code class="docutils literal notranslate"><span class="pre">LZ4</span></code> is the right choice as it
gets excellent ratio per CPU cycle spent. This is why it is the default choice in Cassandra.</p>
<p>For storage critical applications (disk footprint), however, <code class="docutils literal notranslate"><span class="pre">Zstd</span></code> may be a better choice as it can get significant
additional ratio to <code class="docutils literal notranslate"><span class="pre">LZ4</span></code>.</p>
<p><code class="docutils literal notranslate"><span class="pre">Snappy</span></code> is kept for backwards compatibility and <code class="docutils literal notranslate"><span class="pre">LZ4</span></code> will typically be preferable.</p>
<p><code class="docutils literal notranslate"><span class="pre">Deflate</span></code> is kept for backwards compatibility and <code class="docutils literal notranslate"><span class="pre">Zstd</span></code> will typically be preferable.</p>
<div class="section" id="configuring-compression">
<h2>Configuring Compression<a class="headerlink" href="#configuring-compression" title="Permalink to this headline"></a></h2>
<p>Compression is configured on a per-table basis as an optional argument to <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> or <code class="docutils literal notranslate"><span class="pre">ALTER</span> <span class="pre">TABLE</span></code>. Three
options are available for all compressors:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">class</span></code> (default: <code class="docutils literal notranslate"><span class="pre">LZ4Compressor</span></code>): specifies the compression class to use. The two “fast”
compressors are <code class="docutils literal notranslate"><span class="pre">LZ4Compressor</span></code> and <code class="docutils literal notranslate"><span class="pre">SnappyCompressor</span></code> and the two “good” ratio compressors are <code class="docutils literal notranslate"><span class="pre">ZstdCompressor</span></code>
and <code class="docutils literal notranslate"><span class="pre">DeflateCompressor</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">chunk_length_in_kb</span></code> (default: <code class="docutils literal notranslate"><span class="pre">16KiB</span></code>): specifies the number of kilobytes of data per compression chunk. The main
tradeoff here is that larger chunk sizes give compression algorithms more context and improve their ratio, but
require reads to deserialize and read more off disk.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">crc_check_chance</span></code> (default: <code class="docutils literal notranslate"><span class="pre">1.0</span></code>): determines how likely Cassandra is to verify the checksum on each compression
chunk during reads to protect against data corruption. Unless you have profiles indicating this is a performance
problem it is highly encouraged not to turn this off as it is Cassandra’s only protection against bitrot.</p></li>
</ul>
<p>The <code class="docutils literal notranslate"><span class="pre">LZ4Compressor</span></code> supports the following additional options:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">lz4_compressor_type</span></code> (default <code class="docutils literal notranslate"><span class="pre">fast</span></code>): specifies if we should use the <code class="docutils literal notranslate"><span class="pre">high</span></code> (a.k.a <code class="docutils literal notranslate"><span class="pre">LZ4HC</span></code>) ratio version
or the <code class="docutils literal notranslate"><span class="pre">fast</span></code> (a.k.a <code class="docutils literal notranslate"><span class="pre">LZ4</span></code>) version of <code class="docutils literal notranslate"><span class="pre">LZ4</span></code>. The <code class="docutils literal notranslate"><span class="pre">high</span></code> mode supports a configurable level, which can allow
operators to tune the performance &lt;-&gt; ratio tradeoff via the <code class="docutils literal notranslate"><span class="pre">lz4_high_compressor_level</span></code> option. Note that in
<code class="docutils literal notranslate"><span class="pre">4.0</span></code> and above it may be preferable to use the <code class="docutils literal notranslate"><span class="pre">Zstd</span></code> compressor.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">lz4_high_compressor_level</span></code> (default <code class="docutils literal notranslate"><span class="pre">9</span></code>): A number between <code class="docutils literal notranslate"><span class="pre">1</span></code> and <code class="docutils literal notranslate"><span class="pre">17</span></code> inclusive that represents how much
CPU time to spend trying to get more compression ratio. Generally lower levels are “faster” but they get less ratio
and higher levels are slower but get more compression ratio.</p></li>
</ul>
<p>The <code class="docutils literal notranslate"><span class="pre">ZstdCompressor</span></code> supports the following options in addition:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">compression_level</span></code> (default <code class="docutils literal notranslate"><span class="pre">3</span></code>): A number between <code class="docutils literal notranslate"><span class="pre">-131072</span></code> and <code class="docutils literal notranslate"><span class="pre">22</span></code> inclusive that represents how much CPU
time to spend trying to get more compression ratio. The lower the level, the faster the speed (at the cost of ratio).
Values from 20 to 22 are called “ultra levels” and should be used with caution, as they require more memory.
The default of <code class="docutils literal notranslate"><span class="pre">3</span></code> is a good choice for competing with <code class="docutils literal notranslate"><span class="pre">Deflate</span></code> ratios and <code class="docutils literal notranslate"><span class="pre">1</span></code> is a good choice for competing
with <code class="docutils literal notranslate"><span class="pre">LZ4</span></code>.</p></li>
</ul>
<p>Users can set compression using the following syntax:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>CREATE TABLE keyspace.table (id int PRIMARY KEY) WITH compression = {&#39;class&#39;: &#39;LZ4Compressor&#39;};
</pre></div>
</div>
<p>Or</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ALTER TABLE keyspace.table WITH compression = {&#39;class&#39;: &#39;LZ4Compressor&#39;, &#39;chunk_length_in_kb&#39;: 64, &#39;crc_check_chance&#39;: 0.5};
</pre></div>
</div>
<p>Once enabled, compression can be disabled with <code class="docutils literal notranslate"><span class="pre">ALTER</span> <span class="pre">TABLE</span></code> setting <code class="docutils literal notranslate"><span class="pre">enabled</span></code> to <code class="docutils literal notranslate"><span class="pre">false</span></code>:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ALTER TABLE keyspace.table WITH compression = {&#39;enabled&#39;:&#39;false&#39;};
</pre></div>
</div>
<p>Operators should be aware, however, that changing compression is not immediate. The data is compressed when the SSTable
is written, and as SSTables are immutable, the compression will not be modified until the table is compacted. Upon
issuing a change to the compression options via <code class="docutils literal notranslate"><span class="pre">ALTER</span> <span class="pre">TABLE</span></code>, the existing SSTables will not be modified until they
are compacted - if an operator needs compression changes to take effect immediately, the operator can trigger an SSTable
rewrite using <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">scrub</span></code> or <code class="docutils literal notranslate"><span class="pre">nodetool</span> <span class="pre">upgradesstables</span> <span class="pre">-a</span></code>, both of which will rebuild the SSTables on disk,
re-compressing the data in the process.</p>
</div>
<div class="section" id="benefits-and-uses">
<h2>Benefits and Uses<a class="headerlink" href="#benefits-and-uses" title="Permalink to this headline"></a></h2>
<p>Compression’s primary benefit is that it reduces the amount of data written to disk. Not only does the reduced size save
in storage requirements, it often increases read and write throughput, as the CPU overhead of compressing data is faster
than the time it would take to read or write the larger volume of uncompressed data from disk.</p>
<p>Compression is most useful in tables comprised of many rows, where the rows are similar in nature. Tables containing
similar text columns (such as repeated JSON blobs) often compress very well. Tables containing data that has already
been compressed or random data (e.g. benchmark datasets) do not typically compress well.</p>
</div>
<div class="section" id="operational-impact">
<h2>Operational Impact<a class="headerlink" href="#operational-impact" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>Compression metadata is stored off-heap and scales with data on disk. This often requires 1-3GB of off-heap RAM per
terabyte of data on disk, though the exact usage varies with <code class="docutils literal notranslate"><span class="pre">chunk_length_in_kb</span></code> and compression ratios.</p></li>
<li><p>Streaming operations involve compressing and decompressing data on compressed tables - in some code paths (such as
non-vnode bootstrap), the CPU overhead of compression can be a limiting factor.</p></li>
<li><p>To prevent slow compressors (<code class="docutils literal notranslate"><span class="pre">Zstd</span></code>, <code class="docutils literal notranslate"><span class="pre">Deflate</span></code>, <code class="docutils literal notranslate"><span class="pre">LZ4HC</span></code>) from blocking flushes for too long, all three
flush with the default fast <code class="docutils literal notranslate"><span class="pre">LZ4</span></code> compressor and then rely on normal compaction to re-compress the data into the
desired compression strategy. See <cite>CASSANDRA-15379 &lt;https://issues.apache.org/jira/browse/CASSANDRA-15379&gt;</cite> for more
details.</p></li>
<li><p>The compression path checksums data to ensure correctness - while the traditional Cassandra read path does not have a
way to ensure correctness of data on disk, compressed tables allow the user to set <code class="docutils literal notranslate"><span class="pre">crc_check_chance</span></code> (a float from
0.0 to 1.0) to allow Cassandra to probabilistically validate chunks on read to verify bits on disk are not corrupt.</p></li>
</ul>
</div>
<div class="section" id="advanced-use">
<h2>Advanced Use<a class="headerlink" href="#advanced-use" title="Permalink to this headline"></a></h2>
<p>Advanced users can provide their own compression class by implementing the interface at
<code class="docutils literal notranslate"><span class="pre">org.apache.cassandra.io.compress.ICompressor</span></code>.</p>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="cdc.html" class="btn btn-neutral float-right" title="Change Data Capture" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="bloom_filters.html" class="btn btn-neutral float-left" title="Bloom Filters" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright 2020, The Apache Cassandra team
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>