blob: a63e13c1522e3712a6670bc6058143b40a3d83b1 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Apache Druid">
<meta name="keywords" content="druid,kafka,database,analytics,streaming,real-time,real time,apache,open source">
<meta name="author" content="Apache Software Foundation">
<title>Druid | Tutorial: Compacting segments</title>
<link rel="alternate" type="application/atom+xml" href="/feed">
<link rel="shortcut icon" href="/img/favicon.png">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
<link href='//fonts.googleapis.com/css?family=Open+Sans+Condensed:300,700,300italic|Open+Sans:300italic,400italic,600italic,400,300,600,700' rel='stylesheet' type='text/css'>
<link rel="stylesheet" href="/css/bootstrap-pure.css?v=1.1">
<link rel="stylesheet" href="/css/base.css?v=1.1">
<link rel="stylesheet" href="/css/header.css?v=1.1">
<link rel="stylesheet" href="/css/footer.css?v=1.1">
<link rel="stylesheet" href="/css/syntax.css?v=1.1">
<link rel="stylesheet" href="/css/docs.css?v=1.1">
<script>
(function() {
var cx = '000162378814775985090:molvbm0vggm';
var gcse = document.createElement('script');
gcse.type = 'text/javascript';
gcse.async = true;
gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') +
'//cse.google.com/cse.js?cx=' + cx;
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(gcse, s);
})();
</script>
</head>
<body>
<!-- Start page_header include -->
<script src="//ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
<div class="top-navigator">
<div class="container">
<div class="left-cont">
<a class="logo" href="/"><span class="druid-logo"></span></a>
</div>
<div class="right-cont">
<ul class="links">
<li class=""><a href="/technology">Technology</a></li>
<li class=""><a href="/use-cases">Use Cases</a></li>
<li class=""><a href="/druid-powered">Powered By</a></li>
<li class=""><a href="/docs/latest/design/">Docs</a></li>
<li class=""><a href="/community/">Community</a></li>
<li class="header-dropdown">
<a>Apache</a>
<div class="header-dropdown-menu">
<a href="https://www.apache.org/" target="_blank">Foundation</a>
<a href="https://www.apache.org/events/current-event" target="_blank">Events</a>
<a href="https://www.apache.org/licenses/" target="_blank">License</a>
<a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a>
<a href="https://www.apache.org/security/" target="_blank">Security</a>
<a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a>
</div>
</li>
<li class=" button-link"><a href="/downloads.html">Download</a></li>
</ul>
</div>
</div>
<div class="action-button menu-icon">
<span class="fa fa-bars"></span> MENU
</div>
<div class="action-button menu-icon-close">
<span class="fa fa-times"></span> MENU
</div>
</div>
<script type="text/javascript">
var $menu = $('.right-cont');
var $menuIcon = $('.menu-icon');
var $menuIconClose = $('.menu-icon-close');
function showMenu() {
$menu.fadeIn(100);
$menuIcon.fadeOut(100);
$menuIconClose.fadeIn(100);
}
$menuIcon.click(showMenu);
function hideMenu() {
$menu.fadeOut(100);
$menuIconClose.fadeOut(100);
$menuIcon.fadeIn(100);
}
$menuIconClose.click(hideMenu);
$(window).resize(function() {
if ($(window).width() >= 840) {
$menu.fadeIn(100);
$menuIcon.fadeOut(100);
$menuIconClose.fadeOut(100);
}
else {
$menu.fadeOut(100);
$menuIcon.fadeIn(100);
$menuIconClose.fadeOut(100);
}
});
</script>
<!-- Stop page_header include -->
<div class="container doc-container">
<p> Looking for the <a href="/docs/0.23.0/">latest stable documentation</a>?</p>
<div class="row">
<div class="col-md-9 doc-content">
<p>
<a class="btn btn-default btn-xs visible-xs-inline-block visible-sm-inline-block" href="#toc">Table of Contents</a>
</p>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->
<h1 id="tutorial-compacting-segments">Tutorial: Compacting segments</h1>
<p>This tutorial demonstrates how to compact existing segments into fewer but larger segments.</p>
<p>Because there is some per-segment memory and processing overhead, it can sometimes be beneficial to reduce the total number of segments.
Please check <a href="../operations/segment-optimization.html">Segment size optimization</a> for details.</p>
<p>For this tutorial, we&#39;ll assume you&#39;ve already downloaded Apache Druid (incubating) as described in
the <a href="index.html">single-machine quickstart</a> and have it running on your local machine. </p>
<p>It will also be helpful to have finished <a href="../tutorials/tutorial-batch.html">Tutorial: Loading a file</a> and <a href="../tutorials/tutorial-query.html">Tutorial: Querying data</a>.</p>
<h2 id="load-the-initial-data">Load the initial data</h2>
<p>For this tutorial, we&#39;ll be using the Wikipedia edits sample data, with an ingestion task spec that will create 1-3 segments per hour in the input data.</p>
<p>The ingestion spec can be found at <code>quickstart/tutorial/compaction-init-index.json</code>. Let&#39;s submit that spec, which will create a datasource called <code>compaction-tutorial</code>:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/post-index-task --file quickstart/tutorial/compaction-init-index.json
</code></pre></div>
<div class="note caution">
Please note that `maxRowsPerSegment` in the ingestion spec is set to 1000. This is to generate multiple segments per hour and _NOT_ recommended in production.
It's 5000000 by default and may need to be adjusted to make your segments optimized.
</div>
<p>After the ingestion completes, go to <a href="http://localhost:8888/unified-console.html#datasources">http://localhost:8888/unified-console.html#datasources</a> in a browser to see the new datasource in the Druid Console.</p>
<p><img src="../tutorials/img/tutorial-compaction-01.png" alt="compaction-tutorial datasource" title="compaction-tutorial datasource"></p>
<p>Click the <code>51 segments</code> link next to &quot;Fully Available&quot; for the <code>compaction-tutorial</code> datasource to view information about the datasource&#39;s segments:</p>
<p>There will be 51 segments for this datasource, 1-3 segments per hour in the input data:</p>
<p><img src="../tutorials/img/tutorial-compaction-02.png" alt="Original segments" title="Original segments"></p>
<p>Running a COUNT(*) query on this datasource shows that there are 39,244 rows:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>dsql&gt; <span class="k">select</span> count<span class="o">(</span>*<span class="o">)</span> from <span class="s2">&quot;compaction-tutorial&quot;</span><span class="p">;</span>
┌────────┐
│ EXPR<span class="nv">$0</span>
├────────┤
<span class="m">39244</span>
└────────┘
Retrieved <span class="m">1</span> row <span class="k">in</span> <span class="m">1</span>.38s.
</code></pre></div>
<h2 id="compact-the-data">Compact the data</h2>
<p>Let&#39;s now compact these 51 small segments.</p>
<p>We have included a compaction task spec for this tutorial datasource at <code>quickstart/tutorial/compaction-keep-granularity.json</code>:</p>
<div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span>
<span class="nt">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;compact&quot;</span><span class="p">,</span>
<span class="nt">&quot;dataSource&quot;</span><span class="p">:</span> <span class="s2">&quot;compaction-tutorial&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span> <span class="s2">&quot;2015-09-12/2015-09-13&quot;</span><span class="p">,</span>
<span class="nt">&quot;tuningConfig&quot;</span> <span class="p">:</span> <span class="p">{</span>
<span class="nt">&quot;type&quot;</span> <span class="p">:</span> <span class="s2">&quot;index&quot;</span><span class="p">,</span>
<span class="nt">&quot;maxRowsPerSegment&quot;</span> <span class="p">:</span> <span class="mi">5000000</span><span class="p">,</span>
<span class="nt">&quot;maxRowsInMemory&quot;</span> <span class="p">:</span> <span class="mi">25000</span><span class="p">,</span>
<span class="nt">&quot;forceExtendableShardSpecs&quot;</span> <span class="p">:</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This will compact all segments for the interval <code>2015-09-12/2015-09-13</code> in the <code>compaction-tutorial</code> datasource. </p>
<p>The parameters in the <code>tuningConfig</code> control how many segments will be present in the compacted set of segments. </p>
<p>In this tutorial example, only one compacted segment will be created per hour, as each hour has less rows than the 5000000 <code>maxRowsPerSegment</code> (note that the total number of rows is 39244).</p>
<p>Let&#39;s submit this task now:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/post-index-task --file quickstart/tutorial/compaction-keep-granularity.json
</code></pre></div>
<p>After the task finishes, refresh the <a href="http://localhost:8888/unified-console.html#segments">segments view</a>.</p>
<p>The original 51 segments will eventually be marked as &quot;unused&quot; by the Coordinator and removed, with the new compacted segments remaining.</p>
<p>By default, the Druid Coordinator will not mark segments as unused until the Coordinator process has been up for at least 15 minutes, so you may see the old segment set and the new compacted set at the same time in the Druid Console, with 75 total segments:</p>
<p><img src="../tutorials/img/tutorial-compaction-03.png" alt="Compacted segments intermediate state 1" title="Compacted segments intermediate state 1"></p>
<p><img src="../tutorials/img/tutorial-compaction-04.png" alt="Compacted segments intermediate state 2" title="Compacted segments intermediate state 2"></p>
<p>The new compacted segments have a more recent version than the original segments, so even when both sets of segments are shown in the Druid Console, queries will only read from the new compacted segments.</p>
<p>Let&#39;s try running a COUNT(*) on <code>compaction-tutorial</code> again, where the row count should still be 39,244:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>dsql&gt; <span class="k">select</span> count<span class="o">(</span>*<span class="o">)</span> from <span class="s2">&quot;compaction-tutorial&quot;</span><span class="p">;</span>
┌────────┐
│ EXPR<span class="nv">$0</span>
├────────┤
<span class="m">39244</span>
└────────┘
Retrieved <span class="m">1</span> row <span class="k">in</span> <span class="m">1</span>.30s.
</code></pre></div>
<p>After the Coordinator has been running for at least 15 minutes, the <a href="http://localhost:8888/unified-console.html#segments">segments view</a> should show there are 24 segments, one per hour:</p>
<p><img src="../tutorials/img/tutorial-compaction-05.png" alt="Compacted segments hourly granularity 1" title="Compacted segments hourly granularity 1"></p>
<p><img src="../tutorials/img/tutorial-compaction-06.png" alt="Compacted segments hourly granularity 2" title="Compacted segments hourly granularity 2"></p>
<h2 id="compact-the-data-with-new-segment-granularity">Compact the data with new segment granularity</h2>
<p>The compaction task can also produce compacted segments with a granularity different from the granularity of the input segments.</p>
<p>We have included a compaction task spec that will create DAY granularity segments at <code>quickstart/tutorial/compaction-day-granularity.json</code>:</p>
<div class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span>
<span class="nt">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;compact&quot;</span><span class="p">,</span>
<span class="nt">&quot;dataSource&quot;</span><span class="p">:</span> <span class="s2">&quot;compaction-tutorial&quot;</span><span class="p">,</span>
<span class="nt">&quot;interval&quot;</span><span class="p">:</span> <span class="s2">&quot;2015-09-12/2015-09-13&quot;</span><span class="p">,</span>
<span class="nt">&quot;segmentGranularity&quot;</span><span class="p">:</span> <span class="s2">&quot;DAY&quot;</span><span class="p">,</span>
<span class="nt">&quot;tuningConfig&quot;</span> <span class="p">:</span> <span class="p">{</span>
<span class="nt">&quot;type&quot;</span> <span class="p">:</span> <span class="s2">&quot;index&quot;</span><span class="p">,</span>
<span class="nt">&quot;maxRowsPerSegment&quot;</span> <span class="p">:</span> <span class="mi">5000000</span><span class="p">,</span>
<span class="nt">&quot;maxRowsInMemory&quot;</span> <span class="p">:</span> <span class="mi">25000</span><span class="p">,</span>
<span class="nt">&quot;forceExtendableShardSpecs&quot;</span> <span class="p">:</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Note that <code>segmentGranularity</code> is set to <code>DAY</code> in this compaction task spec.</p>
<p>Let&#39;s submit this task now:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/post-index-task --file quickstart/tutorial/compaction-day-granularity.json
</code></pre></div>
<p>It will take a bit of time before the Coordinator marks the old input segments as unused, so you may see an intermediate state with 25 total segments. Eventually, there will only be one DAY granularity segment:</p>
<p><img src="../tutorials/img/tutorial-compaction-07.png" alt="Compacted segments day granularity 1" title="Compacted segments day granularity 1"></p>
<p><img src="../tutorials/img/tutorial-compaction-08.png" alt="Compacted segments day granularity 2" title="Compacted segments day granularity 2"></p>
<h2 id="further-reading">Further reading</h2>
<p><a href="../ingestion/tasks.html">Task documentation</a></p>
<p><a href="../operations/segment-optimization.html">Segment optimization</a></p>
</div>
<div class="col-md-3">
<div class="searchbox">
<gcse:searchbox-only></gcse:searchbox-only>
</div>
<div id="toc" class="nav toc hidden-print">
</div>
</div>
</div>
</div>
<!-- Start page_footer include -->
<footer class="druid-footer">
<div class="container">
<div class="text-center">
<p>
<a href="/technology">Technology</a>&ensp;·&ensp;
<a href="/use-cases">Use Cases</a>&ensp;·&ensp;
<a href="/druid-powered">Powered by Druid</a>&ensp;·&ensp;
<a href="/docs/latest/">Docs</a>&ensp;·&ensp;
<a href="/community/">Community</a>&ensp;·&ensp;
<a href="/downloads.html">Download</a>&ensp;·&ensp;
<a href="/faq">FAQ</a>
</p>
</div>
<div class="text-center">
<a title="Join the user group" href="https://groups.google.com/forum/#!forum/druid-user" target="_blank"><span class="fa fa-comments"></span></a>&ensp;·&ensp;
<a title="Follow Druid" href="https://twitter.com/druidio" target="_blank"><span class="fab fa-twitter"></span></a>&ensp;·&ensp;
<a title="GitHub" href="https://github.com/apache/druid" target="_blank"><span class="fab fa-github"></span></a>
</div>
<div class="text-center license">
Copyright © 2020 <a href="https://www.apache.org/" target="_blank">Apache Software Foundation</a>.<br>
Except where otherwise noted, licensed under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>.<br>
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
</div>
</div>
</footer>
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-131010415-1');
</script>
<script>
function trackDownload(type, url) {
ga('send', 'event', 'download', type, url);
}
</script>
<script src="//code.jquery.com/jquery.min.js"></script>
<script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>
<script src="/assets/js/druid.js"></script>
<!-- stop page_footer include -->
<script>
$(function() {
$(".toc").load("/docs/0.14.0-incubating/toc.html");
// There is no way to tell when .gsc-input will be async loaded into the page so just try to set a placeholder until it works
var tries = 0;
var timer = setInterval(function() {
tries++;
if (tries > 300) clearInterval(timer);
var searchInput = $('input.gsc-input');
if (searchInput.length) {
searchInput.attr('placeholder', 'Search');
clearInterval(timer);
}
}, 100);
});
</script>
</body>
</html>