| The automatic compaction system uses the following syntax:</p><div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"dataSource"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <task_datasource></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"ioConfig"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <IO config></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"dimensionsSpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <custom dimensionsSpec></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"transformSpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <custom transformSpec></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"metricsSpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <custom metricsSpec></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"tuningConfig"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <parallel indexing task tuningConfig></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"granularitySpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <compaction task granularitySpec></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"skipOffsetFromLatest"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <time period to avoid compaction></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"taskPriority"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <compaction task priority></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"taskContext"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <task context></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><div class="theme-admonition theme-admonition-info alert alert--info admonition_LlT9"><div class="admonitionHeading_tbUL"><span class="admonitionIcon_kALy"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Experimental</div><div class="admonitionContent_S0QG"><p>The MSQ task engine is available as a compaction engine when you run automatic compaction as a compaction supervisor. For more information, see <a href="#auto-compaction-using-compaction-supervisors">Auto-compaction using compaction supervisors</a>.</p></div></div><p>For automatic compaction using Coordinator duties, you submit the spec to the <a href="#manage-auto-compaction-using-the-web-console">Compaction config UI</a> or the <a href="#manage-auto-compaction-using-coordinator-apis">Compaction configuration API</a>.</p><p>Most fields in the auto-compaction configuration correlate to a typical <a href="/docs/latest/ingestion/ingestion-spec">Druid ingestion spec</a>. |
| The following example shows how to create a duty group named <code>compaction</code> and set the auto-compaction period to 1 minute:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">druid.coordinator.dutyGroups=["compaction"]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">druid.coordinator.compaction.duties=["compactSegments"]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">druid.coordinator.compaction.period=PT60S</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="view-coordinator-duty-auto-compaction-stats">View Coordinator duty auto-compaction stats<a href="#view-coordinator-duty-auto-compaction-stats" class="hash-link" aria-label="Direct link to View Coordinator duty auto-compaction stats" title="Direct link to View Coordinator duty auto-compaction stats"></a></h3><p>After the Coordinator has initiated auto-compaction, you can view compaction statistics for the datasource, including the number of bytes, segments, and intervals already compacted and those awaiting compaction. The Coordinator also reports the total bytes, segments, and intervals not eligible for compaction in accordance with its <a href="/docs/latest/design/coordinator#segment-search-policy-in-automatic-compaction">segment search policy</a>.</p><p>In the web console, the Datasources view displays auto-compaction statistics. The Tasks view shows the task information for compaction tasks that were triggered by the automatic compaction system.</p><p>To get statistics by API, send a <a href="/docs/latest/api-reference/automatic-compaction-api#view-automatic-compaction-status"><code>GET</code> request</a> to <code>/druid/coordinator/v1/compaction/status</code>. To filter the results to a particular datasource, pass the datasource name as a query parameter to the request—for example, <code>/druid/coordinator/v1/compaction/status?dataSource=wikipedia</code>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="avoid-conflicts-with-ingestion">Avoid conflicts with ingestion<a href="#avoid-conflicts-with-ingestion" class="hash-link" aria-label="Direct link to Avoid conflicts with ingestion" title="Direct link to Avoid conflicts with ingestion"></a></h2><p>Compaction tasks may be interrupted when they interfere with ingestion. For example, this occurs when an ingestion task needs to write data to a segment for a time interval locked for compaction. If there are continuous failures that prevent compaction from making progress, consider one of the following strategies:</p><ul><li>Enable <a href="#enable-concurrent-append-and-replace">concurrent append and replace tasks</a> on your datasource and on the ingestion tasks.</li><li>Set <code>skipOffsetFromLatest</code> to reduce the chance of conflicts between ingestion and compaction. See more details in <a href="#skip-compaction-for-latest-segments">Skip compaction for latest segments</a>.</li><li>Increase the priority value of compaction tasks relative to ingestion tasks. Only recommended for advanced users. This approach can cause ingestion jobs to fail or lag. To change the priority of compaction tasks, set <code>taskPriority</code> to the desired priority value in the auto-compaction configuration. For details on the priority values of different task types, see <a href="/docs/latest/ingestion/tasks#lock-priority">Lock priority</a>.</li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="enable-concurrent-append-and-replace">Enable concurrent append and replace<a href="#enable-concurrent-append-and-replace" class="hash-link" aria-label="Direct link to Enable concurrent append and replace" title="Direct link to Enable concurrent append and replace"></a></h3><p>You can use concurrent append and replace to safely replace the existing data in an interval of a datasource while new data is being appended to that interval even during compaction.</p><p>To do this, you need to update your datasource to allow concurrent append and replace tasks:</p><ul><li>If you're using the API, include the following <code>taskContext</code> property in your API call: <code>"useConcurrentLocks": true</code></li><li>If you're using the UI, enable <strong>Use concurrent locks</strong> in the <strong>Compaction config</strong> for your datasource.</li></ul><p>You'll also need to update your ingestion jobs for the datasource to include the task context <code>"useConcurrentLocks": true</code>.</p><p>For information on how to do this, see <a href="/docs/latest/ingestion/concurrent-append-replace">Concurrent append and replace</a>.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="skip-compaction-for-latest-segments">Skip compaction for latest segments<a href="#skip-compaction-for-latest-segments" class="hash-link" aria-label="Direct link to Skip compaction for latest segments" title="Direct link to Skip compaction for latest segments"></a></h3><p>The Coordinator compacts segments from newest to oldest. In the auto-compaction configuration, you can set a time period, relative to the end time of the most recent segment, for segments that should not be compacted. Assign this value to <code>skipOffsetFromLatest</code>. Note that this offset is not relative to the current time but to the latest segment time. For example, if you want to skip over segments from five days prior to the end time of the most recent segment, assign <code>"skipOffsetFromLatest": "P5D"</code>.</p><p>To set <code>skipOffsetFromLatest</code>, consider how frequently you expect the stream to receive late arriving data. If your stream only occasionally receives late arriving data, the auto-compaction system robustly compacts your data even though data is ingested outside the <code>skipOffsetFromLatest</code> window. For most realtime streaming ingestion use cases, it is reasonable to set <code>skipOffsetFromLatest</code> to a few hours or a day.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="examples">Examples<a href="#examples" class="hash-link" aria-label="Direct link to Examples" title="Direct link to Examples"></a></h2><p>The following examples demonstrate potential use cases in which auto-compaction may improve your Druid performance. See more details in <a href="/docs/latest/data-management/compaction#compaction-guidelines">Compaction strategies</a>. The examples in this section do not change the underlying data.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="change-segment-granularity">Change segment granularity<a href="#change-segment-granularity" class="hash-link" aria-label="Direct link to Change segment granularity" title="Direct link to Change segment granularity"></a></h3><p>You have a stream set up to ingest data with <code>HOUR</code> segment granularity into the <code>wikistream</code> datasource. You notice that your Druid segments are smaller than the <a href="/docs/latest/operations/segment-optimization">recommended segment size</a> of 5 million rows per segment. You wish to automatically compact segments to <code>DAY</code> granularity while leaving the latest week of data <em>not</em> compacted because your stream consistently receives data within that time period.</p><p>The following auto-compaction configuration compacts existing <code>HOUR</code> segments into <code>DAY</code> segments while leaving the latest week of data not compacted:</p><div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"dataSource"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"wikistream"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"granularitySpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"segmentGranularity"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"DAY"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"skipOffsetFromLatest"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"P1W"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="update-partitioning-scheme">Update partitioning scheme<a href="#update-partitioning-scheme" class="hash-link" aria-label="Direct link to Update partitioning scheme" title="Direct link to Update partitioning scheme"></a></h3><p>For your <code>wikipedia</code> datasource, you want to optimize segment access when regularly ingesting data without compromising compute time when querying the data. Your ingestion spec for batch append uses <a href="/docs/latest/ingestion/native-batch#dynamic-partitioning">dynamic partitioning</a> to optimize for write-time operations, while your stream ingestion partitioning is configured by the stream service. You want to implement auto-compaction to reorganize the data with a suitable read-time partitioning using <a href="/docs/latest/ingestion/native-batch#multi-dimension-range-partitioning">multi-dimension range partitioning</a>. Based on the dimensions frequently accessed in queries, you wish to partition on the following dimensions: <code>channel</code>, <code>countryName</code>, <code>namespace</code>.</p><p>The following auto-compaction configuration compacts updates the <code>wikipedia</code> segments to use multi-dimension range partitioning:</p><div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"dataSource"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"wikipedia"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"tuningConfig"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"partitionsSpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"type"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"range"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"partitionDimensions"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"channel"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"countryName"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"namespace"</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"targetRowsPerSegment"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">5000000</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="auto-compaction-using-compaction-supervisors">Auto-compaction using compaction supervisors<a href="#auto-compaction-using-compaction-supervisors" class="hash-link" aria-label="Direct link to Auto-compaction using compaction supervisors" title="Direct link to Auto-compaction using compaction supervisors"></a></h2><div class="theme-admonition theme-admonition-info alert alert--info admonition_LlT9"><div class="admonitionHeading_tbUL"><span class="admonitionIcon_kALy"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg></span>Experimental</div><div class="admonitionContent_S0QG"><p>Compaction supervisors are experimental. For production use, we recommend <a href="#auto-compaction-using-coordinator-duties">auto-compaction using Coordinator duties</a>.</p></div></div><p>You can run automatic compaction using compaction supervisors on the Overlord rather than Coordinator duties. Compaction supervisors provide the following benefits over Coordinator duties:</p><ul><li>Can use the supervisor framework to get information about the auto-compaction, such as status or state</li><li>More easily suspend or resume compaction for a datasource</li><li>Can use either the native compaction engine or the <a href="#use-msq-for-auto-compaction">MSQ task engine</a></li><li>More reactive and submits tasks as soon as a compaction slot is available</li><li>Tracked compaction task status to avoid re-compacting an interval repeatedly</li></ul><p>To use compaction supervisors, set the following properties in your Overlord runtime properties:</p><ul><li><code>druid.supervisor.compaction.enabled</code> to <code>true</code> so that compaction tasks can be run as supervisor tasks</li><li><code>druid.supervisor.compaction.engine</code> to <code>msq</code> to specify the MSQ task engine as the compaction engine or to <code>native</code> to use the native engine. This is the default engine if the <code>engine</code> field is omitted from your compaction config</li></ul><p>Compaction supervisors use the same syntax as auto-compaction using Coordinator duties with one key difference: you submit the auto-compaction as a a supervisor spec. In the spec, set the <code>type</code> to <code>autocompact</code> and include the auto-compaction config in the <code>spec</code>.</p><p>To submit an automatic compaction task, you can submit a supervisor spec through the <a href="#manage-compaction-supervisors-with-the-web-console">web console</a> or the <a href="#manage-compaction-supervisors-with-supervisor-apis">supervisor API</a>.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="manage-compaction-supervisors-with-the-web-console">Manage compaction supervisors with the web console<a href="#manage-compaction-supervisors-with-the-web-console" class="hash-link" aria-label="Direct link to Manage compaction supervisors with the web console" title="Direct link to Manage compaction supervisors with the web console"></a></h3><p>To submit a supervisor spec for MSQ task engine automatic compaction, perform the following steps:</p><ol><li>In the web console, go to the <strong>Supervisors</strong> tab.</li><li>Click <strong>...</strong> > <strong>Submit JSON supervisor</strong>.</li><li>In the dialog, include the following:<ul><li>The type of supervisor spec by setting <code>"type": "autocompact"</code></li><li>The compaction configuration by adding it to the <code>spec</code> field</li></ul><div class="language-json codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-json codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"type"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"autocompact"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"spec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"dataSource"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> YOUR_DATASOURCE</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"tuningConfig"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain">...</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"granularitySpec"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain">...</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> </span><span class="token property">"engine"</span><span class="token operator" style="color:rgb(137, 221, 255)">:</span><span class="token plain"> <native|msq></span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li><li>Submit the supervisor.</li></ol><p>To stop the automatic compaction task, suspend or terminate the supervisor through the UI or API.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="manage-compaction-supervisors-with-supervisor-apis">Manage compaction supervisors with supervisor APIs<a href="#manage-compaction-supervisors-with-supervisor-apis" class="hash-link" aria-label="Direct link to Manage compaction supervisors with supervisor APIs" title="Direct link to Manage compaction supervisors with supervisor APIs"></a></h3><p>Submitting an automatic compaction as a supervisor task uses the same endpoint as supervisor tasks for streaming ingestion.</p><p>The following example configures auto-compaction for the <code>wikipedia</code> datasource:</p><div class="language-sh codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-sh codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">curl --location --request POST 'http://localhost:8081/druid/indexer/v1/supervisor' \</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">--header 'Content-Type: application/json' \</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">--data-raw '{</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "type": "autocompact", // required</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "suspended": false, // optional </span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "spec": { // required</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "dataSource": "wikipedia", // required</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "tuningConfig": {...}, // optional</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "granularitySpec": {...}, // optional</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "engine": <native|msq>, // optional</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> ...</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">}'</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Note that if you omit <code>spec.engine</code>, Druid uses the default compaction engine. You can control the default compaction engine with the <code>druid.supervisor.compaction.engine</code> Overlord runtime property. If <code>spec.engine</code> and <code>druid.supervisor.compaction.engine</code> are omitted, Druid defaults to the native engine.</p><p>To stop the automatic compaction task, suspend or terminate the supervisor through the UI or API.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="use-msq-for-auto-compaction">Use MSQ for auto-compaction<a href="#use-msq-for-auto-compaction" class="hash-link" aria-label="Direct link to Use MSQ for auto-compaction" title="Direct link to Use MSQ for auto-compaction"></a></h3><p>The MSQ task engine is available as a compaction engine if you configure auto-compaction to use compaction supervisors. To use the MSQ task engine for automatic compaction, make sure the following requirements are met:</p><ul><li><a href="/docs/latest/multi-stage-query/#load-the-extension">Load the MSQ task engine extension</a>.</li><li>In your Overlord runtime properties, set the following properties:<ul><li><code>druid.supervisor.compaction.enabled</code> to <code>true</code> so that compaction tasks can be run as a supervisor task.</li><li>Optionally, set <code>druid.supervisor.compaction.engine</code> to <code>msq</code> to specify the MSQ task engine as the default compaction engine. If you don't do this, you'll need to set <code>spec.engine</code> to <code>msq</code> for each compaction supervisor spec where you want to use the MSQ task engine.</li></ul></li><li>Have at least two compaction task slots available or set <code>compactionConfig.taskContext.maxNumTasks</code> to two or more. The MSQ task engine requires at least two tasks to run, one controller task and one worker task.</li></ul><p>You can use <a href="/docs/latest/multi-stage-query/reference#context-parameters">MSQ task engine context parameters</a> in <code>spec.taskContext</code> when configuring your datasource for automatic compaction, such as setting the maximum number of tasks using the <code>spec.taskContext.maxNumTasks</code> parameter. Some of the MSQ task engine context parameters overlap with automatic compaction parameters. When these settings overlap, set one or the other.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="msq-task-engine-limitations">MSQ task engine limitations<a href="#msq-task-engine-limitations" class="hash-link" aria-label="Direct link to MSQ task engine limitations" title="Direct link to MSQ task engine limitations"></a></h4><p>When using the MSQ task engine for auto-compaction, keep the following limitations in mind:</p><ul><li>The <code>metricSpec</code> field is only supported for certain aggregators. For more information, see <a href="#supported-aggregators">Supported aggregators</a>.</li><li>Only dynamic and range-based partitioning are supported.</li><li>Set <code>rollup</code> to <code>true</code> if and only if <code>metricSpec</code> is not empty or null.</li><li>You can only partition on string dimensions. However, multi-valued string dimensions are not supported.</li><li>The <code>maxTotalRows</code> config is not supported in <code>DynamicPartitionsSpec</code>. Use <code>maxRowsPerSegment</code> instead.</li><li>Segments can only be sorted on <code>__time</code> as the first column.</li></ul><h4 class="anchor anchorWithStickyNavbar_LWe7" id="supported-aggregators">Supported aggregators<a href="#supported-aggregators" class="hash-link" aria-label="Direct link to Supported aggregators" title="Direct link to Supported aggregators"></a></h4><p>Auto-compaction using the MSQ task engine supports only aggregators that satisfy the following properties: </p><ul><li><strong>Mergeability</strong>: can combine partial aggregates</li><li><strong>Idempotency</strong>: produces the same results on repeated runs of the aggregator on previously aggregated values in a column</li></ul><p>This is exemplified by the following <code>longSum</code> aggregator:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">{"name": "added", "type": "longSum", "fieldName": "added"}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>where <code>longSum</code> being capable of combining partial results satisfies mergeability, while input and output column being the same (<code>added</code>) ensures idempotency.</p><p>The following are some examples of aggregators that aren't supported since at least one of the required conditions aren't satisfied:</p><ul><li><code>longSum</code> aggregator where the <code>added</code> column rolls up into <code>sum_added</code> column discarding the input <code>added</code> column, violating idempotency, as subsequent runs would no longer find the <code>added</code> column:<div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">{"name": "sum_added", "type": "longSum", "fieldName": "added"}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li><li>Partial sketches which cannot themselves be used to combine partial aggregates and need merging aggregators -- such as <code>HLLSketchMerge</code> required for <code>HLLSketchBuild</code> aggregator below -- violating mergeability:<div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">{"name": "added", "type": "HLLSketchBuild", "fieldName": "added"}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li><li>Count aggregator since it cannot be used to combine partial aggregates and it rolls up into a different <code>count</code> column discarding the input column(s), violating both mergeability and idempotency.<div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">{"type": "count", "name": "count"}</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="learn-more">Learn more<a href="#learn-more" class="hash-link" aria-label="Direct link to Learn more" title="Direct link to Learn more"></a></h2><p>See the following topics for more information:</p><ul><li><a href="/docs/latest/data-management/compaction">Compaction</a> for an overview of compaction in Druid.</li><li><a href="/docs/latest/data-management/manual-compaction">Manual compaction</a> for how to manually perform compaction tasks.</li><li><a href="/docs/latest/operations/segment-optimization">Segment optimization</a> for guidance on evaluating and optimizing Druid segment size.</li><li><a href="/docs/latest/design/coordinator#automatic-compaction">Coordinator process</a> for details on how the Coordinator plans compaction tasks.</li></ul></div></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages"><a class="pagination-nav__link pagination-nav__link--prev" href="/docs/latest/data-management/compaction"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Compaction</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/docs/latest/data-management/manual-compaction"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Manual compaction</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_bqdL thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#auto-compaction-syntax" class="table-of-contents__link toc-highlight">Auto-compaction syntax</a></li><li><a href="#auto-compaction-using-coordinator-duties" class="table-of-contents__link toc-highlight">Auto-compaction using Coordinator duties</a><ul><li><a href="#manage-auto-compaction-using-the-web-console" class="table-of-contents__link toc-highlight">Manage auto-compaction using the web console</a></li><li><a href="#manage-auto-compaction-using-coordinator-apis" class="table-of-contents__link toc-highlight">Manage auto-compaction using Coordinator APIs</a></li><li><a href="#change-compaction-frequency" class="table-of-contents__link toc-highlight">Change compaction frequency</a></li><li><a href="#view-coordinator-duty-auto-compaction-stats" class="table-of-contents__link toc-highlight">View Coordinator duty auto-compaction stats</a></li></ul></li><li><a href="#avoid-conflicts-with-ingestion" class="table-of-contents__link toc-highlight">Avoid conflicts with ingestion</a><ul><li><a href="#enable-concurrent-append-and-replace" class="table-of-contents__link toc-highlight">Enable concurrent append and replace</a></li><li><a href="#skip-compaction-for-latest-segments" class="table-of-contents__link toc-highlight">Skip compaction for latest segments</a></li></ul></li><li><a href="#examples" class="table-of-contents__link toc-highlight">Examples</a><ul><li><a href="#change-segment-granularity" class="table-of-contents__link toc-highlight">Change segment granularity</a></li><li><a href="#update-partitioning-scheme" class="table-of-contents__link toc-highlight">Update partitioning scheme</a></li></ul></li><li><a href="#auto-compaction-using-compaction-supervisors" class="table-of-contents__link toc-highlight">Auto-compaction using compaction supervisors</a><ul><li><a href="#manage-compaction-supervisors-with-the-web-console" class="table-of-contents__link toc-highlight">Manage compaction supervisors with the web console</a></li><li><a href="#manage-compaction-supervisors-with-supervisor-apis" class="table-of-contents__link toc-highlight">Manage compaction supervisors with supervisor APIs</a></li><li><a href="#use-msq-for-auto-compaction" class="table-of-contents__link toc-highlight">Use MSQ for auto-compaction</a></li></ul></li><li><a href="#learn-more" class="table-of-contents__link toc-highlight">Learn more</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="footer__bottom text--center"><div class="margin-bottom--sm"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--light_HNdA footer__logo"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--dark_i4oU footer__logo"></div><div class="footer__copyright">Copyright © 2023 Apache Software Foundation. Except where otherwise noted, licensed under CC BY-SA 4.0. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</div></div></div></footer></div> |
