Google Git
Sign in
apache / druid-website / 2aba6b76f123120b86bf6ff7e30f4004c78949b0 / . / docs / 0.22.1 / ingestion / compaction.html
blob: d9da47488412c91b94d8e844df7e5afc05863cd6 [file] [log] [blame]
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>Compaction · Apache Druid</title><meta name="viewport" content="width=device-width"/><link rel="canonical" href="https://druid.apache.org/docs/0.22.1/ingestion/compaction.html"/><meta name="generator" content="Docusaurus"/><meta name="description" content="Defines compaction and automatic compaction (auto-compaction or autocompaction) for segment optimization. Use cases and strategies for compaction. Describes compaction task configuration."/><meta name="docsearch:language" content="en"/><meta name="docsearch:version" content="0.22.1" /><meta property="og:title" content="Compaction · Apache Druid"/><meta property="og:type" content="website"/><meta property="og:url" content="https://druid.apache.org/index.html"/><meta property="og:description" content="Defines compaction and automatic compaction (auto-compaction or autocompaction) for segment optimization. Use cases and strategies for compaction. Describes compaction task configuration."/><meta property="og:image" content="https://druid.apache.org/img/druid_nav.png"/><meta name="twitter:card" content="summary"/><meta name="twitter:image" content="https://druid.apache.org/img/druid_nav.png"/><link rel="shortcut icon" href="/img/favicon.png"/><link rel="stylesheet" href="https://cdn.jsdelivr.net/docsearch.js/1/docsearch.min.css"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><script async="" src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script><script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments); }
gtag('js', new Date());
gtag('config', 'UA-131010415-1');
</script><link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css"/><link rel="stylesheet" href="/css/code-block-buttons.css"/><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.4/clipboard.min.js"></script><script type="text/javascript" src="/js/code-block-buttons.js"></script><script src="/js/scrollSpy.js"></script><link rel="stylesheet" href="/css/main.css"/><script src="/js/codetabs.js"></script></head><body class="sideNavVisible separateOnPageNav"><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><img class="logo" src="/img/druid_nav.png" alt="Apache Druid"/></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/technology" target="_self">Technology</a></li><li class=""><a href="/use-cases" target="_self">Use Cases</a></li><li class=""><a href="/druid-powered" target="_self">Powered By</a></li><li class="siteNavGroupActive"><a href="/docs/0.22.1/design/index.html" target="_self">Docs</a></li><li class=""><a href="/community/" target="_self">Community</a></li><li class=""><a href="https://www.apache.org" target="_self">Apache</a></li><li class=""><a href="/downloads.html" target="_self">Download</a></li><li class="navSearchWrapper reactNavSearchWrapper"><input type="text" id="search_input_react" placeholder="Search" title="Search"/></li></ul></nav></div></header></div></div><div class="navPusher"><div class="docMainWrapper wrapper"><div class="docsNavContainer" id="docsNav"><nav class="toc"><div class="toggleNav"><section class="navWrapper wrapper"><div class="navBreadcrumb wrapper"><div class="navToggle" id="navToggler"><div class="hamburger-menu"><div class="line1"></div><div class="line2"></div><div class="line3"></div></div></div><h2><i>›</i><span>Ingestion</span></h2><div class="tocToggler" id="tocToggler"><i class="icon-toc"></i></div></div><div class="navGroups"><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Getting started<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/index.html">Introduction to Apache Druid</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/index.html">Quickstart</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/docker.html">Docker</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/single-server.html">Single server deployment</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/cluster.html">Clustered deployment</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Tutorials<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-batch.html">Loading files natively</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-kafka.html">Load from Apache Kafka</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-batch-hadoop.html">Load from Apache Hadoop</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-query.html">Querying data</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-rollup.html">Roll-up</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-retention.html">Configuring data retention</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-update-data.html">Updating existing data</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-compaction.html">Compacting segments</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-delete-data.html">Deleting data</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-ingestion-spec.html">Writing an ingestion spec</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-transform-spec.html">Transforming input data</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/tutorials/tutorial-kerberos-hadoop.html">Kerberized HDFS deep storage</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Design<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/architecture.html">Design</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/segments.html">Segments</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/processes.html">Processes and servers</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/dependencies/deep-storage.html">Deep storage</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/dependencies/metadata-storage.html">Metadata storage</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/dependencies/zookeeper.html">ZooKeeper</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Ingestion<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/index.html">Ingestion</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/data-formats.html">Data formats</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/data-model.html">Data model</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/rollup.html">Data rollup</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/partitioning.html">Partitioning</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/ingestion-spec.html">Ingestion spec</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/schema-design.html">Schema design tips</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/data-management.html">Data management</a></li><li class="navListItem navListItemActive"><a class="navItem" href="/docs/0.22.1/ingestion/compaction.html">Compaction</a></li><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Stream ingestion</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/kafka-ingestion.html">Apache Kafka</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/kinesis-ingestion.html">Amazon Kinesis</a></li></ul></div><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Batch ingestion</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/native-batch.html">Native batch</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/hadoop.html">Hadoop-based</a></li></ul></div><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/tasks.html">Task reference</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/faq.html">Troubleshooting FAQ</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Querying<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/sql.html">Druid SQL</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/querying.html">Native queries</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/query-execution.html">Query execution</a></li><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Concepts</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/datasource.html">Datasources</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/joins.html">Joins</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/lookups.html">Lookups</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/multi-value-dimensions.html">Multi-value dimensions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/multitenancy.html">Multitenancy</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/caching.html">Query caching</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/using-caching.html">Using query caching</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/query-context.html">Context parameters</a></li></ul></div><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Native query types</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/timeseriesquery.html">Timeseries</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/topnquery.html">TopN</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/groupbyquery.html">GroupBy</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/scan-query.html">Scan</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/searchquery.html">Search</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/timeboundaryquery.html">TimeBoundary</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/segmentmetadataquery.html">SegmentMetadata</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/datasourcemetadataquery.html">DatasourceMetadata</a></li></ul></div><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Native query components</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/filters.html">Filters</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/granularities.html">Granularities</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/dimensionspecs.html">Dimensions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/aggregations.html">Aggregations</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/post-aggregations.html">Post-aggregations</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/misc/math-expr.html">Expressions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/having.html">Having filters (groupBy)</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/limitspec.html">Sorting and limiting (groupBy)</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/topnmetricspec.html">Sorting (topN)</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/sorting-orders.html">String comparators</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/virtual-columns.html">Virtual columns</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/geo.html">Spatial filters</a></li></ul></div></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Configuration<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/configuration/index.html">Configuration reference</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions.html">Extensions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/configuration/logging.html">Logging</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Operations<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/druid-console.html">Web console</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/getting-started.html">Getting started with Apache Druid</a></li><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Security</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/security-overview.html">Security overview</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/security-user-auth.html">User authentication and authorization</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/auth-ldap.html">LDAP auth</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/password-provider.html">Password providers</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/dynamic-config-provider.html">Dynamic Config Providers</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/tls-support.html">TLS support</a></li></ul></div><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Performance tuning</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/basic-cluster-tuning.html">Basic cluster tuning</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/segment-optimization.html">Segment Size Optimization</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/http-compression.html">HTTP compression</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/clean-metadata-store.html">Automated metadata cleanup</a></li></ul></div><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/api-reference.html">API reference</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/high-availability.html">High availability</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/rolling-updates.html">Rolling updates</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/rule-configuration.html">Retaining or automatically dropping data</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/metrics.html">Metrics</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/alerts.html">Alerts</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/other-hadoop.html">Working with different versions of Apache Hadoop</a></li><div class="navGroup subNavGroup"><h4 class="navGroupSubcategoryTitle">Misc</h4><ul><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/management-uis.html">Legacy Management UIs</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/dump-segment.html">dump-segment tool</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/reset-cluster.html">reset-cluster tool</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/insert-segment-to-db.html">insert-segment-to-db tool</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/pull-deps.html">pull-deps tool</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/deep-storage-migration.html">Deep storage migration</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/export-metadata.html">Export Metadata Tool</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/metadata-migration.html">Metadata Migration</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/use_sbt_to_build_fat_jar.html">Content for build.sbt</a></li></ul></div></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Development<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/overview.html">Developing on Druid</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/modules.html">Creating extensions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/javascript.html">JavaScript functionality</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/build.html">Build from source</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/versioning.html">Versioning</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/experimental.html">Experimental features</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Misc<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/misc/papers-and-talks.html">Papers</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle collapsible">Hidden<span class="arrow"><svg width="24" height="24" viewBox="0 0 24 24"><path fill="#565656" d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z"></path><path d="M0 0h24v24H0z" fill="none"></path></svg></span></h3><ul class="hide"><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-elasticsearch.html">Apache Druid vs Elasticsearch</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-key-value.html">Apache Druid vs. Key/Value Stores (HBase/Cassandra/OpenTSDB)</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-kudu.html">Apache Druid vs Kudu</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-redshift.html">Apache Druid vs Redshift</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-spark.html">Apache Druid vs Spark</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/comparisons/druid-vs-sql-on-hadoop.html">Apache Druid vs SQL-on-Hadoop</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/auth.html">Authentication and Authorization</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/broker.html">Broker</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/coordinator.html">Coordinator Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/historical.html">Historical Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/indexer.html">Indexer Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/indexing-service.html">Indexing Service</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/middlemanager.html">MiddleManager Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/overlord.html">Overlord Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/router.html">Router Process</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/design/peons.html">Peons</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/approximate-histograms.html">Approximate Histogram aggregators</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/avro.html">Apache Avro</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/azure.html">Microsoft Azure</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/bloom-filter.html">Bloom Filter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/datasketches-extension.html">DataSketches extension</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/datasketches-hll.html">DataSketches HLL Sketch module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/datasketches-quantiles.html">DataSketches Quantiles Sketch module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/datasketches-theta.html">DataSketches Theta Sketch module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/datasketches-tuple.html">DataSketches Tuple Sketch module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-basic-security.html">Basic Security</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-kerberos.html">Kerberos</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-lookups.html">Cached Lookup Module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-ranger-security.html">Apache Ranger Security</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/google.html">Google Cloud Storage</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/hdfs.html">HDFS</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/kafka-extraction-namespace.html">Apache Kafka Lookups</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/lookups-cached-global.html">Globally Cached Lookups</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/mysql.html">MySQL Metadata Store</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/orc.html">ORC Extension</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-pac4j.html">Druid pac4j based Security extension</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/parquet.html">Apache Parquet Extension</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/postgresql.html">PostgreSQL Metadata Store</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/protobuf.html">Protobuf</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/s3.html">S3-compatible</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/simple-client-sslcontext.html">Simple SSLContext Provider Module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/stats.html">Stats aggregator</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/test-stats.html">Test Stats Aggregators</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/druid-aws-rds.html">Druid AWS RDS Module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-core/kubernetes.html">Kubernetes</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/ambari-metrics-emitter.html">Ambari Metrics Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/cassandra.html">Apache Cassandra</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/cloudfiles.html">Rackspace Cloud Files</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/distinctcount.html">DistinctCount Aggregator</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/graphite.html">Graphite Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/influx.html">InfluxDB Line Protocol Parser</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/influxdb-emitter.html">InfluxDB Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/kafka-emitter.html">Kafka Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/materialized-view.html">Materialized View</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/momentsketch-quantiles.html">Moment Sketches for Approximate Quantiles module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/moving-average-query.html">Moving Average Query</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/opentsdb-emitter.html">OpenTSDB Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/redis-cache.html">Druid Redis Cache</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/sqlserver.html">Microsoft SQLServer</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/statsd.html">StatsD Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/tdigestsketch-quantiles.html">T-Digest Quantiles Sketch module</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/thrift.html">Thrift</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/time-min-max.html">Timestamp Min/Max aggregators</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/gce-extensions.html">GCE Extensions</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/aliyun-oss.html">Aliyun OSS</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/development/extensions-contrib/prometheus.html">Prometheus Emitter</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/operations/kubernetes.html">kubernetes</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/hll-old.html">Cardinality/HyperUnique aggregators</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/querying/select-query.html">Select</a></li><li class="navListItem"><a class="navItem" href="/docs/0.22.1/ingestion/standalone-realtime.html">Realtime Process</a></li></ul></div></div></section></div><script>
var coll = document.getElementsByClassName('collapsible');
var checkActiveCategory = true;
for (var i = 0; i < coll.length; i++) {
var links = coll[i].nextElementSibling.getElementsByTagName('*');
if (checkActiveCategory){
for (var j = 0; j < links.length; j++) {
if (links[j].classList.contains('navListItemActive')){
coll[i].nextElementSibling.classList.toggle('hide');
coll[i].childNodes[1].classList.toggle('rotate');
checkActiveCategory = false;
break;
}
}
}
coll[i].addEventListener('click', function() {
var arrow = this.childNodes[1];
arrow.classList.toggle('rotate');
var content = this.nextElementSibling;
content.classList.toggle('hide');
});
}
document.addEventListener('DOMContentLoaded', function() {
createToggler('#navToggler', '#docsNav', 'docsSliderActive');
createToggler('#tocToggler', 'body', 'tocActive');
var headings = document.querySelector('.toc-headings');
headings && headings.addEventListener('click', function(event) {
var el = event.target;
while(el !== headings){
if (el.tagName === 'A') {
document.body.classList.remove('tocActive');
break;
} else{
el = el.parentNode;
}
}
}, false);
function createToggler(togglerSelector, targetSelector, className) {
var toggler = document.querySelector(togglerSelector);
var target = document.querySelector(targetSelector);
if (!toggler) {
return;
}
toggler.onclick = function(event) {
event.preventDefault();
target.classList.toggle(className);
};
}
});
</script></nav></div><div class="container mainContainer docsContainer"><div class="wrapper"><div class="post"><header class="postHeader"><a class="edit-page-link button" href="https://github.com/apache/druid/edit/master/docs/ingestion/compaction.md" target="_blank" rel="noreferrer noopener">Edit</a><h1 id="__docusaurus" class="postHeaderTitle">Compaction</h1></header><article><div><span><!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->
Query performance in Apache Druid depends on optimally sized segments. Compaction is one strategy you can use to optimize segment size for your Druid database. Compaction tasks read an existing set of segments for a given time interval and combine the data into a new "compacted" set of segments. In some cases the compacted segments are larger, but there are fewer of them. In other cases the compacted segments may be smaller. Compaction tends to increase performance because optimized segments require less per-segment processing and less memory overhead for ingestion and for querying paths.
<h2><a class="anchor" aria-hidden="true" id="compaction-strategies"></a><a href="#compaction-strategies" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Compaction strategies</h2>
<p>There are several cases to consider compaction for segment optimization:</p>
<ul>
<li>With streaming ingestion, data can arrive out of chronological order creating lots of small segments.</li>
<li>If you append data using <code>appendToExisting</code> for <a href="/docs/0.22.1/ingestion/native-batch.html">native batch</a> ingestion creating suboptimal segments.</li>
<li>When you use <code>index_parallel</code> for parallel batch indexing and the parallel ingestion tasks create many small segments.</li>
<li>When a misconfigured ingestion task creates oversized segments.</li>
</ul>
<p>By default, compaction does not modify the underlying data of the segments. However, there are cases when you may want to modify data during compaction to improve query performance:</p>
<ul>
<li>If, after ingestion, you realize that data for the time interval is sparse, you can use compaction to increase the segment granularity.</li>
<li>Over time you don't need fine-grained granularity for older data so you want use compaction to change older segments to a coarser query granularity. This reduces the storage space required for older data. For example from <code>minute</code> to <code>hour</code>, or <code>hour</code> to <code>day</code>. You cannot go from coarser granularity to finer granularity.</li>
<li>You can change the dimension order to improve sorting and reduce segment size.</li>
<li>You can remove unused columns in compaction or implement an aggregation metric for older data.</li>
<li>You can change segment rollup from dynamic partitioning with best-effort rollup to hash or range partitioning with perfect rollup. For more information on rollup, see <a href="/docs/0.22.1/ingestion/rollup.html#perfect-rollup-vs-best-effort-rollup">perfect vs best-effort rollup</a>.</li>
</ul>
<p>Compaction does not improve performance in all situations. For example, if you rewrite your data with each ingestion task, you don't need to use compaction. See <a href="/docs/0.22.1/operations/segment-optimization.html">Segment optimization</a> for additional guidance to determine if compaction will help in your environment.</p>
<h2><a class="anchor" aria-hidden="true" id="types-of-compaction"></a><a href="#types-of-compaction" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Types of compaction</h2>
<p>You can configure the Druid Coordinator to perform automatic compaction, also called auto-compaction, for a datasource. Using a segment search policy, the coordinator periodically identifies segments for compaction starting with the newest to oldest. When it discovers segments that have not been compacted or segments that were compacted with a different or changed spec, it submits compaction task for those segments and only those segments.</p>
<p>Automatic compaction works in most use cases and should be your first option. To learn more about automatic compaction, see <a href="/docs/0.22.1/design/coordinator.html#compacting-segments">Compacting Segments</a>.</p>
<p>In cases where you require more control over compaction, you can manually submit compaction tasks. For example:</p>
<ul>
<li>Automatic compaction is running into the limit of task slots available to it, so tasks are waiting for previous automatic compaction tasks to complete. Manual compaction can use all available task slots, therefore you can complete compaction more quickly by submitting more concurrent tasks for more intervals.</li>
<li>You want to force compaction for a specific time range or you want to compact data out of chronological order.</li>
</ul>
<p>See <a href="#setting-up-manual-compaction">Setting up a manual compaction task</a> for more about manual compaction tasks.</p>
<h2><a class="anchor" aria-hidden="true" id="data-handling-with-compaction"></a><a href="#data-handling-with-compaction" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Data handling with compaction</h2>
<p>During compaction, Druid overwrites the original set of segments with the compacted set. Druid also locks the segments for the time interval being compacted to ensure data consistency. By default, compaction tasks do not modify the underlying data. You can configure the compaction task to change the query granularity or add or remove dimensions in the compaction task. This means that the only changes to query results should be the result of intentional, not automatic, changes.</p>
<p>You can set <code>dropExisting</code> in <code>ioConfig</code> to &quot;true&quot; in the compaction task to configure Druid to mark all existing segments fully contained by the interval of the compaction task as unused (drop the segments). See the suggestion for reindexing with finer granularity under <a href="/docs/0.22.1/ingestion/native-batch.html#implementation-considerations">Implementation considerations</a> for an example.</p>
<blockquote>
<p>WARNING: <code>dropExisting</code> in <code>ioConfig</code> is a beta feature. It may cause data within the compaction task interval to become temporarily unavailable.</p>
</blockquote>
<p>If an ingestion task needs to write data to a segment for a time interval locked for compaction, by default the ingestion task supersedes the compaction task and the compaction task fails without finishing. For manual compaction tasks you can adjust the input spec interval to avoid conflicts between ingestion and compaction. For automatic compaction, you can set the <code>skipOffsetFromLatest</code> key to adjustment the auto compaction starting point from the current time to reduce the chance of conflicts between ingestion and compaction. See <a href="/docs/0.22.1/configuration/index.html#compaction-dynamic-configuration">Compaction dynamic configuration</a> for more information. Another option is to set the compaction task to higher priority than the ingestion task.</p>
<h3><a class="anchor" aria-hidden="true" id="segment-granularity-handling"></a><a href="#segment-granularity-handling" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Segment granularity handling</h3>
<p>Unless you modify the segment granularity in the <a href="#compaction-granularity-spec">granularity spec</a>, Druid attempts to retain the granularity for the compacted segments. When segments have different segment granularities with no overlap in interval Druid creates a separate compaction task for each to retain the segment granularity in the compacted segment.</p>
<p>If segments have different segment granularities before compaction but there is some overlap in interval, Druid attempts find start and end of the overlapping interval and uses the closest segment granularity level for the compacted segment. For example consider two overlapping segments: segment &quot;A&quot; for the interval 01/01/2021-01/02/2021 with day granularity and segment &quot;B&quot; for the interval 01/01/2021-02/01/2021. Druid attempts to combine and compacted the overlapped segments. In this example, the earliest start time for the two segments above is 01/01/2020 and the latest end time of the two segments above is 02/01/2020. Druid compacts the segments together even though they have different segment granularity. Druid uses month segment granularity for the newly compacted segment even though segment A's original segment granularity was DAY.</p>
<h3><a class="anchor" aria-hidden="true" id="query-granularity-handling"></a><a href="#query-granularity-handling" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Query granularity handling</h3>
<p>Unless you modify the query granularity in the <a href="#compaction-granularity-spec">granularity spec</a>, Druid retains the query granularity for the compacted segments. If segments have different query granularities before compaction, Druid chooses the finest level of granularity for the resulting compacted segment. For example if a compaction task combines two segments, one with day query granularity and one with minute query granularity, the resulting segment uses minute query granularity.</p>
<blockquote>
<p>In Apache Druid 0.21.0 and prior, Druid sets the granularity for compacted segments to the default granularity of <code>NONE</code> regardless of the query granularity of the original segments.</p>
</blockquote>
<p>If you configure query granularity in compaction to go from a finer granularity like month to a coarser query granularity like year, then Druid overshadows the original segment with coarser granularity. Because the new segments have a coarser granularity, running a kill task to remove the overshadowed segments for those intervals will cause you to permanently lose the finer granularity data.</p>
<h3><a class="anchor" aria-hidden="true" id="dimension-handling"></a><a href="#dimension-handling" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Dimension handling</h3>
<p>Apache Druid supports schema changes. Therefore, dimensions can be different across segments even if they are a part of the same data source. See <a href="/docs/0.22.1/design/segments.html#different-schemas-among-segments">Different schemas among segments</a>. If the input segments have different dimensions, the resulting compacted segment include all dimensions of the input segments.</p>
<p>Even when the input segments have the same set of dimensions, the dimension order or the data type of dimensions can be different. The dimensions of recent segments precede that of old segments in terms of data types and the ordering because more recent segments are more likely to have the preferred order and data types.</p>
<p>If you want to control dimension ordering or ensure specific values for dimension types, you can configure a custom <code>dimensionsSpec</code> in the compaction task spec.</p>
<h3><a class="anchor" aria-hidden="true" id="rollup"></a><a href="#rollup" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rollup</h3>
<p>Druid only rolls up the output segment when <code>rollup</code> is set for all input segments.
See <a href="/docs/0.22.1/ingestion/rollup.html">Roll-up</a> for more details.
You can check that your segments are rolled up or not by using <a href="/docs/0.22.1/querying/segmentmetadataquery.html#analysistypes">Segment Metadata Queries</a>.</p>
<h2><a class="anchor" aria-hidden="true" id="setting-up-manual-compaction"></a><a href="#setting-up-manual-compaction" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setting up manual compaction</h2>
<p>To perform a manual compaction, you submit a compaction task. Compaction tasks merge all segments for the defined interval according to the following syntax:</p>
<pre><code class="hljs css language-json">{
"type": "compact",
"id": &lt;task_id&gt;,
"dataSource": &lt;task_datasource&gt;,
"ioConfig": &lt;IO config&gt;,
"dimensionsSpec" &lt;custom dimensionsSpec&gt;,
"metricsSpec" &lt;custom metricsSpec&gt;,
"tuningConfig" &lt;parallel indexing task tuningConfig&gt;,
"granularitySpec" &lt;compaction task granularitySpec&gt;,
"context": &lt;task context&gt;
}
</code></pre>
<table>
<thead>
<tr><th>Field</th><th>Description</th><th>Required</th></tr>
</thead>
<tbody>
<tr><td><code>type</code></td><td>Task type. Should be <code>compact</code></td><td>Yes</td></tr>
<tr><td><code>id</code></td><td>Task id</td><td>No</td></tr>
<tr><td><code>dataSource</code></td><td>Data source name to compact</td><td>Yes</td></tr>
<tr><td><code>ioConfig</code></td><td>I/O configuration for compaction task. See <a href="#compaction-io-configuration">Compaction I/O configuration</a> for details.</td><td>Yes</td></tr>
<tr><td><code>dimensionsSpec</code></td><td>Custom dimensions spec. The compaction task uses the specified dimensions spec if it exists instead of generating one.</td><td>No</td></tr>
<tr><td><code>metricsSpec</code></td><td>Custom metrics spec. The compaction task uses the specified metrics spec rather than generating one.</td><td>No</td></tr>
<tr><td><code>segmentGranularity</code></td><td>When set, the compaction task changes the segment granularity for the given interval. Deprecated. Use <code>granularitySpec</code>.</td><td>No.</td></tr>
<tr><td><code>tuningConfig</code></td><td><a href="/docs/0.22.1/ingestion/native-batch.html#tuningconfig">Parallel indexing task tuningConfig</a>. <code>awaitSegmentAvailabilityTimeoutMillis</code> in the tuning config is not currently supported for compaction tasks. Do not set it to a non-zero value.</td><td>No</td></tr>
<tr><td><code>context</code></td><td><a href="/docs/0.22.1/ingestion/tasks.html#context">Task context</a></td><td>No</td></tr>
<tr><td><code>granularitySpec</code></td><td>Custom <code>granularitySpec</code> to describe the <code>segmentGranularity</code> and <code>queryGranularity</code> for the compacted segments. See <a href="#compaction-granularity-spec">Compaction granularitySpec</a>.</td><td>No</td></tr>
</tbody>
</table>
<blockquote>
<p>Note: Use <code>granularitySpec</code> over <code>segmentGranularity</code> and only set one of these values. If you specify different values for these in the same compaction spec, the task fails.</p>
</blockquote>
<p>To control the number of result segments per time chunk, you can set <a href="/docs/0.22.1/configuration/index.html#compaction-dynamic-configuration">maxRowsPerSegment</a> or <a href="/docs/0.22.1/ingestion/native-batch.html#tuningconfig">numShards</a>.</p>
<blockquote>
<p>You can run multiple compaction tasks in parallel. For example, if you want to compact the data for a year, you are not limited to running a single task for the entire year. You can run 12 compaction tasks with month-long intervals.</p>
</blockquote>
<p>A compaction task internally generates an <code>index</code> task spec for performing compaction work with some fixed parameters. For example, its <code>inputSource</code> is always the <a href="/docs/0.22.1/ingestion/native-batch.html#druid-input-source">DruidInputSource</a>, and <code>dimensionsSpec</code> and <code>metricsSpec</code> include all dimensions and metrics of the input segments by default.</p>
<p>Compaction tasks exit without doing anything and issue a failure status code in either of the following cases:</p>
<ul>
<li>If the interval you specify has no data segments loaded<br></li>
<li>If the interval you specify is empty.</li>
</ul>
<p>Note that the metadata between input segments and the resulting compacted segments may differ if the metadata among the input segments differs as well. If all input segments have the same metadata, however, the resulting output segment will have the same metadata as all input segments.</p>
<h3><a class="anchor" aria-hidden="true" id="example-compaction-task"></a><a href="#example-compaction-task" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Example compaction task</h3>
<p>The following JSON illustrates a compaction task to compact <em>all segments</em> within the interval <code>2017-01-01/2018-01-01</code> and create new segments:</p>
<pre><code class="hljs css language-json">{
<span class="hljs-attr">"type"</span> : <span class="hljs-string">"compact"</span>,
<span class="hljs-attr">"dataSource"</span> : <span class="hljs-string">"wikipedia"</span>,
<span class="hljs-attr">"ioConfig"</span> : {
<span class="hljs-attr">"type"</span>: <span class="hljs-string">"compact"</span>,
<span class="hljs-attr">"inputSpec"</span>: {
<span class="hljs-attr">"type"</span>: <span class="hljs-string">"interval"</span>,
<span class="hljs-attr">"interval"</span>: <span class="hljs-string">"2020-01-01/2021-01-01"</span>,
}
},
<span class="hljs-attr">"granularitySpec"</span>: {
<span class="hljs-attr">"segmentGranularity"</span>:<span class="hljs-string">"day"</span>,
<span class="hljs-attr">"queryGranularity"</span>:<span class="hljs-string">"hour"</span>
}
}
</code></pre>
<p>This task doesn't specify a <code>granularitySpec</code> so Druid retains the original segment granularity unchanged when compaction is complete.</p>
<h3><a class="anchor" aria-hidden="true" id="compaction-io-configuration"></a><a href="#compaction-io-configuration" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Compaction I/O configuration</h3>
<p>The compaction <code>ioConfig</code> requires specifying <code>inputSpec</code> as follows:</p>
<table>
<thead>
<tr><th>Field</th><th>Description</th><th>Default</th><th>Required?</th></tr>
</thead>
<tbody>
<tr><td><code>type</code></td><td>Task type: <code>compact</code></td><td>none</td><td>Yes</td></tr>
<tr><td><code>inputSpec</code></td><td>Input specification</td><td>none</td><td>Yes</td></tr>
<tr><td><code>dropExisting</code></td><td>If <code>true</code>, when the task publishes newly compacted segments the compaction, it marks all existing segments fully contained by either of the following as unused (drops the segments):<br>- the <code>interval</code> in the <code>interval</code> type <code>inputSpec</code>.<br>- the umbrella interval of the <code>segments</code> in the <code>segment</code> type <code>inputSpec</code>.<br>If compaction fails, Druid does not drop or mark unused any segments.<br><strong>WARNING</strong>: <code>dropExisting</code> in <code>ioConfig</code> is a beta feature. It may cause data within the compaction task interval to become temporarily unavailable.</td><td>false</td><td>no</td></tr>
</tbody>
</table>
<p>Druid supports two supported <code>inputSpec</code> formats:</p>
<h4><a class="anchor" aria-hidden="true" id="interval-inputspec"></a><a href="#interval-inputspec" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Interval <code>inputSpec</code>:</h4>
<pre><code class="hljs"> |Field|Description|Required|
|-----|-----------|--------|
|`type`|Task type. Should be `interval`|Yes|
|`interval`|Interval to compact|Yes|
</code></pre>
<h4><a class="anchor" aria-hidden="true" id="segments-inputspec"></a><a href="#segments-inputspec" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Segments <code>inputSpec</code>:</h4>
<pre><code class="hljs"> |Field|Description|Required|
|-----|-----------|--------|
|`type`|Task type. Should be `segments`|Yes|
|`segments`|A list of segment IDs|Yes|
</code></pre>
<h3><a class="anchor" aria-hidden="true" id="compaction-granularity-spec"></a><a href="#compaction-granularity-spec" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Compaction granularity spec</h3>
<p>You can optionally use the <code>granularitySpec</code> object to configure the segment granularity and the query granularity of the compacted segments. Their syntax is as follows:</p>
<pre><code class="hljs css language-json"> "type": "compact",
"id": &lt;task_id&gt;,
"dataSource": &lt;task_datasource&gt;,
...
,
"granularitySpec": {
"segmentGranularity": &lt;time_period&gt;,
"queryGranularity": &lt;time_period&gt;
}
...
</code></pre>
<p><code>granularitySpec</code> takes the following keys:</p>
<table>
<thead>
<tr><th>Field</th><th>Description</th><th>Required</th></tr>
</thead>
<tbody>
<tr><td><code>segmentGranularity</code></td><td>Time chunking period for the segment granularity. Defaults to 'null', which preserves the original segment granularity. Accepts all <a href="/docs/0.22.1/querying/granularities.html">Query granularity</a> values.</td><td>No</td></tr>
<tr><td><code>queryGranularity</code></td><td>Time chunking period for the query granularity. Defaults to 'null', which preserves the original query granularity. Accepts all <a href="/docs/0.22.1/querying/granularities.html">Query granularity</a> values. Not supported for automatic compaction.</td><td>No</td></tr>
</tbody>
</table>
<p>For example, to set the segment granularity to &quot;day&quot; and the query granularity to &quot;hour&quot;:</p>
<pre><code class="hljs css language-json">{
<span class="hljs-attr">"type"</span> : <span class="hljs-string">"compact"</span>,
<span class="hljs-attr">"dataSource"</span> : <span class="hljs-string">"wikipedia"</span>,
<span class="hljs-attr">"ioConfig"</span> : {
<span class="hljs-attr">"type"</span>: <span class="hljs-string">"compact"</span>,
<span class="hljs-attr">"inputSpec"</span>: {
<span class="hljs-attr">"type"</span>: <span class="hljs-string">"interval"</span>,
<span class="hljs-attr">"interval"</span>: <span class="hljs-string">"2017-01-01/2018-01-01"</span>,
},
<span class="hljs-attr">"granularitySpec"</span>: {
<span class="hljs-attr">"segmentGranularity"</span>:<span class="hljs-string">"day"</span>,
<span class="hljs-attr">"queryGranularity"</span>:<span class="hljs-string">"hour"</span>
}
}
}
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="learn-more"></a><a href="#learn-more" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Learn more</h2>
<p>See the following topics for more information:</p>
<ul>
<li><a href="/docs/0.22.1/operations/segment-optimization.html">Segment optimization</a> for guidance to determine if compaction will help in your case.</li>
<li><a href="/docs/0.22.1/design/coordinator.html#compacting-segments">Compacting Segments</a> for more on automatic compaction.</li>
<li><a href="/docs/0.22.1/operations/api-reference.html#compaction-configuration">Compaction Configuration API</a>
and <a href="/docs/0.22.1/configuration/index.html#compaction-dynamic-configuration">Compaction Configuration</a> for automatic compaction configuration information.</li>
</ul>
</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/0.22.1/ingestion/data-management.html"><span class="arrow-prev">← </span><span>Data management</span></a><a class="docs-next button" href="/docs/0.22.1/development/extensions-core/kafka-ingestion.html"><span>Apache Kafka</span><span class="arrow-next"> →</span></a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#compaction-strategies">Compaction strategies</a></li><li><a href="#types-of-compaction">Types of compaction</a></li><li><a href="#data-handling-with-compaction">Data handling with compaction</a><ul class="toc-headings"><li><a href="#segment-granularity-handling">Segment granularity handling</a></li><li><a href="#query-granularity-handling">Query granularity handling</a></li><li><a href="#dimension-handling">Dimension handling</a></li><li><a href="#rollup">Rollup</a></li></ul></li><li><a href="#setting-up-manual-compaction">Setting up manual compaction</a><ul class="toc-headings"><li><a href="#example-compaction-task">Example compaction task</a></li><li><a href="#compaction-io-configuration">Compaction I/O configuration</a></li><li><a href="#compaction-granularity-spec">Compaction granularity spec</a></li></ul></li><li><a href="#learn-more">Learn more</a></li></ul></nav></div><footer class="nav-footer druid-footer" id="footer"><div class="container"><div class="text-center"><p><a href="/technology">Technology</a> · <a href="/use-cases">Use Cases</a> · <a href="/druid-powered">Powered by Druid</a> · <a href="/docs/0.22.1/latest">Docs</a> · <a href="/community/">Community</a> · <a href="/downloads.html">Download</a> · <a href="/faq">FAQ</a></p></div><div class="text-center"><a title="Join the user group" href="https://groups.google.com/forum/#!forum/druid-user" target="_blank"><span class="fa fa-comments"></span></a> · <a title="Follow Druid" href="https://twitter.com/druidio" target="_blank"><span class="fab fa-twitter"></span></a> · <a title="Download via Apache" href="https://www.apache.org/dyn/closer.cgi?path=/incubator/druid/{{ site.druid_versions[0].versions[0].version }}/apache-druid-{{ site.druid_versions[0].versions[0].version }}-bin.tar.gz" target="_blank"><span class="fas fa-feather"></span></a> · <a title="GitHub" href="https://github.com/apache/druid" target="_blank"><span class="fab fa-github"></span></a></div><div class="text-center license">Copyright © 2019 <a href="https://www.apache.org/" target="_blank">Apache Software Foundation</a>.<br/>Except where otherwise noted, licensed under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>.<br/>Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</div></div></footer></div><script type="text/javascript" src="https://cdn.jsdelivr.net/docsearch.js/1/docsearch.min.js"></script><script>
document.addEventListener('keyup', function(e) {
if (e.target !== document.body) {
return;
}
// keyCode for '/' (slash)
if (e.keyCode === 191) {
const search = document.getElementById('search_input_react');
search && search.focus();
}
});
</script><script>
var search = docsearch({
apiKey: '2de99082a9f38e49dfaa059bbe4c901d',
indexName: 'apache_druid',
inputSelector: '#search_input_react',
algoliaOptions: {"facetFilters":["language:en","version:0.22.1"]}
});
</script></body></html>