| <!doctype html> |
| <html lang="en" dir="ltr" class="docs-wrapper docs-doc-page docs-version-current plugin-docs plugin-id-default docs-doc-id-design/segments"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="generator" content="Docusaurus v2.4.1"> |
| <title data-rh="true">Segments | Apache® Druid</title><meta data-rh="true" name="viewport" content="width=device-width,initial-scale=1"><meta data-rh="true" name="twitter:card" content="summary_large_image"><meta data-rh="true" property="og:image" content="https://druid.apache.org/img/druid_nav.png"><meta data-rh="true" name="twitter:image" content="https://druid.apache.org/img/druid_nav.png"><meta data-rh="true" property="og:url" content="https://druid.apache.org/docs/latest/design/segments"><meta data-rh="true" name="docusaurus_locale" content="en"><meta data-rh="true" name="docsearch:language" content="en"><meta data-rh="true" name="docusaurus_version" content="current"><meta data-rh="true" name="docusaurus_tag" content="docs-default-current"><meta data-rh="true" name="docsearch:version" content="current"><meta data-rh="true" name="docsearch:docusaurus_tag" content="docs-default-current"><meta data-rh="true" property="og:title" content="Segments | Apache® Druid"><meta data-rh="true" name="description" content="<!--"><meta data-rh="true" property="og:description" content="<!--"><link data-rh="true" rel="icon" href="/img/favicon.png"><link data-rh="true" rel="canonical" href="https://druid.apache.org/docs/latest/design/segments"><link data-rh="true" rel="alternate" href="https://druid.apache.org/docs/latest/design/segments" hreflang="en"><link data-rh="true" rel="alternate" href="https://druid.apache.org/docs/latest/design/segments" hreflang="x-default"><link rel="preconnect" href="https://www.google-analytics.com"> |
| <link rel="preconnect" href="https://www.googletagmanager.com"> |
| <script async src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script> |
| <script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag("js",new Date),gtag("config","UA-131010415-1",{})</script> |
| |
| |
| |
| |
| <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css"> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.4/clipboard.min.js"></script><link rel="stylesheet" href="/assets/css/styles.546f39eb.css"> |
| <link rel="preload" href="/assets/js/runtime~main.26d714fb.js" as="script"> |
| <link rel="preload" href="/assets/js/main.bd54ee66.js" as="script"> |
| </head> |
| <body class="navigation-with-keyboard"> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#__docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top navbar--dark"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/druid_nav.png" alt="Apache® Druid" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/druid_nav.png" alt="Apache® Druid" class="themedImage_ToTc themedImage--dark_i4oU"></div></a></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link" href="/technology">Technology</a><a class="navbar__item navbar__link" href="/use-cases">Use Cases</a><a class="navbar__item navbar__link" href="/druid-powered">Powered By</a><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/docs/latest/design/">Docs</a><a class="navbar__item navbar__link" href="/community/">Community</a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Apache®</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://apachecon.com/?ref=druid.apache.org" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><a class="navbar__item navbar__link" href="/downloads/">Download</a><div class="searchBox_ZlJk"><div class="navbar__search"><span aria-label="expand searchbar" role="button" class="search-icon" tabindex="0"></span><input type="search" id="search_input_react" placeholder="Loading..." aria-label="Search" class="navbar__search-input search-bar" disabled=""></div></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="__docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebarViewport_Xe31"><div class="sidebar_njMd"><nav aria-label="Docs sidebar" class="menu thin-scrollbar menu_SIkG"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/design/">Getting started</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/tutorials/tutorial-msq-extern">Tutorials</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/docs/latest/design/architecture">Design</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/design/architecture">Design</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/latest/design/segments">Segments</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/design/processes">Processes and servers</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/design/deep-storage">Deep storage</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/design/metadata-storage">Metadata storage</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/design/zookeeper">ZooKeeper</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/ingestion/">Ingestion</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/data-management/">Data management</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/querying/sql">Querying</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/api-reference/">API reference</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/configuration/">Configuration</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/operations/web-console">Operations</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/development/overview">Development</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/misc/papers-and-talks">Misc</a></div></li></ul></nav></div></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_YNFT"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Design</span><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Segments</span><meta itemprop="position" content="2"></li></ul></nav><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Segments</h1></header><p>Apache Druid stores its data and indexes in <em>segment files</em> partitioned by time. Druid creates a segment for each segment interval that contains data. If an interval is empty—that is, containing no rows—no segment exists for that time interval. Druid may create multiple segments for the same interval if you ingest data for that period via different ingestion jobs. <a href="/docs/latest/data-management/compaction">Compaction</a> is the Druid process that attempts to combine these segments into a single segment per interval for optimal performance.</p><p>The time interval is configurable in the <code>segmentGranularity</code> parameter of the <a href="/docs/latest/ingestion/ingestion-spec#granularityspec"><code>granularitySpec</code></a>.</p><p>For Druid to operate well under heavy query load, it is important for the segment |
| file size to be within the recommended range of 300-700 MB. If your |
| segment files are larger than this range, then consider either |
| changing the granularity of the segment time interval or partitioning your |
| data and/or adjusting the <code>targetRowsPerSegment</code> in your <code>partitionsSpec</code>. |
| A good starting point for this parameter is 5 million rows. |
| See the Sharding section below and the "Partitioning specification" section of |
| the <a href="/docs/latest/ingestion/hadoop#partitionsspec">Batch ingestion</a> documentation |
| for more guidance.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="segment-file-structure">Segment file structure<a href="#segment-file-structure" class="hash-link" aria-label="Direct link to Segment file structure" title="Direct link to Segment file structure"></a></h2><p>Segment files are <em>columnar</em>: the data for each column is laid out in |
| separate data structures. By storing each column separately, Druid decreases query latency by scanning only those columns actually needed for a query. There are three basic column types: timestamp, dimensions, and metrics:</p><p><img loading="lazy" alt="Druid column types" src="/assets/images/druid-column-types-0cd9a011b4cc2b4d05f93b3b31a7e735.png" title="Druid Column Types" width="1936" height="400" class="img_ev3q"></p><p>Timestamp and metrics type columns are arrays of integer or floating point values compressed with |
| <a href="https://github.com/lz4/lz4-java" target="_blank" rel="noopener noreferrer">LZ4</a>. Once a query identifies which rows to select, it decompresses them, pulls out the relevant rows, and applies the |
| desired aggregation operator. If a query doesn’t require a column, Druid skips over that column's data.</p><p>Dimension columns are different because they support filter and |
| group-by operations, so each dimension requires the following |
| three data structures:</p><ul><li><strong>Dictionary</strong>: Maps values (which are always treated as strings) to integer IDs, allowing compact representation of the list and bitmap values.</li><li><strong>List</strong>: The column’s values, encoded using the dictionary. Required for GroupBy and TopN queries. These operators allow queries that solely aggregate metrics based on filters to run without accessing the list of values.</li><li><strong>Bitmap</strong>: One bitmap for each distinct value in the column, to indicate which rows contain that value. Bitmaps allow for quick filtering operations because they are convenient for quickly applying AND and OR operators. Also known as inverted indexes.</li></ul><p>To get a better sense of these data structures, consider the "Page" column from the example data above, represented by the following data structures:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">1: Dictionary</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> {</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "Justin Bieber": 0,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "Ke$ha": 1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">2: List of column data</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> [0,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> 0,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> 1,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> 1]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">3: Bitmaps</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> value="Justin Bieber": [1,1,0,0]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> value="Ke$ha": [0,0,1,1]</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Note that the bitmap is different from the dictionary and list data structures: the dictionary and list grow linearly with the size of the data, but the size of the bitmap section is the product of data size and column cardinality. That is, there is one bitmap per separate column value. Columns with the same value share the same bitmap.</p><p>For each row in the list of column data, there is only a single bitmap that has a non-zero entry. This means that high cardinality columns have extremely sparse, and therefore highly compressible, bitmaps. Druid exploits this using compression algorithms that are specially suited for bitmaps, such as <a href="https://github.com/RoaringBitmap/RoaringBitmap" target="_blank" rel="noopener noreferrer">Roaring bitmap compression</a>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="handling-null-values">Handling null values<a href="#handling-null-values" class="hash-link" aria-label="Direct link to Handling null values" title="Direct link to Handling null values"></a></h2><p>By default, Druid string dimension columns use the values <code>''</code> and <code>null</code> interchangeably. Numeric and metric columns cannot represent <code>null</code> but use nulls to mean <code>0</code>. However, Druid provides a SQL compatible null handling mode, which you can enable at the system level through <code>druid.generic.useDefaultValueForNull</code>. This setting, when set to <code>false</code>, allows Druid to create segments <em>at ingestion time</em> in which the following occurs:</p><ul><li>String columns can distinguish <code>''</code> from <code>null</code>,</li><li>Numeric columns can represent <code>null</code> valued rows instead of <code>0</code>.</li></ul><p>String dimension columns contain no additional column structures in SQL compatible null handling mode. Instead, they reserve an additional dictionary entry for the <code>null</code> value. Numeric columns are stored in the segment with an additional bitmap in which the set bits indicate <code>null</code>-valued rows. </p><p>In addition to slightly increased segment sizes, SQL compatible null handling can incur a performance cost at query time, due to the need to check the null bitmap. This performance cost only occurs for columns that actually contain null values.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="segments-with-different-schemas">Segments with different schemas<a href="#segments-with-different-schemas" class="hash-link" aria-label="Direct link to Segments with different schemas" title="Direct link to Segments with different schemas"></a></h2><p>Druid segments for the same datasource may have different schemas. If a string column (dimension) exists in one segment but not another, queries that involve both segments still work. In default mode, queries for the segment without the dimension behave as if the dimension contains only blank values. In SQL-compatible mode, queries for the segment without the dimension behave as if the dimension contains only null values. Similarly, if one segment has a numeric column (metric) but another does not, queries on the segment without the metric generally operate as expected. Aggregations over the missing metric operate as if the metric doesn't exist.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="column-format">Column format<a href="#column-format" class="hash-link" aria-label="Direct link to Column format" title="Direct link to Column format"></a></h2><p>Each column is stored as two parts:</p><ul><li>A Jackson-serialized <code>ColumnDescriptor</code>.</li><li>The binary data for the column.</li></ul><p>A <code>ColumnDescriptor</code> is Jackson-serialized instance of the internal Druid <code>ColumnDescriptor</code> class . It allows the use of Jackson's polymorphic deserialization to add new and interesting methods of serialization with minimal impact to the code. It consists of some metadata about the column (for example: type, whether it's multi-value) and a list of serialization/deserialization logic that can deserialize the rest of the binary.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="multi-value-columns">Multi-value columns<a href="#multi-value-columns" class="hash-link" aria-label="Direct link to Multi-value columns" title="Direct link to Multi-value columns"></a></h3><p>A multi-value column allows a single row to contain multiple strings for a column. You can think of it as an array of strings. If a datasource uses multi-value columns, then the data structures within the segment files look a bit different. Let's imagine that in the example above, the second row is tagged with both the <code>Ke$ha</code> <em>and</em> <code>Justin Bieber</code> topics, as follows:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">1: Dictionary</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> {</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "Justin Bieber": 0,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> "Ke$ha": 1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">2: List of column data</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> [0,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> [0,1], <--Row value in a multi-value column can contain an array of values</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> 1,</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> 1]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">3: Bitmaps</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> value="Justin Bieber": [1,1,0,0]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> value="Ke$ha": [0,1,1,1]</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> ^</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> |</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain"> Multi-value column contains multiple non-zero entries</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Note the changes to the second row in the list of column data and the <code>Ke$ha</code> |
| bitmap. If a row has more than one value for a column, its entry in |
| the list is an array of values. Additionally, a row with <em>n</em> values in the list has <em>n</em> non-zero valued entries in bitmaps.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="compression">Compression<a href="#compression" class="hash-link" aria-label="Direct link to Compression" title="Direct link to Compression"></a></h2><p>Druid uses LZ4 by default to compress blocks of values for string, long, float, and double columns. Druid uses Roaring to compress bitmaps for string columns and numeric null values. We recommend that you use these defaults unless you've experimented with your data and query patterns suggest that non-default options will perform better in your specific case. </p><p>Druid also supports Concise bitmap compression. For string column bitmaps, the differences between using Roaring and Concise are most pronounced for high cardinality columns. In this case, Roaring is substantially faster on filters that match many values, but in some cases Concise can have a lower footprint due to the overhead of the Roaring format (but is still slower when many values are matched). You configure compression at the segment level, not for individual columns. See <a href="/docs/latest/ingestion/ingestion-spec#indexspec">IndexSpec</a> for more details.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="segment-identification">Segment identification<a href="#segment-identification" class="hash-link" aria-label="Direct link to Segment identification" title="Direct link to Segment identification"></a></h2><p>Segment identifiers typically contain the segment datasource, interval start time (in ISO 8601 format), interval end time (in ISO 8601 format), and version information. If data is additionally sharded beyond a time range, the segment identifier also contains a partition number:</p><p><code>datasource_intervalStart_intervalEnd_version_partitionNum</code></p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="segment-id-examples">Segment ID examples<a href="#segment-id-examples" class="hash-link" aria-label="Direct link to Segment ID examples" title="Direct link to Segment ID examples"></a></h3><p>The increasing partition numbers in the following segments indicate that multiple segments exist for the same interval:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v1_0</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v1_1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v1_2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>If you reindex the data with a new schema, Druid allocates a new version ID to the newly created segments:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v2_0</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v2_1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v2_2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="sharding">Sharding<a href="#sharding" class="hash-link" aria-label="Direct link to Sharding" title="Direct link to Sharding"></a></h2><p>Multiple segments can exist for a single time interval and datasource. These segments form a <code>block</code> for an interval. Depending on the type of <code>shardSpec</code> used to shard the data, Druid queries may only complete if a <code>block</code> is complete. For example, if a block consists of the following three segments:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">sampleData_2011-01-01T02:00:00:00Z_2011-01-01T03:00:00:00Z_v1_0</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">sampleData_2011-01-01T02:00:00:00Z_2011-01-01T03:00:00:00Z_v1_1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">sampleData_2011-01-01T02:00:00:00Z_2011-01-01T03:00:00:00Z_v1_2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>All three segments must load before a query for the interval <code>2011-01-01T02:00:00:00Z_2011-01-01T03:00:00:00Z</code> can complete.</p><p>Linear shard specs are an exception to this rule. Linear shard specs do not enforce "completeness" so queries can complete even if shards are not completely loaded.</p><p>For example, if a real-time ingestion creates three segments that were sharded with linear shard spec, and only two of the segments are loaded, queries return results for those two segments.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="segment-components">Segment components<a href="#segment-components" class="hash-link" aria-label="Direct link to Segment components" title="Direct link to Segment components"></a></h2><p>A segment contains several files:</p><ul><li><p><code>version.bin</code></p><p> 4 bytes representing the current segment version as an integer. For example, for v9 segments the version is 0x0, 0x0, 0x0, 0x9.</p></li><li><p><code>meta.smoosh</code></p><p> A file containing metadata (filenames and offsets) about the contents of the other <code>smoosh</code> files.</p></li><li><p><code>XXXXX.smoosh</code></p><p> Smoosh (<code>.smoosh</code>) files contain concatenated binary data. This file consolidation reduces the number of file descriptors that must be open when accessing data. The files are 2 GB or less in size to remain within the limit of a memory-mapped <code>ByteBuffer</code> in Java. |
| Smoosh files contain the following: </p><ul><li>Individual files for each column in the data, including one for the <code>__time</code> column that refers to the timestamp of the segment. </li><li>An <code>index.drd</code> file that contains additional segment metadata.</li></ul></li></ul><p>In the codebase, segments have an internal format version. The current segment format version is <code>v9</code>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="implications-of-updating-segments">Implications of updating segments<a href="#implications-of-updating-segments" class="hash-link" aria-label="Direct link to Implications of updating segments" title="Direct link to Implications of updating segments"></a></h2><p>Druid uses versioning to manage updates to create a form of multi-version concurrency control (MVCC). These MVCC versions are distinct from the segment format version discussed above.</p><p>Note that updates that span multiple segment intervals are only atomic within each interval. They are not atomic across the entire update. For example, if you have the following segments:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v1_0</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-02/2015-01-03_v1_1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-03/2015-01-04_v1_2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p><code>v2</code> segments are loaded into the cluster as soon as they are built and replace <code>v1</code> segments for the period of time the segments overlap. Before <code>v2</code> segments are completely loaded, the cluster may contain a mixture of <code>v1</code> and <code>v2</code> segments.</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-01/2015-01-02_v1_0</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-02/2015-01-03_v2_1</span><br></span><span class="token-line" style="color:#bfc7d5"><span class="token plain">foo_2015-01-03/2015-01-04_v1_2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In this case, queries may hit a mixture of <code>v1</code> and <code>v2</code> segments.</p></div></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages"><a class="pagination-nav__link pagination-nav__link--prev" href="/docs/latest/design/architecture"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Design</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/docs/latest/design/processes"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Processes and servers</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_bqdL thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#segment-file-structure" class="table-of-contents__link toc-highlight">Segment file structure</a></li><li><a href="#handling-null-values" class="table-of-contents__link toc-highlight">Handling null values</a></li><li><a href="#segments-with-different-schemas" class="table-of-contents__link toc-highlight">Segments with different schemas</a></li><li><a href="#column-format" class="table-of-contents__link toc-highlight">Column format</a><ul><li><a href="#multi-value-columns" class="table-of-contents__link toc-highlight">Multi-value columns</a></li></ul></li><li><a href="#compression" class="table-of-contents__link toc-highlight">Compression</a></li><li><a href="#segment-identification" class="table-of-contents__link toc-highlight">Segment identification</a><ul><li><a href="#segment-id-examples" class="table-of-contents__link toc-highlight">Segment ID examples</a></li></ul></li><li><a href="#sharding" class="table-of-contents__link toc-highlight">Sharding</a></li><li><a href="#segment-components" class="table-of-contents__link toc-highlight">Segment components</a></li><li><a href="#implications-of-updating-segments" class="table-of-contents__link toc-highlight">Implications of updating segments</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="footer__bottom text--center"><div class="margin-bottom--sm"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--light_HNdA footer__logo"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--dark_i4oU footer__logo"></div><div class="footer__copyright">Copyright © 2023 Apache Software Foundation. Except where otherwise noted, licensed under CC BY-SA 4.0. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.26d714fb.js"></script> |
| <script src="/assets/js/main.bd54ee66.js"></script> |
| </body> |
| </html> |
