| <!doctype html> |
| <html lang="en" dir="ltr" class="docs-wrapper docs-doc-page docs-version-current plugin-docs plugin-id-default docs-doc-id-ingestion/faq"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="generator" content="Docusaurus v2.4.1"> |
| <title data-rh="true">Ingestion troubleshooting FAQ | Apache® Druid</title><meta data-rh="true" name="viewport" content="width=device-width,initial-scale=1"><meta data-rh="true" name="twitter:card" content="summary_large_image"><meta data-rh="true" property="og:image" content="https://druid.apache.org/img/druid_nav.png"><meta data-rh="true" name="twitter:image" content="https://druid.apache.org/img/druid_nav.png"><meta data-rh="true" property="og:url" content="https://druid.apache.org/docs/latest/ingestion/faq"><meta data-rh="true" name="docusaurus_locale" content="en"><meta data-rh="true" name="docsearch:language" content="en"><meta data-rh="true" name="docusaurus_version" content="current"><meta data-rh="true" name="docusaurus_tag" content="docs-default-current"><meta data-rh="true" name="docsearch:version" content="current"><meta data-rh="true" name="docsearch:docusaurus_tag" content="docs-default-current"><meta data-rh="true" property="og:title" content="Ingestion troubleshooting FAQ | Apache® Druid"><meta data-rh="true" name="description" content="<!--"><meta data-rh="true" property="og:description" content="<!--"><link data-rh="true" rel="icon" href="/img/favicon.png"><link data-rh="true" rel="canonical" href="https://druid.apache.org/docs/latest/ingestion/faq"><link data-rh="true" rel="alternate" href="https://druid.apache.org/docs/latest/ingestion/faq" hreflang="en"><link data-rh="true" rel="alternate" href="https://druid.apache.org/docs/latest/ingestion/faq" hreflang="x-default"><link rel="preconnect" href="https://www.google-analytics.com"> |
| <link rel="preconnect" href="https://www.googletagmanager.com"> |
| <script async src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script> |
| <script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag("js",new Date),gtag("config","UA-131010415-1",{})</script> |
| |
| |
| |
| |
| <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css"> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.4/clipboard.min.js"></script><link rel="stylesheet" href="/assets/css/styles.f80751b3.css"> |
| <link rel="preload" href="/assets/js/runtime~main.dc5f839a.js" as="script"> |
| <link rel="preload" href="/assets/js/main.a03dfc13.js" as="script"> |
| </head> |
| <body class="navigation-with-keyboard"> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#__docusaurus_skipToContent_fallback">Skip to main content</a></div><nav aria-label="Main" class="navbar navbar--fixed-top navbar--dark"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/druid_nav.png" alt="Apache® Druid" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/druid_nav.png" alt="Apache® Druid" class="themedImage_ToTc themedImage--dark_i4oU"></div></a></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link" href="/technology">Technology</a><a class="navbar__item navbar__link" href="/use-cases">Use Cases</a><a class="navbar__item navbar__link" href="/druid-powered">Powered By</a><a class="navbar__item navbar__link" href="/docs/latest/design/">Docs</a><a class="navbar__item navbar__link" href="/community/">Community</a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Apache®</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://apachecon.com/?ref=druid.apache.org" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship<svg width="12" height="12" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><a class="navbar__item navbar__link" href="/downloads/">Download</a><div class="searchBox_ZlJk"><div class="navbar__search"><span aria-label="expand searchbar" role="button" class="search-icon" tabindex="0"></span><input type="search" id="search_input_react" placeholder="Loading..." aria-label="Search" class="navbar__search-input search-bar" disabled=""></div></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="__docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebarViewport_Xe31"><div class="sidebar_njMd"><nav aria-label="Docs sidebar" class="menu thin-scrollbar menu_SIkG"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/design/">Getting started</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/tutorials/tutorial-msq-extern">Tutorials</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/design/architecture">Design</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/docs/latest/ingestion/">Ingestion</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/ingestion/">Overview</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/latest/ingestion/data-formats">Ingestion concepts</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/latest/multi-stage-query/">SQL-based batch</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/latest/development/extensions-core/kafka-ingestion">Streaming</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" tabindex="0" href="/docs/latest/ingestion/native-batch">Classic batch</a></div></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/ingestion/ingestion-spec">Ingestion spec reference</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/latest/ingestion/schema-design">Schema design tips</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/latest/ingestion/faq">Troubleshooting FAQ</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/data-management/">Data management</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/querying/sql">Querying</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/api-reference/">API reference</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/configuration/">Configuration</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/operations/web-console">Operations</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/development/overview">Development</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/latest/misc/papers-and-talks">Misc</a></div></li></ul></nav></div></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_YNFT"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Ingestion</span><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Troubleshooting FAQ</span><meta itemprop="position" content="2"></li></ul></nav><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Ingestion troubleshooting FAQ</h1></header><h2 class="anchor anchorWithStickyNavbar_LWe7" id="batch-ingestion">Batch Ingestion<a href="#batch-ingestion" class="hash-link" aria-label="Direct link to Batch Ingestion" title="Direct link to Batch Ingestion"></a></h2><p>If you are trying to batch load historical data but no events are being loaded, make sure the interval of your ingestion spec actually encapsulates the interval of your data. Events outside this interval are dropped.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="druid-ingested-my-events-but-they-are-not-in-my-query-results">Druid ingested my events but they are not in my query results<a href="#druid-ingested-my-events-but-they-are-not-in-my-query-results" class="hash-link" aria-label="Direct link to Druid ingested my events but they are not in my query results" title="Direct link to Druid ingested my events but they are not in my query results"></a></h2><p>If the number of ingested events seem correct, make sure your query is correctly formed. If you included a <code>count</code> aggregator in your ingestion spec, you will need to query for the results of this aggregate with a <code>longSum</code> aggregator. Issuing a query with a count aggregator will count the number of Druid rows, which includes <a href="/docs/latest/design/">roll-up</a>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="where-do-my-druid-segments-end-up-after-ingestion">Where do my Druid segments end up after ingestion?<a href="#where-do-my-druid-segments-end-up-after-ingestion" class="hash-link" aria-label="Direct link to Where do my Druid segments end up after ingestion?" title="Direct link to Where do my Druid segments end up after ingestion?"></a></h2><p>Depending on what <code>druid.storage.type</code> is set to, Druid will upload segments to some <a href="/docs/latest/design/deep-storage">Deep Storage</a>. Local disk is used as the default deep storage.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="my-stream-ingest-is-not-handing-segments-off">My stream ingest is not handing segments off<a href="#my-stream-ingest-is-not-handing-segments-off" class="hash-link" aria-label="Direct link to My stream ingest is not handing segments off" title="Direct link to My stream ingest is not handing segments off"></a></h2><p>First, make sure there are no exceptions in the logs of the ingestion process. Also make sure that <code>druid.storage.type</code> is set to a deep storage that isn't <code>local</code> if you are running a distributed cluster.</p><p>Other common reasons that hand-off fails are as follows:</p><p>1) Druid is unable to write to the metadata storage. Make sure your configurations are correct.</p><p>2) Historical processes are out of capacity and cannot download any more segments. You'll see exceptions in the Coordinator logs if this occurs and the web console will show the Historicals are near capacity.</p><p>3) Segments are corrupt and cannot be downloaded. You'll see exceptions in your Historical processes if this occurs.</p><p>4) Deep storage is improperly configured. Make sure that your segment actually exists in deep storage and that the Coordinator logs have no errors.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="how-do-i-get-hdfs-to-work">How do I get HDFS to work?<a href="#how-do-i-get-hdfs-to-work" class="hash-link" aria-label="Direct link to How do I get HDFS to work?" title="Direct link to How do I get HDFS to work?"></a></h2><p>Make sure to include the <code>druid-hdfs-storage</code> and all the hadoop configuration, dependencies (that can be obtained by running command <code>hadoop classpath</code> on a machine where hadoop has been setup) in the classpath. And, provide necessary HDFS settings as described in <a href="/docs/latest/design/deep-storage">deep storage</a> .</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="how-do-i-know-when-i-can-make-query-to-druid-after-submitting-batch-ingestion-task">How do I know when I can make query to Druid after submitting batch ingestion task?<a href="#how-do-i-know-when-i-can-make-query-to-druid-after-submitting-batch-ingestion-task" class="hash-link" aria-label="Direct link to How do I know when I can make query to Druid after submitting batch ingestion task?" title="Direct link to How do I know when I can make query to Druid after submitting batch ingestion task?"></a></h2><p>You can verify if segments created by a recent ingestion task are loaded onto historicals and available for querying using the following workflow.</p><ol><li>Submit your ingestion task.</li><li>Repeatedly poll the <a href="/docs/latest/api-reference/tasks-api">Overlord's tasks API</a> ( <code>/druid/indexer/v1/task/{taskId}/status</code>) until your task is shown to be successfully completed.</li><li>Poll the <a href="/docs/latest/api-reference/legacy-metadata-api#segment-loading-by-datasource">Segment Loading by Datasource API</a> (<code>/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus</code>) with |
| <code>forceMetadataRefresh=true</code> and <code>interval=<INTERVAL_OF_INGESTED_DATA></code> once. |
| (Note: <code>forceMetadataRefresh=true</code> refreshes Coordinator's metadata cache of all datasources. This can be a heavy operation in terms of the load on the metadata store but is necessary to make sure that we verify all the latest segments' load status) |
| If there are segments not yet loaded, continue to step 4, otherwise you can now query the data.</li><li>Repeatedly poll the <a href="/docs/latest/api-reference/legacy-metadata-api#segment-loading-by-datasource">Segment Loading by Datasource API</a> (<code>/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus</code>) with |
| <code>forceMetadataRefresh=false</code> and <code>interval=<INTERVAL_OF_INGESTED_DATA></code>. |
| Continue polling until all segments are loaded. Once all segments are loaded you can now query the data. |
| Note that this workflow only guarantees that the segments are available at the time of the <a href="/docs/latest/api-reference/legacy-metadata-api#segment-loading-by-datasource">Segment Loading by Datasource API</a> call. Segments can still become missing because of historical process failures or any other reasons afterward.</li></ol><h2 class="anchor anchorWithStickyNavbar_LWe7" id="i-dont-see-my-druid-segments-on-my-historical-processes">I don't see my Druid segments on my Historical processes<a href="#i-dont-see-my-druid-segments-on-my-historical-processes" class="hash-link" aria-label="Direct link to I don't see my Druid segments on my Historical processes" title="Direct link to I don't see my Druid segments on my Historical processes"></a></h2><p>You can check the <a href="/docs/latest/operations/web-console">web console</a> to make sure that your segments have actually loaded on <a href="/docs/latest/design/historical">Historical processes</a>. If your segments are not present, check the Coordinator logs for messages about capacity of replication errors. One reason that segments are not downloaded is because Historical processes have maxSizes that are too small, making them incapable of downloading more data. You can change that with (for example):</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#bfc7d5"><span class="token plain">-Ddruid.segmentCache.locations=[{"path":"/tmp/druid/storageLocation","maxSize":"500000000000"}]</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg viewBox="0 0 24 24" class="copyButtonIcon_y97N"><path fill="currentColor" d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg viewBox="0 0 24 24" class="copyButtonSuccessIcon_LjdS"><path fill="currentColor" d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="my-queries-are-returning-empty-results">My queries are returning empty results<a href="#my-queries-are-returning-empty-results" class="hash-link" aria-label="Direct link to My queries are returning empty results" title="Direct link to My queries are returning empty results"></a></h2><p>You can use a <a href="/docs/latest/querying/segmentmetadataquery">segment metadata query</a> for the dimensions and metrics that have been created for your datasource. Make sure that the name of the aggregators you use in your query match one of these metrics. Also make sure that the query interval you specify match a valid time range where data exists.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="real-time-ingestion-seems-to-be-stuck">Real-time ingestion seems to be stuck<a href="#real-time-ingestion-seems-to-be-stuck" class="hash-link" aria-label="Direct link to Real-time ingestion seems to be stuck" title="Direct link to Real-time ingestion seems to be stuck"></a></h2><p>There are a few ways this can occur. Druid will throttle ingestion to prevent out of memory problems if the intermediate persists are taking too long or if hand-off is taking too long. If your process logs indicate certain columns are taking a very long time to build (for example, if your segment granularity is hourly, but creating a single column takes 30 minutes), you should re-evaluate your configuration or scale up your real-time ingestion.</p></div></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages"><a class="pagination-nav__link pagination-nav__link--prev" href="/docs/latest/ingestion/schema-design"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Schema design tips</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/docs/latest/data-management/"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Overview</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_bqdL thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#batch-ingestion" class="table-of-contents__link toc-highlight">Batch Ingestion</a></li><li><a href="#druid-ingested-my-events-but-they-are-not-in-my-query-results" class="table-of-contents__link toc-highlight">Druid ingested my events but they are not in my query results</a></li><li><a href="#where-do-my-druid-segments-end-up-after-ingestion" class="table-of-contents__link toc-highlight">Where do my Druid segments end up after ingestion?</a></li><li><a href="#my-stream-ingest-is-not-handing-segments-off" class="table-of-contents__link toc-highlight">My stream ingest is not handing segments off</a></li><li><a href="#how-do-i-get-hdfs-to-work" class="table-of-contents__link toc-highlight">How do I get HDFS to work?</a></li><li><a href="#how-do-i-know-when-i-can-make-query-to-druid-after-submitting-batch-ingestion-task" class="table-of-contents__link toc-highlight">How do I know when I can make query to Druid after submitting batch ingestion task?</a></li><li><a href="#i-dont-see-my-druid-segments-on-my-historical-processes" class="table-of-contents__link toc-highlight">I don't see my Druid segments on my Historical processes</a></li><li><a href="#my-queries-are-returning-empty-results" class="table-of-contents__link toc-highlight">My queries are returning empty results</a></li><li><a href="#real-time-ingestion-seems-to-be-stuck" class="table-of-contents__link toc-highlight">Real-time ingestion seems to be stuck</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="footer__bottom text--center"><div class="margin-bottom--sm"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--light_HNdA footer__logo"><img src="/img/favicon.png" class="themedImage_ToTc themedImage--dark_i4oU footer__logo"></div><div class="footer__copyright">Copyright © 2023 Apache Software Foundation. Except where otherwise noted, licensed under CC BY-SA 4.0. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.dc5f839a.js"></script> |
| <script src="/assets/js/main.a03dfc13.js"></script> |
| </body> |
| </html> |
