| </span></code></pre></div><p>Navigate to <a target="_blank" rel="noopener noreferrer" href="http://localhost:9000/#/query">http://localhost:9000/#/query</a> and click on the events table. You should see the following:</p><p><img alt="Sample events table containing records" src="https://www.datocms-assets.com/75153/1669668611-image2.png" title="Sample events table containing records"/></p><p>We have records! We can also run our data generator again, and more events will be ingested.</p><h2 id="summary"><a href="#summary" aria-hidden="true" tabindex="-1"><span class="icon icon-link"></span></a>Summary</h2><p>This feature makes real-time data ingestion a bit more forgiving when things go wrong, which has got to be a good thing in my book.</p><p>When you look at the name of this feature, it can seem a bit esoteric and perhaps not something that you’d want to use, but I think you’ll find it to be extremely useful.</p><p>So give it a try and let us know how you get on. If you have any questions about this feature, feel free to join us on <a target="_blank" rel="noopener noreferrer" href="https://stree.ai/slack">Slack</a>, where we’ll be happy to help you out.</p></div></main></div></div><aside class="mt-10 hidden border-l-2 pl-5 lg:sticky lg:top-1 lg:block lg:h-full"><section class="sticky top-0 mb-4 w-[15.375rem]"><div class="flex flex-col space-y-1.5 pb-3"><h3 class="text-sm font-semibold leading-snug text-neutral-500 dark:text-neutral-100">Table of Contents</h3></div><nav class="flex items-center self-start" aria-label="Table of Contents"><ol class="list-none space-y-3"><li class="text-sm leading-tight font-bold"><a href="#how-does-real-time-ingestion-work">How does real-time ingestion work?</a></li><li class="text-sm font-normal leading-tight"><a href="#why-do-we-need-to-pause-and-resume-ingestion">Why do we need to pause and resume ingestion?</a></li><li class="text-sm font-normal leading-tight"><a href="#data-generation">Data Generation</a></li><li class="text-sm font-normal leading-tight"><a href="#pinot-schematable-config">Pinot Schema/Table Config</a></li><li class="text-sm font-normal leading-tight"><a href="#the-pauseresume-flow">The Pause/Resume Flow</a></li><li class="text-sm font-normal leading-tight"><a href="#summary">Summary</a></li></ol></nav></section></aside></div></article></section></main><footer class="border-t bg-sky-100 px-5 py-10 md:px-[6.75rem] md:pb-10 md:pt-16"><div class="mx-auto flex max-w-7xl flex-wrap justify-between"><div class="flex-shrink-0"><svg xmlns="http://www.w3.org/2000/svg" width="120" height="48" fill="none"><g fill="#C7154A" clip-path="url(#logo_svg__a)"><path d="M42.99 18.448c1.032-.553 2.21-.831 3.535-.831 1.542 0 2.938.38 4.187 1.14 1.248.76 2.236 1.841 2.965 3.241.728 1.402 1.091 3.025 1.091 4.872s-.363 3.482-1.091 4.903c-.729 1.424-1.717 2.525-2.965 3.307-1.25.782-2.645 1.173-4.187 1.173-1.325 0-2.493-.271-3.503-.815-1.01-.543-1.83-1.226-2.46-2.053v14.612H36V17.912h4.562v2.606c.586-.825 1.395-1.515 2.426-2.068l.002-.002m6.452 5.605c-.445-.793-1.032-1.395-1.76-1.808a4.72 4.72 0 0 0-2.362-.618c-.847 0-1.602.211-2.33.635-.728.423-1.315 1.038-1.76 1.841-.445.804-.668 1.749-.668 2.835 0 1.087.221 2.032.668 2.835.445.804 1.032 1.417 1.76 1.842a4.557 4.557 0 0 0 2.33.635 4.57 4.57 0 0 0 2.362-.652c.728-.435 1.313-1.053 1.76-1.856.445-.804.668-1.76.668-2.867s-.223-2.025-.668-2.818v-.004M62.947 17.912v18.051h-4.562V17.912h4.562m.551-6.079a2.833 2.833 0 1 1-5.666 0 2.833 2.833 0 0 1 5.666 0M82.954 19.687c1.325 1.358 1.988 3.253 1.988 5.685v10.59H80.38v-9.97c0-1.434-.358-2.537-1.075-3.307-.717-.772-1.695-1.157-2.933-1.157-1.239 0-2.254.387-2.982 1.157-.728.772-1.091 1.873-1.091 3.307v9.97h-4.562V17.91h4.562v2.248a6.322 6.322 0 0 1 2.33-1.841c.944-.445 1.981-.669 3.111-.669 2.15 0 3.889.68 5.214 2.037v.002M92.892 35.098c-1.39-.77-2.482-1.861-3.275-3.275-.794-1.411-1.19-3.041-1.19-4.888s.406-3.475 1.221-4.888a8.502 8.502 0 0 1 3.34-3.275c1.412-.772 2.987-1.157 4.725-1.157 1.739 0 3.312.387 4.725 1.157a8.5 8.5 0 0 1 3.34 3.275c.815 1.411 1.222 3.041 1.222 4.888s-.418 3.475-1.255 4.888a8.708 8.708 0 0 1-3.388 3.275c-1.424.772-3.014 1.157-4.774 1.157-1.76 0-3.301-.385-4.691-1.157m7.021-3.421c.729-.402 1.309-1.005 1.744-1.809.435-.803.651-1.781.651-2.933 0-1.715-.451-3.035-1.351-3.958-.902-.924-2.004-1.385-3.307-1.385s-2.395.461-3.275 1.385c-.88.923-1.32 2.243-1.32 3.958 0 1.715.428 3.035 1.287 3.958.858.924 1.938 1.385 3.241 1.385.825 0 1.602-.2 2.33-.603v.002M115.96 21.658v8.734c0 .608.147 1.048.44 1.32.293.271.787.406 1.482.406H120v3.845h-2.867c-3.845 0-5.766-1.868-5.766-5.605v-8.7h-2.15v-3.746h2.15V13l4.595-1v5.912h4.04v3.746h-4.042M20.03 46.757l-5.538-1.385A1.97 1.97 0 0 1 13 43.46v-5.462c0-.841.349-1.601.907-2.146a12.212 12.212 0 0 0 6.975-3.644c2.602-2.731 3.627-6.578 2.882-10.251L21 9h-4V4a1 1 0 0 0-2 0v7a1 1 0 0 1-2 0v-1a1 1 0 0 0-2 0v6.758a4.489 4.489 0 0 1 2.694-.755c2.278.095 4.156 1.934 4.297 4.21a4.501 4.501 0 0 1-6.992 4.029V29a1 1 0 0 1-2 0V7a1 1 0 0 0-2 0v2h-4L.237 21.957c-.745 3.675.279 7.52 2.882 10.251a12.202 12.202 0 0 0 6.975 3.644c.558.545.907 1.305.907 2.146V43.4c0 .938-.639 1.757-1.55 1.985l-5.48 1.37c-.57.143-.97.655-.97 1.243h18c0-.588-.4-1.1-.97-1.243v.002"></path><path d="M13.5 23a2.5 2.5 0 1 0 0-5 2.5 2.5 0 0 0 0 5M8 5a1 1 0 1 0 0-2 1 1 0 0 0 0 2M12 8a1 1 0 1 0 0-2 1 1 0 0 0 0 2M16 2a1 1 0 1 0 0-2 1 1 0 0 0 0 2"></path></g><defs><clipPath id="logo_svg__a"><path fill="#fff" d="M0 0h120v48H0z"></path></clipPath></defs></svg></div><div class="flex flex-wrap gap-x-16 gap-y-5 py-8 md:pl-24 md:pr-[21.625rem]"> <div><h5 class="mb-4 text-lg font-semibold">Resources</h5><div class="flex justify-between gap-x-10"><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/" class="block py-1 text-gray-600 hover:text-gray-900">Docs</a><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/getting-started" class="block py-1 text-gray-600 hover:text-gray-900">Getting Started</a><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/integrations/thirdeye" class="block py-1 text-gray-600 hover:text-gray-900">ThirdEye</a></div><div class="flex flex-col"><a class="block py-1 text-gray-600 hover:text-gray-900" href="/powered-by">Company Stories</a><a class="block py-1 text-gray-600 hover:text-gray-900" href="/download">Download</a><a class="block py-1 text-gray-600 hover:text-gray-900" href="/blog">Blog</a></div></div></div><div><h5 class="mb-4 text-lg font-semibold">Apache</h5><div class="flex justify-between gap-x-10"><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org" class="block py-1 text-gray-600 hover:text-gray-900">Foundation</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/licenses" class="block py-1 text-gray-600 hover:text-gray-900">License</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/security" class="block py-1 text-gray-600 hover:text-gray-900">Security</a></div><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/foundation/sponsorship.html" class="block py-1 text-gray-600 hover:text-gray-900">Sponsorship</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/events/current-event" class="block py-1 text-gray-600 hover:text-gray-900">Events</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/foundation/thanks.html" class="block py-1 text-gray-600 hover:text-gray-900">Thanks</a></div></div></div></div><div class="mt-4 flex justify-center md:mt-0"><a target="_blank" rel="noopener noreferrer" href="https://join.slack.com/t/apache-pinot/shared_invite/zt-5z7pav2f-yYtjZdVA~EDmrGkho87Vzw" class="mr-4"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-slack fill-gray-900"><rect width="3" height="8" x="13" y="2" rx="1.5"></rect><path d="M19 8.5V10h1.5A1.5 1.5 0 1 0 19 8.5"></path><rect width="3" height="8" x="8" y="14" rx="1.5"></rect><path d="M5 15.5V14H3.5A1.5 1.5 0 1 0 5 15.5"></path><rect width="8" height="3" x="14" y="13" rx="1.5"></rect><path d="M15.5 19H14v1.5a1.5 1.5 0 1 0 1.5-1.5"></path><rect width="8" height="3" x="2" y="8" rx="1.5"></rect><path d="M8.5 5H10V3.5A1.5 1.5 0 1 0 8.5 5"></path></svg></a><a target="_blank" rel="noopener noreferrer" href="https://github.com/apache/pinot"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" fill="currentColor" size="24"><g clip-path="url(#github_svg__a)"><path fill-rule="evenodd" d="M12.01 0C5.369 0 0 5.5 0 12.304c0 5.44 3.44 10.043 8.212 11.673.597.122.815-.265.815-.59 0-.286-.02-1.264-.02-2.283-3.34.734-4.036-1.466-4.036-1.466-.537-1.426-1.332-1.793-1.332-1.793-1.094-.754.08-.754.08-.754 1.212.082 1.849 1.263 1.849 1.263 1.073 1.874 2.803 1.345 3.5 1.019.098-.795.417-1.345.755-1.65-2.665-.285-5.468-1.345-5.468-6.07 0-1.345.477-2.445 1.232-3.3-.119-.306-.537-1.57.12-3.26 0 0 1.014-.326 3.3 1.263.98-.27 1.989-.407 3.003-.408 1.014 0 2.048.143 3.002.408 2.287-1.59 3.301-1.263 3.301-1.263.657 1.69.239 2.954.12 3.26.775.855 1.232 1.955 1.232 3.3 0 4.725-2.803 5.764-5.488 6.07.438.387.815 1.12.815 2.281 0 1.65-.02 2.975-.02 3.382 0 .326.22.713.816.59C20.56 22.347 24 17.744 24 12.305 24.02 5.5 18.63 0 12.01 0" clip-rule="evenodd"></path></g><defs><clipPath id="github_svg__a"><path fill="#fff" d="M0 0h24v24H0z"></path></clipPath></defs></svg></a></div></div><div class="mt-8 border-t border-neutral-300 pt-4 text-left text-sm text-gray-600">Copyright © <!-- -->2024<!-- --> The Apache Software Foundation. Apache Pinot, Pinot, Apache, the Apache feather logo, and the Apache Pinot project logo are registered trademarks of The Apache Software Foundation. This page has references to third party software - Presto, PrestoDB, ThirdEye, Trino, TrinoDB, that are not part of the Apache Software Foundation and are not covered under the Apache License.</div></footer></div><script src="/_next/static/chunks/webpack-dde39ac7c1b4eb4b.js" crossorigin="" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])</script><script>self.__next_f.push([1,"1:HL[\"/_next/static/media/6905431624c34d00-s.p.woff2\",\"font\",{\"crossOrigin\":\"\",\"type\":\"font/woff2\"}]\n2:HL[\"/_next/static/css/9e925a33b1acdac1.css\",\"style\",{\"crossOrigin\":\"\"}]\n0:\"$L3\"\n"])</script><script>self.__next_f.push([1,"4:HL[\"/_next/static/css/c130d1629644f070.css\",\"style\",{\"crossOrigin\":\"\"}]\n"])</script><script>self.__next_f.push([1,"5:I[3728,[],\"\"]\n7:I[9928,[],\"\"]\n8:I[7821,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"ThemeProviders\"]\n9:I[3994,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"\"]\na:I[9640,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js"])</script><script>self.__next_f.push([1,"\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"AlgoliaSearchProvider\"]\nb:I[7975,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"\"]\nc:I[6954,[],\"\"]\nd:I[7264,[],\"\"]\ne:I[8326,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\""])</script><script>self.__next_f.push([1,",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n11:T9fe,"])</script><script>self.__next_f.push([1,"M42.99 18.448c1.032-.553 2.21-.831 3.535-.831 1.542 0 2.938.38 4.187 1.14 1.248.76 2.236 1.841 2.965 3.241.728 1.402 1.091 3.025 1.091 4.872s-.363 3.482-1.091 4.903c-.729 1.424-1.717 2.525-2.965 3.307-1.25.782-2.645 1.173-4.187 1.173-1.325 0-2.493-.271-3.503-.815-1.01-.543-1.83-1.226-2.46-2.053v14.612H36V17.912h4.562v2.606c.586-.825 1.395-1.515 2.426-2.068l.002-.002m6.452 5.605c-.445-.793-1.032-1.395-1.76-1.808a4.72 4.72 0 0 0-2.362-.618c-.847 0-1.602.211-2.33.635-.728.423-1.315 1.038-1.76 1.841-.445.804-.668 1.749-.668 2.835 0 1.087.221 2.032.668 2.835.445.804 1.032 1.417 1.76 1.842a4.557 4.557 0 0 0 2.33.635 4.57 4.57 0 0 0 2.362-.652c.728-.435 1.313-1.053 1.76-1.856.445-.804.668-1.76.668-2.867s-.223-2.025-.668-2.818v-.004M62.947 17.912v18.051h-4.562V17.912h4.562m.551-6.079a2.833 2.833 0 1 1-5.666 0 2.833 2.833 0 0 1 5.666 0M82.954 19.687c1.325 1.358 1.988 3.253 1.988 5.685v10.59H80.38v-9.97c0-1.434-.358-2.537-1.075-3.307-.717-.772-1.695-1.157-2.933-1.157-1.239 0-2.254.387-2.982 1.157-.728.772-1.091 1.873-1.091 3.307v9.97h-4.562V17.91h4.562v2.248a6.322 6.322 0 0 1 2.33-1.841c.944-.445 1.981-.669 3.111-.669 2.15 0 3.889.68 5.214 2.037v.002M92.892 35.098c-1.39-.77-2.482-1.861-3.275-3.275-.794-1.411-1.19-3.041-1.19-4.888s.406-3.475 1.221-4.888a8.502 8.502 0 0 1 3.34-3.275c1.412-.772 2.987-1.157 4.725-1.157 1.739 0 3.312.387 4.725 1.157a8.5 8.5 0 0 1 3.34 3.275c.815 1.411 1.222 3.041 1.222 4.888s-.418 3.475-1.255 4.888a8.708 8.708 0 0 1-3.388 3.275c-1.424.772-3.014 1.157-4.774 1.157-1.76 0-3.301-.385-4.691-1.157m7.021-3.421c.729-.402 1.309-1.005 1.744-1.809.435-.803.651-1.781.651-2.933 0-1.715-.451-3.035-1.351-3.958-.902-.924-2.004-1.385-3.307-1.385s-2.395.461-3.275 1.385c-.88.923-1.32 2.243-1.32 3.958 0 1.715.428 3.035 1.287 3.958.858.924 1.938 1.385 3.241 1.385.825 0 1.602-.2 2.33-.603v.002M115.96 21.658v8.734c0 .608.147 1.048.44 1.32.293.271.787.406 1.482.406H120v3.845h-2.867c-3.845 0-5.766-1.868-5.766-5.605v-8.7h-2.15v-3.746h2.15V13l4.595-1v5.912h4.04v3.746h-4.042M20.03 46.757l-5.538-1.385A1.97 1.97 0 0 1 13 43.46v-5.462c0-.841.349-1.601.907-2.146a12.212 12.212 0 0 0 6.975-3.644c2.602-2.731 3.627-6.578 2.882-10.251L21 9h-4V4a1 1 0 0 0-2 0v7a1 1 0 0 1-2 0v-1a1 1 0 0 0-2 0v6.758a4.489 4.489 0 0 1 2.694-.755c2.278.095 4.156 1.934 4.297 4.21a4.501 4.501 0 0 1-6.992 4.029V29a1 1 0 0 1-2 0V7a1 1 0 0 0-2 0v2h-4L.237 21.957c-.745 3.675.279 7.52 2.882 10.251a12.202 12.202 0 0 0 6.975 3.644c.558.545.907 1.305.907 2.146V43.4c0 .938-.639 1.757-1.55 1.985l-5.48 1.37c-.57.143-.97.655-.97 1.243h18c0-.588-.4-1.1-.97-1.243v.002"])</script><script>self.__next_f.push([1,"3:[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/9e925a33b1acdac1.css\",\"precedence\":\"next\",\"crossOrigin\":\"\"}]],[\"$\",\"$L5\",null,{\"buildId\":\"rmcKjFZ3e9kKdH1iJwCIQ\",\"assetPrefix\":\"\",\"initialCanonicalUrl\":\"/blog/2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\",\"initialTree\":[\"\",{\"children\":[\"blog\",{\"children\":[[\"slug\",\"2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\",\"c\"],{\"children\":[\"__PAGE__?{\\\"slug\\\":[\\\"2022\\\",\\\"11\\\",\\\"28\\\",\\\"Apache-Pinot-Pausing-Real-Time-Ingestion\\\"]}\",{}]}]}]},\"$undefined\",\"$undefined\",true],\"initialHead\":[false,\"$L6\"],\"globalErrorComponent\":\"$7\",\"children\":[null,[\"$\",\"html\",null,{\"lang\":\"en-us\",\"className\":\"__variable_1fc36d scroll-smooth\",\"suppressHydrationWarning\":true,\"children\":[[\"$\",\"head\",null,{\"children\":[[\"$\",\"meta\",null,{\"httpEquiv\":\"Content-Security-Policy\",\"content\":\"default-src 'self';script-src 'self' 'unsafe-eval' 'unsafe-inline' giscus.app analytics.umami.is www.youtube.com www.googletagmanager.com www.google-analytics.com;style-src 'self' 'unsafe-inline';img-src * blob: data:;media-src *.s3.amazonaws.com;connect-src *;font-src 'self';frame-src www.youtube.com youtube.com giscus.app youtu.be https://www.youtube.com https://youtube.com;\"}],[\"$\",\"link\",null,{\"rel\":\"apple-touch-icon\",\"sizes\":\"76x76\",\"href\":\"/static/favicons/apple-touch-icon.png\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"32x32\",\"href\":\"/static/favicons/favicon-32x32.png\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"16x16\",\"href\":\"/static/favicons/favicon-16x16.png\"}],[\"$\",\"link\",null,{\"rel\":\"manifest\",\"href\":\"/static/favicons/site.webmanifest\"}],[\"$\",\"link\",null,{\"rel\":\"mask-icon\",\"href\":\"/static/favicons/safari-pinned-tab.svg\",\"color\":\"#5bbad5\"}],[\"$\",\"meta\",null,{\"name\":\"msapplication-TileColor\",\"content\":\"#000000\"}],[\"$\",\"meta\",null,{\"name\":\"theme-color\",\"media\":\"(prefers-color-scheme: light)\",\"content\":\"#fff\"}],[\"$\",\"meta\",null,{\"name\":\"theme-color\",\"media\":\"(prefers-color-scheme: dark)\",\"content\":\"#000\"}],[\"$\",\"link\",null,{\"rel\":\"alternate\",\"type\":\"application/rss+xml\",\"href\":\"/feed.xml\"}]]}],[\"$\",\"body\",null,{\"className\":\"bg-white text-black antialiased dark:bg-gray-950 dark:text-white\",\"children\":[\"$\",\"$L8\",null,{\"children\":[[\"$undefined\",\"$undefined\",\"$undefined\",\"$undefined\",[[\"$\",\"$L9\",null,{\"strategy\":\"afterInteractive\",\"src\":\"https://www.googletagmanager.com/gtag/js?id=G-ZXG79NJEBY\"}],[\"$\",\"$L9\",null,{\"strategy\":\"afterInteractive\",\"id\":\"ga-script\",\"children\":\"\\n window.dataLayer = window.dataLayer || [];\\n function gtag(){dataLayer.push(arguments);}\\n gtag('js', new Date());\\n gtag('config', 'G-ZXG79NJEBY');\\n \"}]]],[\"$\",\"div\",null,{\"className\":\"mx-auto flex max-w-screen-customDesktop flex-col justify-between font-sans\",\"children\":[\"$\",\"$La\",null,{\"algoliaConfig\":{\"appId\":\"CKRA00L2X9\",\"apiKey\":\"6531f8f7783a88d76629190843f1801e\",\"indexName\":\"prod_apache_pinot_docs\"},\"children\":[[\"$\",\"$Lb\",null,{}],[\"$\",\"main\",null,{\"children\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[\"$\",\"div\",null,{\"className\":\"flex flex-col items-start justify-start md:mt-24 md:flex-row md:items-center md:justify-center md:space-x-6\",\"children\":[[\"$\",\"div\",null,{\"className\":\"space-x-2 pb-8 pt-6 md:space-y-5\",\"children\":[\"$\",\"h1\",null,{\"className\":\"text-6xl font-extrabold leading-9 tracking-tight text-gray-900 dark:text-gray-100 md:border-r-2 md:px-6 md:text-8xl md:leading-14\",\"children\":\"404\"}]}],[\"$\",\"div\",null,{\"className\":\"max-w-md\",\"children\":[[\"$\",\"p\",null,{\"className\":\"mb-4 text-xl font-bold leading-normal md:text-2xl\",\"children\":\"Sorry we couldn't find this page.\"}],[\"$\",\"p\",null,{\"className\":\"mb-8\",\"children\":\"But dont worry, you can find plenty of other things on our homepage.\"}],[\"$\",\"$Le\",null,{\"href\":\"/\",\"className\":\"focus:shadow-outline-blue inline rounded-lg border border-transparent bg-blue-600 px-4 py-2 text-sm font-medium leading-5 text-white shadow transition-colors duration-150 hover:bg-blue-700 focus:outline-none dark:hover:bg-blue-500\",\"children\":\"Back to homepage\"}]]}]]}],\"notFoundStyles\":[],\"initialChildNode\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"initialChildNode\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\",[\"slug\",\"2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\",\"c\"],\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"initialChildNode\":[\"$Lf\",\"$L10\",null],\"childPropSegment\":\"__PAGE__?{\\\"slug\\\":[\\\"2022\\\",\\\"11\\\",\\\"28\\\",\\\"Apache-Pinot-Pausing-Real-Time-Ingestion\\\"]}\",\"styles\":[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/c130d1629644f070.css\",\"precedence\":\"next\",\"crossOrigin\":\"\"}]]}],\"childPropSegment\":[\"slug\",\"2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\",\"c\"],\"styles\":null}],\"childPropSegment\":\"blog\",\"styles\":null}]}],[\"$\",\"footer\",null,{\"className\":\"border-t bg-sky-100 px-5 py-10 md:px-[6.75rem] md:pb-10 md:pt-16\",\"children\":[[\"$\",\"div\",null,{\"className\":\"mx-auto flex max-w-7xl flex-wrap justify-between\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex-shrink-0\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":120,\"height\":48,\"fill\":\"none\",\"children\":[[\"$\",\"g\",null,{\"fill\":\"#C7154A\",\"clipPath\":\"url(#logo_svg__a)\",\"children\":[[\"$\",\"path\",null,{\"d\":\"$11\"}],[\"$\",\"path\",null,{\"d\":\"M13.5 23a2.5 2.5 0 1 0 0-5 2.5 2.5 0 0 0 0 5M8 5a1 1 0 1 0 0-2 1 1 0 0 0 0 2M12 8a1 1 0 1 0 0-2 1 1 0 0 0 0 2M16 2a1 1 0 1 0 0-2 1 1 0 0 0 0 2\"}]]}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"clipPath\",null,{\"id\":\"logo_svg__a\",\"children\":[\"$\",\"path\",null,{\"fill\":\"#fff\",\"d\":\"M0 0h120v48H0z\"}]}]}]]}]}],[\"$\",\"div\",null,{\"className\":\"flex flex-wrap gap-x-16 gap-y-5 py-8 md:pl-24 md:pr-[21.625rem]\",\"children\":[\" \",[[\"$\",\"div\",\"Resources\",{\"children\":[[\"$\",\"h5\",null,{\"className\":\"mb-4 text-lg font-semibold\",\"children\":\"Resources\"}],[\"$\",\"div\",null,{\"className\":\"flex justify-between gap-x-10\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Docs\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/getting-started\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Getting Started\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/integrations/thirdeye\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"ThirdEye\"}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"$Le\",null,{\"href\":\"/powered-by\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Company Stories\"}],[\"$\",\"$Le\",null,{\"href\":\"/download\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Download\"}],[\"$\",\"$Le\",null,{\"href\":\"/blog\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Blog\"}]]}]]}]]}],[\"$\",\"div\",\"Apache\",{\"children\":[[\"$\",\"h5\",null,{\"className\":\"mb-4 text-lg font-semibold\",\"children\":\"Apache\"}],[\"$\",\"div\",null,{\"className\":\"flex justify-between gap-x-10\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Foundation\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/licenses\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"License\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/security\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Security\"}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/foundation/sponsorship.html\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Sponsorship\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/events/current-event\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Events\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/foundation/thanks.html\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Thanks\"}]]}]]}]]}]]]}],[\"$\",\"div\",null,{\"className\":\"mt-4 flex justify-center md:mt-0\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://join.slack.com/t/apache-pinot/shared_invite/zt-5z7pav2f-yYtjZdVA~EDmrGkho87Vzw\",\"className\":\"mr-4\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":24,\"height\":24,\"viewBox\":\"0 0 24 24\",\"fill\":\"none\",\"stroke\":\"currentColor\",\"strokeWidth\":2,\"strokeLinecap\":\"round\",\"strokeLinejoin\":\"round\",\"className\":\"lucide lucide-slack fill-gray-900\",\"children\":[[\"$\",\"rect\",\"diqz80\",{\"width\":\"3\",\"height\":\"8\",\"x\":\"13\",\"y\":\"2\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"183iwg\",{\"d\":\"M19 8.5V10h1.5A1.5 1.5 0 1 0 19 8.5\"}],[\"$\",\"rect\",\"hqg7r1\",{\"width\":\"3\",\"height\":\"8\",\"x\":\"8\",\"y\":\"14\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"76g71w\",{\"d\":\"M5 15.5V14H3.5A1.5 1.5 0 1 0 5 15.5\"}],[\"$\",\"rect\",\"1kmz0a\",{\"width\":\"8\",\"height\":\"3\",\"x\":\"14\",\"y\":\"13\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"jc4sz0\",{\"d\":\"M15.5 19H14v1.5a1.5 1.5 0 1 0 1.5-1.5\"}],[\"$\",\"rect\",\"1omvl4\",{\"width\":\"8\",\"height\":\"3\",\"x\":\"2\",\"y\":\"8\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"16f3cl\",{\"d\":\"M8.5 5H10V3.5A1.5 1.5 0 1 0 8.5 5\"}],\"$undefined\"]}]}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":24,\"height\":24,\"fill\":\"currentColor\",\"size\":24,\"children\":[[\"$\",\"g\",null,{\"clipPath\":\"url(#github_svg__a)\",\"children\":[\"$\",\"path\",null,{\"fillRule\":\"evenodd\",\"d\":\"M12.01 0C5.369 0 0 5.5 0 12.304c0 5.44 3.44 10.043 8.212 11.673.597.122.815-.265.815-.59 0-.286-.02-1.264-.02-2.283-3.34.734-4.036-1.466-4.036-1.466-.537-1.426-1.332-1.793-1.332-1.793-1.094-.754.08-.754.08-.754 1.212.082 1.849 1.263 1.849 1.263 1.073 1.874 2.803 1.345 3.5 1.019.098-.795.417-1.345.755-1.65-2.665-.285-5.468-1.345-5.468-6.07 0-1.345.477-2.445 1.232-3.3-.119-.306-.537-1.57.12-3.26 0 0 1.014-.326 3.3 1.263.98-.27 1.989-.407 3.003-.408 1.014 0 2.048.143 3.002.408 2.287-1.59 3.301-1.263 3.301-1.263.657 1.69.239 2.954.12 3.26.775.855 1.232 1.955 1.232 3.3 0 4.725-2.803 5.764-5.488 6.07.438.387.815 1.12.815 2.281 0 1.65-.02 2.975-.02 3.382 0 .326.22.713.816.59C20.56 22.347 24 17.744 24 12.305 24.02 5.5 18.63 0 12.01 0\",\"clipRule\":\"evenodd\"}]}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"clipPath\",null,{\"id\":\"github_svg__a\",\"children\":[\"$\",\"path\",null,{\"fill\":\"#fff\",\"d\":\"M0 0h24v24H0z\"}]}]}]]}]}]]}]]}],[\"$\",\"div\",null,{\"className\":\"mt-8 border-t border-neutral-300 pt-4 text-left text-sm text-gray-600\",\"children\":[\"Copyright © \",2024,\" The Apache Software Foundation. Apache Pinot, Pinot, Apache, the Apache feather logo, and the Apache Pinot project logo are registered trademarks of The Apache Software Foundation. This page has references to third party software - Presto, PrestoDB, ThirdEye, Trino, TrinoDB, that are not part of the Apache Software Foundation and are not covered under the Apache License.\"]}]]}]]}]}]]}]}]]}],null]}]]\n"])</script><script>self.__next_f.push([1,"12:I[1514,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n13:I[2529,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n14:I[5185,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\""])</script><script>self.__next_f.push([1,"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n"])</script><script>self.__next_f.push([1,"10:[[\"$\",\"script\",null,{\"type\":\"application/ld+json\",\"dangerouslySetInnerHTML\":{\"__html\":\"{\\\"@context\\\":\\\"https://schema.org\\\",\\\"@type\\\":\\\"BlogPosting\\\",\\\"headline\\\":\\\"Apache Pinot™ 0.11 - Pausing Real-Time Ingestion\\\",\\\"datePublished\\\":\\\"2022-11-28T00:00:00.000Z\\\",\\\"dateModified\\\":\\\"2022-11-28T00:00:00.000Z\\\",\\\"description\\\":\\\"Learn about a feature that lets you pause and resume real-time data ingestion in Apache Pinot\\\",\\\"image\\\":\\\"/static/images/twitter-card.png\\\",\\\"url\\\":\\\"https://pinot.apache.org/blog/2022-11-28-Apache-Pinot-Pausing-Real-Time-Ingestion\\\",\\\"author\\\":[{\\\"@type\\\":\\\"Person\\\",\\\"name\\\":\\\"Mark Needham\\\"}]}\"}}],[\"$\",\"section\",null,{\"className\":\" px-5 pt-10 md:px-[13.313rem] md:py-16\",\"children\":[[\"$\",\"$L12\",null,{}],[\"$\",\"article\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"mx-auto lg:flex\",\"children\":[[\"$\",\"div\",null,{\"className\":\"lg:pr-12\",\"children\":[[\"$\",\"header\",null,{\"className\":\"pt-6 md:pr-10\",\"children\":[[\"$\",\"h1\",null,{\"className\":\"text-4xl font-semibold\",\"children\":\"Apache Pinot™ 0.11 - Pausing Real-Time Ingestion\"}],[\"$\",\"p\",null,{\"className\":\"pt-2 text-lg\",\"children\":[\"By: \",\"Mark Needham\"]}],[\"$\",\"p\",null,{\"className\":\"py-2 text-sm\",\"children\":[\"November 28th, 2022\",\" • \",\"7 min read\"]}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col lg:flex-row\",\"children\":[\"$\",\"main\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"prose max-w-[45rem] pb-8 pt-10 dark:prose-invert\",\"children\":[[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://youtu.be/u9CwDpMZRog\",\"children\":[\"$\",\"img\",null,{\"alt\":\"Watch the video\",\"src\":\"https://i3.ytimg.com/vi/u9CwDpMZRog/maxresdefault.jpg\"}]}]}],[\"$\",\"p\",null,{\"children\":[\"The Apache Pinot community recently released version \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://medium.com/apache-pinot-developer-blog/apache-pinot-0-11-released-d564684df5d4\",\"children\":\"0.11.0\"}],\", which has lots of goodies for you to play with.\"]}],[\"$\",\"p\",null,{\"children\":[\"In this post, we will learn about a feature that lets you pause and resume real-time data ingestion. Sajjad Moradi has \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://medium.com/apache-pinot-developer-blog/pause-stream-consumption-on-apache-pinot-772a971ef403\",\"children\":\"also written a blog post about this feature\"}],\", so you can treat this post as a complement to that one.\"]}],[\"$\",\"h2\",null,{\"id\":\"how-does-real-time-ingestion-work\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#how-does-real-time-ingestion-work\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"How does real-time ingestion work?\"]}],[\"$\",\"p\",null,{\"children\":\"Before we get into this feature, let’s first recap how real-time ingestion works.\"}],[\"$\",\"p\",null,{\"children\":\"This only applies to tables that have the REALTIME type. These tables ingest data that comes in from a streaming platform (e.g., Kafka).\"}],[\"$\",\"p\",null,{\"children\":\"Pinot servers ingest rows into consuming segments that reside in volatile memory.\"}],[\"$\",\"p\",null,{\"children\":[\"Once a segment reaches the \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://dev.startree.ai/docs/pinot/recipes/configuring-segment-threshold\",\"children\":\"segment threshold,\"}],\" it will be persisted to disk as a completed segment, and a new consuming segment will be created. This new segment takes over the ingestion of new events from the streaming platform.\"]}],[\"$\",\"p\",null,{\"children\":\"The diagram below shows what things might look like when we’re ingesting data from a Kafka topic that has 3 partitions:\"}],[\"$\",\"p\",null,{\"children\":[\"$\",\"img\",null,{\"alt\":\"Apache pinot 0.11 Real Time Data Ingestion\",\"src\":\"https://www.datocms-assets.com/75153/1669733133-pinot_0-11-realtime_injestion-diagram-v1.png\",\"title\":\"Apache pinot 0.11 Real Time Data Ingestion\"}]}],[\"$\",\"p\",null,{\"children\":\"A table has one consuming segment per partition but would have many completed segments.\"}],[\"$\",\"h2\",null,{\"id\":\"why-do-we-need-to-pause-and-resume-ingestion\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#why-do-we-need-to-pause-and-resume-ingestion\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Why do we need to pause and resume ingestion?\"]}],[\"$\",\"p\",null,{\"children\":\"There are many reasons why you might want to pause and resume ingestion of a stream. Some of the common ones are described below:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"There’s a problem with the underlying stream, and we need to restart the server, reset offsets, or recreate a topic\"}],[\"$\",\"li\",null,{\"children\":\"We want to ingest data from different streams into the same table.\"}],[\"$\",\"li\",null,{\"children\":\"We made a mistake in our ingestion config in Pinot, and it’s now throwing exceptions and isn’t able to ingest any more data.\"}]]}],[\"$\",\"p\",null,{\"children\":\"The 0.11 release adds the following REST API endpoints:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"/tables/{tableName}/pauseCompletion\"}]}],[\"$\",\"li\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"/tables/{tableName}/resumeCompletion\"}]}]]}],[\"$\",\"p\",null,{\"children\":[\"As the names suggest, these endpoints can be used to pause and resume streaming ingestion for a specific table. This release also adds the \",[\"$\",\"code\",null,{\"children\":\"/tables/{tableName}/pauseStatus\"}],\" endpoint, which returns the pause status for a table.\"]}],[\"$\",\"p\",null,{\"children\":\"Let’s see how to use this functionality with help from a worked example.\"}],[\"$\",\"h2\",null,{\"id\":\"data-generation\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#data-generation\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Data Generation\"]}],[\"$\",\"p\",null,{\"children\":\"Let’s imagine that we want to ingest events generated by the following Python script:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-python\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-python\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"import\"}],\" datetime\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"import\"}],\" uuid\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"import\"}],\" random\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"import\"}],\" json\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\"\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"while\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token boolean\",\"children\":\"True\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\":\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" ts \",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"=\"}],\" datetime\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"datetime\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"now\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"strftime\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"%Y-%m-%dT%H:%M:%S.%fZ\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token builtin\",\"children\":\"id\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"=\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token builtin\",\"children\":\"str\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"uuid\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"uuid4\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" count \",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"=\"}],\" random\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"randint\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"0\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"1000\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"print\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" json\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"dumps\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"tsString\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\":\"}],\" ts\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"uuid\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token builtin\",\"children\":\"id\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"count\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\":\"}],\" count\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"We can view the data generated by this script by pasting the above code into a file called datagen.py and then running the following command:\"}],[\"$\",\"p\",null,{\"children\":\"python datagen.py 2\u003e/dev/null | head -n3 | jq\"}],[\"$\",\"p\",null,{\"children\":\"We’ll see the following output:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tsString\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"2022-11-23T12:08:44.127481Z\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"uuid\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"e1c58795-a009-4e21-ae76-cdd66e090797\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"count\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"203\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tsString\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"2022-11-23T12:08:44.127531Z\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"uuid\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"4eedce04-d995-4e99-82ab-6f836b35c580\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"count\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"216\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tsString\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"2022-11-23T12:08:44.127550Z\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"uuid\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"6d72411b-55f5-4f9f-84e4-7c8c5c4581ff\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"count\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"721\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"We’re going to pipe this data into a Kafka stream called ‘events’ like this:\"}],[\"$\",\"p\",null,{\"children\":\"python datagen.py | kcat -P -b localhost:9092 -t events\"}],[\"$\",\"p\",null,{\"children\":[\"We’re not setting a key for these messages in Kafka for simplicity’s sake, but Robin Moffat has an \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://rmoff.net/2020/09/30/setting-key-value-when-piping-from-jq-to-kafkacat/\",\"children\":\"excellent blog post that explains how to do it\"}],\".\"]}],[\"$\",\"h2\",null,{\"id\":\"pinot-schematable-config\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#pinot-schematable-config\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Pinot Schema/Table Config\"]}],[\"$\",\"p\",null,{\"children\":\"We want to ingest this data into a Pinot table with the same name. Let’s first define a schema:\"}],[\"$\",\"p\",null,{\"children\":\"Schema:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"schemaName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"dimensionFieldSpecs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"uuid\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"dataType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"STRING\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"metricFieldSpecs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"count\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"dataType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"INT\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"dateTimeFieldSpecs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"ts\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"dataType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"TIMESTAMP\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"format\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"1:MILLISECONDS:EPOCH\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"granularity\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"1:MILLISECONDS\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Note that the timestamp field is called ts and not tsString, as it is in the Kafka stream. We will transform the DateTime string value held in that field into a proper timestamp using a transformation function.\"}],[\"$\",\"p\",null,{\"children\":\"Our table config is described below:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tableName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tableType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"REALTIME\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"segmentsConfig\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"timeColumnName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"ts\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"schemaName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"replication\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"1\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"replicasPerPartition\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"1\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tableIndexConfig\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"loadMode\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"MMAP\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"streamConfigs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"streamType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"kafka\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.topic.name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.broker.list\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"kafka-pause-resume:9093\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.consumer.type\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"lowlevel\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.consumer.prop.auto.offset.reset\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"smallest\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.consumer.factory.class.name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.decoder.class.name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"ingestionConfig\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"transformConfigs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"columnName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"ts\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"transformFunction\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SS''Z''')\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tenants\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"metadata\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Our transformation has a subtle error. The second parameter passed to the FromDateTime function describes the format of the DateTime string, which we defined as:\"}],[\"$\",\"p\",null,{\"children\":\"YYYY-MM-dd''T''HH:mm:ss.SS''Z''\"}],[\"$\",\"p\",null,{\"children\":\"But tsString has values in the following format:\"}],[\"$\",\"p\",null,{\"children\":\"2022-11-23T12:08:44.127550Z\"}],[\"$\",\"p\",null,{\"children\":\"i.e., we don’t have enough S values - there should be 5 rather than 2.\"}],[\"$\",\"p\",null,{\"children\":\"If we create the table using the following command:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"docker\"}],\" run \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"--network\"}],\" pause-resume \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-v\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token constant environment\",\"children\":\"$$PWD\"}],\"/pinot/config:/config \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" apachepinot/pinot:0.11.0-arm64 AddTable \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-schemaFile\"}],\" /config/schema.json \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-tableConfigFile\"}],\" /config/table.json \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-controllerHost\"}],\" pinot-controller-pause-resume \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-exec\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Pinot will immediately start trying to ingest data from Kafka, and it will throw a lot of exceptions that look like this:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-log\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-log\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"java.lang.RuntimeException:\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"Caught exception while executing function:\"}],\" fromDateTime\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"(\"}],\"tsString\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"'YYYY-MM-dd'\"}],\"T\",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"'HH:mm:ss.SS'\"}],\"Z\",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"''\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\"…\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"Caused by:\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"java.lang.IllegalStateException:\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"Caught exception while invoking method:\"}],\" public static long org\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"apache\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"pinot\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"common\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"function\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"scalar\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"DateTimeFunctions\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"fromDateTime\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"(\"}],\"java\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"lang\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"String\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"java\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"lang\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"String\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\")\"}],\" with arguments\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token number date\",\"children\":\"2022-11-23T\"}],[\"$\",\"span\",null,{\"className\":\"token number time\",\"children\":\"11:12:34.682504Z\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" YYYY\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"-\"}],\"MM\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"-\"}],\"dd\",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"'T'\"}],\"HH\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\"mm\",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\"ss\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"SS\",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"'Z'\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\"\\n\"}]]}]}],[\"$\",\"p\",null,{\"children\":\"At this point, we’d usually be stuck and would need to fix the transformation function and then restart the Pinot server. With the pause/resume feature, we can fix this problem without resorting to such drastic measures.\"}],[\"$\",\"h2\",null,{\"id\":\"the-pauseresume-flow\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#the-pauseresume-flow\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"The Pause/Resume Flow\"]}],[\"$\",\"p\",null,{\"children\":\"Instead, we can follow these steps:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"Pause ingestion for the table\"}],[\"$\",\"li\",null,{\"children\":\"Fix the transformation function\"}],[\"$\",\"li\",null,{\"children\":\"Resume ingestion\"}],[\"$\",\"li\",null,{\"children\":\"Profit $$$\"}]]}],[\"$\",\"p\",null,{\"children\":\"We can pause ingestion by running the following command:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"curl\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-X\"}],\" POST \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"http://localhost:9000/tables/events/pauseConsumption\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"accept: application/json\\\"\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"The response should be something like this:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"pauseFlag\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token boolean\",\"children\":\"true\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"consumingSegments\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events__0__0__20221123T1106Z\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"description\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"Pause flag is set. Consuming segments are being committed. Use /pauseStatus endpoint in a few moments to check if all consuming segments have been committed.\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Let’s follow the response’s advice and check the consuming segments status:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"curl\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-X\"}],\" GET \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"http://localhost:9000/tables/events/pauseStatus\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"accept: application/json\\\"\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"We’ll see the following response:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"pauseFlag\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token boolean\",\"children\":\"true\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"consumingSegments\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"So far, so good. Now we need to fix the table. We have a config, table-fixed.json, that contains a working transformation config. These are the lines of interest:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"ingestionConfig\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"transformConfigs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"columnName\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"ts\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"transformFunction\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SSSSSS''Z''')\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"We now have five S values rather than two, which should sort out our ingestion.\"}],[\"$\",\"p\",null,{\"children\":\"Update the table config:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"curl\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-X\"}],\" PUT \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"http://localhost:9000/tables/events\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"accept: application/json\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"Content-Type: application/json\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-d\"}],\" @pinot/config/table-fixed.json\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"And then resume ingestion. You can pass in the query string parameter consumeFrom, which takes a value of smallest or largest. We’ll pass in smallest since no data has been consumed yet:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"curl\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-X\"}],\" POST \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"http://localhost:9000/tables/events/resumeConsumption?consumeFrom=smallest\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"accept: application/json\\\"\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"The response will be like this:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"pauseFlag\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token boolean\",\"children\":\"false\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"consumingSegments\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"description\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"Pause flag is cleared. Consuming segments are being created. Use /pauseStatus endpoint in a few moments to double check.\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Again, let’s check the consuming segments status:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-bash\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-bash\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"curl\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-X\"}],\" GET \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"http://localhost:9000/tables/events/pauseStatus\\\"\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"\\\\\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token parameter variable\",\"children\":\"-H\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"accept: application/json\\\"\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"This time we will see some consuming segments:\"}],[\"$\",\"$L13\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"pauseFlag\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token boolean\",\"children\":\"false\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"consumingSegments\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"[\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"events__0__22__20221123T1124Z\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"]\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":[\"Navigate to \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"http://localhost:9000/#/query\",\"children\":\"http://localhost:9000/#/query\"}],\" and click on the events table. You should see the following:\"]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"img\",null,{\"alt\":\"Sample events table containing records\",\"src\":\"https://www.datocms-assets.com/75153/1669668611-image2.png\",\"title\":\"Sample events table containing records\"}]}],[\"$\",\"p\",null,{\"children\":\"We have records! We can also run our data generator again, and more events will be ingested.\"}],[\"$\",\"h2\",null,{\"id\":\"summary\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#summary\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Summary\"]}],[\"$\",\"p\",null,{\"children\":\"This feature makes real-time data ingestion a bit more forgiving when things go wrong, which has got to be a good thing in my book.\"}],[\"$\",\"p\",null,{\"children\":\"When you look at the name of this feature, it can seem a bit esoteric and perhaps not something that you’d want to use, but I think you’ll find it to be extremely useful.\"}],[\"$\",\"p\",null,{\"children\":[\"So give it a try and let us know how you get on. If you have any questions about this feature, feel free to join us on \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://stree.ai/slack\",\"children\":\"Slack\"}],\", where we’ll be happy to help you out.\"]}]]}]}]}]]}],[\"$\",\"aside\",null,{\"className\":\"mt-10 hidden border-l-2 pl-5 lg:sticky lg:top-1 lg:block lg:h-full\",\"children\":[\"$\",\"section\",null,{\"className\":\"sticky top-0 mb-4 w-[15.375rem]\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col space-y-1.5 pb-3\",\"children\":[\"$\",\"h3\",null,{\"className\":\"text-sm font-semibold leading-snug text-neutral-500 dark:text-neutral-100\",\"children\":\"Table of Contents\"}]}],[\"$\",\"$L14\",null,{\"chapters\":[{\"value\":\"How does real-time ingestion work?\",\"url\":\"#how-does-real-time-ingestion-work\",\"depth\":2},{\"value\":\"Why do we need to pause and resume ingestion?\",\"url\":\"#why-do-we-need-to-pause-and-resume-ingestion\",\"depth\":2},{\"value\":\"Data Generation\",\"url\":\"#data-generation\",\"depth\":2},{\"value\":\"Pinot Schema/Table Config\",\"url\":\"#pinot-schematable-config\",\"depth\":2},{\"value\":\"The Pause/Resume Flow\",\"url\":\"#the-pauseresume-flow\",\"depth\":2},{\"value\":\"Summary\",\"url\":\"#summary\",\"depth\":2}]}]]}]}]]}]}]]}]]\n"])</script><script>self.__next_f.push([1,"6:[[\"$\",\"meta\",\"0\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"meta\",\"1\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"2\",{\"children\":\"Apache Pinot™ 0.11 - Pausing Real-Time Ingestion | Apache Pinot™\"}],[\"$\",\"meta\",\"3\",{\"name\":\"description\",\"content\":\"Learn about a feature that lets you pause and resume real-time data ingestion in Apache Pinot\"}],[\"$\",\"meta\",\"4\",{\"name\":\"robots\",\"content\":\"index, follow\"}],[\"$\",\"meta\",\"5\",{\"name\":\"googlebot\",\"content\":\"index, follow, max-video-preview:-1, max-image-preview:large, max-snippet:-1\"}],[\"$\",\"link\",\"6\",{\"rel\":\"canonical\",\"href\":\"https://pinot.apache.org/blog/2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\"}],[\"$\",\"link\",\"7\",{\"rel\":\"alternate\",\"type\":\"application/rss+xml\",\"href\":\"https://pinot.apache.org/feed.xml\"}],[\"$\",\"meta\",\"8\",{\"property\":\"og:title\",\"content\":\"Apache Pinot™ 0.11 - Pausing Real-Time Ingestion\"}],[\"$\",\"meta\",\"9\",{\"property\":\"og:description\",\"content\":\"Learn about a feature that lets you pause and resume real-time data ingestion in Apache Pinot\"}],[\"$\",\"meta\",\"10\",{\"property\":\"og:url\",\"content\":\"https://pinot.apache.org/blog/2022/11/28/Apache-Pinot-Pausing-Real-Time-Ingestion\"}],[\"$\",\"meta\",\"11\",{\"property\":\"og:site_name\",\"content\":\"Apache Pinot™\"}],[\"$\",\"meta\",\"12\",{\"property\":\"og:locale\",\"content\":\"en_US\"}],[\"$\",\"meta\",\"13\",{\"property\":\"og:image\",\"content\":\"https://pinot.apache.org/static/images/twitter-card.png\"}],[\"$\",\"meta\",\"14\",{\"property\":\"og:type\",\"content\":\"article\"}],[\"$\",\"meta\",\"15\",{\"property\":\"article:published_time\",\"content\":\"2022-11-28T00:00:00.000Z\"}],[\"$\",\"meta\",\"16\",{\"property\":\"article:modified_time\",\"content\":\"2022-11-28T00:00:00.000Z\"}],[\"$\",\"meta\",\"17\",{\"property\":\"article:author\",\"content\":\"Mark Needham\"}],[\"$\",\"meta\",\"18\",{\"name\":\"twitter:card\",\"content\":\"summary_large_image\"}],[\"$\",\"meta\",\"19\",{\"name\":\"twitter:title\",\"content\":\"Apache Pinot™ 0.11 - Pausing Real-Time Ingestion\"}],[\"$\",\"meta\",\"20\",{\"name\":\"twitter:description\",\"content\":\"Learn about a feature that lets you pause and resume real-time data ingestion in Apache Pinot\"}],[\"$\",\"meta\",\"21\",{\"name\":\"twitter:image\",\"content\":\"https://pinot.apache.org/static/images/twitter-card.png\"}],[\"$\",\"meta\",\"22\",{\"name\":\"next-size-adjust\"}]]\n"])</script><script>self.__next_f.push([1,"f:null\n"])</script><script>self.__next_f.push([1,""])</script></body></html> |
