| </span></code></pre></div><p><code><field-names-1 and 2></code> are a comma-separated list of fields you wish to encode with CLP.</p><p>You can read the design <a target="_blank" rel="noopener noreferrer" href="https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU/edit">document</a> for more details into why and how this feature was implemented.</p><h2 id="summary"><a href="#summary" aria-hidden="true" tabindex="-1"><span class="icon icon-link"></span></a>Summary</h2><p>Apache Pinot’s evolution is expressly due to the humans behind the code, and in reaching 1.0 release status it is proper and fitting to give credit to the open source project’s key committers. Since its early days, over <a target="_blank" rel="noopener noreferrer" href="https://github.com/apache/pinot/graphs/contributors">three hundred contributors</a> have produced more than 1.3 million source lines of code (SLOC).</p><p><img alt="Image test" src="https://pinot.apache.org/blogs/apache-pinot-1-0-name-cloud.png"/></p><p>The introduction of Apache Pinot 1.0 represents an exceptional stride forward in real-time online analytical processing (OLAP) capabilities, marking a watershed moment in the evolution of real-time analytics systems. This release redefines the limits of what can be achieved in the realm of instant data analysis, presenting a game-changing solution for organizations seeking high throughput and low latency in their OLAP queries. If you would like to get started with Apache Pinot 1.0, you can check out the documentation, and download it now.</p><h2 id="resources"><a href="#resources" aria-hidden="true" tabindex="-1"><span class="icon icon-link"></span></a>Resources</h2><p>If you want to try out Apache Pinot, the following resources will help you get started:</p><p><a target="_blank" rel="noopener noreferrer" href="https://pinot.apache.org/download/">Download page</a></p><p><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/getting-started">Getting started</a></p><p><a target="_blank" rel="noopener noreferrer" href="https://communityinviter.com/apps/apache-pinot/apache-pinot">Join our Slack channel</a></p><p><a target="_blank" rel="noopener noreferrer" href="https://www.meetup.com/apache-pinot">See our upcoming events</a></p><p><a target="_blank" rel="noopener noreferrer" href="https://twitter.com/ApachePinot">Follow us on social media</a></p></div></main></div></div><aside class="mt-10 hidden border-l-2 pl-5 lg:sticky lg:top-1 lg:block lg:h-full"><section class="sticky top-0 mb-4 w-[15.375rem]"><div class="flex flex-col space-y-1.5 pb-3"><h3 class="text-sm font-semibold leading-snug text-neutral-500 dark:text-neutral-100">Table of Contents</h3></div><nav class="flex items-center self-start" aria-label="Table of Contents"><ol class="list-none space-y-3"><li class="text-sm leading-tight font-bold"><a href="#what-makes-a-10-release">What Makes a “1.0 Release?”</a></li><li class="text-sm font-normal leading-tight"><a href="#join-support">Join Support</a></li><li class="text-sm font-normal leading-tight"><a href="#upsert-improvements">Upsert Improvements</a></li><li class="text-sm leading-tight ml-6 font-normal"><a href="#segment-compaction-for-upsert-tables">Segment Compaction for Upsert Tables</a></li><li class="text-sm leading-tight ml-6 font-normal"><a href="#delete-support-for-upsert-tables">DELETE Support for Upsert Tables</a></li><li class="text-sm font-normal leading-tight"><a href="#null-value-support">NULL Value Support</a></li><li class="text-sm leading-tight ml-6 font-normal"><a href="#aggregations">Aggregations</a></li><li class="text-sm leading-tight ml-6 font-normal"><a href="#group-by">Group By</a></li><li class="text-sm leading-tight ml-6 font-normal"><a href="#sorting">Sorting</a></li><li class="text-sm font-normal leading-tight"><a href="#index-pluggability">Index Pluggability</a></li><li class="text-sm font-normal leading-tight"><a href="#apache-pinot-spark-3-connector-and-passing-pinot-options">Apache Pinot Spark 3 Connector and Passing Pinot Options</a></li><li class="text-sm font-normal leading-tight"><a href="#petabyte-scale-log-storage-and-search-in-pinot-with-clp">Petabyte-Scale Log Storage and Search in Pinot with CLP</a></li><li class="text-sm font-normal leading-tight"><a href="#summary">Summary</a></li><li class="text-sm font-normal leading-tight"><a href="#resources">Resources</a></li></ol></nav></section></aside></div></article></section></main><footer class="border-t bg-sky-100 px-5 py-10 md:px-[6.75rem] md:pb-10 md:pt-16"><div class="mx-auto flex max-w-7xl flex-wrap justify-between"><div class="flex-shrink-0"><svg xmlns="http://www.w3.org/2000/svg" width="120" height="48" fill="none"><g fill="#C7154A" clip-path="url(#logo_svg__a)"><path d="M42.99 18.448c1.032-.553 2.21-.831 3.535-.831 1.542 0 2.938.38 4.187 1.14 1.248.76 2.236 1.841 2.965 3.241.728 1.402 1.091 3.025 1.091 4.872s-.363 3.482-1.091 4.903c-.729 1.424-1.717 2.525-2.965 3.307-1.25.782-2.645 1.173-4.187 1.173-1.325 0-2.493-.271-3.503-.815-1.01-.543-1.83-1.226-2.46-2.053v14.612H36V17.912h4.562v2.606c.586-.825 1.395-1.515 2.426-2.068l.002-.002m6.452 5.605c-.445-.793-1.032-1.395-1.76-1.808a4.72 4.72 0 0 0-2.362-.618c-.847 0-1.602.211-2.33.635-.728.423-1.315 1.038-1.76 1.841-.445.804-.668 1.749-.668 2.835 0 1.087.221 2.032.668 2.835.445.804 1.032 1.417 1.76 1.842a4.557 4.557 0 0 0 2.33.635 4.57 4.57 0 0 0 2.362-.652c.728-.435 1.313-1.053 1.76-1.856.445-.804.668-1.76.668-2.867s-.223-2.025-.668-2.818v-.004M62.947 17.912v18.051h-4.562V17.912h4.562m.551-6.079a2.833 2.833 0 1 1-5.666 0 2.833 2.833 0 0 1 5.666 0M82.954 19.687c1.325 1.358 1.988 3.253 1.988 5.685v10.59H80.38v-9.97c0-1.434-.358-2.537-1.075-3.307-.717-.772-1.695-1.157-2.933-1.157-1.239 0-2.254.387-2.982 1.157-.728.772-1.091 1.873-1.091 3.307v9.97h-4.562V17.91h4.562v2.248a6.322 6.322 0 0 1 2.33-1.841c.944-.445 1.981-.669 3.111-.669 2.15 0 3.889.68 5.214 2.037v.002M92.892 35.098c-1.39-.77-2.482-1.861-3.275-3.275-.794-1.411-1.19-3.041-1.19-4.888s.406-3.475 1.221-4.888a8.502 8.502 0 0 1 3.34-3.275c1.412-.772 2.987-1.157 4.725-1.157 1.739 0 3.312.387 4.725 1.157a8.5 8.5 0 0 1 3.34 3.275c.815 1.411 1.222 3.041 1.222 4.888s-.418 3.475-1.255 4.888a8.708 8.708 0 0 1-3.388 3.275c-1.424.772-3.014 1.157-4.774 1.157-1.76 0-3.301-.385-4.691-1.157m7.021-3.421c.729-.402 1.309-1.005 1.744-1.809.435-.803.651-1.781.651-2.933 0-1.715-.451-3.035-1.351-3.958-.902-.924-2.004-1.385-3.307-1.385s-2.395.461-3.275 1.385c-.88.923-1.32 2.243-1.32 3.958 0 1.715.428 3.035 1.287 3.958.858.924 1.938 1.385 3.241 1.385.825 0 1.602-.2 2.33-.603v.002M115.96 21.658v8.734c0 .608.147 1.048.44 1.32.293.271.787.406 1.482.406H120v3.845h-2.867c-3.845 0-5.766-1.868-5.766-5.605v-8.7h-2.15v-3.746h2.15V13l4.595-1v5.912h4.04v3.746h-4.042M20.03 46.757l-5.538-1.385A1.97 1.97 0 0 1 13 43.46v-5.462c0-.841.349-1.601.907-2.146a12.212 12.212 0 0 0 6.975-3.644c2.602-2.731 3.627-6.578 2.882-10.251L21 9h-4V4a1 1 0 0 0-2 0v7a1 1 0 0 1-2 0v-1a1 1 0 0 0-2 0v6.758a4.489 4.489 0 0 1 2.694-.755c2.278.095 4.156 1.934 4.297 4.21a4.501 4.501 0 0 1-6.992 4.029V29a1 1 0 0 1-2 0V7a1 1 0 0 0-2 0v2h-4L.237 21.957c-.745 3.675.279 7.52 2.882 10.251a12.202 12.202 0 0 0 6.975 3.644c.558.545.907 1.305.907 2.146V43.4c0 .938-.639 1.757-1.55 1.985l-5.48 1.37c-.57.143-.97.655-.97 1.243h18c0-.588-.4-1.1-.97-1.243v.002"></path><path d="M13.5 23a2.5 2.5 0 1 0 0-5 2.5 2.5 0 0 0 0 5M8 5a1 1 0 1 0 0-2 1 1 0 0 0 0 2M12 8a1 1 0 1 0 0-2 1 1 0 0 0 0 2M16 2a1 1 0 1 0 0-2 1 1 0 0 0 0 2"></path></g><defs><clipPath id="logo_svg__a"><path fill="#fff" d="M0 0h120v48H0z"></path></clipPath></defs></svg></div><div class="flex flex-wrap gap-x-16 gap-y-5 py-8 md:pl-24 md:pr-[21.625rem]"> <div><h5 class="mb-4 text-lg font-semibold">Resources</h5><div class="flex justify-between gap-x-10"><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/" class="block py-1 text-gray-600 hover:text-gray-900">Docs</a><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/getting-started" class="block py-1 text-gray-600 hover:text-gray-900">Getting Started</a><a target="_blank" rel="noopener noreferrer" href="https://docs.pinot.apache.org/integrations/thirdeye" class="block py-1 text-gray-600 hover:text-gray-900">ThirdEye</a></div><div class="flex flex-col"><a class="block py-1 text-gray-600 hover:text-gray-900" href="/powered-by">Company Stories</a><a class="block py-1 text-gray-600 hover:text-gray-900" href="/download">Download</a><a class="block py-1 text-gray-600 hover:text-gray-900" href="/blog">Blog</a></div></div></div><div><h5 class="mb-4 text-lg font-semibold">Apache</h5><div class="flex justify-between gap-x-10"><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org" class="block py-1 text-gray-600 hover:text-gray-900">Foundation</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/licenses" class="block py-1 text-gray-600 hover:text-gray-900">License</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/security" class="block py-1 text-gray-600 hover:text-gray-900">Security</a></div><div class="flex flex-col"><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/foundation/sponsorship.html" class="block py-1 text-gray-600 hover:text-gray-900">Sponsorship</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/events/current-event" class="block py-1 text-gray-600 hover:text-gray-900">Events</a><a target="_blank" rel="noopener noreferrer" href="https://www.apache.org/foundation/thanks.html" class="block py-1 text-gray-600 hover:text-gray-900">Thanks</a></div></div></div></div><div class="mt-4 flex justify-center md:mt-0"><a target="_blank" rel="noopener noreferrer" href="https://join.slack.com/t/apache-pinot/shared_invite/zt-5z7pav2f-yYtjZdVA~EDmrGkho87Vzw" class="mr-4"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-slack fill-gray-900"><rect width="3" height="8" x="13" y="2" rx="1.5"></rect><path d="M19 8.5V10h1.5A1.5 1.5 0 1 0 19 8.5"></path><rect width="3" height="8" x="8" y="14" rx="1.5"></rect><path d="M5 15.5V14H3.5A1.5 1.5 0 1 0 5 15.5"></path><rect width="8" height="3" x="14" y="13" rx="1.5"></rect><path d="M15.5 19H14v1.5a1.5 1.5 0 1 0 1.5-1.5"></path><rect width="8" height="3" x="2" y="8" rx="1.5"></rect><path d="M8.5 5H10V3.5A1.5 1.5 0 1 0 8.5 5"></path></svg></a><a target="_blank" rel="noopener noreferrer" href="https://github.com/apache/pinot"><svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" fill="currentColor" size="24"><g clip-path="url(#github_svg__a)"><path fill-rule="evenodd" d="M12.01 0C5.369 0 0 5.5 0 12.304c0 5.44 3.44 10.043 8.212 11.673.597.122.815-.265.815-.59 0-.286-.02-1.264-.02-2.283-3.34.734-4.036-1.466-4.036-1.466-.537-1.426-1.332-1.793-1.332-1.793-1.094-.754.08-.754.08-.754 1.212.082 1.849 1.263 1.849 1.263 1.073 1.874 2.803 1.345 3.5 1.019.098-.795.417-1.345.755-1.65-2.665-.285-5.468-1.345-5.468-6.07 0-1.345.477-2.445 1.232-3.3-.119-.306-.537-1.57.12-3.26 0 0 1.014-.326 3.3 1.263.98-.27 1.989-.407 3.003-.408 1.014 0 2.048.143 3.002.408 2.287-1.59 3.301-1.263 3.301-1.263.657 1.69.239 2.954.12 3.26.775.855 1.232 1.955 1.232 3.3 0 4.725-2.803 5.764-5.488 6.07.438.387.815 1.12.815 2.281 0 1.65-.02 2.975-.02 3.382 0 .326.22.713.816.59C20.56 22.347 24 17.744 24 12.305 24.02 5.5 18.63 0 12.01 0" clip-rule="evenodd"></path></g><defs><clipPath id="github_svg__a"><path fill="#fff" d="M0 0h24v24H0z"></path></clipPath></defs></svg></a></div></div><div class="mt-8 border-t border-neutral-300 pt-4 text-left text-sm text-gray-600">Copyright © <!-- -->2024<!-- --> The Apache Software Foundation. Apache Pinot, Pinot, Apache, the Apache feather logo, and the Apache Pinot project logo are registered trademarks of The Apache Software Foundation. This page has references to third party software - Presto, PrestoDB, ThirdEye, Trino, TrinoDB, that are not part of the Apache Software Foundation and are not covered under the Apache License.</div></footer></div><script src="/_next/static/chunks/webpack-dde39ac7c1b4eb4b.js" crossorigin="" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])</script><script>self.__next_f.push([1,"1:HL[\"/_next/static/media/6905431624c34d00-s.p.woff2\",\"font\",{\"crossOrigin\":\"\",\"type\":\"font/woff2\"}]\n2:HL[\"/_next/static/css/9e925a33b1acdac1.css\",\"style\",{\"crossOrigin\":\"\"}]\n0:\"$L3\"\n"])</script><script>self.__next_f.push([1,"4:HL[\"/_next/static/css/c130d1629644f070.css\",\"style\",{\"crossOrigin\":\"\"}]\n"])</script><script>self.__next_f.push([1,"5:I[3728,[],\"\"]\n7:I[9928,[],\"\"]\n8:I[7821,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"ThemeProviders\"]\n9:I[3994,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"\"]\na:I[9640,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js"])</script><script>self.__next_f.push([1,"\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"AlgoliaSearchProvider\"]\nb:I[7975,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"702\",\"static/chunks/702-a2bf9fe707814b79.js\",\"185\",\"static/chunks/app/layout-776a485845c720ef.js\"],\"\"]\nc:I[6954,[],\"\"]\nd:I[7264,[],\"\"]\ne:I[8326,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\""])</script><script>self.__next_f.push([1,",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n11:T9fe,"])</script><script>self.__next_f.push([1,"M42.99 18.448c1.032-.553 2.21-.831 3.535-.831 1.542 0 2.938.38 4.187 1.14 1.248.76 2.236 1.841 2.965 3.241.728 1.402 1.091 3.025 1.091 4.872s-.363 3.482-1.091 4.903c-.729 1.424-1.717 2.525-2.965 3.307-1.25.782-2.645 1.173-4.187 1.173-1.325 0-2.493-.271-3.503-.815-1.01-.543-1.83-1.226-2.46-2.053v14.612H36V17.912h4.562v2.606c.586-.825 1.395-1.515 2.426-2.068l.002-.002m6.452 5.605c-.445-.793-1.032-1.395-1.76-1.808a4.72 4.72 0 0 0-2.362-.618c-.847 0-1.602.211-2.33.635-.728.423-1.315 1.038-1.76 1.841-.445.804-.668 1.749-.668 2.835 0 1.087.221 2.032.668 2.835.445.804 1.032 1.417 1.76 1.842a4.557 4.557 0 0 0 2.33.635 4.57 4.57 0 0 0 2.362-.652c.728-.435 1.313-1.053 1.76-1.856.445-.804.668-1.76.668-2.867s-.223-2.025-.668-2.818v-.004M62.947 17.912v18.051h-4.562V17.912h4.562m.551-6.079a2.833 2.833 0 1 1-5.666 0 2.833 2.833 0 0 1 5.666 0M82.954 19.687c1.325 1.358 1.988 3.253 1.988 5.685v10.59H80.38v-9.97c0-1.434-.358-2.537-1.075-3.307-.717-.772-1.695-1.157-2.933-1.157-1.239 0-2.254.387-2.982 1.157-.728.772-1.091 1.873-1.091 3.307v9.97h-4.562V17.91h4.562v2.248a6.322 6.322 0 0 1 2.33-1.841c.944-.445 1.981-.669 3.111-.669 2.15 0 3.889.68 5.214 2.037v.002M92.892 35.098c-1.39-.77-2.482-1.861-3.275-3.275-.794-1.411-1.19-3.041-1.19-4.888s.406-3.475 1.221-4.888a8.502 8.502 0 0 1 3.34-3.275c1.412-.772 2.987-1.157 4.725-1.157 1.739 0 3.312.387 4.725 1.157a8.5 8.5 0 0 1 3.34 3.275c.815 1.411 1.222 3.041 1.222 4.888s-.418 3.475-1.255 4.888a8.708 8.708 0 0 1-3.388 3.275c-1.424.772-3.014 1.157-4.774 1.157-1.76 0-3.301-.385-4.691-1.157m7.021-3.421c.729-.402 1.309-1.005 1.744-1.809.435-.803.651-1.781.651-2.933 0-1.715-.451-3.035-1.351-3.958-.902-.924-2.004-1.385-3.307-1.385s-2.395.461-3.275 1.385c-.88.923-1.32 2.243-1.32 3.958 0 1.715.428 3.035 1.287 3.958.858.924 1.938 1.385 3.241 1.385.825 0 1.602-.2 2.33-.603v.002M115.96 21.658v8.734c0 .608.147 1.048.44 1.32.293.271.787.406 1.482.406H120v3.845h-2.867c-3.845 0-5.766-1.868-5.766-5.605v-8.7h-2.15v-3.746h2.15V13l4.595-1v5.912h4.04v3.746h-4.042M20.03 46.757l-5.538-1.385A1.97 1.97 0 0 1 13 43.46v-5.462c0-.841.349-1.601.907-2.146a12.212 12.212 0 0 0 6.975-3.644c2.602-2.731 3.627-6.578 2.882-10.251L21 9h-4V4a1 1 0 0 0-2 0v7a1 1 0 0 1-2 0v-1a1 1 0 0 0-2 0v6.758a4.489 4.489 0 0 1 2.694-.755c2.278.095 4.156 1.934 4.297 4.21a4.501 4.501 0 0 1-6.992 4.029V29a1 1 0 0 1-2 0V7a1 1 0 0 0-2 0v2h-4L.237 21.957c-.745 3.675.279 7.52 2.882 10.251a12.202 12.202 0 0 0 6.975 3.644c.558.545.907 1.305.907 2.146V43.4c0 .938-.639 1.757-1.55 1.985l-5.48 1.37c-.57.143-.97.655-.97 1.243h18c0-.588-.4-1.1-.97-1.243v.002"])</script><script>self.__next_f.push([1,"3:[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/9e925a33b1acdac1.css\",\"precedence\":\"next\",\"crossOrigin\":\"\"}]],[\"$\",\"$L5\",null,{\"buildId\":\"rmcKjFZ3e9kKdH1iJwCIQ\",\"assetPrefix\":\"\",\"initialCanonicalUrl\":\"/blog/2023/09/19/Annoucing-Apache-Pinot-1-0\",\"initialTree\":[\"\",{\"children\":[\"blog\",{\"children\":[[\"slug\",\"2023/09/19/Annoucing-Apache-Pinot-1-0\",\"c\"],{\"children\":[\"__PAGE__?{\\\"slug\\\":[\\\"2023\\\",\\\"09\\\",\\\"19\\\",\\\"Annoucing-Apache-Pinot-1-0\\\"]}\",{}]}]}]},\"$undefined\",\"$undefined\",true],\"initialHead\":[false,\"$L6\"],\"globalErrorComponent\":\"$7\",\"children\":[null,[\"$\",\"html\",null,{\"lang\":\"en-us\",\"className\":\"__variable_1fc36d scroll-smooth\",\"suppressHydrationWarning\":true,\"children\":[[\"$\",\"head\",null,{\"children\":[[\"$\",\"meta\",null,{\"httpEquiv\":\"Content-Security-Policy\",\"content\":\"default-src 'self';script-src 'self' 'unsafe-eval' 'unsafe-inline' giscus.app analytics.umami.is www.youtube.com www.googletagmanager.com www.google-analytics.com;style-src 'self' 'unsafe-inline';img-src * blob: data:;media-src *.s3.amazonaws.com;connect-src *;font-src 'self';frame-src www.youtube.com youtube.com giscus.app youtu.be https://www.youtube.com https://youtube.com;\"}],[\"$\",\"link\",null,{\"rel\":\"apple-touch-icon\",\"sizes\":\"76x76\",\"href\":\"/static/favicons/apple-touch-icon.png\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"32x32\",\"href\":\"/static/favicons/favicon-32x32.png\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"16x16\",\"href\":\"/static/favicons/favicon-16x16.png\"}],[\"$\",\"link\",null,{\"rel\":\"manifest\",\"href\":\"/static/favicons/site.webmanifest\"}],[\"$\",\"link\",null,{\"rel\":\"mask-icon\",\"href\":\"/static/favicons/safari-pinned-tab.svg\",\"color\":\"#5bbad5\"}],[\"$\",\"meta\",null,{\"name\":\"msapplication-TileColor\",\"content\":\"#000000\"}],[\"$\",\"meta\",null,{\"name\":\"theme-color\",\"media\":\"(prefers-color-scheme: light)\",\"content\":\"#fff\"}],[\"$\",\"meta\",null,{\"name\":\"theme-color\",\"media\":\"(prefers-color-scheme: dark)\",\"content\":\"#000\"}],[\"$\",\"link\",null,{\"rel\":\"alternate\",\"type\":\"application/rss+xml\",\"href\":\"/feed.xml\"}]]}],[\"$\",\"body\",null,{\"className\":\"bg-white text-black antialiased dark:bg-gray-950 dark:text-white\",\"children\":[\"$\",\"$L8\",null,{\"children\":[[\"$undefined\",\"$undefined\",\"$undefined\",\"$undefined\",[[\"$\",\"$L9\",null,{\"strategy\":\"afterInteractive\",\"src\":\"https://www.googletagmanager.com/gtag/js?id=G-ZXG79NJEBY\"}],[\"$\",\"$L9\",null,{\"strategy\":\"afterInteractive\",\"id\":\"ga-script\",\"children\":\"\\n window.dataLayer = window.dataLayer || [];\\n function gtag(){dataLayer.push(arguments);}\\n gtag('js', new Date());\\n gtag('config', 'G-ZXG79NJEBY');\\n \"}]]],[\"$\",\"div\",null,{\"className\":\"mx-auto flex max-w-screen-customDesktop flex-col justify-between font-sans\",\"children\":[\"$\",\"$La\",null,{\"algoliaConfig\":{\"appId\":\"CKRA00L2X9\",\"apiKey\":\"6531f8f7783a88d76629190843f1801e\",\"indexName\":\"prod_apache_pinot_docs\"},\"children\":[[\"$\",\"$Lb\",null,{}],[\"$\",\"main\",null,{\"children\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[\"$\",\"div\",null,{\"className\":\"flex flex-col items-start justify-start md:mt-24 md:flex-row md:items-center md:justify-center md:space-x-6\",\"children\":[[\"$\",\"div\",null,{\"className\":\"space-x-2 pb-8 pt-6 md:space-y-5\",\"children\":[\"$\",\"h1\",null,{\"className\":\"text-6xl font-extrabold leading-9 tracking-tight text-gray-900 dark:text-gray-100 md:border-r-2 md:px-6 md:text-8xl md:leading-14\",\"children\":\"404\"}]}],[\"$\",\"div\",null,{\"className\":\"max-w-md\",\"children\":[[\"$\",\"p\",null,{\"className\":\"mb-4 text-xl font-bold leading-normal md:text-2xl\",\"children\":\"Sorry we couldn't find this page.\"}],[\"$\",\"p\",null,{\"className\":\"mb-8\",\"children\":\"But dont worry, you can find plenty of other things on our homepage.\"}],[\"$\",\"$Le\",null,{\"href\":\"/\",\"className\":\"focus:shadow-outline-blue inline rounded-lg border border-transparent bg-blue-600 px-4 py-2 text-sm font-medium leading-5 text-white shadow transition-colors duration-150 hover:bg-blue-700 focus:outline-none dark:hover:bg-blue-500\",\"children\":\"Back to homepage\"}]]}]]}],\"notFoundStyles\":[],\"initialChildNode\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"initialChildNode\":[\"$\",\"$Lc\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\",[\"slug\",\"2023/09/19/Annoucing-Apache-Pinot-1-0\",\"c\"],\"children\"],\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"loadingScripts\":\"$undefined\",\"hasLoading\":false,\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$Ld\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"initialChildNode\":[\"$Lf\",\"$L10\",null],\"childPropSegment\":\"__PAGE__?{\\\"slug\\\":[\\\"2023\\\",\\\"09\\\",\\\"19\\\",\\\"Annoucing-Apache-Pinot-1-0\\\"]}\",\"styles\":[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/c130d1629644f070.css\",\"precedence\":\"next\",\"crossOrigin\":\"\"}]]}],\"childPropSegment\":[\"slug\",\"2023/09/19/Annoucing-Apache-Pinot-1-0\",\"c\"],\"styles\":null}],\"childPropSegment\":\"blog\",\"styles\":null}]}],[\"$\",\"footer\",null,{\"className\":\"border-t bg-sky-100 px-5 py-10 md:px-[6.75rem] md:pb-10 md:pt-16\",\"children\":[[\"$\",\"div\",null,{\"className\":\"mx-auto flex max-w-7xl flex-wrap justify-between\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex-shrink-0\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":120,\"height\":48,\"fill\":\"none\",\"children\":[[\"$\",\"g\",null,{\"fill\":\"#C7154A\",\"clipPath\":\"url(#logo_svg__a)\",\"children\":[[\"$\",\"path\",null,{\"d\":\"$11\"}],[\"$\",\"path\",null,{\"d\":\"M13.5 23a2.5 2.5 0 1 0 0-5 2.5 2.5 0 0 0 0 5M8 5a1 1 0 1 0 0-2 1 1 0 0 0 0 2M12 8a1 1 0 1 0 0-2 1 1 0 0 0 0 2M16 2a1 1 0 1 0 0-2 1 1 0 0 0 0 2\"}]]}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"clipPath\",null,{\"id\":\"logo_svg__a\",\"children\":[\"$\",\"path\",null,{\"fill\":\"#fff\",\"d\":\"M0 0h120v48H0z\"}]}]}]]}]}],[\"$\",\"div\",null,{\"className\":\"flex flex-wrap gap-x-16 gap-y-5 py-8 md:pl-24 md:pr-[21.625rem]\",\"children\":[\" \",[[\"$\",\"div\",\"Resources\",{\"children\":[[\"$\",\"h5\",null,{\"className\":\"mb-4 text-lg font-semibold\",\"children\":\"Resources\"}],[\"$\",\"div\",null,{\"className\":\"flex justify-between gap-x-10\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Docs\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/getting-started\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Getting Started\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/integrations/thirdeye\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"ThirdEye\"}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"$Le\",null,{\"href\":\"/powered-by\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Company Stories\"}],[\"$\",\"$Le\",null,{\"href\":\"/download\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Download\"}],[\"$\",\"$Le\",null,{\"href\":\"/blog\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Blog\"}]]}]]}]]}],[\"$\",\"div\",\"Apache\",{\"children\":[[\"$\",\"h5\",null,{\"className\":\"mb-4 text-lg font-semibold\",\"children\":\"Apache\"}],[\"$\",\"div\",null,{\"className\":\"flex justify-between gap-x-10\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Foundation\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/licenses\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"License\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/security\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Security\"}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/foundation/sponsorship.html\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Sponsorship\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/events/current-event\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Events\"}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.apache.org/foundation/thanks.html\",\"className\":\"block py-1 text-gray-600 hover:text-gray-900\",\"children\":\"Thanks\"}]]}]]}]]}]]]}],[\"$\",\"div\",null,{\"className\":\"mt-4 flex justify-center md:mt-0\",\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://join.slack.com/t/apache-pinot/shared_invite/zt-5z7pav2f-yYtjZdVA~EDmrGkho87Vzw\",\"className\":\"mr-4\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":24,\"height\":24,\"viewBox\":\"0 0 24 24\",\"fill\":\"none\",\"stroke\":\"currentColor\",\"strokeWidth\":2,\"strokeLinecap\":\"round\",\"strokeLinejoin\":\"round\",\"className\":\"lucide lucide-slack fill-gray-900\",\"children\":[[\"$\",\"rect\",\"diqz80\",{\"width\":\"3\",\"height\":\"8\",\"x\":\"13\",\"y\":\"2\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"183iwg\",{\"d\":\"M19 8.5V10h1.5A1.5 1.5 0 1 0 19 8.5\"}],[\"$\",\"rect\",\"hqg7r1\",{\"width\":\"3\",\"height\":\"8\",\"x\":\"8\",\"y\":\"14\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"76g71w\",{\"d\":\"M5 15.5V14H3.5A1.5 1.5 0 1 0 5 15.5\"}],[\"$\",\"rect\",\"1kmz0a\",{\"width\":\"8\",\"height\":\"3\",\"x\":\"14\",\"y\":\"13\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"jc4sz0\",{\"d\":\"M15.5 19H14v1.5a1.5 1.5 0 1 0 1.5-1.5\"}],[\"$\",\"rect\",\"1omvl4\",{\"width\":\"8\",\"height\":\"3\",\"x\":\"2\",\"y\":\"8\",\"rx\":\"1.5\"}],[\"$\",\"path\",\"16f3cl\",{\"d\":\"M8.5 5H10V3.5A1.5 1.5 0 1 0 8.5 5\"}],\"$undefined\"]}]}],[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot\",\"children\":[\"$\",\"svg\",null,{\"xmlns\":\"http://www.w3.org/2000/svg\",\"width\":24,\"height\":24,\"fill\":\"currentColor\",\"size\":24,\"children\":[[\"$\",\"g\",null,{\"clipPath\":\"url(#github_svg__a)\",\"children\":[\"$\",\"path\",null,{\"fillRule\":\"evenodd\",\"d\":\"M12.01 0C5.369 0 0 5.5 0 12.304c0 5.44 3.44 10.043 8.212 11.673.597.122.815-.265.815-.59 0-.286-.02-1.264-.02-2.283-3.34.734-4.036-1.466-4.036-1.466-.537-1.426-1.332-1.793-1.332-1.793-1.094-.754.08-.754.08-.754 1.212.082 1.849 1.263 1.849 1.263 1.073 1.874 2.803 1.345 3.5 1.019.098-.795.417-1.345.755-1.65-2.665-.285-5.468-1.345-5.468-6.07 0-1.345.477-2.445 1.232-3.3-.119-.306-.537-1.57.12-3.26 0 0 1.014-.326 3.3 1.263.98-.27 1.989-.407 3.003-.408 1.014 0 2.048.143 3.002.408 2.287-1.59 3.301-1.263 3.301-1.263.657 1.69.239 2.954.12 3.26.775.855 1.232 1.955 1.232 3.3 0 4.725-2.803 5.764-5.488 6.07.438.387.815 1.12.815 2.281 0 1.65-.02 2.975-.02 3.382 0 .326.22.713.816.59C20.56 22.347 24 17.744 24 12.305 24.02 5.5 18.63 0 12.01 0\",\"clipRule\":\"evenodd\"}]}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"clipPath\",null,{\"id\":\"github_svg__a\",\"children\":[\"$\",\"path\",null,{\"fill\":\"#fff\",\"d\":\"M0 0h24v24H0z\"}]}]}]]}]}]]}]]}],[\"$\",\"div\",null,{\"className\":\"mt-8 border-t border-neutral-300 pt-4 text-left text-sm text-gray-600\",\"children\":[\"Copyright © \",2024,\" The Apache Software Foundation. Apache Pinot, Pinot, Apache, the Apache feather logo, and the Apache Pinot project logo are registered trademarks of The Apache Software Foundation. This page has references to third party software - Presto, PrestoDB, ThirdEye, Trino, TrinoDB, that are not part of the Apache Software Foundation and are not covered under the Apache License.\"]}]]}]]}]}]]}]}]]}],null]}]]\n"])</script><script>self.__next_f.push([1,"12:I[1514,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n13:I[7246,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n14:I[2529,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\""])</script><script>self.__next_f.push([1,"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n15:I[5185,[\"326\",\"static/chunks/326-3a90a6443b9c824c.js\",\"413\",\"static/chunks/413-f9f40b83f7bb3f22.js\",\"980\",\"static/chunks/980-6e243f9cd384c7d2.js\",\"797\",\"static/chunks/app/blog/%5B...slug%5D/page-502e08b6677b55da.js\"],\"\"]\n"])</script><script>self.__next_f.push([1,"10:[[\"$\",\"script\",null,{\"type\":\"application/ld+json\",\"dangerouslySetInnerHTML\":{\"__html\":\"{\\\"@context\\\":\\\"https://schema.org\\\",\\\"@type\\\":\\\"BlogPosting\\\",\\\"headline\\\":\\\"Announcing Apache Pinot 1.0™\\\",\\\"datePublished\\\":\\\"2023-09-19T00:00:00.000Z\\\",\\\"dateModified\\\":\\\"2023-09-19T00:00:00.000Z\\\",\\\"description\\\":\\\"Introducing Apache Pinot 1.0 Release\\\",\\\"image\\\":\\\"/static/images/twitter-card.png\\\",\\\"url\\\":\\\"https://pinot.apache.org/blog/2023-09-19-Annoucing-Apache-Pinot-1-0\\\",\\\"author\\\":[{\\\"@type\\\":\\\"Person\\\",\\\"name\\\":\\\"Hubert Dulay\\\"},{\\\"@type\\\":\\\"Person\\\",\\\"name\\\":\\\"Mayank Shrivastava\\\"},{\\\"@type\\\":\\\"Person\\\",\\\"name\\\":\\\"Neha Pawar\\\"}]}\"}}],[\"$\",\"section\",null,{\"className\":\" px-5 pt-10 md:px-[13.313rem] md:py-16\",\"children\":[[\"$\",\"$L12\",null,{}],[\"$\",\"article\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"mx-auto lg:flex\",\"children\":[[\"$\",\"div\",null,{\"className\":\"lg:pr-12\",\"children\":[[\"$\",\"header\",null,{\"className\":\"pt-6 md:pr-10\",\"children\":[[\"$\",\"h1\",null,{\"className\":\"text-4xl font-semibold\",\"children\":\"Announcing Apache Pinot 1.0™\"}],[\"$\",\"p\",null,{\"className\":\"pt-2 text-lg\",\"children\":[\"By: \",\"Hubert Dulay, Mayank Shrivastava, Neha Pawar\"]}],[\"$\",\"p\",null,{\"className\":\"py-2 text-sm\",\"children\":[\"September 19th, 2023\",\" • \",\"13 min read\"]}]]}],[\"$\",\"div\",null,{\"className\":\"flex flex-col lg:flex-row\",\"children\":[\"$\",\"main\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"prose max-w-[45rem] pb-8 pt-10 dark:prose-invert\",\"children\":[[\"$\",\"h2\",null,{\"id\":\"what-makes-a-10-release\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#what-makes-a-10-release\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"What Makes a “1.0 Release?”\"]}],[\"$\",\"p\",null,{\"children\":\"Apache Pinot has continuously evolved since the project’s inception within LinkedIn in 2013. Back then it was developed at a single company with a single use case in mind: to power “who viewed my profile?” Over the ensuing decade the Apache Pinot community expanded to be embraced by many other organizations, and those organizations have expanded its capabilities to address new use cases. Apache Pinot in 2023 is continuously evolving to address emerging needs in the real-time analytics community. Let’s look at how much innovation has gone into Apache Pinot over the years:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"Upserts — data-in-motion tends to stay in motion, and one of the cornerstone capabilities of Apache Pinot is upsert support to handle upsert mutations in real-time.\"}],[\"$\",\"li\",null,{\"children\":\"Query-time Native JOINs — it was important to get this right, so that they were performant and scalable, allowing high QPS. This we will discuss in more detail below.\"}],[\"$\",\"li\",null,{\"children\":\"Pluggable architecture — a broad user base requires the ability to extend the database with new customizable index types, routing strategies and storage options\"}],[\"$\",\"li\",null,{\"children\":\"Handling Semi-Structured/Unstructured Data — Pinot can easily index JSON and text data types at scale.\"}],[\"$\",\"li\",null,{\"children\":\"Improving ANSI SQL Compliance — to that end, we’ve added better NULL handling, window functions, and as stated above, the capability for native JOINs.\"}]]}],[\"$\",\"p\",null,{\"children\":\"With all of these features and capabilities, Apache Pinot moves farther and farther from mere database status, and becomes more of a complete platform that can tackle entire new classes of use cases that were beyond its capabilities in earlier days.\"}],[\"$\",\"p\",null,{\"children\":\"First let’s look at what Apache Pinot 1.0 itself is delivering. The first foundational pillar of what makes something worthy of a “1.0” release is software quality. Over the past year, since September 2022, engineers across the Apache Pinot community have closed over 300 issues to provide new features, optimize performance, expand test coverage, and squash bugs.\"}],[\"$\",\"p\",null,{\"children\":[\"Features are also a key thing that makes a new release worthy of “1.0” status. The most critical part of the 1.0 release is undoubtedly the \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/developers/advanced/v2-multi-stage-query-engine\",\"children\":\"Multi-Stage Query Engine\"}],\", which permits Apache Pinot users to do \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://startree.ai/blog/apache-pinot-native-join-support\",\"children\":\"performant and scalable query-time JOINs\"}],\".\"]}],[\"$\",\"p\",null,{\"children\":\"The original engine works very well for simpler filter-and-aggregate queries, but the broker could become a bottleneck for more complex queries. The new engine also resolves this by introducing intermediary compute stages on the query servers, and brings Apache Pinot closer to full ANSI SQL semantics. While this query engine has been available within Apache Pinot already (since release 0.11.0), with the release of Apache Pinot 1.0 this feature is functionally complete.\"}],[\"$\",\"p\",null,{\"children\":[\"(While you can read more below, check out the accompanying blog by Apache Pinot PMC Neha Pawar about using query-time JOINs \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://startree.ai/blog/query-time-joins-in-apache-pinot-1-0\",\"children\":\"here\"}],\").\"]}],[\"$\",\"p\",null,{\"children\":[\"This post is a summary of the high points, but you can find a full list of everything included in the release notes. And if you’d like a \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://youtu.be/2cwRHM4J7kI?si=hEtl6W2eNlMkWqag\",\"children\":\"video treatment of many of the main features in 1.0\"}],\", including some helpful animations, watch here:\"]}],[\"$\",\"$L13\",null,{\"src\":\"https://www.youtube.com/embed/2cwRHM4J7kI\",\"title\":\"YouTube video player\"}],[\"$\",\"p\",null,{\"children\":\"Otherwise, let’s have a look at some of the highlighted changes:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"Join Support - Part of the Multi-Stage Query Engine\"}],[\"$\",\"li\",null,{\"children\":\"Improved Upserts - Deletion and Compaction Support\"}],[\"$\",\"li\",null,{\"children\":\"Encode User-Specified Compressed Log Processor (CLP) During Ingestion\"}],[\"$\",\"li\",null,{\"children\":\"NULL Support\"}],[\"$\",\"li\",null,{\"children\":\"Pluggable Index Types [Index Service Provider Interface (SPI)]\"}],[\"$\",\"li\",null,{\"children\":\"Improved Pinot-Spark Integration - Spark3 Compatibility\"}]]}],[\"$\",\"h2\",null,{\"id\":\"join-support\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#join-support\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Join Support\"]}],[\"$\",\"p\",null,{\"children\":\"Apache Pinot 1.0 introduces native query-time JOIN support equipping Pinot to handle a broad spectrum of JOIN scenarios providing full coverage from user-facing analytics all the way up to ad hoc analytics. Underpinning this innovation is the multi-stage query engine, introduced a year ago, which efficiently manages complex analytical queries, including JOIN operations. This engine alleviates computational burdens by offloading tasks from brokers to a dedicated intermediate compute stage. Additionally, a new planner supporting full SQL semantics enhances Pinot's analytical capabilities.\"}],[\"$\",\"p\",null,{\"children\":\"JOIN optimization strategies play a pivotal role in Apache Pinot 1.0. These include predicate push-down to individual tables and using indexing and pruning to reduce scanning which speeds up query processing, smart data layout considerations to minimize data shuffling, and query hints for fine-tuning JOIN operations. With support for all JOIN types and three JOIN algorithms, including broadcast join, shuffle distributed hash join, and lookup join, Apache Pinot delivers versatility and scalability. By significantly reducing query latency and simplifying architecture, Apache Pinot 1.0 is a game-changer for real-time OLAP systems.\"}],[\"$\",\"p\",null,{\"children\":[\"For more detailed information on JOINs, please visit this blog \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://startree.ai/blog/query-time-joins-in-apache-pinot-1-0\",\"children\":\"post\"}],\".\"]}],[\"$\",\"p\",null,{\"children\":\"Discover How Uber is using Joins in Apache Pinot For a real-world use case, Uber is already using the new join capabilities of Apache Pinot at scale in production. You can watch this video to learn more.\"}],[\"$\",\"$L13\",null,{\"src\":\"https://www.youtube.com/embed/z4Chhref1BM\",\"title\":\"YouTube video player\"}],[\"$\",\"h2\",null,{\"id\":\"upsert-improvements\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#upsert-improvements\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Upsert Improvements\"]}],[\"$\",\"p\",null,{\"children\":\"Support for upserts is one of the key capabilities Apache Pinot offers that differentiates it from other real-time analytics databases. It is a vital feature when real-time streaming data is prone to frequent updates. While upserts have been available in Apache Pinot since 0.6.0, with 1.0 they include two major new enhancements: segment compaction and delete support for upsert tables.\"}],[\"$\",\"h3\",null,{\"id\":\"segment-compaction-for-upsert-tables\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#segment-compaction-for-upsert-tables\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Segment Compaction for Upsert Tables\"]}],[\"$\",\"p\",null,{\"children\":\"Pinot’s Upsert tables store all versions of a record ingested into immutable segments on disk. Older records unnecessarily consume valuable storage space when they’re no longer used in query results. Pinot’s Segment Compaction reclaims this valuable storage space by introducing a periodic process that replaces completed segments with compacted segments which only contain the latest version of the records.\"}],[\"$\",\"$L14\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"task\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"taskTypeConfigsMap\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"UpsertCompactionTask\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"schedule\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"0 */5 * ? * *\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"bufferTimePeriod\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"7d\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"invalidRecordsThresholdPercent\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"30\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"invalidRecordsThresholdCount\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"100000\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"The example above, bufferTimePeriod is set to “7d” which means that any segment that was completed over 7 days ago may be eligible for compaction. However, if you want to ensure that segments are compacted without any additional delay this config can be set to “0d”.\"}],[\"$\",\"p\",null,{\"children\":\"invalidRecordsThresholdPercent is an optional limit to the amount of older records allowed in the completed segment represented as a percentage of the total number of records in the segment (i.e. old records / total records). In the example, this property is set to “30” which means that if more than 30% of the records in the completed segment are old, then the segment may be selected for compaction.\"}],[\"$\",\"p\",null,{\"children\":\"invalidRecordsThresholdCount is also a limit similar to the previous property, but allows you to express the threshold as a record count. In the example above, this property is set to “100000” which means that if the segment contains more than 100K records then it may be selected for compaction.\"}],[\"$\",\"p\",null,{\"children\":[[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://robert-zych.medium.com/segment-compaction-for-upsert-enabled-tables-in-apache-pinot-3f30657aa077\",\"children\":\"Read more\"}],\" about the design of this feature.\"]}],[\"$\",\"h3\",null,{\"id\":\"delete-support-for-upsert-tables\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#delete-support-for-upsert-tables\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"DELETE Support for Upsert Tables\"]}],[\"$\",\"p\",null,{\"children\":\"Apache Pinot upsert tables now support deleting records. Supporting delete with upsert avoids the need for the user to explicitly filter out invalid records in the query. SELECT _FROM table WHERE deleted_column != true becomes as simple as SELECT _ FROM table. Pinot will only return the latest non-deleted records from the table. This feature opens up the support to ingest Change Data Capture (CDC) data like Debezium where the changes from a source (typically, mutable) will contain DELETE events.\"}],[\"$\",\"p\",null,{\"children\":[\"Deletes itself is implemented as a soft-delete in Apache Pinot with a dedicated boolean column that serves as a delete marker for the record. Pinot automatically filters out records that are marked in this column. For more details, please see the \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/basics/data-import/upsert#delete-column\",\"children\":\"documentation\"}],\".\"]}],[\"$\",\"h2\",null,{\"id\":\"null-value-support\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#null-value-support\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"NULL Value Support\"]}],[\"$\",\"p\",null,{\"children\":\"This feature enables Postgres compatible NULL semantics in Apache Pinot queries. The NULL semantics are important for usability for full SQL compatibility which many BI applications like Tableau rely upon when invoking queries to render dashboards. Previously in Pinot, we could not represent NULL. The workaround was to use special values like Integer.MIN_VALUE to represent NULL. Now Pinot 1.0 has full support to represent NULL values. By adding NULL support, Pinot 1.0 has increased the Tableau certification pass rate by 90%.\"}],[\"$\",\"p\",null,{\"children\":\"Here are some examples of how NULLs will work in Pinot 1.0.\"}],[\"$\",\"h3\",null,{\"id\":\"aggregations\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#aggregations\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Aggregations\"]}],[\"$\",\"p\",null,{\"children\":\"Given the following table below, aggregating columns with NULL values will have this behavior.\"}],[\"$\",\"div\",null,{\"className\":\"w-full overflow-x-auto\",\"children\":[\"$\",\"table\",null,{\"children\":[[\"$\",\"thead\",null,{\"children\":[\"$\",\"tr\",null,{\"children\":[[\"$\",\"th\",null,{\"children\":\"col1\"}],[\"$\",\"th\",null,{\"children\":\"col2\"}]]}]}],[\"$\",\"tbody\",null,{\"children\":[[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"1\"}],[\"$\",\"td\",null,{\"children\":\"NULL\"}]]}],[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"2\"}],[\"$\",\"td\",null,{\"children\":\"NULL\"}]]}],[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"3\"}],[\"$\",\"td\",null,{\"children\":\"1\"}]]}]]}]]}]}],[\"$\",\"p\",null,{\"children\":\"Since col1 does not contain NULL values, all the values are included in the aggregation.\"}],[\"$\",\"$L14\",null,{\"className\":\"language-sql\",\"children\":[\"$\",\"code\",null,{\"className\":\"language-sql code-highlight\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"sum\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"col1\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token comment\",\"children\":\"-- returns 6\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"count\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"col1\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token comment\",\"children\":\"-- returns 3\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":\"In the select statement below, the NULL values in col2 are not included in the aggregation.\"}],[\"$\",\"$L14\",null,{\"className\":\"language-sql\",\"children\":[\"$\",\"code\",null,{\"className\":\"language-sql code-highlight\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"sum\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"col2\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token comment\",\"children\":\"-- returns 1\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token function\",\"children\":\"count\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],\"col2\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token comment\",\"children\":\"-- returns 1\"}],\"\\n\"]}]]}]}],[\"$\",\"h3\",null,{\"id\":\"group-by\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#group-by\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Group By\"]}],[\"$\",\"p\",null,{\"children\":\"Pinot now supports grouping by NULL. In the example below, we are grouping by col1 which contains a NULL value. Given the table below, grouping by columns with NULL value will have this behavior.\"}],[\"$\",\"div\",null,{\"className\":\"w-full overflow-x-auto\",\"children\":[\"$\",\"table\",null,{\"children\":[[\"$\",\"thead\",null,{\"children\":[\"$\",\"tr\",null,{\"children\":[\"$\",\"th\",null,{\"children\":\"col1\"}]}]}],[\"$\",\"tbody\",null,{\"children\":[[\"$\",\"tr\",null,{\"children\":[\"$\",\"td\",null,{\"children\":\"a\"}]}],[\"$\",\"tr\",null,{\"children\":[\"$\",\"td\",null,{\"children\":\"NULL\"}]}],[\"$\",\"tr\",null,{\"children\":[\"$\",\"td\",null,{\"children\":\"b\"}]}],[\"$\",\"tr\",null,{\"children\":[\"$\",\"td\",null,{\"children\":\"a\"}]}]]}]]}]}],[\"$\",\"p\",null,{\"children\":\"The following select statement will output the following result.\"}],[\"$\",\"p\",null,{\"children\":\"select col1, count(*) from table group by col1\"}],[\"$\",\"div\",null,{\"className\":\"w-full overflow-x-auto\",\"children\":[\"$\",\"table\",null,{\"children\":[[\"$\",\"thead\",null,{\"children\":[\"$\",\"tr\",null,{\"children\":[[\"$\",\"th\",null,{\"children\":\"col1\"}],[\"$\",\"th\",null,{\"children\":\"count()\"}]]}]}],[\"$\",\"tbody\",null,{\"children\":[[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"a\"}],[\"$\",\"td\",null,{\"children\":\"2\"}]]}],[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"b\"}],[\"$\",\"td\",null,{\"children\":\"1\"}]]}],[\"$\",\"tr\",null,{\"children\":[[\"$\",\"td\",null,{\"children\":\"NULL\"}],[\"$\",\"td\",null,{\"children\":\"1\"}]]}]]}]]}]}],[\"$\",\"h3\",null,{\"id\":\"sorting\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#sorting\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Sorting\"]}],[\"$\",\"p\",null,{\"children\":\"Pinot now allows you to specify the location of NULL values when sorting records. The default is to act as though NULLs are larger than non-NULLs.\"}],[\"$\",\"p\",null,{\"children\":\"Given this list of values, sorting them will result in the following.\"}],[\"$\",\"p\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"values: 1, 2, 3, NULL\"}]}],[\"$\",\"p\",null,{\"children\":\"Example 1:\"}],[\"$\",\"p\",null,{\"children\":\"NULL values sort BEFORE all non-NULL values.\"}],[\"$\",\"p\",null,{\"children\":\"SQL:\"}],[\"$\",\"$L14\",null,{\"className\":\"language-sql\",\"children\":[\"$\",\"code\",null,{\"className\":\"language-sql code-highlight\",\"children\":[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" col \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"from\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"table\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"order\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"by\"}],\" col NULLS \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"FIRST\"}],\"\\n\"]}]}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"RESULT: NULL, 1, 2, 3\"}]}],[\"$\",\"p\",null,{\"children\":\"Example 2:\"}],[\"$\",\"p\",null,{\"children\":\"NULL values sort AFTER all non-NULL values.\"}],[\"$\",\"p\",null,{\"children\":\"SQL:\"}],[\"$\",\"$L14\",null,{\"className\":\"language-sql\",\"children\":[\"$\",\"code\",null,{\"className\":\"language-sql code-highlight\",\"children\":[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" col \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"from\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"table\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"order\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"by\"}],\" col \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"ASC\"}],\" NULLS \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"LAST\"}],\"\\n\"]}]}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"RESULT: 1, 2, 3, NULL\"}]}],[\"$\",\"p\",null,{\"children\":\"Example 3:\"}],[\"$\",\"p\",null,{\"children\":\"Default behavior is NULL LAST.\"}],[\"$\",\"p\",null,{\"children\":\"SQL:\"}],[\"$\",\"$L14\",null,{\"className\":\"language-sql\",\"children\":[\"$\",\"code\",null,{\"className\":\"language-sql code-highlight\",\"children\":[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"select\"}],\" col \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"from\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"table\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"order\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"by\"}],\" col\\n\"]}]}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"code\",null,{\"children\":\"RESULT: 1, 2, 3, NULL\"}]}],[\"$\",\"h2\",null,{\"id\":\"index-pluggability\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#index-pluggability\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Index Pluggability\"]}],[\"$\",\"p\",null,{\"children\":\"Today, Pinot supports multiple index types, like forward index, bloom filter, and range index. Before Pinot 1.0, index types were all statically defined, which means that in order to create a new index type, you’d need to rebuild Pinot from source. Ideally that shouldn’t be the case.\"}],[\"$\",\"p\",null,{\"children\":[\"To increase speed of development, \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot/issues/10183\",\"children\":\"Index Service Provider Interface (SPI)\"}],\", or index-spi, reduces friction by adding the ability to include new index types at runtime in Pinot. This opens the ability of adding third party indexes by including an external jar in the classpath and adding some configuration. This opens up Pinot indexing to lower-friction innovation from the community.\"]}],[\"$\",\"p\",null,{\"children\":\"For now, SPI-accessible indexes are limited to single field index types. Features like the star-tree index or other multi-column approaches are not yet supported.\"}],[\"$\",\"h2\",null,{\"id\":\"apache-pinot-spark-3-connector-and-passing-pinot-options\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#apache-pinot-spark-3-connector-and-passing-pinot-options\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Apache Pinot Spark 3 Connector and Passing Pinot Options\"]}],[\"$\",\"p\",null,{\"children\":[\"Apache Spark users can now take advantage of Pinot’s ability to provide high scalability, low latency, and high concurrency within the context of a Spark 3 cluster using the \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/README.md\",\"children\":\"Apache Pinot Spark 3 Connector\"}],\".\"]}],[\"$\",\"p\",null,{\"children\":\"This connector supports Apache Spark (2.x and 3.x) as a processor to create and push segment files to the database and can read realtime, offline, and hybrid tables from Pinot.\"}],[\"$\",\"p\",null,{\"children\":\"Now you can merge your streaming and batch datasets together in Spark to provide a full view of real-time and historical data for your machine learning algorithms and feature stores.\"}],[\"$\",\"p\",null,{\"children\":\"Performance Features\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"Distributed, parallel scan\"}],[\"$\",\"li\",null,{\"children\":\"Streaming reads using gRPC (optional)\"}],[\"$\",\"li\",null,{\"children\":\"Column and filter push down to optimize performance\"}],[\"$\",\"li\",null,{\"children\":\"Support for Pinot’s Query Options that include: maxExecutionThreads, enableNullHandling, skipUpsert, etc.\"}]]}],[\"$\",\"p\",null,{\"children\":\"Usability Features\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"SQL support instead of PQL\"}],[\"$\",\"li\",null,{\"children\":\"Overlap between realtime and offline segments is queried exactly once for hybrid tables\"}],[\"$\",\"li\",null,{\"children\":[\"Schema discovery - If schema is not specified, the \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/documentation/read_model.md\",\"children\":\"connector reads the table schema\"}],\" from the Pinot controller, and then converts to the Spark schema.\"]}]]}],[\"$\",\"p\",null,{\"children\":[\"Here is an example that reads a Pinot table, by setting the format to “pinot” spark will automatically load the Pinot connector and read the “airlinesStats” table. The queryOptions property allows you to provide \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/users/user-guide-query/query-options\",\"children\":\"Pinot Query Options\"}],\".\"]}],[\"$\",\"$L14\",null,{\"className\":\"language-scala\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-scala\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token keyword\",\"children\":\"val\"}],\" data \",[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\"=\"}],\" spark\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"read\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"format\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"pinot\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"option\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"table\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"airlineStats\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"option\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"tableType\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"offline\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"option\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"queryOptions\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"enableNullHandling=true,maxExecutionThreads=1\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"load\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"sql\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"SELECT * FROM airlineStats WHERE DEST = ‘SFO’\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\"\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\"data\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\".\"}],\"show\",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"(\"}],[\"$\",\"span\",null,{\"className\":\"token number\",\"children\":\"100\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\")\"}],\"\\n\"]}]]}]}],[\"$\",\"h2\",null,{\"id\":\"petabyte-scale-log-storage-and-search-in-pinot-with-clp\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#petabyte-scale-log-storage-and-search-in-pinot-with-clp\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Petabyte-Scale Log Storage and Search in Pinot with CLP\"]}],[\"$\",\"p\",null,{\"children\":[\"Compressed Log Processor (CLP) is a tool capable of losslessly compressing text logs and searching them in their compressed state. It achieves a better compression ratio than general purpose compressors alone, while retaining the ability to search the compressed log events without incurring the performance penalty of fully decompressing them. Part of CLP’s algorithm was deployed within \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.uber.com/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/\",\"children\":\"Uber\"}],\" to compress unstructured Spark logs, as they are generated, achieving an unprecedented compression of 169×.\"]}],[\"$\",\"p\",null,{\"children\":\"Log events generated as JSON objects with user-defined schemas, meaning each event may have different keys. Such user-defined schemas make these events challenging to store in a table with a set schema. With Log Storage and Search in Pinot with CLP, users would be able to:\"}],[\"$\",\"ul\",null,{\"children\":[[\"$\",\"li\",null,{\"children\":\"Store their log events losslessly (without dropping fields)\"}],[\"$\",\"li\",null,{\"children\":\"Store their logs with some amount of compression\"}],[\"$\",\"li\",null,{\"children\":\"Query their logs efficiently\"}]]}],[\"$\",\"p\",null,{\"children\":\"The CLP ingestion pipeline can be used on log events from a stream, such as JSON log events ingested from Kafka. The plugin takes two inputs: a JSON record and a list of fields to encode with CLP.\"}],[\"$\",\"p\",null,{\"children\":\"The fields to encode can be configured as shown:\"}],[\"$\",\"$L14\",null,{\"className\":\"language-json\",\"children\":[\"$\",\"code\",null,{\"className\":\"code-highlight language-json\",\"children\":[[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\" ...\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"tableIndexConfig\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\" ...\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"streamConfigs\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"{\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":\" ...\\n\"}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.decoder.class.name\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"org.apache.pinot.plugin.inputformat.clplog.CLPLogMessageDecoder\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\",\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token property\",\"children\":\"\\\"stream.kafka.decoder.prop.fieldsForClpEncoding\\\"\"}],[\"$\",\"span\",null,{\"className\":\"token operator\",\"children\":\":\"}],\" \",[\"$\",\"span\",null,{\"className\":\"token string\",\"children\":\"\\\"\u003cfield-name-1\u003e,\u003cfield-name-2\u003e\\\"\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[\" \",[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}],[\"$\",\"span\",null,{\"className\":\"code-line\",\"children\":[[\"$\",\"span\",null,{\"className\":\"token punctuation\",\"children\":\"}\"}],\"\\n\"]}]]}]}],[\"$\",\"p\",null,{\"children\":[[\"$\",\"code\",null,{\"children\":\"\u003cfield-names-1 and 2\u003e\"}],\" are a comma-separated list of fields you wish to encode with CLP.\"]}],[\"$\",\"p\",null,{\"children\":[\"You can read the design \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU/edit\",\"children\":\"document\"}],\" for more details into why and how this feature was implemented.\"]}],[\"$\",\"h2\",null,{\"id\":\"summary\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#summary\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Summary\"]}],[\"$\",\"p\",null,{\"children\":[\"Apache Pinot’s evolution is expressly due to the humans behind the code, and in reaching 1.0 release status it is proper and fitting to give credit to the open source project’s key committers. Since its early days, over \",[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://github.com/apache/pinot/graphs/contributors\",\"children\":\"three hundred contributors\"}],\" have produced more than 1.3 million source lines of code (SLOC).\"]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"img\",null,{\"alt\":\"Image test\",\"src\":\"https://pinot.apache.org/blogs/apache-pinot-1-0-name-cloud.png\"}]}],[\"$\",\"p\",null,{\"children\":\"The introduction of Apache Pinot 1.0 represents an exceptional stride forward in real-time online analytical processing (OLAP) capabilities, marking a watershed moment in the evolution of real-time analytics systems. This release redefines the limits of what can be achieved in the realm of instant data analysis, presenting a game-changing solution for organizations seeking high throughput and low latency in their OLAP queries. If you would like to get started with Apache Pinot 1.0, you can check out the documentation, and download it now.\"}],[\"$\",\"h2\",null,{\"id\":\"resources\",\"children\":[[\"$\",\"a\",null,{\"href\":\"#resources\",\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],\"Resources\"]}],[\"$\",\"p\",null,{\"children\":\"If you want to try out Apache Pinot, the following resources will help you get started:\"}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://pinot.apache.org/download/\",\"children\":\"Download page\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://docs.pinot.apache.org/getting-started\",\"children\":\"Getting started\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://communityinviter.com/apps/apache-pinot/apache-pinot\",\"children\":\"Join our Slack channel\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://www.meetup.com/apache-pinot\",\"children\":\"See our upcoming events\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"target\":\"_blank\",\"rel\":\"noopener noreferrer\",\"href\":\"https://twitter.com/ApachePinot\",\"children\":\"Follow us on social media\"}]}]]}]}]}]]}],[\"$\",\"aside\",null,{\"className\":\"mt-10 hidden border-l-2 pl-5 lg:sticky lg:top-1 lg:block lg:h-full\",\"children\":[\"$\",\"section\",null,{\"className\":\"sticky top-0 mb-4 w-[15.375rem]\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex flex-col space-y-1.5 pb-3\",\"children\":[\"$\",\"h3\",null,{\"className\":\"text-sm font-semibold leading-snug text-neutral-500 dark:text-neutral-100\",\"children\":\"Table of Contents\"}]}],[\"$\",\"$L15\",null,{\"chapters\":[{\"value\":\"What Makes a “1.0 Release?”\",\"url\":\"#what-makes-a-10-release\",\"depth\":2},{\"value\":\"Join Support\",\"url\":\"#join-support\",\"depth\":2},{\"value\":\"Upsert Improvements\",\"url\":\"#upsert-improvements\",\"depth\":2},{\"value\":\"Segment Compaction for Upsert Tables\",\"url\":\"#segment-compaction-for-upsert-tables\",\"depth\":3},{\"value\":\"DELETE Support for Upsert Tables\",\"url\":\"#delete-support-for-upsert-tables\",\"depth\":3},{\"value\":\"NULL Value Support\",\"url\":\"#null-value-support\",\"depth\":2},{\"value\":\"Aggregations\",\"url\":\"#aggregations\",\"depth\":3},{\"value\":\"Group By\",\"url\":\"#group-by\",\"depth\":3},{\"value\":\"Sorting\",\"url\":\"#sorting\",\"depth\":3},{\"value\":\"Index Pluggability\",\"url\":\"#index-pluggability\",\"depth\":2},{\"value\":\"Apache Pinot Spark 3 Connector and Passing Pinot Options\",\"url\":\"#apache-pinot-spark-3-connector-and-passing-pinot-options\",\"depth\":2},{\"value\":\"Petabyte-Scale Log Storage and Search in Pinot with CLP\",\"url\":\"#petabyte-scale-log-storage-and-search-in-pinot-with-clp\",\"depth\":2},{\"value\":\"Summary\",\"url\":\"#summary\",\"depth\":2},{\"value\":\"Resources\",\"url\":\"#resources\",\"depth\":2}]}]]}]}]]}]}]]}]]\n"])</script><script>self.__next_f.push([1,"6:[[\"$\",\"meta\",\"0\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"meta\",\"1\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"2\",{\"children\":\"Announcing Apache Pinot 1.0™ | Apache Pinot™\"}],[\"$\",\"meta\",\"3\",{\"name\":\"description\",\"content\":\"Introducing Apache Pinot 1.0 Release\"}],[\"$\",\"meta\",\"4\",{\"name\":\"robots\",\"content\":\"index, follow\"}],[\"$\",\"meta\",\"5\",{\"name\":\"googlebot\",\"content\":\"index, follow, max-video-preview:-1, max-image-preview:large, max-snippet:-1\"}],[\"$\",\"link\",\"6\",{\"rel\":\"canonical\",\"href\":\"https://pinot.apache.org/blog/2023/09/19/Annoucing-Apache-Pinot-1-0\"}],[\"$\",\"link\",\"7\",{\"rel\":\"alternate\",\"type\":\"application/rss+xml\",\"href\":\"https://pinot.apache.org/feed.xml\"}],[\"$\",\"meta\",\"8\",{\"property\":\"og:title\",\"content\":\"Announcing Apache Pinot 1.0™\"}],[\"$\",\"meta\",\"9\",{\"property\":\"og:description\",\"content\":\"Introducing Apache Pinot 1.0 Release\"}],[\"$\",\"meta\",\"10\",{\"property\":\"og:url\",\"content\":\"https://pinot.apache.org/blog/2023/09/19/Annoucing-Apache-Pinot-1-0\"}],[\"$\",\"meta\",\"11\",{\"property\":\"og:site_name\",\"content\":\"Apache Pinot™\"}],[\"$\",\"meta\",\"12\",{\"property\":\"og:locale\",\"content\":\"en_US\"}],[\"$\",\"meta\",\"13\",{\"property\":\"og:image\",\"content\":\"https://pinot.apache.org/static/images/twitter-card.png\"}],[\"$\",\"meta\",\"14\",{\"property\":\"og:type\",\"content\":\"article\"}],[\"$\",\"meta\",\"15\",{\"property\":\"article:published_time\",\"content\":\"2023-09-19T00:00:00.000Z\"}],[\"$\",\"meta\",\"16\",{\"property\":\"article:modified_time\",\"content\":\"2023-09-19T00:00:00.000Z\"}],[\"$\",\"meta\",\"17\",{\"property\":\"article:author\",\"content\":\"Hubert Dulay\"}],[\"$\",\"meta\",\"18\",{\"property\":\"article:author\",\"content\":\"Mayank Shrivastava\"}],[\"$\",\"meta\",\"19\",{\"property\":\"article:author\",\"content\":\"Neha Pawar\"}],[\"$\",\"meta\",\"20\",{\"name\":\"twitter:card\",\"content\":\"summary_large_image\"}],[\"$\",\"meta\",\"21\",{\"name\":\"twitter:title\",\"content\":\"Announcing Apache Pinot 1.0™\"}],[\"$\",\"meta\",\"22\",{\"name\":\"twitter:description\",\"content\":\"Introducing Apache Pinot 1.0 Release\"}],[\"$\",\"meta\",\"23\",{\"name\":\"twitter:image\",\"content\":\"https://pinot.apache.org/static/images/twitter-card.png\"}],[\"$\",\"meta\",\"24\",{\"name\":\"next-size-adjust\"}]]\n"])</script><script>self.__next_f.push([1,"f:null\n"])</script><script>self.__next_f.push([1,""])</script></body></html> |
