docs/FixSparkArtifactVersion/performance/index.html

<!doctype html><html lang=en dir=ltr>
<head>
<meta charset=utf-8>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta name=description content="Performance #   Iceberg is designed for huge tables and is used in production where a single table can contain tens of petabytes of data. Even multi-petabyte tables can be read from a single node, without needing a distributed SQL engine to sift through table metadata.  Scan planning #  Scan planning is the process of finding the files in a table that are needed for a query.">
<meta name=theme-color content="#FFFFFF">
<meta name=color-scheme content="light dark"><meta property="og:title" content>
<meta property="og:description" content="Performance #   Iceberg is designed for huge tables and is used in production where a single table can contain tens of petabytes of data. Even multi-petabyte tables can be read from a single node, without needing a distributed SQL engine to sift through table metadata.  Scan planning #  Scan planning is the process of finding the files in a table that are needed for a query.">
<meta property="og:type" content="article">
<meta property="og:url" content="https://iceberg.apache.org/docs/FixSparkArtifactVersion/performance/"><meta property="article:section" content="docs">
<title>Performance | Apache Iceberg</title>
<link rel=manifest href=/docs/FixSparkArtifactVersion/manifest.json>
<link rel=icon href=/docs/FixSparkArtifactVersion/favicon.png type=image/x-icon>
<link rel=stylesheet href=/docs/FixSparkArtifactVersion/book.min.179e158d24f3ef709534173fd8b1c1e541a4fa3e23c1b5d8e887464c58949cc9.css integrity="sha256-F54VjSTz73CVNBc/2LHB5UGk+j4jwbXY6IdGTFiUnMk=" crossorigin=anonymous>
<script defer src=/docs/FixSparkArtifactVersion/flexsearch.min.js></script>
<script defer src=/docs/FixSparkArtifactVersion/en.search.min.22e67e1125667c60fd7b9b24ba8c036274712b6ce8b2ee4cf165beead757bc10.js integrity="sha256-IuZ+ESVmfGD9e5skuowDYnRxK2zosu5M8WW+6tdXvBA=" crossorigin=anonymous></script>
</head>
<body dir=ltr>
<input type=checkbox class="hidden toggle" id=menu-control>
<input type=checkbox class="hidden toggle" id=toc-control>
<main class="container flex">
<aside class=book-menu>
<div class=book-menu-content>
<nav>
<h2 class=book-brand>
<a class="flex align-center" href=/docs/FixSparkArtifactVersion/../../><img src=/docs/FixSparkArtifactVersion/img/iceberg-logo-icon.png alt=Logo><span>Apache Iceberg</span>
</a>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../releases>
<img id=version-shield src=https://img.shields.io/badge/version-FixSparkArtifactVersion-blue alt>
</a>
</h2>
<div class=book-search>
<input type=text id=book-search-input placeholder=Search aria-label=Search maxlength=64 data-hotkeys=s/>
<div class="book-search-spinner hidden"></div>
<ul id=book-search-results></ul>
<a href=https://github.com/apache/iceberg target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/GitHub-Mark.png target=_blank class=top-external-icon>
</a>
<a href=https://join.slack.com/t/apache-iceberg/shared_invite/zt-tlv0zjz6-jGJEkHfb1~heMCJA3Uycrg target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/Slack_Mark_Web.png target=_blank class=top-external-icon>
</a>
</div>
<ul>
<li class=book-section-flats>
<span>
<i class="fa fa-table fa-fw"></i>
Tables</span>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/configuration/>
Configuration</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/evolution/>
Evolution</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/maintenance/>
Maintenance</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/partitioning/>
Partitioning</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/performance/ class=active>
Performance</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/reliability/>
Reliability</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/schemas/>
Schemas</a>
</li>
</ul>
</li>
<li class=book-section-flats>
<span>
<i class="fa fa-star-o fa-fw"></i>
Spark</span>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/getting-started/>
Getting Started</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-configuration/>
Configuration</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-ddl/>
DDL</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-procedures/>
Procedures</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-queries/>
Queries</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-structured-streaming/>
Structured Streaming</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-writes/>
Writes</a>
</li>
</ul>
</li>
<li class=book-section-flats>
<span>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/flink-logo.png class="navigation-icon fa-fw">Flink</span>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink/>
Getting Started</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink-connector/>
Flink Connector</a>
</li>
</ul>
</li>
<li class=book-section-flats>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/hive/>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/hive-logo.png class="navigation-icon fa-fw">Hive</a>
<ul>
</ul>
</li>
<li>
<a href=https://trino.io/docs/current/connector/iceberg.html target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/trino-logo.png class="navigation-icon fa-fw">
Trino
</a>
</li>
<li>
<a href=https://prestodb.io/docs/current/connector/iceberg.html target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/prestodb-logo.png class="navigation-icon fa-fw">
Presto
</a>
</li>
<li>
<a href=https://docs.dremio.com/data-formats/apache-iceberg/ target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/dremio-logo.png class="navigation-icon fa-fw">
Dremio
</a>
</li>
<li>
<a href=https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/athena-logo.png class="navigation-icon fa-fw">
Amazon Athena
</a>
</li>
<li>
<a href=https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-create-cluster.html target=_blank>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/emr-logo.png class="navigation-icon fa-fw">
Amazon EMR
</a>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-56605d8e971a871885e28ee5142728bf class=toggle>
<label for=section-56605d8e971a871885e28ee5142728bf class="flex justify-between">
<a role=button>
<i class="fa fa-handshake-o fa-fw"></i>
Integrations</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/aws/>
AWS</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/jdbc/>
JDBC</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/nessie/>
Nessie</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-bf7b3283e3790c00c8caaa140299052b class=toggle>
<label for=section-bf7b3283e3790c00c8caaa140299052b class="flex justify-between">
<a role=button>
<i class="fa fa-connectdevelop fa-fw"></i>
API</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/java-api-quickstart/>
Java Quickstart</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/api/>
Java API</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/custom-catalog/>
Java Custom Catalog</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../javadoc/FixSparkArtifactVersion>
Javadocs
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-quickstart/>
Python Quickstart</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-api-intro/>
Python API</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-feature-support/>
Python Feature Support</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-7e66f1754ca5d93e20ecdc89df5b8b28 class=toggle>
<label for=section-7e66f1754ca5d93e20ecdc89df5b8b28 class="flex justify-between">
<a role=button>
<i class="fa fa-users fa-fw"></i>
Community</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../blogs>
Blogs
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../community>
Join
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../talks>
Talks
</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-87dda23e9104fe3231cee3bc88a2d754 class=toggle>
<label for=section-87dda23e9104fe3231cee3bc88a2d754 class="flex justify-between">
<a role=button>
<i class="fa fa-object-ungroup fa-fw"></i>
Format</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../spec>
Spec
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../terms>
Terms
</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-2e5d3f5f142758d8dd368e9c281dd08e class=toggle>
<label for=section-2e5d3f5f142758d8dd368e9c281dd08e class="flex justify-between">
<a role=button>
<i class="fa fa-wrench fa-fw"></i>
Project</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../how-to-release>
How to Release
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../roadmap>
Roadmap
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../security>
Security
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../trademarks>
Trademarks
</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-4ddb27a8612bc8118c0b36386905d332 class=toggle>
<label for=section-4ddb27a8612bc8118c0b36386905d332 class="flex justify-between">
<a role=button>
<i class="fa fa-code-fork fa-fw"></i>
Releases</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../latest>
Latest
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.13.0>
0.13.0
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.12.1>
0.12.1
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../releases>
Release Notes
</a>
</li>
</ul>
</li>
<li class=book-section-collapsed>
<input type=checkbox id=section-296746d27808aa768e500824aaf2adea class=toggle>
<label for=section-296746d27808aa768e500824aaf2adea class="flex justify-between">
<a role=button>
<img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/asf.png class="navigation-icon fa-fw">ASF</a>
</label>
<ul>
<li class=navigation-icon-pad>
<a href=https://www.apache.org/licenses/ target=_blank>
<i class="fa fa-external-link fa-fw"></i>
License
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://www.apache.org/security/ target=_blank>
<i class="fa fa-external-link fa-fw"></i>
Security
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://www.apache.org/foundation/thanks.html target=_blank>
<i class="fa fa-external-link fa-fw"></i>
Sponsors
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://www.apache.org/foundation/sponsorship.html target=_blank>
<i class="fa fa-external-link fa-fw"></i>
Donate
</a>
</li>
<li class=navigation-icon-pad>
<a href=https://www.apache.org/events/current-event.html target=_blank>
<i class="fa fa-external-link fa-fw"></i>
Events
</a>
</li>
</ul>
</li>
</ul>
</nav>
<script>(function(){var a=document.querySelector("aside .book-menu-content");addEventListener("beforeunload",function(b){localStorage.setItem("menu.scrollTop",a.scrollTop)}),a.scrollTop=localStorage.getItem("menu.scrollTop")})()</script>
</div>
</aside>
<div class=book-page>
<header class=book-header>
<div class="flex align-center justify-between">
<link rel=stylesheet href=/docs/FixSparkArtifactVersion/fontawesome/css/font-awesome.min.css>
<label for=menu-control>
<img src=/docs/FixSparkArtifactVersion/svg/menu.svg class=book-icon alt=Menu>
</label>
<strong>Performance</strong>
<label for=toc-control>
<img src=/docs/FixSparkArtifactVersion/svg/toc.svg class=book-icon alt="Table of Contents">
</label>
</div>
<aside class="hidden clearfix">
<nav id=TableOfContents>
<ul>
<li><a href=#scan-planning>Scan planning</a>
<ul>
<li><a href=#metadata-filtering>Metadata filtering</a></li>
<li><a href=#data-filtering>Data filtering</a></li>
</ul>
</li>
</ul>
</nav>
</aside>
</header>
<article class=markdown>
<h1 id=performance>
Performance
<a class=anchor href=#performance>#</a>
</h1>
<ul>
<li>Iceberg is designed for huge tables and is used in production where a <em>single table</em> can contain tens of petabytes of data.</li>
<li>Even multi-petabyte tables can be read from a single node, without needing a distributed SQL engine to sift through table metadata.</li>
</ul>
<h2 id=scan-planning>
Scan planning
<a class=anchor href=#scan-planning>#</a>
</h2>
<p>Scan planning is the process of finding the files in a table that are needed for a query.</p>
<p>Planning in an Iceberg table fits on a single node because Iceberg&rsquo;s metadata can be used to prune <em>metadata</em> files that aren&rsquo;t needed, in addition to filtering <em>data</em> files that don&rsquo;t contain matching data.</p>
<p>Fast scan planning from a single node enables:</p>
<ul>
<li>Lower latency SQL queries &ndash; by eliminating a distributed scan to plan a distributed scan</li>
<li>Access from any client &ndash; stand-alone processes can read data directly from Iceberg tables</li>
</ul>
<h3 id=metadata-filtering>
Metadata filtering
<a class=anchor href=#metadata-filtering>#</a>
</h3>
<p>Iceberg uses two levels of metadata to track the files in a snapshot.</p>
<ul>
<li><strong>Manifest files</strong> store a list of data files, along each data file&rsquo;s partition data and column-level stats</li>
<li>A <strong>manifest list</strong> stores the snapshot&rsquo;s list of manifests, along with the range of values for each partition field</li>
</ul>
<p>For fast scan planning, Iceberg first filters manifests using the partition value ranges in the manifest list. Then, it reads each manifest to get data files. With this scheme, the manifest list acts as an index over the manifest files, making it possible to plan without reading all manifests.</p>
<p>In addition to partition value ranges, a manifest list also stores the number of files added or deleted in a manifest to speed up operations like snapshot expiration.</p>
<h3 id=data-filtering>
Data filtering
<a class=anchor href=#data-filtering>#</a>
</h3>
<p>Manifest files include a tuple of partition data and column-level stats for each data file.</p>
<p>During planning, query predicates are automatically converted to predicates on the partition data and applied first to filter data files. Next, column-level value counts, null counts, lower bounds, and upper bounds are used to eliminate files that cannot match the query predicate.</p>
<p>By using upper and lower bounds to filter data files at planning time, Iceberg uses clustered data to eliminate splits without running tasks. In some cases, this is a <a href=https://conferences.oreilly.com/strata/strata-ny-2018/cdn.oreillystatic.com/en/assets/1/event/278/Introducing%20Iceberg_%20Tables%20designed%20for%20object%20stores%20Presentation.pdf>10x performance improvement</a>.</p>
</article>
<footer class=book-footer>
<div class="flex flex-wrap justify-between">
</div>
<script>(function(){function a(c){const a=window.getSelection(),b=document.createRange();b.selectNodeContents(c),a.removeAllRanges(),a.addRange(b)}document.querySelectorAll("pre code").forEach(b=>{b.addEventListener("click",function(c){a(b.parentElement),navigator.clipboard&&navigator.clipboard.writeText(b.parentElement.textContent)})})})()</script>
</footer>
<div class=book-comments>
</div>
<label for=menu-control class="hidden book-menu-overlay"></label>
</div>
<aside class=book-toc>
<div class=book-toc-content>
<nav id=TableOfContents>
<ul>
<li><a href=#scan-planning>Scan planning</a>
<ul>
<li><a href=#metadata-filtering>Metadata filtering</a></li>
<li><a href=#data-filtering>Data filtering</a></li>
</ul>
</li>
</ul>
</nav>
</div>
</aside>
</main>
</body>
</html>