| <!doctype html><html lang=en dir=ltr> |
| <head> |
| <meta charset=utf-8> |
| <meta name=viewport content="width=device-width,initial-scale=1"> |
| <meta name=description content="Iceberg Java API # Tables # The main purpose of the Iceberg API is to manage table metadata, like schema, partition spec, metadata, and data files that store table data. |
| Table metadata and operations are accessed through the Table interface. This interface will return table information. |
| Table metadata # The Table interface provides access to the table metadata: |
| schema returns the current table schema spec returns the current table partition spec properties returns a map of key-value properties currentSnapshot returns the current table snapshot snapshots returns all valid snapshots for the table snapshot(id) returns a specific snapshot by ID location returns the table’s base location Tables also provide refresh to update the table to the latest version, and expose helpers:"> |
| <meta name=theme-color content="#FFFFFF"> |
| <meta name=color-scheme content="light dark"><meta property="og:title" content="Java API"> |
| <meta property="og:description" content="Iceberg Java API # Tables # The main purpose of the Iceberg API is to manage table metadata, like schema, partition spec, metadata, and data files that store table data. |
| Table metadata and operations are accessed through the Table interface. This interface will return table information. |
| Table metadata # The Table interface provides access to the table metadata: |
| schema returns the current table schema spec returns the current table partition spec properties returns a map of key-value properties currentSnapshot returns the current table snapshot snapshots returns all valid snapshots for the table snapshot(id) returns a specific snapshot by ID location returns the table’s base location Tables also provide refresh to update the table to the latest version, and expose helpers:"> |
| <meta property="og:type" content="article"> |
| <meta property="og:url" content="https://iceberg.apache.org/docs/FixSparkArtifactVersion/api/"><meta property="article:section" content="docs"> |
| <title>Java API | Apache Iceberg</title> |
| <link rel=manifest href=/docs/FixSparkArtifactVersion/manifest.json> |
| <link rel=icon href=/docs/FixSparkArtifactVersion/favicon.png type=image/x-icon> |
| <link rel=stylesheet href=/docs/FixSparkArtifactVersion/book.min.179e158d24f3ef709534173fd8b1c1e541a4fa3e23c1b5d8e887464c58949cc9.css integrity="sha256-F54VjSTz73CVNBc/2LHB5UGk+j4jwbXY6IdGTFiUnMk=" crossorigin=anonymous> |
| <script defer src=/docs/FixSparkArtifactVersion/flexsearch.min.js></script> |
| <script defer src=/docs/FixSparkArtifactVersion/en.search.min.22e67e1125667c60fd7b9b24ba8c036274712b6ce8b2ee4cf165beead757bc10.js integrity="sha256-IuZ+ESVmfGD9e5skuowDYnRxK2zosu5M8WW+6tdXvBA=" crossorigin=anonymous></script> |
| </head> |
| <body dir=ltr> |
| <input type=checkbox class="hidden toggle" id=menu-control> |
| <input type=checkbox class="hidden toggle" id=toc-control> |
| <main class="container flex"> |
| <aside class=book-menu> |
| <div class=book-menu-content> |
| <nav> |
| <h2 class=book-brand> |
| <a class="flex align-center" href=/docs/FixSparkArtifactVersion/../../><img src=/docs/FixSparkArtifactVersion/img/iceberg-logo-icon.png alt=Logo><span>Apache Iceberg</span> |
| </a> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../releases> |
| <img id=version-shield src=https://img.shields.io/badge/version-FixSparkArtifactVersion-blue alt> |
| </a> |
| </h2> |
| <div class=book-search> |
| <input type=text id=book-search-input placeholder=Search aria-label=Search maxlength=64 data-hotkeys=s/> |
| <div class="book-search-spinner hidden"></div> |
| <ul id=book-search-results></ul> |
| <a href=https://github.com/apache/iceberg target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/GitHub-Mark.png target=_blank class=top-external-icon> |
| </a> |
| <a href=https://join.slack.com/t/apache-iceberg/shared_invite/zt-tlv0zjz6-jGJEkHfb1~heMCJA3Uycrg target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/Slack_Mark_Web.png target=_blank class=top-external-icon> |
| </a> |
| </div> |
| <ul> |
| <li class=book-section-flats> |
| <span> |
| <i class="fa fa-table fa-fw"></i> |
| Tables</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/configuration/> |
| Configuration</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/evolution/> |
| Evolution</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/maintenance/> |
| Maintenance</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/partitioning/> |
| Partitioning</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/performance/> |
| Performance</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/reliability/> |
| Reliability</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/schemas/> |
| Schemas</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <span> |
| <i class="fa fa-star-o fa-fw"></i> |
| Spark</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/getting-started/> |
| Getting Started</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-configuration/> |
| Configuration</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-ddl/> |
| DDL</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-procedures/> |
| Procedures</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-queries/> |
| Queries</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-structured-streaming/> |
| Structured Streaming</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-writes/> |
| Writes</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <span> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/flink-logo.png class="navigation-icon fa-fw">Flink</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink/> |
| Getting Started</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink-connector/> |
| Flink Connector</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/hive/> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/hive-logo.png class="navigation-icon fa-fw">Hive</a> |
| <ul> |
| </ul> |
| </li> |
| <li> |
| <a href=https://trino.io/docs/current/connector/iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/trino-logo.png class="navigation-icon fa-fw"> |
| Trino |
| </a> |
| </li> |
| <li> |
| <a href=https://prestodb.io/docs/current/connector/iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/prestodb-logo.png class="navigation-icon fa-fw"> |
| Presto |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.dremio.com/data-formats/apache-iceberg/ target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/dremio-logo.png class="navigation-icon fa-fw"> |
| Dremio |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/athena-logo.png class="navigation-icon fa-fw"> |
| Amazon Athena |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-create-cluster.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/emr-logo.png class="navigation-icon fa-fw"> |
| Amazon EMR |
| </a> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-56605d8e971a871885e28ee5142728bf class=toggle> |
| <label for=section-56605d8e971a871885e28ee5142728bf class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-handshake-o fa-fw"></i> |
| Integrations</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/aws/> |
| AWS</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/jdbc/> |
| JDBC</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/nessie/> |
| Nessie</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-bf7b3283e3790c00c8caaa140299052b class=toggle checked> |
| <label for=section-bf7b3283e3790c00c8caaa140299052b class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-connectdevelop fa-fw"></i> |
| API</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/java-api-quickstart/> |
| Java Quickstart</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/api/ class=active> |
| Java API</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/custom-catalog/> |
| Java Custom Catalog</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../javadoc/FixSparkArtifactVersion> |
| Javadocs |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-quickstart/> |
| Python Quickstart</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-api-intro/> |
| Python API</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-feature-support/> |
| Python Feature Support</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-7e66f1754ca5d93e20ecdc89df5b8b28 class=toggle> |
| <label for=section-7e66f1754ca5d93e20ecdc89df5b8b28 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-users fa-fw"></i> |
| Community</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../blogs> |
| Blogs |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../community> |
| Join |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../talks> |
| Talks |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-87dda23e9104fe3231cee3bc88a2d754 class=toggle> |
| <label for=section-87dda23e9104fe3231cee3bc88a2d754 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-object-ungroup fa-fw"></i> |
| Format</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../spec> |
| Spec |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../terms> |
| Terms |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-2e5d3f5f142758d8dd368e9c281dd08e class=toggle> |
| <label for=section-2e5d3f5f142758d8dd368e9c281dd08e class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-wrench fa-fw"></i> |
| Project</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../how-to-release> |
| How to Release |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../roadmap> |
| Roadmap |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../security> |
| Security |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../trademarks> |
| Trademarks |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-4ddb27a8612bc8118c0b36386905d332 class=toggle> |
| <label for=section-4ddb27a8612bc8118c0b36386905d332 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-code-fork fa-fw"></i> |
| Releases</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../latest> |
| Latest |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.13.0> |
| 0.13.0 |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.12.1> |
| 0.12.1 |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../releases> |
| Release Notes |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-296746d27808aa768e500824aaf2adea class=toggle> |
| <label for=section-296746d27808aa768e500824aaf2adea class="flex justify-between"> |
| <a role=button> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/asf.png class="navigation-icon fa-fw">ASF</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/licenses/ target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| License |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/security/ target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Security |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/foundation/thanks.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Sponsors |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/foundation/sponsorship.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Donate |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/events/current-event.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Events |
| </a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| <script>(function(){var a=document.querySelector("aside .book-menu-content");addEventListener("beforeunload",function(b){localStorage.setItem("menu.scrollTop",a.scrollTop)}),a.scrollTop=localStorage.getItem("menu.scrollTop")})()</script> |
| </div> |
| </aside> |
| <div class=book-page> |
| <header class=book-header> |
| <div class="flex align-center justify-between"> |
| <link rel=stylesheet href=/docs/FixSparkArtifactVersion/fontawesome/css/font-awesome.min.css> |
| <label for=menu-control> |
| <img src=/docs/FixSparkArtifactVersion/svg/menu.svg class=book-icon alt=Menu> |
| </label> |
| <strong>Java API</strong> |
| <label for=toc-control> |
| <img src=/docs/FixSparkArtifactVersion/svg/toc.svg class=book-icon alt="Table of Contents"> |
| </label> |
| </div> |
| <aside class="hidden clearfix"> |
| <nav id=TableOfContents> |
| <ul> |
| <li><a href=#tables>Tables</a> |
| <ul> |
| <li><a href=#table-metadata>Table metadata</a></li> |
| <li><a href=#scanning>Scanning</a></li> |
| <li><a href=#update-operations>Update operations</a></li> |
| <li><a href=#transactions>Transactions</a></li> |
| </ul> |
| </li> |
| <li><a href=#types>Types</a> |
| <ul> |
| <li><a href=#primitives>Primitives</a></li> |
| <li><a href=#nested-types>Nested types</a></li> |
| </ul> |
| </li> |
| <li><a href=#expressions>Expressions</a> |
| <ul> |
| <li><a href=#expression-binding>Expression binding</a></li> |
| <li><a href=#expression-example>Expression example</a></li> |
| </ul> |
| </li> |
| <li><a href=#modules>Modules</a></li> |
| </ul> |
| </nav> |
| </aside> |
| </header> |
| <article class=markdown> |
| <h1 id=iceberg-java-api> |
| Iceberg Java API |
| <a class=anchor href=#iceberg-java-api>#</a> |
| </h1> |
| <h2 id=tables> |
| Tables |
| <a class=anchor href=#tables>#</a> |
| </h2> |
| <p>The main purpose of the Iceberg API is to manage table metadata, like schema, partition spec, metadata, and data files that store table data.</p> |
| <p>Table metadata and operations are accessed through the <code>Table</code> interface. This interface will return table information.</p> |
| <h3 id=table-metadata> |
| Table metadata |
| <a class=anchor href=#table-metadata>#</a> |
| </h3> |
| <p>The <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/Table.html><code>Table</code> interface</a> provides access to the table metadata:</p> |
| <ul> |
| <li><code>schema</code> returns the current table <a href=../schemas>schema</a></li> |
| <li><code>spec</code> returns the current table partition spec</li> |
| <li><code>properties</code> returns a map of key-value <a href=../configuration>properties</a></li> |
| <li><code>currentSnapshot</code> returns the current table snapshot</li> |
| <li><code>snapshots</code> returns all valid snapshots for the table</li> |
| <li><code>snapshot(id)</code> returns a specific snapshot by ID</li> |
| <li><code>location</code> returns the table’s base location</li> |
| </ul> |
| <p>Tables also provide <code>refresh</code> to update the table to the latest version, and expose helpers:</p> |
| <ul> |
| <li><code>io</code> returns the <code>FileIO</code> used to read and write table files</li> |
| <li><code>locationProvider</code> returns a <code>LocationProvider</code> used to create paths for data and metadata files</li> |
| </ul> |
| <h3 id=scanning> |
| Scanning |
| <a class=anchor href=#scanning>#</a> |
| </h3> |
| <h4 id=file-level> |
| File level |
| <a class=anchor href=#file-level>#</a> |
| </h4> |
| <p>Iceberg table scans start by creating a <code>TableScan</code> object with <code>newScan</code>.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>TableScan scan <span style=color:#f92672>=</span> table<span style=color:#f92672>.</span><span style=color:#a6e22e>newScan</span><span style=color:#f92672>();</span> |
| </code></pre></div><p>To configure a scan, call <code>filter</code> and <code>select</code> on the <code>TableScan</code> to get a new <code>TableScan</code> with those changes.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>TableScan filteredScan <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span> |
| </code></pre></div><p>Calls to configuration methods create a new <code>TableScan</code> so that each <code>TableScan</code> is immutable and won’t change unexpectedly if shared across threads.</p> |
| <p>When a scan is configured, <code>planFiles</code>, <code>planTasks</code>, and <code>schema</code> are used to return files, tasks, and the read projection.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>TableScan scan <span style=color:#f92672>=</span> table<span style=color:#f92672>.</span><span style=color:#a6e22e>newScan</span><span style=color:#f92672>()</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>select</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>);</span> |
| |
| Schema projection <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>schema</span><span style=color:#f92672>();</span> |
| Iterable<span style=color:#f92672><</span>CombinedScanTask<span style=color:#f92672>></span> tasks <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>planTasks</span><span style=color:#f92672>();</span> |
| </code></pre></div><p>Use <code>asOfTime</code> or <code>useSnapshot</code> to configure the table snapshot for time travel queries.</p> |
| <h4 id=row-level> |
| Row level |
| <a class=anchor href=#row-level>#</a> |
| </h4> |
| <p>Iceberg table scans start by creating a <code>ScanBuilder</code> object with <code>IcebergGenerics.read</code>.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>ScanBuilder scanBuilder <span style=color:#f92672>=</span> IcebergGenerics<span style=color:#f92672>.</span><span style=color:#a6e22e>read</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span> |
| </code></pre></div><p>To configure a scan, call <code>where</code> and <code>select</code> on the <code>ScanBuilder</code> to get a new <code>ScanBuilder</code> with those changes.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>scanBuilder<span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span> |
| </code></pre></div><p>When a scan is configured, call method <code>build</code> to execute scan. <code>build</code> return <code>CloseableIterable<Record></code></p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>CloseableIterable<span style=color:#f92672><</span>Record<span style=color:#f92672>></span> result <span style=color:#f92672>=</span> IcebergGenerics<span style=color:#f92672>.</span><span style=color:#a6e22e>read</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span> |
| </code></pre></div><p>where <code>Record</code> is Iceberg record for iceberg-data module <code>org.apache.iceberg.data.Record</code>.</p> |
| <h3 id=update-operations> |
| Update operations |
| <a class=anchor href=#update-operations>#</a> |
| </h3> |
| <p><code>Table</code> also exposes operations that update the table. These operations use a builder pattern, <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/PendingUpdate.html><code>PendingUpdate</code></a>, that commits when <code>PendingUpdate#commit</code> is called.</p> |
| <p>For example, updating the table schema is done by calling <code>updateSchema</code>, adding updates to the builder, and finally calling <code>commit</code> to commit the pending changes to the table:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>table<span style=color:#f92672>.</span><span style=color:#a6e22e>updateSchema</span><span style=color:#f92672>()</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>addColumn</span><span style=color:#f92672>(</span><span style=color:#e6db74>"count"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>LongType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>())</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span> |
| </code></pre></div><p>Available operations to update a table are:</p> |
| <ul> |
| <li><code>updateSchema</code> – update the table schema</li> |
| <li><code>updateProperties</code> – update table properties</li> |
| <li><code>updateLocation</code> – update the table’s base location</li> |
| <li><code>newAppend</code> – used to append data files</li> |
| <li><code>newFastAppend</code> – used to append data files, will not compact metadata</li> |
| <li><code>newOverwrite</code> – used to append data files and remove files that are overwritten</li> |
| <li><code>newDelete</code> – used to delete data files</li> |
| <li><code>newRewrite</code> – used to rewrite data files; will replace existing files with new versions</li> |
| <li><code>newTransaction</code> – create a new table-level transaction</li> |
| <li><code>rewriteManifests</code> – rewrite manifest data by clustering files, for faster scan planning</li> |
| <li><code>rollback</code> – rollback the table state to a specific snapshot</li> |
| </ul> |
| <h3 id=transactions> |
| Transactions |
| <a class=anchor href=#transactions>#</a> |
| </h3> |
| <p>Transactions are used to commit multiple table changes in a single atomic operation. A transaction is used to create individual operations using factory methods, like <code>newAppend</code>, just like working with a <code>Table</code>. Operations created by a transaction are committed as a group when <code>commitTransaction</code> is called.</p> |
| <p>For example, deleting and appending a file in the same transaction:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>Transaction t <span style=color:#f92672>=</span> table<span style=color:#f92672>.</span><span style=color:#a6e22e>newTransaction</span><span style=color:#f92672>();</span> |
| |
| <span style=color:#75715e>// commit operations to the transaction |
| </span><span style=color:#75715e></span>t<span style=color:#f92672>.</span><span style=color:#a6e22e>newDelete</span><span style=color:#f92672>().</span><span style=color:#a6e22e>deleteFromRowFilter</span><span style=color:#f92672>(</span>filter<span style=color:#f92672>).</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span> |
| t<span style=color:#f92672>.</span><span style=color:#a6e22e>newAppend</span><span style=color:#f92672>().</span><span style=color:#a6e22e>appendFile</span><span style=color:#f92672>(</span>data<span style=color:#f92672>).</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span> |
| |
| <span style=color:#75715e>// commit all the changes to the table |
| </span><span style=color:#75715e></span>t<span style=color:#f92672>.</span><span style=color:#a6e22e>commitTransaction</span><span style=color:#f92672>();</span> |
| </code></pre></div><h2 id=types> |
| Types |
| <a class=anchor href=#types>#</a> |
| </h2> |
| <p>Iceberg data types are located in the <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/types/package-summary.html><code>org.apache.iceberg.types</code> package</a>.</p> |
| <h3 id=primitives> |
| Primitives |
| <a class=anchor href=#primitives>#</a> |
| </h3> |
| <p>Primitive type instances are available from static methods in each type class. Types without parameters use <code>get</code>, and types like <code>decimal</code> use factory methods:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span> <span style=color:#75715e>// int |
| </span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DoubleType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span> <span style=color:#75715e>// double |
| </span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DecimalType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span>9<span style=color:#f92672>,</span> 2<span style=color:#f92672>)</span> <span style=color:#75715e>// decimal(9, 2) |
| </span></code></pre></div><h3 id=nested-types> |
| Nested types |
| <a class=anchor href=#nested-types>#</a> |
| </h3> |
| <p>Structs, maps, and lists are created using factory methods in type classes.</p> |
| <p>Like struct fields, map keys or values and list elements are tracked as nested fields. Nested fields track <a href=../evolution#correctness>field IDs</a> and nullability.</p> |
| <p>Struct fields are created using <code>NestedField.optional</code> or <code>NestedField.required</code>. Map value and list element nullability is set in the map and list factory methods.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#75715e>// struct<1 id: int, 2 data: optional string> |
| </span><span style=color:#75715e></span>StructType struct <span style=color:#f92672>=</span> Struct<span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()),</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>())</span> |
| <span style=color:#f92672>)</span> |
| </code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#75715e>// map<1 key: int, 2 value: optional string> |
| </span><span style=color:#75715e></span>MapType map <span style=color:#f92672>=</span> MapType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofOptional</span><span style=color:#f92672>(</span> |
| 1<span style=color:#f92672>,</span> 2<span style=color:#f92672>,</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>(),</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span> |
| <span style=color:#f92672>)</span> |
| </code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#75715e>// array<1 element: int> |
| </span><span style=color:#75715e></span>ListType list <span style=color:#f92672>=</span> ListType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> IntegerType<span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>());</span> |
| </code></pre></div><h2 id=expressions> |
| Expressions |
| <a class=anchor href=#expressions>#</a> |
| </h2> |
| <p>Iceberg’s expressions are used to configure table scans. To create expressions, use the factory methods in <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/expressions/Expressions.html><code>Expressions</code></a>.</p> |
| <p>Supported predicate expressions are:</p> |
| <ul> |
| <li><code>isNull</code></li> |
| <li><code>notNull</code></li> |
| <li><code>equal</code></li> |
| <li><code>notEqual</code></li> |
| <li><code>lessThan</code></li> |
| <li><code>lessThanOrEqual</code></li> |
| <li><code>greaterThan</code></li> |
| <li><code>greaterThanOrEqual</code></li> |
| <li><code>in</code></li> |
| <li><code>notIn</code></li> |
| <li><code>startsWith</code></li> |
| <li><code>notStartsWith</code></li> |
| </ul> |
| <p>Supported expression operations are:</p> |
| <ul> |
| <li><code>and</code></li> |
| <li><code>or</code></li> |
| <li><code>not</code></li> |
| </ul> |
| <p>Constant expressions are:</p> |
| <ul> |
| <li><code>alwaysTrue</code></li> |
| <li><code>alwaysFalse</code></li> |
| </ul> |
| <h3 id=expression-binding> |
| Expression binding |
| <a class=anchor href=#expression-binding>#</a> |
| </h3> |
| <p>When created, expressions are unbound. Before an expression is used, it will be bound to a data type to find the field ID the expression name represents, and to convert predicate literals.</p> |
| <p>For example, before using the expression <code>lessThan("x", 10)</code>, Iceberg needs to determine which column <code>"x"</code> refers to and convert <code>10</code> to that column’s data type.</p> |
| <p>If the expression could be bound to the type <code>struct<1 x: long, 2 y: long></code> or to <code>struct<11 x: int, 12 y: int></code>.</p> |
| <h3 id=expression-example> |
| Expression example |
| <a class=anchor href=#expression-example>#</a> |
| </h3> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java>table<span style=color:#f92672>.</span><span style=color:#a6e22e>newScan</span><span style=color:#f92672>()</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>greaterThanOrEqual</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 10<span style=color:#f92672>))</span> |
| </code></pre></div><h2 id=modules> |
| Modules |
| <a class=anchor href=#modules>#</a> |
| </h2> |
| <p>Iceberg table support is organized in library modules:</p> |
| <ul> |
| <li><code>iceberg-common</code> contains utility classes used in other modules</li> |
| <li><code>iceberg-api</code> contains the public Iceberg API, including expressions, types, tables, and operations</li> |
| <li><code>iceberg-arrow</code> is an implementation of the Iceberg type system for reading and writing data stored in Iceberg tables using Apache Arrow as the in-memory data format</li> |
| <li><code>iceberg-aws</code> contains implementations of the Iceberg API to be used with tables stored on AWS S3 and/or for tables defined using the AWS Glue data catalog</li> |
| <li><code>iceberg-core</code> contains implementations of the Iceberg API and support for Avro data files, <strong>this is what processing engines should depend on</strong></li> |
| <li><code>iceberg-parquet</code> is an optional module for working with tables backed by Parquet files</li> |
| <li><code>iceberg-orc</code> is an optional module for working with tables backed by ORC files (<em>experimental</em>)</li> |
| <li><code>iceberg-hive-metastore</code> is an implementation of Iceberg tables backed by the Hive metastore Thrift client</li> |
| </ul> |
| <p>This project Iceberg also has modules for adding Iceberg support to processing engines and associated tooling:</p> |
| <ul> |
| <li><code>iceberg-spark2</code> is an implementation of Spark’s Datasource V2 API in 2.4 for Iceberg (use iceberg-spark-runtime for a shaded version)</li> |
| <li><code>iceberg-spark3</code> is an implementation of Spark’s Datasource V2 API in 3.0 for Iceberg (use iceberg-spark3-runtime for a shaded version)</li> |
| <li><code>iceberg-spark-3.1</code> is an implementation of Spark’s Datasource V2 API in 3.1 for Iceberg (use iceberg-spark-runtime-3.1 for a shaded version)</li> |
| <li><code>iceberg-spark-3.2</code> is an implementation of Spark’s Datasource V2 API in 3.2 for Iceberg (use iceberg-spark-runtime-3.2 for a shaded version)</li> |
| <li><code>iceberg-flink</code> is an implementation of Flink’s Table and DataStream API for Iceberg (use iceberg-flink-runtime for a shaded version)</li> |
| <li><code>iceberg-hive3</code> is an implementation of Hive 3 specific SerDe’s for Timestamp, TimestampWithZone, and Date object inspectors (use iceberg-hive-runtime for a shaded version).</li> |
| <li><code>iceberg-mr</code> is an implementation of MapReduce and Hive InputFormats and SerDes for Iceberg (use iceberg-hive-runtime for a shaded version for use with Hive)</li> |
| <li><code>iceberg-nessie</code> is a module used to integrate Iceberg table metadata history and operations with <a href=https://projectnessie.org/>Project Nessie</a></li> |
| <li><code>iceberg-data</code> is a client library used to read Iceberg tables from JVM applications</li> |
| <li><code>iceberg-pig</code> is an implementation of Pig’s LoadFunc API for Iceberg</li> |
| <li><code>iceberg-runtime</code> generates a shaded runtime jar for Spark to integrate with iceberg tables</li> |
| </ul> |
| </article> |
| <footer class=book-footer> |
| <div class="flex flex-wrap justify-between"> |
| </div> |
| <script>(function(){function a(c){const a=window.getSelection(),b=document.createRange();b.selectNodeContents(c),a.removeAllRanges(),a.addRange(b)}document.querySelectorAll("pre code").forEach(b=>{b.addEventListener("click",function(c){a(b.parentElement),navigator.clipboard&&navigator.clipboard.writeText(b.parentElement.textContent)})})})()</script> |
| </footer> |
| <div class=book-comments> |
| </div> |
| <label for=menu-control class="hidden book-menu-overlay"></label> |
| </div> |
| <aside class=book-toc> |
| <div class=book-toc-content> |
| <nav id=TableOfContents> |
| <ul> |
| <li><a href=#tables>Tables</a> |
| <ul> |
| <li><a href=#table-metadata>Table metadata</a></li> |
| <li><a href=#scanning>Scanning</a></li> |
| <li><a href=#update-operations>Update operations</a></li> |
| <li><a href=#transactions>Transactions</a></li> |
| </ul> |
| </li> |
| <li><a href=#types>Types</a> |
| <ul> |
| <li><a href=#primitives>Primitives</a></li> |
| <li><a href=#nested-types>Nested types</a></li> |
| </ul> |
| </li> |
| <li><a href=#expressions>Expressions</a> |
| <ul> |
| <li><a href=#expression-binding>Expression binding</a></li> |
| <li><a href=#expression-example>Expression example</a></li> |
| </ul> |
| </li> |
| <li><a href=#modules>Modules</a></li> |
| </ul> |
| </nav> |
| </div> |
| </aside> |
| </main> |
| </body> |
| </html> |