| <!doctype html><html lang=en dir=ltr> |
| <head> |
| <meta charset=utf-8> |
| <meta name=viewport content="width=device-width,initial-scale=1"> |
| <meta name=description content="Java API Quickstart # Create a table # Tables are created using either a Catalog or an implementation of the Tables interface. |
| Using a Hive catalog # The Hive catalog connects to a Hive metastore to keep track of Iceberg tables. You can initialize a Hive catalog with a name and some properties. (see: Catalog properties) |
| Note: Currently, setConf is always required for hive catalogs, but this will change in the future."> |
| <meta name=theme-color content="#FFFFFF"> |
| <meta name=color-scheme content="light dark"><meta property="og:title" content="Java Quickstart"> |
| <meta property="og:description" content="Java API Quickstart # Create a table # Tables are created using either a Catalog or an implementation of the Tables interface. |
| Using a Hive catalog # The Hive catalog connects to a Hive metastore to keep track of Iceberg tables. You can initialize a Hive catalog with a name and some properties. (see: Catalog properties) |
| Note: Currently, setConf is always required for hive catalogs, but this will change in the future."> |
| <meta property="og:type" content="article"> |
| <meta property="og:url" content="https://iceberg.apache.org/docs/FixSparkArtifactVersion/java-api-quickstart/"><meta property="article:section" content="docs"> |
| <title>Java Quickstart | Apache Iceberg</title> |
| <link rel=manifest href=/docs/FixSparkArtifactVersion/manifest.json> |
| <link rel=icon href=/docs/FixSparkArtifactVersion/favicon.png type=image/x-icon> |
| <link rel=stylesheet href=/docs/FixSparkArtifactVersion/book.min.179e158d24f3ef709534173fd8b1c1e541a4fa3e23c1b5d8e887464c58949cc9.css integrity="sha256-F54VjSTz73CVNBc/2LHB5UGk+j4jwbXY6IdGTFiUnMk=" crossorigin=anonymous> |
| <script defer src=/docs/FixSparkArtifactVersion/flexsearch.min.js></script> |
| <script defer src=/docs/FixSparkArtifactVersion/en.search.min.22e67e1125667c60fd7b9b24ba8c036274712b6ce8b2ee4cf165beead757bc10.js integrity="sha256-IuZ+ESVmfGD9e5skuowDYnRxK2zosu5M8WW+6tdXvBA=" crossorigin=anonymous></script> |
| </head> |
| <body dir=ltr> |
| <input type=checkbox class="hidden toggle" id=menu-control> |
| <input type=checkbox class="hidden toggle" id=toc-control> |
| <main class="container flex"> |
| <aside class=book-menu> |
| <div class=book-menu-content> |
| <nav> |
| <h2 class=book-brand> |
| <a class="flex align-center" href=/docs/FixSparkArtifactVersion/../../><img src=/docs/FixSparkArtifactVersion/img/iceberg-logo-icon.png alt=Logo><span>Apache Iceberg</span> |
| </a> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../releases> |
| <img id=version-shield src=https://img.shields.io/badge/version-FixSparkArtifactVersion-blue alt> |
| </a> |
| </h2> |
| <div class=book-search> |
| <input type=text id=book-search-input placeholder=Search aria-label=Search maxlength=64 data-hotkeys=s/> |
| <div class="book-search-spinner hidden"></div> |
| <ul id=book-search-results></ul> |
| <a href=https://github.com/apache/iceberg target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/GitHub-Mark.png target=_blank class=top-external-icon> |
| </a> |
| <a href=https://join.slack.com/t/apache-iceberg/shared_invite/zt-tlv0zjz6-jGJEkHfb1~heMCJA3Uycrg target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/Slack_Mark_Web.png target=_blank class=top-external-icon> |
| </a> |
| </div> |
| <ul> |
| <li class=book-section-flats> |
| <span> |
| <i class="fa fa-table fa-fw"></i> |
| Tables</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/configuration/> |
| Configuration</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/evolution/> |
| Evolution</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/maintenance/> |
| Maintenance</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/partitioning/> |
| Partitioning</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/performance/> |
| Performance</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/reliability/> |
| Reliability</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/schemas/> |
| Schemas</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <span> |
| <i class="fa fa-star-o fa-fw"></i> |
| Spark</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/getting-started/> |
| Getting Started</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-configuration/> |
| Configuration</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-ddl/> |
| DDL</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-procedures/> |
| Procedures</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-queries/> |
| Queries</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-structured-streaming/> |
| Structured Streaming</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/spark-writes/> |
| Writes</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <span> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/flink-logo.png class="navigation-icon fa-fw">Flink</span> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink/> |
| Getting Started</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/flink-connector/> |
| Flink Connector</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-flats> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/hive/> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/hive-logo.png class="navigation-icon fa-fw">Hive</a> |
| <ul> |
| </ul> |
| </li> |
| <li> |
| <a href=https://trino.io/docs/current/connector/iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/trino-logo.png class="navigation-icon fa-fw"> |
| Trino |
| </a> |
| </li> |
| <li> |
| <a href=https://prestodb.io/docs/current/connector/iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/prestodb-logo.png class="navigation-icon fa-fw"> |
| Presto |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.dremio.com/data-formats/apache-iceberg/ target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/dremio-logo.png class="navigation-icon fa-fw"> |
| Dremio |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/athena-logo.png class="navigation-icon fa-fw"> |
| Amazon Athena |
| </a> |
| </li> |
| <li> |
| <a href=https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-create-cluster.html target=_blank> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/emr-logo.png class="navigation-icon fa-fw"> |
| Amazon EMR |
| </a> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-56605d8e971a871885e28ee5142728bf class=toggle> |
| <label for=section-56605d8e971a871885e28ee5142728bf class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-handshake-o fa-fw"></i> |
| Integrations</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/aws/> |
| AWS</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/jdbc/> |
| JDBC</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/nessie/> |
| Nessie</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-bf7b3283e3790c00c8caaa140299052b class=toggle checked> |
| <label for=section-bf7b3283e3790c00c8caaa140299052b class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-connectdevelop fa-fw"></i> |
| API</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/java-api-quickstart/ class=active> |
| Java Quickstart</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/api/> |
| Java API</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/custom-catalog/> |
| Java Custom Catalog</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../javadoc/FixSparkArtifactVersion> |
| Javadocs |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-quickstart/> |
| Python Quickstart</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-api-intro/> |
| Python API</a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/python-feature-support/> |
| Python Feature Support</a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-7e66f1754ca5d93e20ecdc89df5b8b28 class=toggle> |
| <label for=section-7e66f1754ca5d93e20ecdc89df5b8b28 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-users fa-fw"></i> |
| Community</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../blogs> |
| Blogs |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../community> |
| Join |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../talks> |
| Talks |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-87dda23e9104fe3231cee3bc88a2d754 class=toggle> |
| <label for=section-87dda23e9104fe3231cee3bc88a2d754 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-object-ungroup fa-fw"></i> |
| Format</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../spec> |
| Spec |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../terms> |
| Terms |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-2e5d3f5f142758d8dd368e9c281dd08e class=toggle> |
| <label for=section-2e5d3f5f142758d8dd368e9c281dd08e class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-wrench fa-fw"></i> |
| Project</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../how-to-release> |
| How to Release |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../roadmap> |
| Roadmap |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../security> |
| Security |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../trademarks> |
| Trademarks |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-4ddb27a8612bc8118c0b36386905d332 class=toggle> |
| <label for=section-4ddb27a8612bc8118c0b36386905d332 class="flex justify-between"> |
| <a role=button> |
| <i class="fa fa-code-fork fa-fw"></i> |
| Releases</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../latest> |
| Latest |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.13.0> |
| 0.13.0 |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../0.12.1> |
| 0.12.1 |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://iceberg.apache.org/docs/FixSparkArtifactVersion/../../../releases> |
| Release Notes |
| </a> |
| </li> |
| </ul> |
| </li> |
| <li class=book-section-collapsed> |
| <input type=checkbox id=section-296746d27808aa768e500824aaf2adea class=toggle> |
| <label for=section-296746d27808aa768e500824aaf2adea class="flex justify-between"> |
| <a role=button> |
| <img src=https://iceberg.apache.org/docs/FixSparkArtifactVersion/img/../img/asf.png class="navigation-icon fa-fw">ASF</a> |
| </label> |
| <ul> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/licenses/ target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| License |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/security/ target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Security |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/foundation/thanks.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Sponsors |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/foundation/sponsorship.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Donate |
| </a> |
| </li> |
| <li class=navigation-icon-pad> |
| <a href=https://www.apache.org/events/current-event.html target=_blank> |
| <i class="fa fa-external-link fa-fw"></i> |
| Events |
| </a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| <script>(function(){var a=document.querySelector("aside .book-menu-content");addEventListener("beforeunload",function(b){localStorage.setItem("menu.scrollTop",a.scrollTop)}),a.scrollTop=localStorage.getItem("menu.scrollTop")})()</script> |
| </div> |
| </aside> |
| <div class=book-page> |
| <header class=book-header> |
| <div class="flex align-center justify-between"> |
| <link rel=stylesheet href=/docs/FixSparkArtifactVersion/fontawesome/css/font-awesome.min.css> |
| <label for=menu-control> |
| <img src=/docs/FixSparkArtifactVersion/svg/menu.svg class=book-icon alt=Menu> |
| </label> |
| <strong>Java Quickstart</strong> |
| <label for=toc-control> |
| <img src=/docs/FixSparkArtifactVersion/svg/toc.svg class=book-icon alt="Table of Contents"> |
| </label> |
| </div> |
| <aside class="hidden clearfix"> |
| <nav id=TableOfContents> |
| <ul> |
| <li><a href=#create-a-table>Create a table</a> |
| <ul> |
| <li><a href=#using-a-hive-catalog>Using a Hive catalog</a></li> |
| <li><a href=#using-a-hadoop-catalog>Using a Hadoop catalog</a></li> |
| <li><a href=#using-hadoop-tables>Using Hadoop tables</a></li> |
| <li><a href=#tables-in-spark>Tables in Spark</a></li> |
| </ul> |
| </li> |
| <li><a href=#schemas>Schemas</a> |
| <ul> |
| <li><a href=#create-a-schema>Create a schema</a></li> |
| <li><a href=#convert-a-schema-from-avro>Convert a schema from Avro</a></li> |
| <li><a href=#convert-a-schema-from-spark>Convert a schema from Spark</a></li> |
| </ul> |
| </li> |
| <li><a href=#partitioning>Partitioning</a> |
| <ul> |
| <li><a href=#create-a-partition-spec>Create a partition spec</a></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| </aside> |
| </header> |
| <article class=markdown> |
| <h1 id=java-api-quickstart> |
| Java API Quickstart |
| <a class=anchor href=#java-api-quickstart>#</a> |
| </h1> |
| <h2 id=create-a-table> |
| Create a table |
| <a class=anchor href=#create-a-table>#</a> |
| </h2> |
| <p>Tables are created using either a <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/catalog/Catalog.html><code>Catalog</code></a> or an implementation of the <a href=../../../javadoc/FixSparkArtifactVersion/index.html?org/apache/iceberg/Tables.html><code>Tables</code></a> interface.</p> |
| <h3 id=using-a-hive-catalog> |
| Using a Hive catalog |
| <a class=anchor href=#using-a-hive-catalog>#</a> |
| </h3> |
| <p>The Hive catalog connects to a Hive metastore to keep track of Iceberg tables. |
| You can initialize a Hive catalog with a name and some properties. |
| (see: <a href=../configuration/#catalog-properties>Catalog properties</a>)</p> |
| <p><strong>Note:</strong> Currently, <code>setConf</code> is always required for hive catalogs, but this will change in the future.</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.hive.HiveCatalog<span style=color:#f92672>;</span> |
| |
| Catalog catalog <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> HiveCatalog<span style=color:#f92672>();</span> |
| catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>setConf</span><span style=color:#f92672>(</span>spark<span style=color:#f92672>.</span><span style=color:#a6e22e>sparkContext</span><span style=color:#f92672>().</span><span style=color:#a6e22e>hadoopConfiguration</span><span style=color:#f92672>());</span> <span style=color:#75715e>// Configure using Spark's Hadoop configuration |
| </span><span style=color:#75715e></span> |
| Map <span style=color:#f92672><</span>String<span style=color:#f92672>,</span> String<span style=color:#f92672>></span> properties <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> HashMap<span style=color:#f92672><</span>String<span style=color:#f92672>,</span> String<span style=color:#f92672>>();</span> |
| properties<span style=color:#f92672>.</span><span style=color:#a6e22e>put</span><span style=color:#f92672>(</span><span style=color:#e6db74>"warehouse"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"..."</span><span style=color:#f92672>);</span> |
| properties<span style=color:#f92672>.</span><span style=color:#a6e22e>put</span><span style=color:#f92672>(</span><span style=color:#e6db74>"uri"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"..."</span><span style=color:#f92672>);</span> |
| |
| catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>initialize</span><span style=color:#f92672>(</span><span style=color:#e6db74>"hive"</span><span style=color:#f92672>,</span> properties<span style=color:#f92672>);</span> |
| </code></pre></div><p>The <code>Catalog</code> interface defines methods for working with tables, like <code>createTable</code>, <code>loadTable</code>, <code>renameTable</code>, and <code>dropTable</code>.</p> |
| <p>To create a table, pass an <code>Identifier</code> and a <code>Schema</code> along with other initial metadata:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.Table<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.catalog.TableIdentifier<span style=color:#f92672>;</span> |
| |
| TableIdentifier name <span style=color:#f92672>=</span> TableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span><span style=color:#e6db74>"logging"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"logs"</span><span style=color:#f92672>);</span> |
| Table table <span style=color:#f92672>=</span> catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>createTable</span><span style=color:#f92672>(</span>name<span style=color:#f92672>,</span> schema<span style=color:#f92672>,</span> spec<span style=color:#f92672>);</span> |
| |
| <span style=color:#75715e>// or to load an existing table, use the following line |
| </span><span style=color:#75715e>// Table table = catalog.loadTable(name); |
| </span></code></pre></div><p>The logs <a href=#create-a-schema>schema</a> and <a href=#create-a-partition-spec>partition spec</a> are created below.</p> |
| <h3 id=using-a-hadoop-catalog> |
| Using a Hadoop catalog |
| <a class=anchor href=#using-a-hadoop-catalog>#</a> |
| </h3> |
| <p>A Hadoop catalog doesn’t need to connect to a Hive MetaStore, but can only be used with HDFS or similar file systems that support atomic rename. Concurrent writes with a Hadoop catalog are not safe with a local FS or S3. To create a Hadoop catalog:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.hadoop.conf.Configuration<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.hadoop.HadoopCatalog<span style=color:#f92672>;</span> |
| |
| Configuration conf <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Configuration<span style=color:#f92672>();</span> |
| String warehousePath <span style=color:#f92672>=</span> <span style=color:#e6db74>"hdfs://host:8020/warehouse_path"</span><span style=color:#f92672>;</span> |
| HadoopCatalog catalog <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> HadoopCatalog<span style=color:#f92672>(</span>conf<span style=color:#f92672>,</span> warehousePath<span style=color:#f92672>);</span> |
| </code></pre></div><p>Like the Hive catalog, <code>HadoopCatalog</code> implements <code>Catalog</code>, so it also has methods for working with tables, like <code>createTable</code>, <code>loadTable</code>, and <code>dropTable</code>.</p> |
| <p>This example creates a table with Hadoop catalog:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.Table<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.catalog.TableIdentifier<span style=color:#f92672>;</span> |
| |
| TableIdentifier name <span style=color:#f92672>=</span> TableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span><span style=color:#e6db74>"logging"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"logs"</span><span style=color:#f92672>);</span> |
| Table table <span style=color:#f92672>=</span> catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>createTable</span><span style=color:#f92672>(</span>name<span style=color:#f92672>,</span> schema<span style=color:#f92672>,</span> spec<span style=color:#f92672>);</span> |
| |
| <span style=color:#75715e>// or to load an existing table, use the following line |
| </span><span style=color:#75715e>// Table table = catalog.loadTable(name); |
| </span></code></pre></div><p>The logs <a href=#create-a-schema>schema</a> and <a href=#create-a-partition-spec>partition spec</a> are created below.</p> |
| <h3 id=using-hadoop-tables> |
| Using Hadoop tables |
| <a class=anchor href=#using-hadoop-tables>#</a> |
| </h3> |
| <p>Iceberg also supports tables that are stored in a directory in HDFS. Concurrent writes with a Hadoop tables are not safe when stored in the local FS or S3. Directory tables don’t support all catalog operations, like rename, so they use the <code>Tables</code> interface instead of <code>Catalog</code>.</p> |
| <p>To create a table in HDFS, use <code>HadoopTables</code>:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.hadoop.conf.Configuration<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.hadoop.HadoopTables<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.Table<span style=color:#f92672>;</span> |
| |
| Configuration conf <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Configuration<span style=color:#f92672>();</span> |
| HadoopTables tables <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> HadoopTables<span style=color:#f92672>(</span>conf<span style=color:#f92672>);</span> |
| Table table <span style=color:#f92672>=</span> tables<span style=color:#f92672>.</span><span style=color:#a6e22e>create</span><span style=color:#f92672>(</span>schema<span style=color:#f92672>,</span> spec<span style=color:#f92672>,</span> table_location<span style=color:#f92672>);</span> |
| |
| <span style=color:#75715e>// or to load an existing table, use the following line |
| </span><span style=color:#75715e>// Table table = tables.load(table_location); |
| </span></code></pre></div><blockquote class="book-hint danger"> |
| Hadoop tables shouldn’t be used with file systems that do not support atomic rename. Iceberg relies on rename to synchronize concurrent commits for directory tables. |
| </blockquote> |
| <h3 id=tables-in-spark> |
| Tables in Spark |
| <a class=anchor href=#tables-in-spark>#</a> |
| </h3> |
| <p>Spark uses both <code>HiveCatalog</code> and <code>HadoopTables</code> to load tables. Hive is used when the identifier passed to <code>load</code> or <code>save</code> is not a path, otherwise Spark assumes it is a path-based table.</p> |
| <p>To read and write to tables from Spark see:</p> |
| <ul> |
| <li><a href=../spark-queries#querying-with-sql>SQL queries in Spark</a></li> |
| <li><a href=../spark-writes#insert-into><code>INSERT INTO</code> in Spark</a></li> |
| <li><a href=../spark-writes#merge-into><code>MERGE INTO</code> in Spark</a></li> |
| </ul> |
| <h2 id=schemas> |
| Schemas |
| <a class=anchor href=#schemas>#</a> |
| </h2> |
| <h3 id=create-a-schema> |
| Create a schema |
| <a class=anchor href=#create-a-schema>#</a> |
| </h3> |
| <p>This example creates a schema for a <code>logs</code> table:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.Schema<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.types.Types<span style=color:#f92672>;</span> |
| |
| Schema schema <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Schema<span style=color:#f92672>(</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"level"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()),</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"event_time"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>TimestampType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>withZone</span><span style=color:#f92672>()),</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>3<span style=color:#f92672>,</span> <span style=color:#e6db74>"message"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()),</span> |
| Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>4<span style=color:#f92672>,</span> <span style=color:#e6db74>"call_stack"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>ListType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span>5<span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()))</span> |
| <span style=color:#f92672>);</span> |
| </code></pre></div><p>When using the Iceberg API directly, type IDs are required. Conversions from other schema formats, like Spark, Avro, and Parquet will automatically assign new IDs.</p> |
| <p>When a table is created, all IDs in the schema are re-assigned to ensure uniqueness.</p> |
| <h3 id=convert-a-schema-from-avro> |
| Convert a schema from Avro |
| <a class=anchor href=#convert-a-schema-from-avro>#</a> |
| </h3> |
| <p>To create an Iceberg schema from an existing Avro schema, use converters in <code>AvroSchemaUtil</code>:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.avro.Schema<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.avro.Schema.Parser<span style=color:#f92672>;</span> |
| <span style=color:#f92672>import</span> org.apache.iceberg.avro.AvroSchemaUtil<span style=color:#f92672>;</span> |
| |
| Schema avroSchema <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Parser<span style=color:#f92672>().</span><span style=color:#a6e22e>parse</span><span style=color:#f92672>(</span><span style=color:#e6db74>"{\"type\": \"record\" , ... }"</span><span style=color:#f92672>);</span> |
| Schema icebergSchema <span style=color:#f92672>=</span> AvroSchemaUtil<span style=color:#f92672>.</span><span style=color:#a6e22e>toIceberg</span><span style=color:#f92672>(</span>avroSchema<span style=color:#f92672>);</span> |
| </code></pre></div><h3 id=convert-a-schema-from-spark> |
| Convert a schema from Spark |
| <a class=anchor href=#convert-a-schema-from-spark>#</a> |
| </h3> |
| <p>To create an Iceberg schema from an existing table, use converters in <code>SparkSchemaUtil</code>:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.spark.SparkSchemaUtil<span style=color:#f92672>;</span> |
| |
| Schema schema <span style=color:#f92672>=</span> SparkSchemaUtil<span style=color:#f92672>.</span><span style=color:#a6e22e>schemaForTable</span><span style=color:#f92672>(</span>sparkSession<span style=color:#f92672>,</span> table_name<span style=color:#f92672>);</span> |
| </code></pre></div><h2 id=partitioning> |
| Partitioning |
| <a class=anchor href=#partitioning>#</a> |
| </h2> |
| <h3 id=create-a-partition-spec> |
| Create a partition spec |
| <a class=anchor href=#create-a-partition-spec>#</a> |
| </h3> |
| <p>Partition specs describe how Iceberg should group records into data files. Partition specs are created for a table’s schema using a builder.</p> |
| <p>This example creates a partition spec for the <code>logs</code> table that partitions records by the hour of the log event’s timestamp and by log level:</p> |
| <div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=color:#f92672>import</span> org.apache.iceberg.PartitionSpec<span style=color:#f92672>;</span> |
| |
| PartitionSpec spec <span style=color:#f92672>=</span> PartitionSpec<span style=color:#f92672>.</span><span style=color:#a6e22e>builderFor</span><span style=color:#f92672>(</span>schema<span style=color:#f92672>)</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>hour</span><span style=color:#f92672>(</span><span style=color:#e6db74>"event_time"</span><span style=color:#f92672>)</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>identity</span><span style=color:#f92672>(</span><span style=color:#e6db74>"level"</span><span style=color:#f92672>)</span> |
| <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span> |
| </code></pre></div><p>For more information on the different partition transforms that Iceberg offers, visit <a href=../../../spec#partitioning>this page</a>.</p> |
| </article> |
| <footer class=book-footer> |
| <div class="flex flex-wrap justify-between"> |
| </div> |
| <script>(function(){function a(c){const a=window.getSelection(),b=document.createRange();b.selectNodeContents(c),a.removeAllRanges(),a.addRange(b)}document.querySelectorAll("pre code").forEach(b=>{b.addEventListener("click",function(c){a(b.parentElement),navigator.clipboard&&navigator.clipboard.writeText(b.parentElement.textContent)})})})()</script> |
| </footer> |
| <div class=book-comments> |
| </div> |
| <label for=menu-control class="hidden book-menu-overlay"></label> |
| </div> |
| <aside class=book-toc> |
| <div class=book-toc-content> |
| <nav id=TableOfContents> |
| <ul> |
| <li><a href=#create-a-table>Create a table</a> |
| <ul> |
| <li><a href=#using-a-hive-catalog>Using a Hive catalog</a></li> |
| <li><a href=#using-a-hadoop-catalog>Using a Hadoop catalog</a></li> |
| <li><a href=#using-hadoop-tables>Using Hadoop tables</a></li> |
| <li><a href=#tables-in-spark>Tables in Spark</a></li> |
| </ul> |
| </li> |
| <li><a href=#schemas>Schemas</a> |
| <ul> |
| <li><a href=#create-a-schema>Create a schema</a></li> |
| <li><a href=#convert-a-schema-from-avro>Convert a schema from Avro</a></li> |
| <li><a href=#convert-a-schema-from-spark>Convert a schema from Spark</a></li> |
| </ul> |
| </li> |
| <li><a href=#partitioning>Partitioning</a> |
| <ul> |
| <li><a href=#create-a-partition-spec>Create a partition spec</a></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| </div> |
| </aside> |
| </main> |
| </body> |
| </html> |