| {"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"about/","title":"About","text":"<p>Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.</p> <ul> <li> Learn More </li> <ul>"},{"location":"benchmarks/","title":"Benchmarks","text":""},{"location":"benchmarks/#available-benchmarks-and-how-to-run-them","title":"Available Benchmarks and how to run them","text":"<p>Benchmarks are located under <code><project-name>/jmh</code>. It is generally favorable to only run the tests of interest rather than running all available benchmarks. Also note that JMH benchmarks run within the same JVM as the system-under-test, so results might vary between runs.</p>"},{"location":"benchmarks/#running-benchmarks-on-github","title":"Running Benchmarks on GitHub","text":"<p>It is possible to run one or more Benchmarks via the JMH Benchmarks GH action on your own fork of the Iceberg repo. This GH action takes the following inputs: * The repository name where those benchmarks should be run against, such as <code>apache/iceberg</code> or <code><user>/iceberg</code> * The branch name to run benchmarks against, such as <code>master</code> or <code>my-cool-feature-branch</code> * A list of comma-separated double-quoted Benchmark names, such as <code>\"IcebergSourceFlatParquetDataReadBenchmark\", \"IcebergSourceFlatParquetDataFilterBenchmark\", \"IcebergSourceNestedListParquetDataWriteBenchmark\"</code></p> <p>Benchmark results will be uploaded once all benchmarks are done.</p> <p>It is worth noting that the GH runners have limited resources so the benchmark results should rather be seen as an indicator to guide developers in understanding code changes. It is likely that there is variability in results across different runs, therefore the benchmark results shouldn't be used to form assumptions around production choices.</p>"},{"location":"benchmarks/#running-benchmarks-locally","title":"Running Benchmarks locally","text":"<p>Below are the existing benchmarks shown with the actual commands on how to run them locally.</p>"},{"location":"benchmarks/#icebergsourcenestedlistparquetdatawritebenchmark","title":"IcebergSourceNestedListParquetDataWriteBenchmark","text":"<p>A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedListParquetDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-list-parquet-data-write-benchmark-result.txt</code></p>"},{"location":"benchmarks/#sparkparquetreadersnesteddatabenchmark","title":"SparkParquetReadersNestedDataBenchmark","text":"<p>A benchmark that evaluates the performance of reading nested Parquet data using Iceberg and Spark Parquet readers. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=SparkParquetReadersNestedDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-readers-nested-data-benchmark-result.txt</code></p>"},{"location":"benchmarks/#sparkparquetwritersflatdatabenchmark","title":"SparkParquetWritersFlatDataBenchmark","text":"<p>A benchmark that evaluates the performance of writing Parquet data with a flat schema using Iceberg and Spark Parquet writers. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=SparkParquetWritersFlatDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-writers-flat-data-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourceflatorcdatareadbenchmark","title":"IcebergSourceFlatORCDataReadBenchmark","text":"<p>A benchmark that evaluates the performance of reading ORC data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceFlatORCDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-orc-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#sparkparquetreadersflatdatabenchmark","title":"SparkParquetReadersFlatDataBenchmark","text":"<p>A benchmark that evaluates the performance of reading Parquet data with a flat schema using Iceberg and Spark Parquet readers. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=SparkParquetReadersFlatDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-readers-flat-data-benchmark-result.txt</code></p>"},{"location":"benchmarks/#vectorizedreaddictionaryencodedflatparquetdatabenchmark","title":"VectorizedReadDictionaryEncodedFlatParquetDataBenchmark","text":"<p>A benchmark to compare performance of reading Parquet dictionary encoded data with a flat schema using vectorized Iceberg read path and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=VectorizedReadDictionaryEncodedFlatParquetDataBenchmark -PjmhOutputPath=benchmark/vectorized-read-dict-encoded-flat-parquet-data-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedlistorcdatawritebenchmark","title":"IcebergSourceNestedListORCDataWriteBenchmark","text":"<p>A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedListORCDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-list-orc-data-write-benchmark-result.txt</code></p>"},{"location":"benchmarks/#vectorizedreadflatparquetdatabenchmark","title":"VectorizedReadFlatParquetDataBenchmark","text":"<p>A benchmark to compare performance of reading Parquet data with a flat schema using vectorized Iceberg read path and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=VectorizedReadFlatParquetDataBenchmark -PjmhOutputPath=benchmark/vectorized-read-flat-parquet-data-result.txt</code></p>"},{"location":"benchmarks/#icebergsourceflatparquetdatawritebenchmark","title":"IcebergSourceFlatParquetDataWriteBenchmark","text":"<p>A benchmark that evaluates the performance of writing Parquet data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceFlatParquetDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-parquet-data-write-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedavrodatareadbenchmark","title":"IcebergSourceNestedAvroDataReadBenchmark","text":"<p>A benchmark that evaluates the performance of reading Avro data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedAvroDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-avro-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourceflatavrodatareadbenchmark","title":"IcebergSourceFlatAvroDataReadBenchmark","text":"<p>A benchmark that evaluates the performance of reading Avro data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceFlatAvroDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-avro-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedparquetdatawritebenchmark","title":"IcebergSourceNestedParquetDataWriteBenchmark","text":"<p>A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedParquetDataWriteBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-parquet-data-write-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedparquetdatareadbenchmark","title":"IcebergSourceNestedParquetDataReadBenchmark","text":"<ul> <li>A benchmark that evaluates the performance of reading nested Parquet data using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</li> </ul> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedParquetDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-parquet-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedorcdatareadbenchmark","title":"IcebergSourceNestedORCDataReadBenchmark","text":"<p>A benchmark that evaluates the performance of reading ORC data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedORCDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-orc-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourceflatparquetdatareadbenchmark","title":"IcebergSourceFlatParquetDataReadBenchmark","text":"<p>A benchmark that evaluates the performance of reading Parquet data with a flat schema using Iceberg and the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceFlatParquetDataReadBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-parquet-data-read-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourceflatparquetdatafilterbenchmark","title":"IcebergSourceFlatParquetDataFilterBenchmark","text":"<p>A benchmark that evaluates the file skipping capabilities in the Spark data source for Iceberg. This class uses a dataset with a flat schema, where the records are clustered according to the column used in the filter predicate. The performance is compared to the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3:</p> <p><code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceFlatParquetDataFilterBenchmark -PjmhOutputPath=benchmark/iceberg-source-flat-parquet-data-filter-benchmark-result.txt</code></p>"},{"location":"benchmarks/#icebergsourcenestedparquetdatafilterbenchmark","title":"IcebergSourceNestedParquetDataFilterBenchmark","text":"<p>A benchmark that evaluates the file skipping capabilities in the Spark data source for Iceberg. This class uses a dataset with nested data, where the records are clustered according to the column used in the filter predicate. The performance is compared to the built-in file source in Spark. To run this benchmark for either spark-2 or spark-3: <code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=IcebergSourceNestedParquetDataFilterBenchmark -PjmhOutputPath=benchmark/iceberg-source-nested-parquet-data-filter-benchmark-result.txt</code></p>"},{"location":"benchmarks/#sparkparquetwritersnesteddatabenchmark","title":"SparkParquetWritersNestedDataBenchmark","text":"<ul> <li>A benchmark that evaluates the performance of writing nested Parquet data using Iceberg and Spark Parquet writers. To run this benchmark for either spark-2 or spark-3: <code>./gradlew :iceberg-spark:iceberg-spark[2|3]:jmh -PjmhIncludeRegex=SparkParquetWritersNestedDataBenchmark -PjmhOutputPath=benchmark/spark-parquet-writers-nested-data-benchmark-result.txt</code></li> </ul>"},{"location":"blogs/","title":"Blogs","text":""},{"location":"blogs/#iceberg-blogs","title":"Iceberg Blogs","text":"<p>Here is a list of company blogs that talk about Iceberg. The blogs are ordered from most recent to oldest.</p>"},{"location":"blogs/#apache-hive-4x-with-iceberg-branches-tags","title":"Apache Hive-4.x with Iceberg Branches & Tags","text":"<p>Date: October 12th, 2023, Company: Cloudera</p> <p>Authors: Ayush Saxena</p>"},{"location":"blogs/#apache-hive-4x-with-apache-iceberg","title":"Apache Hive 4.x With Apache Iceberg","text":"<p>Date: October 12th, 2023, Company: Cloudera</p> <p>Authors: Ayush Saxena</p>"},{"location":"blogs/#from-hive-tables-to-iceberg-tables-hassle-free","title":"From Hive Tables to Iceberg Tables: Hassle-Free","text":"<p>Date: July 14th, 2023, Company: Cloudera</p> <p>Authors: Srinivas Rishindra Pothireddi</p>"},{"location":"blogs/#from-hive-tables-to-iceberg-tables-hassle-free_1","title":"From Hive Tables to Iceberg Tables: Hassle-Free","text":"<p>Date: July 14th, 2023, Company: Cloudera</p> <p>Authors: Srinivas Rishindra Pothireddi</p>"},{"location":"blogs/#12-times-faster-query-planning-with-iceberg-manifest-caching-in-impala","title":"12 Times Faster Query Planning With Iceberg Manifest Caching in Impala","text":"<p>Date: July 13th, 2023, Company: Cloudera</p> <p>Authors: Riza Suminto</p>"},{"location":"blogs/#how-bilibili-builds-olap-data-lakehouse-with-apache-iceberg","title":"How Bilibili Builds OLAP Data Lakehouse with Apache Iceberg","text":"<p>Date: June 14th, 2023, Company: Bilibili</p> <p>Authors: Rui Li</p>"},{"location":"blogs/#introducing-the-apache-iceberg-catalog-migration-tool","title":"Introducing the Apache Iceberg Catalog Migration Tool","text":"<p>Date: May 12th, 2022, Company: Dremio</p> <p>Authors: Dipankar Mazumdar & Ajantha Bhat</p>"},{"location":"blogs/#3-ways-to-use-python-with-apache-iceberg","title":"3 Ways to Use Python with Apache Iceberg","text":"<p>Date: April 12th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#3-ways-to-convert-a-delta-lake-table-into-an-apache-iceberg-table","title":"3 Ways to Convert a Delta Lake Table Into an Apache Iceberg Table","text":"<p>Date: April 3rd, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#how-to-convert-csv-files-into-an-apache-iceberg-table-with-dremio","title":"How to Convert CSV Files into an Apache Iceberg table with Dremio","text":"<p>Date: April 3rd, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#open-data-lakehouse-powered-by-iceberg-for-all-your-data-warehouse-needs","title":"Open Data Lakehouse powered by Iceberg for all your Data Warehouse needs","text":"<p>Date: April 3rd, 2023, Company: Cloudera</p> <p>Authors: Zoltan Borok-Nagy, Ayush Saxena, Tamas Mate, Simhadri Govindappa</p>"},{"location":"blogs/#exploring-branch-tags-in-apache-iceberg-using-spark","title":"Exploring Branch & Tags in Apache Iceberg using Spark","text":"<p>Date: March 29th, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#iceberg-tables-catalog-support-now-available","title":"Iceberg Tables: Catalog Support Now Available","text":"<p>Date: March 29th, 2023, Company: Snowflake</p> <p>Authors: Ron Ortloff, Dennis Huo</p>"},{"location":"blogs/#dealing-with-data-incidents-using-the-rollback-feature-in-apache-iceberg","title":"Dealing with Data Incidents Using the Rollback Feature in Apache Iceberg","text":"<p>Date: February 24th, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#partition-and-file-pruning-for-dremios-apache-iceberg-backed-reflections","title":"Partition and File Pruning for Dremio\u2019s Apache Iceberg-backed Reflections","text":"<p>Date: February 8th, 2022, Company: Dremio</p> <p>Author: Benny Chow</p>"},{"location":"blogs/#understanding-iceberg-table-metadata","title":"Understanding Iceberg Table Metadata","text":"<p>Date: January 30st, 2023, Company: Snowflake</p> <p>Author: Phani Raj</p>"},{"location":"blogs/#creating-and-managing-apache-iceberg-tables-using-serverless-features-and-without-coding","title":"Creating and managing Apache Iceberg tables using serverless features and without coding","text":"<p>Date: January 27th, 2023, Company: Snowflake</p> <p>Author: Parag Jain</p>"},{"location":"blogs/#getting-started-with-apache-iceberg","title":"Getting started with Apache Iceberg","text":"<p>Date: January 27th, 2023, Company: Snowflake</p> <p>Author: Jedidiah Rajbhushan</p>"},{"location":"blogs/#how-apache-iceberg-enables-acid-compliance-for-data-lakes","title":"How Apache Iceberg enables ACID compliance for data lakes","text":"<p>Date: January 13th, 2023, Company: Snowflake</p> <p>Authors: Sumeet Tandure</p>"},{"location":"blogs/#multi-cloud-open-lakehouse-with-apache-iceberg-in-cloudera-data-platform","title":"Multi-Cloud Open Lakehouse with Apache Iceberg in Cloudera Data Platform","text":"<p>Date: December 15th, 2022, Company: Cloudera</p> <p>Authors: Bill Zhang, Shaun Ahmadian, Zoltan Borok-Nagy, Vincent Kulandaisamy</p>"},{"location":"blogs/#connecting-tableau-to-apache-iceberg-tables-with-dremio","title":"Connecting Tableau to Apache Iceberg Tables with Dremio","text":"<p>Date: December 15th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#getting-started-with-project-nessie-apache-iceberg-and-apache-spark-using-docker","title":"Getting Started with Project Nessie, Apache Iceberg, and Apache Spark Using Docker","text":"<p>Date: December 15th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#apache-iceberg-faq","title":"Apache Iceberg FAQ","text":"<p>Date: December 14th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#a-notebook-for-getting-started-with-project-nessie-apache-iceberg-and-apache-spark","title":"A Notebook for getting started with Project Nessie, Apache Iceberg, and Apache Spark","text":"<p>Date: December 5th, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#time-travel-with-dremio-and-apache-iceberg","title":"Time Travel with Dremio and Apache Iceberg","text":"<p>Date: November 29th, 2022, Company: Dremio</p> <p>Author: Michael Flower</p>"},{"location":"blogs/#compaction-in-apache-iceberg-fine-tuning-your-iceberg-tables-data-files","title":"Compaction in Apache Iceberg: Fine-Tuning Your Iceberg Table's Data Files","text":"<p>Date: November 9th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#the-life-of-a-read-query-for-apache-iceberg-tables","title":"The Life of a Read Query for Apache Iceberg Tables","text":"<p>Date: October 31st, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#puffins-and-icebergs-additional-stats-for-apache-iceberg-tables","title":"Puffins and Icebergs: Additional Stats for Apache Iceberg Tables","text":"<p>Date: October 17th, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#apache-iceberg-and-the-right-to-be-forgotten","title":"Apache Iceberg and the Right to be Forgotten","text":"<p>Date: September 30th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#streaming-data-into-apache-iceberg-tables-using-aws-kinesis-and-aws-glue","title":"Streaming Data into Apache Iceberg tables using AWS Kinesis and AWS Glue","text":"<p>Date: September 26th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#iceberg-flink-sink-stream-directly-into-your-data-warehouse-tables","title":"Iceberg Flink Sink: Stream Directly into your Data Warehouse Tables","text":"<p>Date: October 12, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#partitioning-for-correctness-and-performance","title":"Partitioning for Correctness (and Performance)","text":"<p>Date: September 28, 2022, Company: Tabular</p> <p>Author: Jason Reid</p>"},{"location":"blogs/#ensuring-high-performance-at-any-scale-with-apache-icebergs-object-store-file-layout","title":"Ensuring High Performance at Any Scale with Apache Iceberg\u2019s Object Store File Layout","text":"<p>Date: September 20, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#introduction-to-apache-iceberg-using-spark","title":"Introduction to Apache Iceberg Using Spark","text":"<p>Date: September 15, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#how-z-ordering-in-apache-iceberg-helps-improve-performance","title":"How Z-Ordering in Apache Iceberg Helps Improve Performance","text":"<p>Date: September 13th, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#apache-iceberg-101-your-guide-to-learning-apache-iceberg-concepts-and-practices","title":"Apache Iceberg 101 \u2013 Your Guide to Learning Apache Iceberg Concepts and Practices","text":"<p>Date: September 12th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#a-hands-on-look-at-the-structure-of-an-apache-iceberg-table","title":"A Hands-On Look at the Structure of an Apache Iceberg Table","text":"<p>Date: August 24, 2022, Company: Dremio</p> <p>Author: Dipankar Mazumdar</p>"},{"location":"blogs/#future-proof-partitioning-and-fewer-table-rewrites-with-apache-iceberg","title":"Future-Proof Partitioning and Fewer Table Rewrites with Apache Iceberg","text":"<p>Date: August 18, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#how-to-use-apache-iceberg-in-cdps-open-lakehouse","title":"How to use Apache Iceberg in CDP's Open Lakehouse","text":"<p>Date: August 8th, 2022, Company: Cloudera</p> <p>Authors: Bill Zhang, Peter Ableda, Shaun Ahmadian, Manish Maheshwari</p>"},{"location":"blogs/#near-real-time-ingestion-for-trino","title":"Near Real-Time Ingestion For Trino","text":"<p>Date: August 4th, 2022, Company: Starburst</p> <p>Authors: Eric Hwang, Monica Miller, Brian Zhan</p>"},{"location":"blogs/#how-to-implement-apache-iceberg-in-aws-athena","title":"How to implement Apache Iceberg in AWS Athena","text":"<p>Date: July 28th, 2022</p> <p>Author: [Shneior Dicastro]</p>"},{"location":"blogs/#supercharge-your-data-lakehouse-with-apache-iceberg-in-cloudera-data-platform","title":"Supercharge your Data Lakehouse with Apache Iceberg in Cloudera Data Platform","text":"<p>Date: June 30th, 2022, Company: Cloudera</p> <p>Authors: Bill Zhang, Shaun Ahmadian</p>"},{"location":"blogs/#migrating-a-hive-table-to-an-iceberg-table-hands-on-tutorial","title":"Migrating a Hive Table to an Iceberg Table Hands-on Tutorial","text":"<p>Date: June 6th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#fewer-accidental-full-table-scans-brought-to-you-by-apache-icebergs-hidden-partitioning","title":"Fewer Accidental Full Table Scans Brought to You by Apache Iceberg\u2019s Hidden Partitioning","text":"<p>Date: May 21st, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#an-introduction-to-the-iceberg-java-api-part-2-table-scans","title":"An Introduction To The Iceberg Java API Part 2 - Table Scans","text":"<p>Date: May 11th, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#icebergs-guiding-light-the-iceberg-open-table-format-specification","title":"Iceberg's Guiding Light: The Iceberg Open Table Format Specification","text":"<p>Date: April 26th, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#how-to-migrate-a-hive-table-to-an-iceberg-table","title":"How to Migrate a Hive Table to an Iceberg Table","text":"<p>Date: April 15th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#using-icebergs-s3fileio-implementation-to-store-your-data-in-minio","title":"Using Iceberg's S3FileIO Implementation To Store Your Data In MinIO","text":"<p>Date: April 14th, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#maintaining-iceberg-tables-compaction-expiring-snapshots-and-more","title":"Maintaining Iceberg Tables \u2013 Compaction, Expiring Snapshots, and More","text":"<p>Date: April 7th, 2022, Company: Dremio</p> <p>Author: Alex Merced</p>"},{"location":"blogs/#an-introduction-to-the-iceberg-java-api-part-1","title":"An Introduction To The Iceberg Java API - Part 1","text":"<p>Date: April 1st, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#integrated-audits-streamlined-data-observability-with-apache-iceberg","title":"Integrated Audits: Streamlined Data Observability With Apache Iceberg","text":"<p>Date: March 2nd, 2022, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#introducing-apache-iceberg-in-cloudera-data-platform","title":"Introducing Apache Iceberg in Cloudera Data Platform","text":"<p>Date: February 23rd, 2022, Company: Cloudera</p> <p>Authors: Bill Zhang, Peter Vary, Marton Bod, Wing Yew Poon</p>"},{"location":"blogs/#whats-new-in-iceberg-013","title":"What's new in Iceberg 0.13","text":"<p>Date: February 22nd, 2022, Company: Tabular</p> <p>Author: Ryan Blue</p>"},{"location":"blogs/#apache-iceberg-becomes-industry-open-standard-with-ecosystem-adoption","title":"Apache Iceberg Becomes Industry Open Standard with Ecosystem Adoption","text":"<p>Date: February 3rd, 2022, Company: Dremio</p> <p>Author: Mark Lyons</p>"},{"location":"blogs/#docker-spark-and-iceberg-the-fastest-way-to-try-iceberg","title":"Docker, Spark, and Iceberg: The Fastest Way to Try Iceberg!","text":"<p>Date: February 2nd, 2022, Company: Tabular</p> <p>Author: Sam Redai, Kyle Bendickson</p>"},{"location":"blogs/#expanding-the-data-cloud-with-apache-iceberg","title":"Expanding the Data Cloud with Apache Iceberg","text":"<p>Date: January 21st, 2022, Company: Snowflake</p> <p>Author: James Malone</p>"},{"location":"blogs/#iceberg-fileio-cloud-native-tables","title":"Iceberg FileIO: Cloud Native Tables","text":"<p>Date: December 16th, 2021, Company: Tabular</p> <p>Author: Daniel Weeks</p>"},{"location":"blogs/#using-spark-in-emr-with-apache-iceberg","title":"Using Spark in EMR with Apache Iceberg","text":"<p>Date: December 10th, 2021, Company: Tabular</p> <p>Author: Sam Redai</p>"},{"location":"blogs/#using-flink-cdc-to-synchronize-data-from-mysql-sharding-tables-and-build-real-time-data-lake","title":"Using Flink CDC to synchronize data from MySQL sharding tables and build real-time data lake","text":"<p>Date: November 11th, 2021, Company: Ververica, Alibaba Cloud</p> <p>Author: Yuxia Luo, Jark Wu, Zheng Hu</p>"},{"location":"blogs/#metadata-indexing-in-iceberg","title":"Metadata Indexing in Iceberg","text":"<p>Date: October 10th, 2021, Company: Tabular</p> <p>Author: Ryan Blue</p>"},{"location":"blogs/#using-debezium-to-create-a-data-lake-with-apache-iceberg","title":"Using Debezium to Create a Data Lake with Apache Iceberg","text":"<p>Date: October 20th, 2021, Company: Memiiso Community</p> <p>Author: Ismail Simsek</p>"},{"location":"blogs/#how-to-analyze-cdc-data-in-iceberg-data-lake-using-flink","title":"How to Analyze CDC Data in Iceberg Data Lake Using Flink","text":"<p>Date: June 15th, 2021, Company: Alibaba Cloud Community</p> <p>Author: Li Jinsong, Hu Zheng, Yang Weihai, Peidan Li</p>"},{"location":"blogs/#apache-iceberg-an-architectural-look-under-the-covers","title":"Apache Iceberg: An Architectural Look Under the Covers","text":"<p>Date: July 6th, 2021, Company: Dremio</p> <p>Author: Jason Hughes</p>"},{"location":"blogs/#migrating-to-apache-iceberg-at-adobe-experience-platform","title":"Migrating to Apache Iceberg at Adobe Experience Platform","text":"<p>Date: Jun 17th, 2021, Company: Adobe</p> <p>Author: Romin Parekh, Miao Wang, Shone Sadler</p>"},{"location":"blogs/#flink-iceberg-how-to-construct-a-whole-scenario-real-time-data-warehouse","title":"Flink + Iceberg: How to Construct a Whole-scenario Real-time Data Warehouse","text":"<p>Date: Jun 8th, 2021, Company: Tencent</p> <p>Author Shu (Simon Su) Su</p>"},{"location":"blogs/#trino-on-ice-iii-iceberg-concurrency-model-snapshots-and-the-iceberg-spec","title":"Trino on Ice III: Iceberg Concurrency Model, Snapshots, and the Iceberg Spec","text":"<p>Date: May 25th, 2021, Company: Starburst</p> <p>Author: Brian Olsen</p>"},{"location":"blogs/#trino-on-ice-ii-in-place-table-evolution-and-cloud-compatibility-with-iceberg","title":"Trino on Ice II: In-Place Table Evolution and Cloud Compatibility with Iceberg","text":"<p>Date: May 11th, 2021, Company: Starburst</p> <p>Author: Brian Olsen</p>"},{"location":"blogs/#trino-on-ice-i-a-gentle-introduction-to-iceberg","title":"Trino On Ice I: A Gentle Introduction To Iceberg","text":"<p>Date: Apr 27th, 2021, Company: Starburst</p> <p>Author: Brian Olsen</p>"},{"location":"blogs/#apache-iceberg-a-different-table-design-for-big-data","title":"Apache Iceberg: A Different Table Design for Big Data","text":"<p>Date: Feb 1st, 2021, Company: thenewstack.io</p> <p>Author: Susan Hall</p>"},{"location":"blogs/#a-short-introduction-to-apache-iceberg","title":"A Short Introduction to Apache Iceberg","text":"<p>Date: Jan 26th, 2021, Company: Expedia</p> <p>Author: Christine Mathiesen</p>"},{"location":"blogs/#taking-query-optimizations-to-the-next-level-with-iceberg","title":"Taking Query Optimizations to the Next Level with Iceberg","text":"<p>Date: Jan 14th, 2021, Company: Adobe</p> <p>Author: Gautam Kowshik, Xabriel J. Collazo Mojica</p>"},{"location":"blogs/#fastingest-low-latency-gobblin-with-apache-iceberg-and-orc-format","title":"FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format","text":"<p>Date: Jan 6th, 2021, Company: Linkedin</p> <p>Author: Zihan Li, Sudarshan Vasudevan, Lei Sun, Shirshanka Das</p>"},{"location":"blogs/#high-throughput-ingestion-with-iceberg","title":"High Throughput Ingestion with Iceberg","text":"<p>Date: Dec 22nd, 2020, Company: Adobe</p> <p>Author: Andrei Ionescu, Shone Sadler, Anil Malkani</p>"},{"location":"blogs/#optimizing-data-warehouse-storage","title":"Optimizing data warehouse storage","text":"<p>Date: Dec 21st, 2020, Company: Netflix</p> <p>Author: Anupom Syam</p>"},{"location":"blogs/#iceberg-at-adobe","title":"Iceberg at Adobe","text":"<p>Date: Dec 3rd, 2020, Company: Adobe</p> <p>Author: Shone Sadler, Romin Parekh, Anil Malkani</p>"},{"location":"blogs/#bulldozer-batch-data-moving-from-data-warehouse-to-online-key-value-stores","title":"Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value Stores","text":"<p>Date: Oct 27th, 2020, Company: Netflix</p> <p>Author: Tianlong Chen, Ioannis Papapanagiotou</p>"},{"location":"catalog/","title":"Iceberg Catalogs","text":""},{"location":"catalog/#iceberg-catalogs","title":"Iceberg Catalogs","text":""},{"location":"catalog/#overview","title":"Overview","text":"<p>You may think of Iceberg as a format for managing data in a single table, but the Iceberg library needs a way to keep track of those tables by name. Tasks like creating, dropping, and renaming tables are the responsibility of a catalog. Catalogs manage a collection of tables that are usually grouped into namespaces. The most important responsibility of a catalog is tracking a table's current metadata, which is provided by the catalog when you load a table.</p> <p>The first step when using an Iceberg client is almost always initializing and configuring a catalog. The configured catalog is then used by compute engines to execute catalog operations. Multiple types of compute engines using a shared Iceberg catalog allows them to share a common data layer. </p> <p>A catalog is almost always configured through the processing engine which passes along a set of properties during initialization. Different processing engines have different ways to configure a catalog. When configuring a catalog, it\u2019s always best to refer to the Iceberg documentation as well as the docs for the specific processing engine being used. Ultimately, these configurations boil down to a common set of catalog properties that will be passed to configure the Iceberg catalog.</p>"},{"location":"catalog/#catalog-implementations","title":"Catalog Implementations","text":"<p>Iceberg catalogs are flexible and can be implemented using almost any backend system. They can be plugged into any Iceberg runtime, and allow any processing engine that supports Iceberg to load the tracked Iceberg tables. Iceberg also comes with a number of catalog implementations that are ready to use out of the box.</p> <p>This includes: - REST - a server-side catalog that\u2019s exposed through a REST API - Hive Metastore - tracks namespaces and tables using a Hive metastore - JDBC - tracks namespaces and tables in a simple JDBC database - Nessie - a transactional catalog that tracks namespaces and tables in a database with git-like version control</p> <p>There are more catalog types in addition to the ones listed here as well as custom catalogs that are developed to include specialized functionality.</p>"},{"location":"catalog/#decoupling-using-the-rest-catalog","title":"Decoupling Using the REST Catalog","text":"<p>The REST catalog was introduced in the Iceberg 0.14.0 release and provides greater control over how Iceberg catalogs are implemented. Instead of using technology-specific logic contained in the catalog clients, the implementation details of a REST catalog lives on the catalog server. If you\u2019re familiar with Hive, this is somewhat similar to the Hive thrift service that allows access to a hive server over a single port. The server-side logic can be written in any language and use any custom technology, as long as the API follows the Iceberg REST Open API specification.</p> <p>A great benefit of the REST catalog is that it allows you to use a single client to talk to any catalog backend. This increased flexibility makes it easier to make custom catalogs compatible with engines like Athena or Starburst without requiring the inclusion of a Jar into the classpath.</p>"},{"location":"community/","title":"Community","text":""},{"location":"community/#welcome","title":"Welcome!","text":"<p>Apache Iceberg tracks issues in GitHub and prefers to receive contributions as pull requests.</p> <p>Community discussions happen primarily on the dev mailing list, on apache-iceberg Slack workspace, and on specific GitHub issues.</p>"},{"location":"community/#contribute","title":"Contribute","text":"<p>See Contributing for more details on how to contribute to Iceberg.</p>"},{"location":"community/#issues","title":"Issues","text":"<p>Issues are tracked in GitHub:</p> <ul> <li>View open issues</li> <li>Open a new issue</li> </ul>"},{"location":"community/#slack","title":"Slack","text":"<p>We use the Apache Iceberg workspace on Slack. To be invited, follow this invite link.</p> <p>Please note that this link may occasionally break when Slack does an upgrade. If you encounter problems using it, please let us know by sending an email to dev@iceberg.apache.org.</p>"},{"location":"community/#iceberg-community-events","title":"Iceberg Community Events","text":"<p>This calendar contains two calendar feeds:</p> <ul> <li>Iceberg Community Events - Events such as conferences and meetups, aimed to educate and inspire Iceberg users.</li> <li>Iceberg Dev Events - Events such as the triweekly Iceberg sync, aimed to discuss the project roadmap and how to implement features.</li> </ul> <p>You can subscribe to either or both of these calendars by clicking the \"+ Google Calendar\" icon on the bottom right.</p>"},{"location":"community/#mailing-lists","title":"Mailing Lists","text":"<p>Iceberg has four mailing lists:</p> <ul> <li>Developers: dev@iceberg.apache.org -- used for community discussions<ul> <li>Subscribe</li> <li>Unsubscribe</li> <li>Archive</li> </ul> </li> <li>Commits: commits@iceberg.apache.org -- distributes commit notifications<ul> <li>Subscribe</li> <li>Unsubscribe</li> <li>Archive</li> </ul> </li> <li>Issues: issues@iceberg.apache.org -- Github issue tracking<ul> <li>Subscribe</li> <li>Unsubscribe</li> <li>Archive</li> </ul> </li> <li>Private: private@iceberg.apache.org -- private list for the PMC to discuss sensitive issues related to the health of the project<ul> <li>Archive</li> </ul> </li> </ul>"},{"location":"community/#community-guidelines","title":"Community Guidelines","text":""},{"location":"community/#apache-iceberg-community-guidelines","title":"Apache Iceberg Community Guidelines","text":"<p>The Apache Iceberg community is built on the principles described in the Apache Way and all who engage with the community are expected to be respectful, open, come with the best interests of the community in mind, and abide by the Apache Foundation Code of Conduct. </p>"},{"location":"community/#participants-with-corporate-interests","title":"Participants with Corporate Interests","text":"<p>A wide range of corporate entities have interests that overlap in both features and frameworks related to Iceberg and while we encourage engagement and contributions, the community is not a venue for marketing, solicitation, or recruitment.</p> <p>Any vendor who wants to participate in the Apache Iceberg community Slack workspace should create a dedicated vendor channel for their organization prefixed by <code>vendor-</code>. </p> <p>This space can be used to discuss features and integration with Iceberg related to the vendor offering. This space should not be used to promote competing vendor products/services or disparage other vendor offerings. Discussion should be focused on questions asked by the community and not to expand/introduce/redirect users to alternate offerings.</p>"},{"location":"community/#marketing-solicitation-recruiting","title":"Marketing / Solicitation / Recruiting","text":"<p>The Apache Iceberg community is a space for everyone to operate free of influence. The development lists, slack workspace, and github should not be used to market products or services. Solicitation or overt promotion should not be performed in common channels or through direct messages.</p> <p>Recruitment of community members should not be conducted through direct messages or community channels, but opportunities related to contributing to or using Iceberg can be posted to the <code>#jobs</code> channel. </p> <p>For questions regarding any of the guidelines above, please contact a PMC member</p>"},{"location":"contribute/","title":"Contribute","text":""},{"location":"contribute/#contributing","title":"Contributing","text":"<p>In this page, you will find some guidelines on contributing to Apache Iceberg. Please keep in mind that none of these are hard rules and they're meant as a collection of helpful suggestions to make contributing as seamless of an experience as possible.</p> <p>If you are thinking of contributing but first would like to discuss the change you wish to make, we welcome you to head over to the Community page on the official Iceberg documentation site to find a number of ways to connect with the community, including slack and our mailing lists. Of course, always feel free to just open a new issue in the GitHub repo. You can also check the following for a good first issue.</p> <p>The Iceberg Project is hosted on GitHub at https://github.com/apache/iceberg.</p>"},{"location":"contribute/#pull-request-process","title":"Pull Request Process","text":"<p>The Iceberg community prefers to receive contributions as Github pull requests.</p> <p>View open pull requests</p> <ul> <li>PRs are automatically labeled based on the content by our github-actions labeling action</li> <li>It's helpful to include a prefix in the summary that provides context to PR reviewers, such as <code>Build:</code>, <code>Docs:</code>, <code>Spark:</code>, <code>Flink:</code>, <code>Core:</code>, <code>API:</code></li> <li>If a PR is related to an issue, adding <code>Closes #1234</code> in the PR description will automatically close the issue and helps keep the project clean</li> <li>If a PR is posted for visibility and isn't necessarily ready for review or merging, be sure to convert the PR to a draft</li> </ul>"},{"location":"contribute/#building-the-project-locally","title":"Building the Project Locally","text":"<p>Iceberg is built using Gradle with Java 8 or Java 11.</p> <ul> <li>To invoke a build and run tests: <code>./gradlew build</code></li> <li>To skip tests: <code>./gradlew build -x test -x integrationTest</code></li> <li>To fix code style: <code>./gradlew spotlessApply</code></li> <li>To build particular Spark/Flink Versions: <code>./gradlew build -DsparkVersions=3.4,3.5 -DflinkVersions=1.14</code></li> </ul> <p>Iceberg table support is organized in library modules:</p> <ul> <li><code>iceberg-common</code> contains utility classes used in other modules</li> <li><code>iceberg-api</code> contains the public Iceberg API</li> <li><code>iceberg-core</code> contains implementations of the Iceberg API and support for Avro data files, this is what processing engines should depend on</li> <li><code>iceberg-parquet</code> is an optional module for working with tables backed by Parquet files</li> <li><code>iceberg-arrow</code> is an optional module for reading Parquet into Arrow memory</li> <li><code>iceberg-orc</code> is an optional module for working with tables backed by ORC files</li> <li><code>iceberg-hive-metastore</code> is an implementation of Iceberg tables backed by the Hive metastore Thrift client</li> <li><code>iceberg-data</code> is an optional module for working with tables directly from JVM applications</li> </ul> <p>This project Iceberg also has modules for adding Iceberg support to processing engines:</p> <ul> <li><code>iceberg-spark</code> is an implementation of Spark's Datasource V2 API for Iceberg with submodules for each spark versions (use runtime jars for a shaded version)</li> <li><code>iceberg-flink</code> contains classes for integrating with Apache Flink (use iceberg-flink-runtime for a shaded version)</li> <li><code>iceberg-mr</code> contains an InputFormat and other classes for integrating with Apache Hive</li> <li><code>iceberg-pig</code> is an implementation of Pig's LoadFunc API for Iceberg</li> </ul>"},{"location":"contribute/#setting-up-ide-and-code-style","title":"Setting up IDE and Code Style","text":""},{"location":"contribute/#configuring-code-formatter-for-eclipseintellij","title":"Configuring Code Formatter for Eclipse/IntelliJ","text":"<p>Follow the instructions for Eclipse or IntelliJ to install the google-java-format plugin (note the required manual actions for IntelliJ).</p>"},{"location":"contribute/#semantic-versioning","title":"Semantic Versioning","text":"<p>Apache Iceberg leverages semantic versioning to ensure compatibility for developers and users of the iceberg libraries as APIs and implementations evolve. The requirements and guarantees provided depend on the subproject as described below:</p>"},{"location":"contribute/#major-version-deprecations-required","title":"Major Version Deprecations Required","text":"<p>Modules <code>iceberg-api</code></p> <p>The API subproject is the main interface for developers and users of the Iceberg API and therefore has the strongest guarantees. Evolution of the interfaces in this subproject are enforced by Revapi and require explicit acknowledgement of API changes. All public interfaces and classes require one major version for deprecation cycle. Any backward incompatible changes should be annotated as <code>@Deprecated</code> and removed for the next major release. Backward compatible changes are allowed within major versions.</p>"},{"location":"contribute/#minor-version-deprecations-required","title":"Minor Version Deprecations Required","text":"<p>Modules <code>iceberg-common</code> <code>iceberg-core</code> <code>iceberg-data</code> <code>iceberg-orc</code> <code>iceberg-parquet</code></p> <p>Changes to public interfaces and classes in the subprojects listed above require a deprecation cycle of one minor release. These projects contain common and internal code used by other projects and can evolve within a major release. Minor release deprecation will provide other subprojects and external projects notice and opportunity to transition to new implementations.</p>"},{"location":"contribute/#minor-version-deprecations-discretionary","title":"Minor Version Deprecations Discretionary","text":"<p>modules (All modules not referenced above)</p> <p>Other modules are less likely to be extended directly and modifications should make a good faith effort to follow a minor version deprecation cycle. If there are significant structural or design changes that result in deprecations being difficult to orchestrate, it is up to the committers to decide if deprecation is necessary.</p>"},{"location":"contribute/#deprecation-notices","title":"Deprecation Notices","text":"<p>All interfaces, classes, and methods targeted for deprecation must include the following:</p> <ol> <li><code>@Deprecated</code> annotation on the appropriate element</li> <li><code>@depreceted</code> javadoc comment including: the version for removal, the appropriate alternative for usage</li> <li>Replacement of existing code paths that use the deprecated behavior</li> </ol> <p>Example:</p> <pre><code> /**\n * Set the sequence number for this manifest entry.\n *\n * @param sequenceNumber a sequence number\n * @deprecated since 1.0.0, will be removed in 1.1.0; use dataSequenceNumber() instead.\n */\n @Deprecated\n void sequenceNumber(long sequenceNumber);\n</code></pre>"},{"location":"contribute/#iceberg-code-contribution-guidelines","title":"Iceberg Code Contribution Guidelines","text":""},{"location":"contribute/#style","title":"Style","text":"<p>Java code adheres to the Google style, which will be verified via <code>./gradlew spotlessCheck</code> during builds. In order to automatically fix Java code style issues, please use <code>./gradlew spotlessApply</code>.</p> <p>NOTE: The google-java-format plugin will always use the latest version of the google-java-format. However, <code>spotless</code> itself is configured to use google-java-format 1.7 since that version is compatible with JDK 8. When formatting the code in the IDE, there is a slight chance that it will produce slightly different results. In such a case please run <code>./gradlew spotlessApply</code> as CI will check the style against google-java-format 1.7.</p>"},{"location":"contribute/#copyright","title":"Copyright","text":"<p>Each file must include the Apache license information as a header.</p> <pre><code>Licensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements. See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership. The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License. You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied. See the License for the\nspecific language governing permissions and limitations\nunder the License.\n</code></pre>"},{"location":"contribute/#configuring-copyright-for-intellij-idea","title":"Configuring Copyright for IntelliJ IDEA","text":"<p>Every file needs to include the Apache license as a header. This can be automated in IntelliJ by adding a Copyright profile:</p> <ol> <li>In the Settings/Preferences dialog go to Editor \u2192 Copyright \u2192 Copyright Profiles.</li> <li>Add a new profile and name it Apache.</li> <li>Add the following text as the license text:</li> </ol> <p><pre><code>Licensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements. See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership. The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License. You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied. See the License for the\nspecific language governing permissions and limitations\nunder the License.\n</code></pre> 4. Go to Editor \u2192 Copyright and choose the Apache profile as the default profile for this project. 5. Click Apply.</p>"},{"location":"contribute/#java-style-guidelines","title":"Java style guidelines","text":""},{"location":"contribute/#method-naming","title":"Method naming","text":"<ol> <li>Make method names as short as possible, while being clear. Omit needless words.</li> <li>Avoid <code>get</code> in method names, unless an object must be a Java bean.<ul> <li>In most cases, replace <code>get</code> with a more specific verb that describes what is happening in the method, like <code>find</code> or <code>fetch</code>.</li> <li>If there isn't a more specific verb or the method is a getter, omit <code>get</code> because it isn't helpful to readers and makes method names longer.</li> </ul> </li> <li>Where possible, use words and conjugations that form correct sentences in English when read<ul> <li>For example, <code>Transform.preservesOrder()</code> reads correctly in an if statement: <code>if (transform.preservesOrder()) { ... }</code></li> </ul> </li> </ol>"},{"location":"contribute/#boolean-arguments","title":"Boolean arguments","text":"<p>Avoid boolean arguments to methods that are not <code>private</code> to avoid confusing invocations like <code>sendMessage(false)</code>. It is better to create two methods with names and behavior, even if both are implemented by one internal method.</p> <pre><code> // prefer exposing suppressFailure in method names\n public void sendMessageIgnoreFailure() {\n sendMessageInternal(true);\n }\n\n public void sendMessage() {\n sendMessageInternal(false);\n }\n\n private void sendMessageInternal(boolean suppressFailure) {\n ...\n }\n</code></pre> <p>When passing boolean arguments to existing or external methods, use inline comments to help the reader understand actions without an IDE.</p> <pre><code> // BAD: it is not clear what false controls\n dropTable(identifier, false);\n\n // GOOD: these uses of dropTable are clear to the reader\n dropTable(identifier, true /* purge data */);\n dropTable(identifier, purge);\n</code></pre>"},{"location":"contribute/#config-naming","title":"Config naming","text":"<ol> <li>Use <code>-</code> to link words in one concept<ul> <li>For example, preferred convection <code>access-key-id</code> rather than <code>access.key.id</code></li> </ul> </li> <li>Use <code>.</code> to create a hierarchy of config groups<ul> <li>For example, <code>s3</code> in <code>s3.access-key-id</code>, <code>s3.secret-access-key</code></li> </ul> </li> </ol>"},{"location":"contribute/#testing","title":"Testing","text":""},{"location":"contribute/#assertj","title":"AssertJ","text":"<p>Prefer using AssertJ assertions as those provide a rich and intuitive set of strongly-typed assertions. Checks can be expressed in a fluent way and AssertJ provides rich context when assertions fail. Additionally, AssertJ has powerful testing capabilities on collections and exceptions. Please refer to the usage guide for additional examples.</p> <p><pre><code>// bad: will only say true != false when check fails\nassertTrue(x instanceof Xyz);\n\n// better: will show type of x when check fails\nassertThat(x).isInstanceOf(Xyz.class);\n\n// bad: will only say true != false when check fails\nassertTrue(catalog.listNamespaces().containsAll(expected));\n\n// better: will show content of expected and of catalog.listNamespaces() if check fails\nassertThat(catalog.listNamespaces()).containsAll(expected);\n</code></pre> <pre><code>// ok\nassertNotNull(metadataFileLocations);\nassertEquals(metadataFileLocations.size(), 4);\n\n// better: will show the content of metadataFileLocations if check fails\nassertThat(metadataFileLocations).isNotNull().hasSize(4);\n\n// or\nassertThat(metadataFileLocations).isNotNull().hasSameSizeAs(expected).hasSize(4);\n</code></pre></p> <p><pre><code>// bad\ntry {\n catalog.createNamespace(deniedNamespace);\n Assert.fail(\"this should fail\");\n} catch (Exception e) {\n assertEquals(AccessDeniedException.class, e.getClass());\n assertEquals(\"User 'testUser' has no permission to create namespace\", e.getMessage());\n}\n\n// better\nassertThatThrownBy(() -> catalog.createNamespace(deniedNamespace))\n .isInstanceOf(AccessDeniedException.class)\n .hasMessage(\"User 'testUser' has no permission to create namespace\");\n</code></pre> Checks on exceptions should always make sure to assert that a particular exception message has occurred.</p>"},{"location":"contribute/#awaitility","title":"Awaitility","text":"<p>Avoid using <code>Thread.sleep()</code> in tests as it leads to long test durations and flaky behavior if a condition takes slightly longer than expected.</p> <pre><code>deleteTablesAsync();\nThread.sleep(3000L);\nassertThat(tables()).isEmpty();\n</code></pre> <p>A better alternative is using Awaitility to make sure <code>tables()</code> are eventually empty. The below example will run the check with a default polling interval of 100 millis:</p> <pre><code>deleteTablesAsync();\nAwaitility.await(\"Tables were not deleted\")\n .atMost(5, TimeUnit.SECONDS)\n .untilAsserted(() -> assertThat(tables()).isEmpty());\n</code></pre> <p>Please refer to the usage guide of Awaitility for more usage examples.</p>"},{"location":"contribute/#junit4-junit5","title":"JUnit4 / JUnit5","text":"<p>Iceberg currently uses a mix of JUnit4 (<code>org.junit</code> imports) and JUnit5 (<code>org.junit.jupiter.api</code> imports) tests. To allow an easier migration to JUnit5 in the future, new test classes that are being added to the codebase should be written purely in JUnit5 where possible.</p>"},{"location":"contribute/#running-benchmarks","title":"Running Benchmarks","text":"<p>Some PRs/changesets might require running benchmarks to determine whether they are affecting the baseline performance. Currently there is no \"push a single button to get a performance comparison\" solution available, therefore one has to run JMH performance tests on their local machine and post the results on the PR.</p> <p>See Benchmarks for a summary of available benchmarks and how to run them.</p>"},{"location":"contribute/#website-and-documentation-updates","title":"Website and Documentation Updates","text":"<p>Currently, there is an iceberg-docs repository which contains the HTML/CSS and other files needed for the Iceberg website. The docs folder in the Iceberg repository contains the markdown content for the documentation site. All markdown changes should still be made to this repository.</p>"},{"location":"contribute/#submitting-pull-requests","title":"Submitting Pull Requests","text":"<p>Changes to the markdown contents should be submitted directly to this repository.</p> <p>Changes to the website appearance (e.g. HTML, CSS changes) should be submitted to the iceberg-docs repository against the <code>main</code> branch.</p> <p>Changes to the documentation of old Iceberg versions should be submitted to the iceberg-docs repository against the specific version branch.</p>"},{"location":"contribute/#reporting-issues","title":"Reporting Issues","text":"<p>All issues related to the doc website should still be submitted to the Iceberg repository. The GitHub Issues feature of the iceberg-docs repository is disabled.</p>"},{"location":"contribute/#running-locally","title":"Running Locally","text":"<p>Clone the iceberg-docs repository to run the website locally: <pre><code>git clone git@github.com:apache/iceberg-docs.git\ncd iceberg-docs\n</code></pre></p> <p>To start the landing page site locally, run: <pre><code>cd landing-page && hugo serve\n</code></pre></p> <p>To start the documentation site locally, run: <pre><code>cd docs && hugo serve\n</code></pre></p> <p>If you would like to see how the latest website looks based on the documentation in the Iceberg repository, you can copy docs to the iceberg-docs repository by: <pre><code>rm -rf docs/content/docs\nrm -rf landing-page/content/common\ncp -r <path to iceberg repo>/docs/versioned docs/content/docs\ncp -r <path to iceberg repo>/docs/common landing-page/content/common\n</code></pre></p>"},{"location":"fileio/","title":"FileIO","text":""},{"location":"fileio/#iceberg-fileio","title":"Iceberg FileIO","text":""},{"location":"fileio/#overview","title":"Overview","text":"<p>Iceberg comes with a flexible abstraction around reading and writing data and metadata files. The FileIO interface allows the Iceberg library to communicate with the underlying storage layer. FileIO is used for all metadata operations during the job planning and commit stages.</p>"},{"location":"fileio/#iceberg-files","title":"Iceberg Files","text":"<p>The metadata for an Iceberg table tracks the absolute path for data files which allows greater abstraction over the physical layout. Additionally, changes to table state are performed by writing new metadata files and never involve renaming files. This allows a much smaller set of requirements for file operations. The essential functionality for a FileIO implementation is that it can read files, write files, and seek to any position within a stream.</p>"},{"location":"fileio/#usage-in-processing-engines","title":"Usage in Processing Engines","text":"<p>The responsibility of reading and writing data files lies with the processing engines and happens during task execution. However, after data files are written, processing engines use FileIO to write new Iceberg metadata files that capture the new state of the table. A blog post that provides a deeper understanding of FileIO is Iceberg FileIO: Cloud Native Tables</p> <p>Different FileIO implementations are used depending on the type of storage. Iceberg comes with a set of FileIO implementations for popular storage providers. - Amazon S3 - Google Cloud Storage - Object Service Storage (including https) - Dell Enterprise Cloud Storage - Hadoop (adapts any Hadoop FileSystem implementation)</p> <p>As an example, take a look at the blog post Using Iceberg's S3FileIO Implementation to Store Your Data in MinIO which walks through how to use the Amazon S3 FileIO with MinIO.</p>"},{"location":"gcm-stream-spec/","title":"AES GCM Stream Spec","text":""},{"location":"gcm-stream-spec/#aes-gcm-stream-file-format-extension","title":"AES GCM Stream file format extension","text":""},{"location":"gcm-stream-spec/#background-and-motivation","title":"Background and Motivation","text":"<p>Iceberg supports a number of data file formats. Two of these formats (Parquet and ORC) have built-in encryption capabilities, that allow to protect sensitive information in the data files. However, besides the data files, Iceberg tables also have metadata files, that keep sensitive information too (e.g., min/max values in manifest files, or bloom filter bitsets in puffin files). Metadata file formats (AVRO, JSON, Puffin) don't have encryption support.</p> <p>Moreover, with the exception of Parquet, no Iceberg data or metadata file format supports integrity verification, required for end-to-end tamper proofing of Iceberg tables.</p> <p>This document specifies details of a simple file format extension that adds encryption and tamper-proofing to any existing file format.</p>"},{"location":"gcm-stream-spec/#goals","title":"Goals","text":"<ul> <li>Metadata encryption: enable encryption of manifests, manifest lists, snapshots and stats.</li> <li>Avro data encryption: enable encryption of data files in tables that use the Avro format.</li> <li>Support read splitting: enable seekable decrypting streams that can be used with splittable formats like Avro.</li> <li>Tamper proofing of Iceberg data and metadata files.</li> </ul>"},{"location":"gcm-stream-spec/#overview","title":"Overview","text":"<p>The output stream, produced by a metadata or data writer, is split into equal-size blocks (plus last block that can be shorter). Each block is enciphered (encrypted/signed) with a given encryption key, and stored in a file in the AES GCM Stream format. Upon reading, the stored cipherblocks are verified for integrity; then decrypted and passed to metadata or data readers.</p>"},{"location":"gcm-stream-spec/#encryption-algorithm","title":"Encryption algorithm","text":"<p>AES GCM Stream uses the standard AEG GCM cipher, and supports all AES key sizes: 128, 192 and 256 bits.</p> <p>AES GCM is an authenticated encryption. Besides data confidentiality (encryption), it supports two levels of integrity verification (authentication): of the data (default), and of the data combined with an optional AAD (\u201cadditional authenticated data\u201d). An AAD is a free text to be authenticated, together with the data. The structure of AES GCM Stream AADs is described below.</p> <p>AES GCM requires a unique vector to be provided for each encrypted block. In this document, the unique input to GCM encryption is called nonce (\u201cnumber used once\u201d). AES GCM Stream encryption uses the RBG-based (random bit generator) nonce construction as defined in the section 8.2.2 of the NIST SP 800-38D document. For each encrypted block, AES GCM Stream generates a unique nonce with a length of 12 bytes (96 bits).</p>"},{"location":"gcm-stream-spec/#format-specification","title":"Format specification","text":""},{"location":"gcm-stream-spec/#file-structure","title":"File structure","text":"<p>The AES GCM Stream files have the following structure</p> <pre><code>Magic BlockLength CipherBlock\u2081 CipherBlock\u2082 ... CipherBlock\u2099\n</code></pre> <p>where</p> <ul> <li><code>Magic</code> is four bytes 0x41, 0x47, 0x53, 0x31 (\"AGS1\", short for: AES GCM Stream, version 1)</li> <li><code>BlockLength</code> is four bytes (little endian) integer keeping the length of the equal-size split blocks before encryption. The length is specified in bytes.</li> <li><code>CipherBlock\u1d62</code> is the i-th enciphered block in the file, with the structure defined below.</li> </ul>"},{"location":"gcm-stream-spec/#cipher-block-structure","title":"Cipher Block structure","text":"<p>Cipher blocks have the following structure</p> nonce ciphertext tag <p>where</p> <ul> <li><code>nonce</code> is the AES GCM nonce, with a length of 12 bytes.</li> <li><code>ciphertext</code> is the encrypted block. Its length is identical to the length of the block before encryption (\"plaintext\"). The length of all plaintext blocks, except the last, is <code>BlockLength</code> bytes. The last block has a non-zero length <= <code>BlockLength</code>.</li> <li><code>tag</code> is the AES GCM tag, with a length of 16 bytes.</li> </ul> <p>AES GCM Stream encrypts all blocks by the GCM cipher, without padding. The AES GCM cipher must be implemented by a cryptographic provider according to the NIST SP 800-38D specification. In AES GCM Stream, an input to the GCM cipher is an AES encryption key, a nonce, a plaintext and an AAD (described below). The output is a ciphertext with the length equal to that of plaintext, and a 16-byte authentication tag used to verify the ciphertext and AAD integrity.</p>"},{"location":"gcm-stream-spec/#additional-authenticated-data","title":"Additional Authenticated Data","text":"<p>The AES GCM cipher protects against byte replacement inside a ciphertext block - but, without an AAD, it can't prevent replacement of one ciphertext block with another (encrypted with the same key). AES GCM Stream leverages AADs to protect against swapping ciphertext blocks inside a file or between files. AES GCM Stream can also protect against swapping full files - for example, replacement of a metadata file with an old version. AADs are built to reflects the identity of a file and of the blocks inside the file.</p> <p>AES GCM Stream constructs a block AAD from two components: an AAD prefix - a string provided by Iceberg for the file (with the file ID), and an AAD suffix - the block sequence number in the file, as an int in a 4-byte little-endian form. The block AAD is a direct concatenation of the prefix and suffix parts.</p>"},{"location":"hive-quickstart/","title":"Hive and Iceberg Quickstart","text":""},{"location":"hive-quickstart/#hive-and-iceberg-quickstart","title":"Hive and Iceberg Quickstart","text":"<p>This guide will get you up and running with an Iceberg and Hive environment, including sample code to highlight some powerful features. You can learn more about Iceberg's Hive runtime by checking out the Hive section.</p> <ul> <li>Docker Images</li> <li>Creating a Table</li> <li>Writing Data to a Table</li> <li>Reading Data from a Table</li> <li>Next Steps</li> </ul>"},{"location":"hive-quickstart/#docker-images","title":"Docker Images","text":"<p>The fastest way to get started is to use Apache Hive images which provides a SQL-like interface to create and query Iceberg tables from your laptop. You need to install the Docker Desktop.</p> <p>Take a look at the Tags tab in Apache Hive docker images to see the available Hive versions.</p> <p>Set the version variable. <pre><code>export HIVE_VERSION=4.0.0-beta-1\n</code></pre></p> <p>Start the container, using the option <code>--platform linux/amd64</code> for a Mac with an M-Series chip: <pre><code>docker run -d --platform linux/amd64 -p 10000:10000 -p 10002:10002 --env SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}\n</code></pre></p> <p>The docker run command above configures Hive to use the embedded derby database for Hive Metastore. Hive Metastore functions as the Iceberg catalog to locate Iceberg files, which can be anywhere. </p> <p>Give HiveServer (HS2) a little time to come up in the docker container, and then start the Hive Beeline client using the following command to connect with the HS2 containers you already started: <pre><code>docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:10000/'\n</code></pre></p> <p>The hive prompt appears: <pre><code>0: jdbc:hive2://localhost:10000>\n</code></pre></p> <p>You can now run SQL queries to create Iceberg tables and query the tables. <pre><code>show databases;\n</code></pre></p>"},{"location":"hive-quickstart/#creating-a-table","title":"Creating a Table","text":"<p>To create your first Iceberg table in Hive, run a <code>CREATE TABLE</code> command. Let's create a table using <code>nyc.taxis</code> where <code>nyc</code> is the database name and <code>taxis</code> is the table name. <pre><code>CREATE DATABASE nyc;\n</code></pre> <pre><code>CREATE TABLE nyc.taxis\n(\n trip_id bigint,\n trip_distance float,\n fare_amount double,\n store_and_fwd_flag string\n)\nPARTITIONED BY (vendor_id bigint) STORED BY ICEBERG;\n</code></pre> Iceberg catalogs support the full range of SQL DDL commands, including:</p> <ul> <li><code>CREATE TABLE</code></li> <li><code>CREATE TABLE AS SELECT</code></li> <li><code>CREATE TABLE LIKE TABLE</code></li> <li><code>ALTER TABLE</code></li> <li><code>DROP TABLE</code></li> </ul>"},{"location":"hive-quickstart/#writing-data-to-a-table","title":"Writing Data to a Table","text":"<p>After your table is created, you can insert records. <pre><code>INSERT INTO nyc.taxis\nVALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1);\n</code></pre></p>"},{"location":"hive-quickstart/#reading-data-from-a-table","title":"Reading Data from a Table","text":"<p>To read a table, simply use the Iceberg table's name. <pre><code>SELECT * FROM nyc.taxis;\n</code></pre></p>"},{"location":"hive-quickstart/#next-steps","title":"Next steps","text":""},{"location":"hive-quickstart/#adding-iceberg-to-hive","title":"Adding Iceberg to Hive","text":"<p>If you already have a Hive 4.0.0-alpha-1, or later, environment, it comes with the Iceberg 0.13.1 included. No additional downloads or jars are needed. If you have a Hive 2.3.x or Hive 3.1.x environment see Enabling Iceberg support in Hive.</p>"},{"location":"hive-quickstart/#learn-more","title":"Learn More","text":"<p>To learn more about setting up a database other than Derby, see Apache Hive Quick Start. You can also set up a standalone metastore, HS2 and Postgres. Now that you're up and running with Iceberg and Hive, check out the Iceberg-Hive docs to learn more!</p>"},{"location":"how-to-release/","title":"How To Release","text":""},{"location":"how-to-release/#introduction","title":"Introduction","text":"<p>This page walks you through the release process of the Iceberg project. Here you can read about the release process in general for an Apache project.</p> <p>Decisions about releases are made by three groups:</p> <ul> <li>Release Manager: Does the work of creating the release, signing it, counting votes, announcing the release and so on. Requires the assistance of a committer for some steps.</li> <li>The community: Performs the discussion of whether it is the right time to create a release and what that release should contain. The community can also cast non-binding votes on the release.</li> <li>PMC: Gives binding votes on the release.</li> </ul> <p>This page describes the procedures that the release manager and voting PMC members take during the release process.</p>"},{"location":"how-to-release/#setup","title":"Setup","text":"<p>To create a release candidate, you will need:</p> <ul> <li>Apache LDAP credentals for Nexus and SVN</li> <li>A GPG key for signing, published in KEYS</li> </ul> <p>If you have not published your GPG key yet, you must publish it before sending the vote email by doing:</p> <pre><code>svn co https://dist.apache.org/repos/dist/dev/iceberg icebergsvn\ncd icebergsvn\necho \"\" >> KEYS # append a newline\ngpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures\ngpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block\nsvn commit -m \"add key for <YOUR NAME HERE>\"\n</code></pre>"},{"location":"how-to-release/#nexus-access","title":"Nexus access","text":"<p>Nexus credentials are configured in your personal <code>~/.gradle/gradle.properties</code> file using <code>mavenUser</code> and <code>mavenPassword</code>:</p> <pre><code>mavenUser=yourApacheID\nmavenPassword=SomePassword\n</code></pre>"},{"location":"how-to-release/#pgp-signing","title":"PGP signing","text":"<p>The release scripts use the command-line <code>gpg</code> utility so that signing can use the gpg-agent and does not require writing your private key's passphrase to a configuration file.</p> <p>To configure gradle to sign convenience binary artifacts, add the following settings to <code>~/.gradle/gradle.properties</code>:</p> <pre><code>signing.gnupg.keyName=Your Name (CODE SIGNING KEY)\n</code></pre> <p>To use <code>gpg</code> instead of <code>gpg2</code>, also set <code>signing.gnupg.executable=gpg</code></p> <p>For more information, see the Gradle signing documentation.</p>"},{"location":"how-to-release/#apache-repository","title":"Apache repository","text":"<p>The release should be executed against <code>https://github.com/apache/iceberg.git</code> instead of any fork. Set it as remote with name <code>apache</code> for release if it is not already set up.</p>"},{"location":"how-to-release/#creating-a-release-candidate","title":"Creating a release candidate","text":""},{"location":"how-to-release/#initiate-a-discussion-about-the-release-with-the-community","title":"Initiate a discussion about the release with the community","text":"<p>This step can be useful to gather ongoing patches that the community thinks should be in the upcoming release.</p> <p>The communication can be started via a [DISCUSS] mail on the dev@ channel and the desired tickets can be added to the github milestone of the next release.</p> <p>Note, creating a milestone in github requires a committer. However, a non-committer can assign tasks to a milestone if added to the list of collaborators in .asf.yaml</p> <p>The release status is discussed during each community sync meeting. Release manager should join the meeting to report status and discuss any release blocker.</p>"},{"location":"how-to-release/#build-the-source-release","title":"Build the source release","text":"<p>To create the source release artifacts, run the <code>source-release.sh</code> script with the release version and release candidate number:</p> <pre><code>dev/source-release.sh -v 0.13.0 -r 0 -k <YOUR KEY ID HERE>\n</code></pre> <p>Example console output:</p> <pre><code>Preparing source for apache-iceberg-0.13.0-rc1\nAdding version.txt and tagging release...\n[master ca8bb7d0] Add version.txt for release 0.13.0\n 1 file changed, 1 insertion(+)\n create mode 100644 version.txt\nPushing apache-iceberg-0.13.0-rc1 to origin...\nEnumerating objects: 5, done.\nCounting objects: 100% (5/5), done.\nDelta compression using up to 12 threads\nCompressing objects: 100% (3/3), done.\nWriting objects: 100% (4/4), 433 bytes | 433.00 KiB/s, done.\nTotal 4 (delta 1), reused 0 (delta 0)\nremote: Resolving deltas: 100% (1/1), completed with 1 local object.\nTo https://github.com/apache/iceberg.git\n * [new tag] apache-iceberg-0.13.0-rc1 -> apache-iceberg-0.13.0-rc1\nCreating tarball using commit ca8bb7d0821f35bbcfa79a39841be8fb630ac3e5\nSigning the tarball...\nChecking out Iceberg RC subversion repo...\nChecked out revision 52260.\nAdding tarball to the Iceberg distribution Subversion repo...\nA tmp/apache-iceberg-0.13.0-rc1\nA tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.asc\nA (bin) tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz\nA tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.sha512\nAdding tmp/apache-iceberg-0.13.0-rc1\nAdding (bin) tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz\nAdding tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.asc\nAdding tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.sha512\nTransmitting file data ...done\nCommitting transaction...\nCommitted revision 52261.\nCreating release-announcement-email.txt...\nSuccess! The release candidate is available here:\n https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.0-rc1\n\nCommit SHA1: ca8bb7d0821f35bbcfa79a39841be8fb630ac3e5\n\nWe have generated a release announcement email for you here:\n/Users/jackye/iceberg/release-announcement-email.txt\n\nPlease note that you must update the Nexus repository URL\ncontained in the mail before sending it out.\n</code></pre> <p>The source release script will create a candidate tag based on the HEAD revision in git and will prepare the release tarball, signature, and checksum files. It will also upload the source artifacts to SVN.</p> <p>Note the commit SHA1 and candidate location because those will be added to the vote thread.</p> <p>Once the source release is ready, use it to stage convenience binary artifacts in Nexus.</p>"},{"location":"how-to-release/#build-and-stage-convenience-binaries","title":"Build and stage convenience binaries","text":"<p>Convenience binaries are created using the source release tarball from in the last step.</p> <p>Untar the source release and go into the release directory:</p> <pre><code>tar xzf apache-iceberg-0.13.0.tar.gz\ncd apache-iceberg-0.13.0\n</code></pre> <p>To build and publish the convenience binaries, run the <code>dev/stage-binaries.sh</code> script. This will push to a release staging repository.</p> <p>Disable gradle parallelism by setting <code>org.gradle.parallel=false</code> in <code>gradle.properties</code>.</p> <pre><code>dev/stage-binaries.sh\n</code></pre> <p>Next, you need to close the staging repository:</p> <ol> <li>Go to Nexus and log in</li> <li>In the menu on the left, choose \"Staging Repositories\"</li> <li>Select the Iceberg repository</li> <li>If multiple staging repositories are created after running the script, verify that gradle parallelism is disabled and try again.</li> <li>At the top, select \"Close\" and follow the instructions</li> <li>In the comment field use \"Apache Iceberg <version> RC<num>\"</li> </ol>"},{"location":"how-to-release/#start-a-vote-thread","title":"Start a VOTE thread","text":"<p>The last step for a candidate is to create a VOTE thread on the dev mailing list. The email template is already generated in <code>release-announcement-email.txt</code> with some details filled.</p> <p>Example title subject:</p> <pre><code>[VOTE] Release Apache Iceberg <VERSION> RC<NUM>\n</code></pre> <p>Example content:</p> <pre><code>Hi everyone,\n\nI propose the following RC to be released as official Apache Iceberg <VERSION> release.\n\nThe commit id is <SHA1>\n* This corresponds to the tag: apache-iceberg-<VERSION>-rc<NUM>\n* https://github.com/apache/iceberg/commits/apache-iceberg-<VERSION>-rc<NUM>\n* https://github.com/apache/iceberg/tree/<SHA1>\n\nThe release tarball, signature, and checksums are here:\n* https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-<VERSION>-rc<NUM>/\n\nYou can find the KEYS file here:\n* https://dist.apache.org/repos/dist/dev/iceberg/KEYS\n\nConvenience binary artifacts are staged in Nexus. The Maven repository URL is:\n* https://repository.apache.org/content/repositories/orgapacheiceberg-<ID>/\n\nThis release includes important changes that I should have summarized here, but I'm lazy.\n\nPlease download, verify, and test.\n\nPlease vote in the next 72 hours. (Weekends excluded)\n\n[ ] +1 Release this as Apache Iceberg <VERSION>\n[ ] +0\n[ ] -1 Do not release this because...\n\nOnly PMC members have binding votes, but other community members are encouraged to cast\nnon-binding votes. This vote will pass if there are 3 binding +1 votes and more binding\n+1 votes than -1 votes.\n</code></pre> <p>When a candidate is passed or rejected, reply with the voting result:</p> <pre><code>Subject: [RESULT][VOTE] Release Apache Iceberg <VERSION> RC<NUM>\n</code></pre> <pre><code>Thanks everyone who participated in the vote for Release Apache Iceberg <VERSION> RC<NUM>.\n\nThe vote result is:\n\n+1: 3 (binding), 5 (non-binding)\n+0: 0 (binding), 0 (non-binding)\n-1: 0 (binding), 0 (non-binding)\n\nTherefore, the release candidate is passed/rejected.\n</code></pre>"},{"location":"how-to-release/#finishing-the-release","title":"Finishing the release","text":"<p>After the release vote has passed, you need to release the last candidate's artifacts.</p> <p>But note that releasing the artifacts should happen around the same time the new docs are released so make sure the documentation changes are prepared when going through the below steps.</p>"},{"location":"how-to-release/#publishing-the-release","title":"Publishing the release","text":"<p>First, copy the source release directory to releases:</p> <pre><code>mkdir iceberg\ncd iceberg\nsvn co https://dist.apache.org/repos/dist/dev/iceberg candidates\nsvn co https://dist.apache.org/repos/dist/release/iceberg releases\ncp -r candidates/apache-iceberg-<VERSION>-rcN/ releases/apache-iceberg-<VERSION>\ncd releases\nsvn add apache-iceberg-<VERSION>\nsvn ci -m 'Iceberg: Add release <VERSION>'\n</code></pre> <p>Note</p> <p>The above step requires PMC privileges to execute.</p> <p>Next, add a release tag to the git repository based on the passing candidate tag:</p> <pre><code>git tag -am 'Release Apache Iceberg <VERSION>' apache-iceberg-<VERSION> apache-iceberg-<VERSION>-rcN\n</code></pre> <p>Then release the candidate repository in Nexus.</p>"},{"location":"how-to-release/#announcing-the-release","title":"Announcing the release","text":"<p>To announce the release, wait until Maven central has mirrored the Apache binaries, then update the Iceberg site and send an announcement email:</p> <p><pre><code>[ANNOUNCE] Apache Iceberg release <VERSION>\n</code></pre> <pre><code>I'm pleased to announce the release of Apache Iceberg <VERSION>!\n\nApache Iceberg is an open table format for huge analytic datasets. Iceberg\ndelivers high query performance for tables with tens of petabytes of data,\nalong with atomic commits, concurrent writes, and SQL-compatible table\nevolution.\n\nThis release can be downloaded from: https://www.apache.org/dyn/closer.cgi/iceberg/<TARBALL NAME WITHOUT .tar.gz>/<TARBALL NAME>\n\nJava artifacts are available from Maven Central.\n\nThanks to everyone for contributing!\n</code></pre></p>"},{"location":"how-to-release/#update-revapi","title":"Update revapi","text":"<p>Create a PR in the <code>iceberg</code> repo to make revapi run on the new release. For an example see this PR.</p>"},{"location":"how-to-release/#update-github","title":"Update GitHub","text":"<ul> <li>Create a PR in the <code>iceberg</code> repo to add the new version to the github issue template. For an example see this PR.</li> <li>Draft a new release to update Github to show the latest release. A changelog can be generated automatically using Github.</li> </ul>"},{"location":"how-to-release/#documentation-release","title":"Documentation Release","text":"<p>Documentation needs to be updated as a part of an Iceberg release after a release candidate is passed. The commands described below assume you are in a directory containing a local clone of the <code>iceberg-docs</code> repository and <code>iceberg</code> repository. Adjust the commands accordingly if it is not the case. Note that all changes in <code>iceberg</code> need to happen against the <code>master</code> branch and changes in <code>iceberg-docs</code> need to happen against the <code>main</code> branch. </p>"},{"location":"how-to-release/#common-documentation-update","title":"Common documentation update","text":"<ol> <li>To start the release process, run the following steps in the <code>iceberg-docs</code> repository to copy docs over: <pre><code>cp -r ../iceberg/format/* ../iceberg-docs/landing-page/content/common/\n</code></pre></li> <li>Change into the <code>iceberg-docs</code> repository and create a branch. <pre><code>cd ../iceberg-docs\ngit checkout -b <BRANCH NAME>\n</code></pre></li> <li>Commit, push, and open a PR against the <code>iceberg-docs</code> repo (<code><BRANCH NAME></code> -> <code>main</code>)</li> </ol>"},{"location":"how-to-release/#versioned-documentation-update","title":"Versioned documentation update","text":"<p>Once the common docs changes have been merged into <code>main</code>, the next step is to update the versioned docs.</p> <ol> <li>In the <code>iceberg-docs</code> repository, cut a new branch using the version number as the branch name <pre><code>cd ../iceberg-docs\ngit checkout -b <VERSION>\ngit push --set-upstream apache <VERSION>\n</code></pre></li> <li>Copy the versioned docs from the <code>iceberg</code> repo into the <code>iceberg-docs</code> repo <pre><code>rm -rf ../iceberg-docs/docs/content\ncp -r ../iceberg/docs ../iceberg-docs/docs/content\n</code></pre></li> <li>Commit the changes and open a PR against the <code><VERSION></code> branch in the <code>iceberg-docs</code> repo</li> </ol>"},{"location":"how-to-release/#javadoc-update","title":"Javadoc update","text":"<p>In the <code>iceberg</code> repository, generate the javadoc for your release and copy it to the <code>javadoc</code> folder in <code>iceberg-docs</code> repo: <pre><code>cd ../iceberg\n./gradlew refreshJavadoc\nrm -rf ../iceberg-docs/javadoc\ncp -r site/docs/javadoc/<VERSION NUMBER> ../iceberg-docs/javadoc\n</code></pre></p> <p>This resulted changes in <code>iceberg-docs</code> should be approved in a separate PR.</p>"},{"location":"how-to-release/#update-the-latest-branch","title":"Update the latest branch","text":"<p>Since <code>main</code> is currently the same as the version branch, one needs to rebase <code>latest</code> branch against <code>main</code>:</p> <pre><code>git checkout latest\ngit rebase main\ngit push apache latest\n</code></pre>"},{"location":"how-to-release/#set-latest-version-in-iceberg-docs-repo","title":"Set latest version in iceberg-docs repo","text":"<p>The last step is to update the <code>main</code> branch in <code>iceberg-docs</code> to set the latest version. A PR needs to be published in the <code>iceberg-docs</code> repository with the following changes: 1. Update variable <code>latestVersions.iceberg</code> to the new release version in <code>landing-page/config.toml</code> 2. Update variable <code>latestVersions.iceberg</code> to the new release version and <code>versions.nessie</code> to the version of <code>org.projectnessie.nessie:*</code> from versions.props in <code>docs/config.toml</code> 3. Update list <code>versions</code> with the new release in <code>landing-page/config.toml</code> 4. Update list <code>versions</code> with the new release in <code>docs/config.toml</code> 5. Mark the current latest release notes to past releases under <code>landing-page/content/common/release-notes.md</code> 6. Add release notes for the new release version in <code>landing-page/content/common/release-notes.md</code></p>"},{"location":"how-to-release/#how-to-verify-a-release","title":"How to Verify a Release","text":"<p>Each Apache Iceberg release is validated by the community by holding a vote. A community release manager will prepare a release candidate and call a vote on the Iceberg dev list. To validate the release candidate, community members will test it out in their downstream projects and environments. It's recommended to report the Java, Scala, Spark, Flink and Hive versions you have tested against when you vote.</p> <p>In addition to testing in downstream projects, community members also check the release's signatures, checksums, and license documentation.</p>"},{"location":"how-to-release/#validating-a-source-release-candidate","title":"Validating a source release candidate","text":"<p>Release announcements include links to the following:</p> <ul> <li>A source tarball</li> <li>A signature (.asc)</li> <li>A checksum (.sha512)</li> <li>KEYS file</li> <li>GitHub change comparison</li> </ul> <p>After downloading the source tarball, signature, checksum, and KEYS file, here are instructions on how to verify signatures, checksums, and documentation.</p>"},{"location":"how-to-release/#verifying-signatures","title":"Verifying Signatures","text":"<p>First, import the keys. <pre><code>curl https://dist.apache.org/repos/dist/dev/iceberg/KEYS -o KEYS\ngpg --import KEYS\n</code></pre></p> <p>Next, verify the <code>.asc</code> file. <pre><code>gpg --verify apache-iceberg-1.4.3.tar.gz.asc\n</code></pre></p>"},{"location":"how-to-release/#verifying-checksums","title":"Verifying Checksums","text":"<pre><code>shasum -a 512 --check apache-iceberg-1.4.3.tar.gz.sha512\n</code></pre>"},{"location":"how-to-release/#verifying-license-documentation","title":"Verifying License Documentation","text":"<p>Untar the archive and change into the source directory. <pre><code>tar xzf apache-iceberg-1.4.3.tar.gz\ncd apache-iceberg-1.4.3\n</code></pre></p> <p>Run RAT checks to validate license headers. <pre><code>dev/check-license\n</code></pre></p>"},{"location":"how-to-release/#verifying-build-and-test","title":"Verifying Build and Test","text":"<p>To verify that the release candidate builds properly, run the following command. <pre><code>./gradlew build\n</code></pre></p>"},{"location":"how-to-release/#testing-release-binaries","title":"Testing release binaries","text":"<p>Release announcements will also include a maven repository location. You can use this location to test downstream dependencies by adding it to your maven or gradle build.</p> <p>To use the release in your maven build, add the following to your <code>POM</code> or <code>settings.xml</code>: <pre><code>...\n <repositories>\n <repository>\n <id>iceberg-release-candidate</id>\n <name>Iceberg Release Candidate</name>\n <url>${MAVEN_URL}</url>\n </repository>\n </repositories>\n...\n</code></pre></p> <p>To use the release in your gradle build, add the following to your <code>build.gradle</code>: <pre><code>repositories {\n mavenCentral()\n maven {\n url \"${MAVEN_URL}\"\n }\n}\n</code></pre></p> <p>Note</p> <p>Replace <code>${MAVEN_URL}</code> with the URL provided in the release announcement</p>"},{"location":"how-to-release/#verifying-with-spark","title":"Verifying with Spark","text":"<p>To verify using spark, start a <code>spark-shell</code> with a command like the following command (use the appropriate spark-runtime jar for the Spark installation): <pre><code>spark-shell \\\n --conf spark.jars.repositories=${MAVEN_URL} \\\n --packages org.apache.iceberg:iceberg-spark3-runtime:1.4.3 \\\n --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\\n --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \\\n --conf spark.sql.catalog.local.type=hadoop \\\n --conf spark.sql.catalog.local.warehouse=${LOCAL_WAREHOUSE_PATH} \\\n --conf spark.sql.catalog.local.default-namespace=default \\\n --conf spark.sql.defaultCatalog=local\n</code></pre></p>"},{"location":"how-to-release/#verifying-with-flink","title":"Verifying with Flink","text":"<p>To verify using Flink, start a Flink SQL Client with the following command: <pre><code>wget ${MAVEN_URL}/iceberg-flink-runtime/1.4.3/iceberg-flink-runtime-1.4.3.jar\n\nsql-client.sh embedded \\\n -j iceberg-flink-runtime-1.4.3.jar \\\n -j ${FLINK_CONNECTOR_PACKAGE}-${HIVE_VERSION}_${SCALA_VERSION}-${FLINK_VERSION}.jar \\\n shell\n</code></pre></p>"},{"location":"how-to-release/#voting","title":"Voting","text":"<p>Votes are cast by replying to the release candidate announcement email on the dev mailing list with either <code>+1</code>, <code>0</code>, or <code>-1</code>.</p> <p>[ ] +1 Release this as Apache Iceberg 1.4.3 [ ] +0 [ ] -1 Do not release this because...</p> <p>In addition to your vote, it's customary to specify if your vote is binding or non-binding. Only members of the Project Management Committee have formally binding votes. If you're unsure, you can specify that your vote is non-binding. To read more about voting in the Apache framework, checkout the Voting information page on the Apache foundation's website.</p>"},{"location":"multi-engine-support/","title":"Multi-Engine Support","text":""},{"location":"multi-engine-support/#multi-engine-support","title":"Multi-Engine Support","text":"<p>Apache Iceberg is an open standard for huge analytic tables that can be used by any processing engine. The community continuously improves Iceberg core library components to enable integrations with different compute engines that power analytics, business intelligence, machine learning, etc. Connectors for Spark, Flink and Hive are maintained in the main Iceberg repository.</p>"},{"location":"multi-engine-support/#multi-version-support","title":"Multi-Version Support","text":"<p>Processing engine connectors maintained in the iceberg repository are built for multiple versions.</p> <p>For Spark and Flink, each new version that introduces backwards incompatible upgrade has its dedicated integration codebase and release artifacts. For example, the code for Iceberg Spark 3.4 integration is under <code>/spark/v3.4</code> and the code for Iceberg Spark 3.5 integration is under <code>/spark/v3.5</code>. Different artifacts (<code>iceberg-spark-3.4_2.12</code> and <code>iceberg-spark-3.5_2.12</code>) are released for users to consume. By doing this, changes across versions are isolated. New features in Iceberg could be developed against the latest features of an engine without breaking support of old APIs in past engine versions.</p> <p>For Hive, Hive 2 uses the <code>iceberg-mr</code> package for Iceberg integration, and Hive 3 requires an additional dependency of the <code>iceberg-hive3</code> package.</p>"},{"location":"multi-engine-support/#runtime-jar","title":"Runtime Jar","text":"<p>Iceberg provides a runtime connector jar for each supported version of Spark, Flink and Hive. When using Iceberg with these engines, the runtime jar is the only addition to the classpath needed in addition to vendor dependencies. For example, to use Iceberg with Spark 3.5 and AWS integrations, <code>iceberg-spark-runtime-3.5_2.12</code> and AWS SDK dependencies are needed for the Spark installation.</p> <p>Spark and Flink provide different runtime jars for each supported engine version. Hive 2 and Hive 3 currently share the same runtime jar. The runtime jar names and latest version download links are listed in the tables below.</p>"},{"location":"multi-engine-support/#engine-version-lifecycle","title":"Engine Version Lifecycle","text":"<p>Each engine version undergoes the following lifecycle stages:</p> <ol> <li>Beta: a new engine version is supported, but still in the experimental stage. Maybe the engine version itself is still in preview (e.g. Spark <code>3.0.0-preview</code>), or the engine does not yet have full feature compatibility compared to old versions yet. This stage allows Iceberg to release an engine version support without the need to wait for feature parity, shortening the release time.</li> <li>Maintained: an engine version is actively maintained by the community. Users can expect parity for most features across all the maintained versions. If a feature has to leverage some new engine functionalities that older versions don't have, then feature parity across maintained versions is not guaranteed.</li> <li>Deprecated: an engine version is no longer actively maintained. People who are still interested in the version can backport any necessary feature or bug fix from newer versions, but the community will not spend effort in achieving feature parity. Iceberg recommends users to move towards a newer version. Contributions to a deprecated version is expected to diminish over time, so that eventually no change is added to a deprecated version.</li> <li>End-of-life: a vote can be initiated in the community to fully remove a deprecated version out of the Iceberg repository to mark as its end of life.</li> </ol>"},{"location":"multi-engine-support/#current-engine-version-lifecycle-status","title":"Current Engine Version Lifecycle Status","text":""},{"location":"multi-engine-support/#apache-spark","title":"Apache Spark","text":"Version Lifecycle Stage Initial Iceberg Support Latest Iceberg Support Latest Runtime Jar 2.4 End of Life 0.7.0-incubating 1.2.1 iceberg-spark-runtime-2.4 3.0 End of Life 0.9.0 1.0.0 iceberg-spark-runtime-3.0_2.12 3.1 End of Life 0.12.0 1.3.1 iceberg-spark-runtime-3.1_2.12 [1] 3.2 Deprecated 0.13.0 1.4.3 iceberg-spark-runtime-3.2_2.12 3.3 Maintained 0.14.0 1.4.3 iceberg-spark-runtime-3.3_2.12 3.4 Maintained 1.3.0 1.4.3 iceberg-spark-runtime-3.4_2.12 <ul> <li>[1] Spark 3.1 shares the same runtime jar <code>iceberg-spark3-runtime</code> with Spark 3.0 before Iceberg 0.13.0</li> </ul>"},{"location":"multi-engine-support/#apache-flink","title":"Apache Flink","text":"<p>Based on the guideline of the Flink community, only the latest 2 minor versions are actively maintained. Users should continuously upgrade their Flink version to stay up-to-date.</p> Version Lifecycle Stage Initial Iceberg Support Latest Iceberg Support Latest Runtime Jar 1.11 End of Life 0.9.0 0.12.1 iceberg-flink-runtime 1.12 End of Life 0.12.0 0.13.1 iceberg-flink-runtime-1.12 [3] 1.13 End of Life 0.13.0 1.0.0 iceberg-flink-runtime-1.13 1.14 End of Life 0.13.0 1.2.0 iceberg-flink-runtime-1.14 1.15 End of Life 0.14.0 1.4.3 iceberg-flink-runtime-1.15 1.16 Deprecated 1.1.0 1.4.3 iceberg-flink-runtime-1.16 1.17 Maintained 1.3.0 1.4.3 iceberg-flink-runtime-1.17 1.18 Maintained 1.5.0 1.4.3 iceberg-flink-runtime-1.18 <ul> <li>[3] Flink 1.12 shares the same runtime jar <code>iceberg-flink-runtime</code> with Flink 1.11 before Iceberg 0.13.0</li> </ul>"},{"location":"multi-engine-support/#apache-hive","title":"Apache Hive","text":"Version Recommended minor version Lifecycle Stage Initial Iceberg Support Latest Iceberg Support Latest Runtime Jar 2 2.3.8 Maintained 0.8.0-incubating 1.4.3 iceberg-hive-runtime 3 3.1.2 Maintained 0.10.0 1.4.3 iceberg-hive-runtime"},{"location":"multi-engine-support/#developer-guide","title":"Developer Guide","text":""},{"location":"multi-engine-support/#maintaining-existing-engine-versions","title":"Maintaining existing engine versions","text":"<p>Iceberg recommends the following for developers who are maintaining existing engine versions:</p> <ol> <li>New features should always be prioritized first in the latest version, which is either a maintained or beta version.</li> <li>For features that could be backported, contributors are encouraged to either perform backports to all maintained versions, or at least create some issues to track the backport.</li> <li>If the change is small enough, updating all versions in a single PR is acceptable. Otherwise, using separated PRs for each version is recommended.</li> </ol>"},{"location":"multi-engine-support/#supporting-new-engines","title":"Supporting new engines","text":"<p>Iceberg recommends new engines to build support by importing the Iceberg libraries to the engine's project. This allows the Iceberg support to evolve with the engine. Projects such as Trino and Presto are good examples of such support strategy.</p> <p>In this approach, an Iceberg version upgrade is needed for an engine to consume new Iceberg features. To facilitate engine development against unreleased Iceberg features, a daily snapshot is published in the Apache snapshot repository.</p> <p>If bringing an engine directly to the Iceberg main repository is needed, please raise a discussion thread in the Iceberg community.</p>"},{"location":"puffin-spec/","title":"Puffin Spec","text":""},{"location":"puffin-spec/#puffin-file-format","title":"Puffin file format","text":"<p>This is a specification for Puffin, a file format designed to store information such as indexes and statistics about data managed in an Iceberg table that cannot be stored directly within the Iceberg manifest. A Puffin file contains arbitrary pieces of information (here called \"blobs\"), along with metadata necessary to interpret them. The blobs supported by Iceberg are documented at Blob types.</p>"},{"location":"puffin-spec/#format-specification","title":"Format specification","text":"<p>A file conforming to the Puffin file format specification should have the structure as described below.</p>"},{"location":"puffin-spec/#versions","title":"Versions","text":"<p>Currently, there is a single version of the Puffin file format, described below.</p>"},{"location":"puffin-spec/#file-structure","title":"File structure","text":"<p>The Puffin file has the following structure</p> <pre><code>Magic Blob\u2081 Blob\u2082 ... Blob\u2099 Footer\n</code></pre> <p>where</p> <ul> <li><code>Magic</code> is four bytes 0x50, 0x46, 0x41, 0x31 (short for: Puffin Fratercula arctica, version 1),</li> <li><code>Blob\u1d62</code> is i-th blob contained in the file, to be interpreted by application according to the footer,</li> <li><code>Footer</code> is defined below.</li> </ul>"},{"location":"puffin-spec/#footer-structure","title":"Footer structure","text":"<p>Footer has the following structure</p> <pre><code>Magic FooterPayload FooterPayloadSize Flags Magic\n</code></pre> <p>where</p> <ul> <li><code>Magic</code>: four bytes, same as at the beginning of the file</li> <li><code>FooterPayload</code>: optionally compressed, UTF-8 encoded JSON payload describing the blobs in the file, with the structure described below</li> <li><code>FooterPayloadSize</code>: a length in bytes of the <code>FooterPayload</code> (after compression, if compressed), stored as 4 byte integer</li> <li><code>Flags</code>: 4 bytes for boolean flags</li> <li>byte 0 (first)<ul> <li>bit 0 (lowest bit): whether <code>FooterPayload</code> is compressed</li> <li>all other bits are reserved for future use and should be set to 0 on write</li> </ul> </li> <li>all other bytes are reserved for future use and should be set to 0 on write</li> </ul> <p>A 4 byte integer is always signed, in a two's complement representation, stored little-endian.</p>"},{"location":"puffin-spec/#footer-payload","title":"Footer Payload","text":"<p>Footer payload bytes is either uncompressed or LZ4-compressed (as a single LZ4 compression frame with content size present), UTF-8 encoded JSON payload representing a single <code>FileMetadata</code> object.</p>"},{"location":"puffin-spec/#filemetadata","title":"FileMetadata","text":"<p><code>FileMetadata</code> has the following fields</p> Field Name Field Type Required Description blobs list of BlobMetadata objects yes properties JSON object with string property values no storage for arbitrary meta-information, like writer identification/version. See Common properties for properties that are recommended to be set by a writer."},{"location":"puffin-spec/#blobmetadata","title":"BlobMetadata","text":"<p><code>BlobMetadata</code> has the following fields</p> Field Name Field Type Required Description type JSON string yes See Blob types fields JSON list of ints yes List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob. snapshot-id JSON long yes ID of the Iceberg table's snapshot the blob was computed from. sequence-number JSON long yes Sequence number of the Iceberg table's snapshot the blob was computed from. offset JSON long yes The offset in the file where the blob contents start length JSON long yes The length of the blob stored in the file (after compression, if compressed) compression-codec JSON string no See Compression codecs. If omitted, the data is assumed to be uncompressed. properties JSON object with string property values no storage for arbitrary meta-information about the blob"},{"location":"puffin-spec/#blob-types","title":"Blob types","text":"<p>The blobs can be of a type listed below</p>"},{"location":"puffin-spec/#apache-datasketches-theta-v1-blob-type","title":"<code>apache-datasketches-theta-v1</code> blob type","text":"<p>A serialized form of a \"compact\" Theta sketch produced by the Apache DataSketches library. The sketch is obtained by constructing Alpha family sketch with default seed, and feeding it with individual distinct values converted to bytes using Iceberg's single-value serialization.</p> <p>The blob metadata for this blob may include following properties:</p> <ul> <li><code>ndv</code>: estimate of number of distinct values, derived from the sketch.</li> </ul>"},{"location":"puffin-spec/#compression-codecs","title":"Compression codecs","text":"<p>The data can also be uncompressed. If it is compressed the codec should be one of codecs listed below. For maximal interoperability, other codecs are not supported.</p> Codec name Description lz4 Single LZ4 compression frame, with content size present zstd Single Zstandard compression frame, with content size present __"},{"location":"puffin-spec/#common-properties","title":"Common properties","text":"<p>When writing a Puffin file it is recommended to set the following fields in the FileMetadata's <code>properties</code> field.</p> <ul> <li><code>created-by</code> - human-readable identification of the application writing the file, along with its version. Example \"Trino version 381\".</li> </ul>"},{"location":"releases/","title":"Releases","text":""},{"location":"releases/#downloads","title":"Downloads","text":"<p>The latest version of Iceberg is 1.4.3.</p> <ul> <li>1.4.3 source tar.gz -- signature -- sha512</li> <li>1.4.3 Spark 3.5_2.12 runtime Jar -- 3.5_2.13</li> <li>1.4.3 Spark 3.4_2.12 runtime Jar -- 3.4_2.13</li> <li>1.4.3 Spark 3.3_2.12 runtime Jar -- 3.3_2.13</li> <li>1.4.3 Spark 3.2_2.12 runtime Jar -- 3.2_2.13</li> <li>1.4.3 Flink 1.17 runtime Jar</li> <li>1.4.3 Flink 1.16 runtime Jar</li> <li>1.4.3 Flink 1.15 runtime Jar</li> <li>1.4.3 Hive runtime Jar</li> <li>1.4.3 aws-bundle Jar</li> <li>1.4.3 gcp-bundle Jar</li> <li>1.4.3 azure-bundle Jar</li> </ul> <p>To use Iceberg in Spark or Flink, download the runtime JAR for your engine version and add it to the jars folder of your installation.</p> <p>To use Iceberg in Hive 2 or Hive 3, download the Hive runtime JAR and add it to Hive using <code>ADD JAR</code>.</p>"},{"location":"releases/#gradle","title":"Gradle","text":"<p>To add a dependency on Iceberg in Gradle, add the following to <code>build.gradle</code>:</p> <pre><code>dependencies {\n compile 'org.apache.iceberg:iceberg-core:1.4.3'\n}\n</code></pre> <p>You may also want to include <code>iceberg-parquet</code> for Parquet file support.</p>"},{"location":"releases/#maven","title":"Maven","text":"<p>To add a dependency on Iceberg in Maven, add the following to your <code>pom.xml</code>:</p> <pre><code><dependencies>\n ...\n <dependency>\n <groupId>org.apache.iceberg</groupId>\n <artifactId>iceberg-core</artifactId>\n <version>1.4.3</version>\n </dependency>\n ...\n</dependencies>\n</code></pre>"},{"location":"releases/#143-release","title":"1.4.3 Release","text":"<p>Apache Iceberg 1.4.3 was released on December 27, 2023. The main issue it solves is missing files from a transaction retry with conflicting manifests. It is recommended to upgrade if you use transactions.</p> <ul> <li>Core: Scan only live entries in partitions table (#8969) by @Fokko in #9197</li> <li>Core: Fix missing files from transaction retries with conflicting manifest merges by @nastra in #9337</li> <li>JDBC Catalog: Fix namespaceExists check with special characters by @ismailsimsek in #9291</li> <li>Core: Expired Snapshot files in a transaction should be deleted by @bartash in #9223</li> <li>Core: Fix missing delete files from transaction by @nastra in #9356</li> </ul>"},{"location":"releases/#past-releases","title":"Past releases","text":""},{"location":"releases/#142-release","title":"1.4.2 Release","text":"<p>Apache Iceberg 1.4.2 was released on November 2, 2023. The 1.4.2 patch release addresses fixing a remaining case where split offsets should be ignored when they are deemed invalid.</p> <ul> <li>Core</li> <li>Core: Ignore split offsets array when split offset is past file length (#8925)</li> </ul>"},{"location":"releases/#141-release","title":"1.4.1 Release","text":"<p>Apache Iceberg 1.4.1 was released on October 23, 2023. The 1.4.1 release addresses various issues identified in the 1.4.0 release.</p> <ul> <li>Core</li> <li>Core: Do not use a lazy split offset list in manifests (#8834)</li> <li>Core: Ignore split offsets when the last split offset is past the file length (#8860)</li> <li>AWS</li> <li>Avoid static global credentials provider which doesn't play well with lifecycle management (#8677)</li> <li>Flink</li> <li>Reverting the default custom partitioner for bucket column (#8848)</li> </ul>"},{"location":"releases/#140-release","title":"1.4.0 release","text":"<p>Apache Iceberg 1.4.0 was released on October 4, 2023. The 1.4.0 release adds a variety of new features and bug fixes.</p> <ul> <li>API</li> <li>Implement bound expression sanitization (#8149)</li> <li>Remove overflow checks in <code>DefaultCounter</code> causing performance issues (#8297)</li> <li>Support incremental scanning with branch (#5984)</li> <li>Add a validation API to <code>DeleteFiles</code> which validates files exist (#8525)</li> <li>Core</li> <li>Use V2 format by default in new tables (#8381)</li> <li>Use <code>zstd</code> compression for Parquet by default in new tables (#8593)</li> <li>Add strict metadata cleanup mode and enable it by default (#8397) (#8599)</li> <li>Avoid generating huge manifests during commits (#6335)</li> <li>Add a writer for unordered position deletes (#7692)</li> <li>Optimize <code>DeleteFileIndex</code> (#8157)</li> <li>Optimize lookup in <code>DeleteFileIndex</code> without useful bounds (#8278)</li> <li>Optimize split offsets handling (#8336)</li> <li>Optimize computing user-facing state in data tasks (#8346)</li> <li>Don't persist useless file and position bounds for deletes (#8360)</li> <li>Don't persist counts for paths and positions in position delete files (#8590)</li> <li>Support setting system-level properties via environmental variables (#5659)</li> <li>Add JSON parser for <code>ContentFile</code> and <code>FileScanTask</code> (#6934)</li> <li>Add REST spec and request for commits to multiple tables (#7741)</li> <li>Add REST API for committing changes against multiple tables (#7569)</li> <li>Default to exponential retry strategy in REST client (#8366)</li> <li>Support registering tables with REST session catalog (#6512)</li> <li>Add last updated timestamp and snapshot ID to partitions metadata table (#7581)</li> <li>Add total data size to partitions metadata table (#7920)</li> <li>Extend <code>ResolvingFileIO</code> to support bulk operations (#7976)</li> <li>Key metadata in Avro format (#6450)</li> <li>Add AES GCM encryption stream (#3231)</li> <li>Fix a connection leak in streaming delete filters (#8132)</li> <li>Fix lazy snapshot loading history (#8470)</li> <li>Fix unicode handling in HTTPClient (#8046)</li> <li>Fix paths for unpartitioned specs in writers (#7685)</li> <li>Fix OOM caused by Avro decoder caching (#7791)</li> <li>Spark</li> <li>Added support for Spark 3.5<ul> <li>Code for DELETE, UPDATE, and MERGE commands has moved to Spark, and all related extensions have been dropped from Iceberg.</li> <li>Support for WHEN NOT MATCHED BY SOURCE clause in MERGE.</li> <li>Column pruning in merge-on-read operations.</li> <li>Ability to request a bigger advisory partition size for the final write to produce well-sized output files without harming the job parallelism.</li> </ul> </li> <li>Dropped support for Spark 3.1</li> <li>Deprecated support for Spark 3.2</li> <li>Support vectorized reads for merge-on-read operations in Spark 3.4 and 3.5 (#8466)</li> <li>Increase default advisory partition size for writes in Spark 3.5 (#8660)</li> <li>Support distributed planning in Spark 3.4 and 3.5 (#8123)</li> <li>Support pushing down system functions by V2 filters in Spark 3.4 and 3.5 (#7886)</li> <li>Support fanout position delta writers in Spark 3.4 and 3.5 (#7703)</li> <li>Use fanout writers for unsorted tables by default in Spark 3.5 (#8621)</li> <li>Support multiple shuffle partitions per file in compaction in Spark 3.4 and 3.5 (#7897)</li> <li>Output net changes across snapshots for carryover rows in CDC (#7326)</li> <li>Display read metrics on Spark SQL UI (#7447) (#8445)</li> <li>Adjust split size to benefit from cluster parallelism in Spark 3.4 and 3.5 (#7714)</li> <li>Add <code>fast_forward</code> procedure (#8081)</li> <li>Support filters when rewriting position deletes (#7582)</li> <li>Support setting current snapshot with ref (#8163)</li> <li>Make backup table name configurable during migration (#8227)</li> <li>Add write and SQL options to override compression config (#8313)</li> <li>Correct partition transform functions to match the spec (#8192)</li> <li>Enable extra commit properties with metadata delete (#7649)</li> <li>Flink</li> <li>Add possibility of ordering the splits based on the file sequence number (#7661)</li> <li>Fix serialization in <code>TableSink</code> with anonymous object (#7866)</li> <li>Switch to <code>FileScanTaskParser</code> for JSON serialization of <code>IcebergSourceSplit</code> (#7978)</li> <li>Custom partitioner for bucket partitions (#7161)</li> <li>Implement data statistics coordinator to aggregate data statistics from operator subtasks (#7360)</li> <li>Support alter table column (#7628)</li> <li>Parquet</li> <li>Add encryption config to read and write builders (#2639)</li> <li>Skip writing bloom filters for deletes (#7617)</li> <li>Cache codecs by name and level (#8182)</li> <li>Fix decimal data reading from <code>ParquetAvroValueReaders</code> (#8246)</li> <li>Handle filters with transforms by assuming data must be scanned (#8243)</li> <li>ORC</li> <li>Handle filters with transforms by assuming the filter matches (#8244)</li> <li>Vendor Integrations </li> <li>GCP: Fix single byte read in <code>GCSInputStream</code> (#8071)</li> <li>GCP: Add properties for OAtuh2 and update library (#8073)</li> <li>GCP: Add prefix and bulk operations to <code>GCSFileIO</code> (#8168)</li> <li>GCP: Add bundle jar for GCP-related dependencies (#8231)</li> <li>GCP: Add range reads to <code>GCSInputStream</code> (#8301)</li> <li>AWS: Add bundle jar for AWS-related dependencies (#8261)</li> <li>AWS: support config storage class for <code>S3FileIO</code> (#8154)</li> <li>AWS: Add <code>FileIO</code> tracker/closer to Glue catalog (#8315)</li> <li>AWS: Update S3 signer spec to allow an optional string body in <code>S3SignRequest</code> (#8361)</li> <li>Azure: Add <code>FileIO</code> that supports ADLSv2 storage (#8303)</li> <li>Azure: Make <code>ADLSFileIO</code> implement <code>DelegateFileIO</code> (#8563)</li> <li>Nessie: Provide better commit message on table registration (#8385)</li> <li>Dependencies</li> <li>Bump Nessie to 0.71.0</li> <li>Bump ORC to 1.9.1</li> <li>Bump Arrow to 12.0.1</li> <li>Bump AWS Java SDK to 2.20.131</li> </ul>"},{"location":"releases/#131-release","title":"1.3.1 release","text":"<p>Apache Iceberg 1.3.1 was released on July 25, 2023. The 1.3.1 release addresses various issues identified in the 1.3.0 release.</p> <ul> <li>Core</li> <li>Table Metadata parser now accepts null for fields: current-snapshot-id, properties, and snapshots (#8064)</li> <li>Hive</li> <li>Fix HiveCatalog deleting metadata on failures in checking lock status (#7931)</li> <li>Spark</li> <li>Fix RewritePositionDeleteFiles failure for certain partition types (#8059)</li> <li>Fix RewriteDataFiles concurrency edge-case on commit timeouts (#7933)</li> <li>Fix partition-level DELETE operations for WAP branches (#7900)</li> <li>Flink</li> <li>FlinkCatalog creation no longer creates the default database (#8039)</li> </ul>"},{"location":"releases/#130-release","title":"1.3.0 release","text":"<p>Apache Iceberg 1.3.0 was released on May 30th, 2023. The 1.3.0 release adds a variety of new features and bug fixes.</p> <ul> <li>Core</li> <li>Expose file and data sequence numbers in ContentFile (#7555)</li> <li>Improve bit density in object storage layout (#7128)</li> <li>Store split offsets for delete files (#7011)</li> <li>Readable metrics in entries metadata table (#7539)</li> <li>Delete file stats in partitions metadata table (#6661)</li> <li>Optimized vectorized reads for Parquet Decimal (#3249)</li> <li>Vectorized reads for Parquet INT96 timestamps in imported data (#6962)</li> <li>Support selected vector with ORC row and batch readers (#7197)</li> <li>Clean up expired metastore clients (#7310)</li> <li>Support for deleting old partition spec columns in V1 tables (#7398)</li> <li>Spark</li> <li>Initial support for Spark 3.4</li> <li>Removed integration for Spark 2.4</li> <li>Support for storage-partitioned joins with mismatching keys in Spark 3.4 (MERGE commands) (#7424)</li> <li>Support for TimestampNTZ in Spark 3.4 (#7553)</li> <li>Ability to handle skew during writes in Spark 3.4 (#7520)</li> <li>Ability to coalesce small tasks during writes in Spark 3.4 (#7532)</li> <li>Distribution and ordering enhancements in Spark 3.4 (#7637)</li> <li>Action for rewriting position deletes (#7389)</li> <li>Procedure for rewriting position deletes (#7572)</li> <li>Avoid local sort for MERGE cardinality check (#7558)</li> <li>Support for rate limits in Structured Streaming (#4479)</li> <li>Read and write support for UUIDs (#7399)</li> <li>Concurrent compaction is enabled by default (#6907)</li> <li>Support for metadata columns in changelog tables (#7152)</li> <li>Add file group failure info for data compaction (#7361)</li> <li>Flink</li> <li>Initial support for Flink 1.17</li> <li>Removed integration for Flink 1.14</li> <li>Data statistics operator to collect traffic distribution for guiding smart shuffling (#6382)</li> <li>Data statistics operator sends local data statistics to coordinator and receives aggregated data statistics from coordinator for smart shuffling (#7269)</li> <li>Exposed write parallelism in SQL hints (#7039)</li> <li>Row-level filtering (#7109)</li> <li>Use starting sequence number by default when rewriting data files (#7218)</li> <li>Config for max allowed consecutive planning failures in IcebergSource before failing the job (#7571)</li> <li>Vendor Integrations</li> <li>AWS: Use Apache HTTP client as default AWS HTTP client (#7119)</li> <li>AWS: Prevent token refresh scheduling on every sign request (#7270)</li> <li>AWS: Disable local credentials if remote signing is enabled (#7230)</li> <li>Dependencies</li> <li>Bump Arrow to 12.0.0</li> <li>Bump ORC to 1.8.3</li> <li>Bump Parquet to 1.13.1</li> <li>Bump Nessie to 0.59.0</li> </ul>"},{"location":"releases/#121-release","title":"1.2.1 release","text":"<p>Apache Iceberg 1.2.1 was released on April 11th, 2023. The 1.2.1 release is a patch release to address various issues identified in the prior release. Here is an overview:</p> <ul> <li>CORE</li> <li>REST: fix previous locations for refs-only load #7284</li> <li>Parse snapshot-id as long in remove-statistics update #7235</li> <li>Spark</li> <li>Broadcast table instead of file IO in rewrite manifests #7263</li> <li>Revert \"Spark: Add \"Iceberg\" prefix to SparkTable name string for SparkUI #7273</li> <li>AWS</li> <li>Make AuthSession cache static #7289</li> <li>Abort S3 input stream on close if not EOS #7262</li> <li>Disable local credentials if remote signing is enabled #7230</li> <li>Prevent token refresh scheduling on every sign request #7270</li> <li>S3 Credentials provider support in DefaultAwsClientFactory #7066</li> </ul>"},{"location":"releases/#120-release","title":"1.2.0 release","text":"<p>Apache Iceberg 1.2.0 was released on March 20th, 2023. The 1.2.0 release adds a variety of new features and bug fixes. Here is an overview:</p> <ul> <li>Core</li> <li>Added AES GCM encrpytion stream spec (#5432)</li> <li>Added support for Delta Lake to Iceberg table conversion (#6449, #6880)</li> <li>Added support for <code>position_deletes</code> metadata table (#6365, #6716)</li> <li>Added support for scan and commit metrics reporter that is pluggable through catalog (#6404, #6246, #6410) </li> <li>Added support for branch commit for all operations (#4926, #5010)</li> <li>Added <code>FileIO</code> support for ORC readers and writers (#6293)</li> <li>Updated all actions to leverage bulk delete whenever possible (#6682)</li> <li>Updated snapshot ID definition in Puffin spec to support statistics file reuse (#6272)</li> <li>Added human-readable metrics information in <code>files</code> metadata table (#5376)</li> <li>Fixed incorrect Parquet row group skipping when min and max values are <code>NaN</code> (#6517)</li> <li>Fixed a bug that location provider could generate paths with double slash (<code>//</code>) which is not compatible in a Hadoop file system (#6777)</li> <li>Fixed metadata table time travel failure for tables that performed schema evolution (#6980)</li> <li>Spark</li> <li>Added time range query support for changelog table (#6350)</li> <li>Added changelog view procedure for v1 table (#6012)</li> <li>Added support for storage partition joins to improve read and write performance (#6371)</li> <li>Updated default Arrow environment settings to improve read performance (#6550)</li> <li>Added aggregate pushdown support for <code>min</code>, <code>max</code> and <code>count</code> to improve read performance (#6622)</li> <li>Updated default distribution mode settings to improve write performance (#6828, #6838)</li> <li>Updated DELETE to perform metadata-only update whenever possible to improve write performance (#6899)</li> <li>Improved predicate pushdown support for write operations (#6636)</li> <li>Added support for reading a branch or tag through table identifier and <code>VERSION AS OF</code> (a.k.a. <code>FOR SYSTEM_VERSION AS OF</code>) SQL syntax (#6717, #6575)</li> <li>Added support for writing to a branch through identifier or through write-audit-publish (WAP) workflow settings (#6965, #7050)</li> <li>Added DDL SQL extensions to create, replace and drop a branch or tag (#6638, #6637, #6752, #6807)</li> <li>Added UDFs for <code>years</code>, <code>months</code>, <code>days</code> and <code>hours</code> transforms (#6207, #6261, #6300, #6339)</li> <li>Added partition related stats for <code>add_files</code> procedure result (#6797)</li> <li>Fixed a bug that <code>rewrite_manifests</code> procedure produced a new manifest even when there was no rewrite performed (#6659)</li> <li>Fixed a bug that statistics files were not cleaned up in <code>expire_snapshots</code> procedure (#6090)</li> <li>Flink</li> <li>Added support for metadata tables (#6222)</li> <li>Added support for read options in Flink source (#5967)</li> <li>Added support for reading and writing Avro <code>GenericRecord</code> (#6557, #6584)</li> <li>Added support for reading a branch or tag and write to a branch (#6660, #5029)</li> <li>Added throttling support for streaming read (#6299)</li> <li>Added support for multiple sinks for the same table in the same job (#6528)</li> <li>Fixed a bug that metrics config was not applied to equality and position deletes (#6271, #6313)</li> <li>Vendor Integrations</li> <li>Added Snowflake catalog integration (#6428)</li> <li>Added AWS sigV4 authentication support for REST catalog (#6951)</li> <li>Added support for AWS S3 remote signing (#6169, #6835, #7080)</li> <li>Updated AWS Glue catalog to skip table version archive by default (#6919)</li> <li>Updated AWS Glue catalog to not require a warehouse location (#6586)</li> <li>Fixed a bug that a bucket-only AWS S3 location such as <code>s3://my-bucket</code> could not be parsed (#6352)</li> <li>Fixed a bug that unnecessary HTTP client dependencies had to be included to use any AWS integration (#6746)</li> <li>Fixed a bug that AWS Glue catalog did not respect custom catalog ID when determining default warehouse location (#6223)</li> <li>Fixes a bug that AWS DynamoDB catalog namespace listing result was incomplete (#6823)</li> <li>Dependencies</li> <li>Upgraded ORC to 1.8.1 (#6349)</li> <li>Upgraded Jackson to 2.14.1 (#6168)</li> <li>Upgraded AWS SDK V2 to 2.20.18 (#7003)</li> <li>Upgraded Nessie to 0.50.0 (#6875)</li> </ul> <p>For more details, please visit Github.</p>"},{"location":"releases/#110-release","title":"1.1.0 release","text":"<p>Apache Iceberg 1.1.0 was released on November 28th, 2022. The 1.1.0 release deprecates various pre-1.0.0 methods, and adds a variety of new features. Here is an overview:</p> <ul> <li>Core</li> <li>Puffin statistics have been added to the Table API</li> <li>Support for Table scan reporting, which enables collection of statistics of the table scans.</li> <li>Add file sequence number to ManifestEntry</li> <li>Support register table for all the catalogs (previously it was only for Hive)</li> <li>Support performing merge appends and delete files on branches</li> <li>Improved Expire Snapshots FileCleanupStrategy</li> <li>SnapshotProducer supports branch writes</li> <li>Spark</li> <li>Support for aggregate expressions</li> <li>SparkChangelogTable for querying changelogs</li> <li>Dropped support for Apache Spark 3.0</li> <li>Flink</li> <li>FLIP-27 reader is supported in SQL</li> <li>Added support for Flink 1.16, dropped support for Flink 1.13</li> <li>Dependencies</li> <li>AWS SDK: 2.17.257</li> <li>Nessie: 0.44</li> <li>Apache ORC: 1.8.0 (Also, supports setting bloom filters on row groups)</li> </ul> <p>For more details, please visit Github.</p>"},{"location":"releases/#100-release","title":"1.0.0 release","text":"<p>The 1.0.0 release officially guarantees the stability of the Iceberg API.</p> <p>Iceberg's API has been largely stable since very early releases and has been integrated with many processing engines, but was still released under a 0.y.z version number indicating that breaking changes may happen. From 1.0.0 forward, the project will follow semver in the public API module, iceberg-api.</p> <p>This release removes deprecated APIs that are no longer part of the API. To make transitioning to the new release easier, it is based on the 0.14.1 release with only important bug fixes:</p> <ul> <li>Increase metrics limit to 100 columns (#5933)</li> <li>Bump Spark patch versions for CVE-2022-33891 (#5292)</li> <li>Exclude Scala from Spark runtime Jars (#5884)</li> </ul>"},{"location":"releases/#0141-release","title":"0.14.1 release","text":"<p>This release includes all bug fixes from the 0.14.x patch releases.</p>"},{"location":"releases/#notable-bug-fixes","title":"Notable bug fixes","text":"<ul> <li>API</li> <li>API: Fix ID assignment in schema merging (#5395)</li> <li>Core</li> <li>Core: Fix snapshot log with intermediate transaction snapshots (#5568)</li> <li>Core: Fix exception handling in BaseTaskWriter (#5683)</li> <li>Core: Support deleting tables without metadata files (#5510)</li> <li>Core: Add CommitStateUnknownException handling to REST (#5694)</li> <li>Spark</li> <li>Spark: Fix stats in rewrite metadata action (#5691)</li> <li>File Formats</li> <li>Parquet: Close zstd input stream early to avoid memory pressure (#5681)</li> <li>Vendor Integrations</li> <li>Core, AWS: Fix Kryo serialization failure for FileIO (#5437)</li> <li>AWS: S3OutputStream - failure to close should persist on subsequent close calls (#5311)</li> </ul>"},{"location":"releases/#0140-release","title":"0.14.0 release","text":"<p>Apache Iceberg 0.14.0 was released on 16 July 2022.</p>"},{"location":"releases/#highlights","title":"Highlights","text":"<ul> <li>Added several performance improvements for scan planning and Spark queries</li> <li>Added a common REST catalog client that uses change-based commits to resolve commit conflicts on the service side</li> <li>Added support for Spark 3.3, including <code>AS OF</code> syntax for SQL time travel queries</li> <li>Added support for Scala 2.13 with Spark 3.2 or later</li> <li>Added merge-on-read support for MERGE and UPDATE queries in Spark 3.2 or later</li> <li>Added support to rewrite partitions using zorder</li> <li>Added support for Flink 1.15 and dropped support for Flink 1.12</li> <li>Added a spec and implementation for Puffin, a format for large stats and index blobs, like Theta sketches or bloom filters</li> <li>Added new interfaces for consuming data incrementally (both append and changelog scans)</li> <li>Added support for bulk operations and ranged reads to FileIO interfaces</li> <li>Added more metadata tables to show delete files in the metadata tree</li> </ul>"},{"location":"releases/#high-level-features","title":"High-level features","text":"<ul> <li>API</li> <li>Added IcebergBuild to expose Iceberg version and build information</li> <li>Added binary compatibility checking to the build (#4638, #4798)</li> <li>Added a new IncrementalAppendScan interface and planner implementation (#4580)</li> <li>Added a new IncrementalChangelogScan interface (#4870)</li> <li>Refactored the ScanTask hierarchy to create new task types for changelog scans (#5077)</li> <li>Added expression sanitizer (#4672)</li> <li>Added utility to check expression equivalence (#4947)</li> <li>Added support for serializing FileIO instances using initialization properties (#5178)</li> <li>Updated Snapshot methods to accept a FileIO to read metadata files, deprecated old methods (#4873)</li> <li>Added optional interfaces to FileIO, for batch deletes (#4052), prefix operations (#5096), and ranged reads (#4608)</li> <li>Core</li> <li>Added a common client for REST-based catalog services that uses a change-based protocol (#4320, #4319)</li> <li>Added Puffin, a file format for statistics and index payloads or sketches (#4944, #4537)</li> <li>Added snapshot references to track tags and branches (#4019)</li> <li>ManageSnapshots now supports multiple operations using transactions, and added branch and tag operations (#4128, #4071)</li> <li>ReplacePartitions and OverwriteFiles now support serializable isolation (#2925, #4052)</li> <li>Added new metadata tables: <code>data_files</code> (#4336), <code>delete_files</code> (#4243), <code>all_delete_files</code>, and <code>all_files</code> (#4694)</li> <li>Added deleted files to the <code>files</code> metadata table (#4336) and delete file counts to the <code>manifests</code> table (#4764)</li> <li>Added support for predicate pushdown for the <code>all_data_files</code> metadata table (#4382) and the <code>all_manifests</code> table (#4736)</li> <li>Added support for catalogs to default table properties on creation (#4011)</li> <li>Updated sort order construction to ensure all partition fields are added to avoid partition closed failures (#5131)</li> <li>Spark</li> <li>Spark 3.3 is now supported (#5056)</li> <li>Added SQL time travel using <code>AS OF</code> syntax in Spark 3.3 (#5156)</li> <li>Scala 2.13 is now supported for Spark 3.2 and 3.3 (#4009)</li> <li>Added support for the <code>mergeSchema</code> option for DataFrame writes (#4154)</li> <li>MERGE and UPDATE queries now support the lazy / merge-on-read strategy (#3984, #4047)</li> <li>Added zorder rewrite strategy to the <code>rewrite_data_files</code> stored procedure and action (#3983, #4902)</li> <li>Added a <code>register_table</code> stored procedure to create tables from metadata JSON files (#4810)</li> <li>Added a <code>publish_changes</code> stored procedure to publish staged commits by ID (#4715)</li> <li>Added <code>CommitMetadata</code> helper class to set snapshot summary properties from SQL (#4956)</li> <li>Added support to supply a file listing to remove orphan data files procedure and action (#4503)</li> <li>Added FileIO metrics to the Spark UI (#4030, #4050)</li> <li>DROP TABLE now supports the PURGE flag (#3056)</li> <li>Added support for custom isolation level for dynamic partition overwrites (#2925) and filter overwrites (#4293)</li> <li>Schema identifier fields are now shown in table properties (#4475)</li> <li>Abort cleanup now supports parallel execution (#4704)</li> <li>Flink</li> <li>Flink 1.15 is now supported (#4553)</li> <li>Flink 1.12 support was removed (#4551)</li> <li>Added a FLIP-27 source and builder to 1.14 and 1.15 (#5109)</li> <li>Added an option to set the monitor interval (#4887) and an option to limit the number of snapshots in a streaming read planning operation (#4943)</li> <li>Added support for write options, like <code>write-format</code> to Flink sink builder (#3998)</li> <li>Added support for task locality when reading from HDFS (#3817)</li> <li>Use Hadoop configuration files from <code>hadoop-conf-dir</code> property (#4622)</li> <li>Vendor integrations</li> <li>Added Dell ECS integration (#3376, #4221)</li> <li>JDBC catalog now supports namespace properties (#3275)</li> <li>AWS Glue catalog supports native Glue locking (#4166)</li> <li>AWS S3FileIO supports using S3 access points (#4334), bulk operations (#4052, #5096), ranged reads (#4608), and tagging at write time or in place of deletes (#4259, #4342)</li> <li>AWS GlueCatalog supports passing LakeFormation credentials (#4280) </li> <li>AWS DynamoDB catalog and lock supports overriding the DynamoDB endpoint (#4726)</li> <li>Nessie now supports namespaces and namespace properties (#4385, #4610)</li> <li>Nessie now passes most common catalog tests (#4392)</li> <li>Parquet</li> <li>Added support for row group skipping using Parquet bloom filters (#4938)</li> <li>Added table configuration options for writing Parquet bloom filters (#5035)</li> <li>ORC</li> <li>Support file rolling at a target file size (#4419)</li> <li>Support table compression settings, <code>write.orc.compression-codec</code> and <code>write.orc.compression-strategy</code> (#4273)</li> </ul>"},{"location":"releases/#performance-improvements","title":"Performance improvements","text":"<ul> <li>Core</li> <li>Fixed manifest file handling in scan planning to open manifests in the planning threadpool (#5206)</li> <li>Avoided an extra S3 HEAD request by passing file length when opening manifest files (#5207)</li> <li>Refactored Arrow vectorized readers to avoid extra dictionary copies (#5137)</li> <li>Improved Arrow decimal handling to improve decimal performance (#5168, #5198)</li> <li>Added support for Avro files with Zstd compression (#4083)</li> <li>Column metrics are now disabled by default after the first 32 columns (#3959, #5215)</li> <li>Updated delete filters to copy row wrappers to avoid expensive type analysis (#5249)</li> <li>Snapshot expiration supports parallel execution (#4148)</li> <li>Manifest updates can use a custom thread pool (#4146)</li> <li>Spark</li> <li>Parquet vectorized reads are enabled by default (#4196)</li> <li>Scan statistics now adjust row counts for split data files (#4446)</li> <li>Implemented <code>SupportsReportStatistics</code> in <code>ScanBuilder</code> to work around SPARK-38962 (#5136)</li> <li>Updated Spark tables to avoid expensive (and inaccurate) size estimation (#5225)</li> <li>Flink</li> <li>Operators will now use a worker pool per job (#4177)</li> <li>Fixed <code>ClassCastException</code> thrown when reading arrays from Parquet (#4432)</li> <li>Hive</li> <li>Added vectorized Parquet reads for Hive 3 (#3980)</li> <li>Improved generic reader performance using copy instead of create (#4218)</li> </ul>"},{"location":"releases/#notable-bug-fixes_1","title":"Notable bug fixes","text":"<p>This release includes all bug fixes from the 0.13.x patch releases.</p> <ul> <li>Core</li> <li>Fixed an exception thrown when metadata-only deletes encounter delete files that are partially matched (#4304)</li> <li>Fixed transaction retries for changes without validations, like schema updates, that could ignore an update (#4464)</li> <li>Fixed failures when reading metadata tables with evolved partition specs (#4520, #4560)</li> <li>Fixed delete files dropped when a manifest is rewritten following a format version upgrade (#4514)</li> <li>Fixed missing metadata files resulting from an OOM during commit cleanup (#4673)</li> <li>Updated logging to use sanitized expressions to avoid leaking values (#4672)</li> <li>Spark</li> <li>Fixed Spark to skip calling abort when CommitStateUnknownException is thrown (#4687)</li> <li>Fixed MERGE commands with mixed case identifiers (#4848)</li> <li>Flink</li> <li>Fixed table property update failures when tables have a primary key (#4561)</li> <li>Integrations</li> <li>JDBC catalog behavior has been updated to pass common catalog tests (#4220, #4231)</li> </ul>"},{"location":"releases/#dependency-changes","title":"Dependency changes","text":"<ul> <li>Updated Apache Avro to 1.10.2 (previously 1.10.1)</li> <li>Updated Apache Parquet to 1.12.3 (previously 1.12.2)</li> <li>Updated Apache ORC to 1.7.5 (previously 1.7.2)</li> <li>Updated Apache Arrow to 7.0.0 (previously 6.0.0)</li> <li>Updated AWS SDK to 2.17.131 (previously 2.15.7)</li> <li>Updated Nessie to 0.30.0 (previously 0.18.0)</li> <li>Updated Caffeine to 2.9.3 (previously 2.8.4)</li> </ul>"},{"location":"releases/#0132","title":"0.13.2","text":"<p>Apache Iceberg 0.13.2 was released on June 15th, 2022.</p> <ul> <li>Git tag: 0.13.2</li> <li>0.13.2 source tar.gz -- signature -- sha512</li> <li>0.13.2 Spark 3.2 runtime Jar</li> <li>0.13.2 Spark 3.1 runtime Jar</li> <li>0.13.2 Spark 3.0 runtime Jar</li> <li>0.13.2 Spark 2.4 runtime Jar</li> <li>0.13.2 Flink 1.14 runtime Jar</li> <li>0.13.2 Flink 1.13 runtime Jar</li> <li>0.13.2 Flink 1.12 runtime Jar</li> <li>0.13.2 Hive runtime Jar</li> </ul> <p>Important bug fixes and changes:</p> <ul> <li>Core</li> <li>#4673 fixes table corruption from OOM during commit cleanup</li> <li>#4514 row delta delete files were dropped in sequential commits after table format updated to v2</li> <li>#4464 fixes an issue were conflicting transactions have been ignored during a commit</li> <li>#4520 fixes an issue with wrong table predicate filtering with evolved partition specs</li> <li>Spark</li> <li>#4663 fixes NPEs in Spark value converter</li> <li>#4687 fixes an issue with incorrect aborts when non-runtime exceptions were thrown in Spark</li> <li>Flink</li> <li>Note that there's a correctness issue when using upsert mode in Flink 1.12. Given that Flink 1.12 is deprecated, it was decided to not fix this bug but rather log a warning (see also #4754).</li> <li>Nessie</li> <li>#4509 fixes a NPE that occurred when accessing refreshed tables in NessieCatalog</li> </ul> <p>A more exhaustive list of changes is available under the 0.13.2 release milestone.</p>"},{"location":"releases/#0131","title":"0.13.1","text":"<p>Apache Iceberg 0.13.1 was released on February 14th, 2022.</p> <ul> <li>Git tag: 0.13.1</li> <li>0.13.1 source tar.gz -- signature -- sha512</li> <li>0.13.1 Spark 3.2 runtime Jar</li> <li>0.13.1 Spark 3.1 runtime Jar</li> <li>0.13.1 Spark 3.0 runtime Jar</li> <li>0.13.1 Spark 2.4 runtime Jar</li> <li>0.13.1 Flink 1.14 runtime Jar</li> <li>0.13.1 Flink 1.13 runtime Jar</li> <li>0.13.1 Flink 1.12 runtime Jar</li> <li>0.13.1 Hive runtime Jar</li> </ul> <p>Important bug fixes:</p> <ul> <li>Spark</li> <li>#4023 fixes predicate pushdown in row-level operations for merge conditions in Spark 3.2. Prior to the fix, filters would not be extracted and targeted merge conditions were not pushed down leading to degraded performance for these targeted merge operations.</li> <li> <p>#4024 fixes table creation in the root namespace of a Hadoop Catalog.</p> </li> <li> <p>Flink</p> </li> <li>#3986 fixes manifest location collisions when there are multiple committers in the same Flink job.</li> </ul>"},{"location":"releases/#0130","title":"0.13.0","text":"<p>Apache Iceberg 0.13.0 was released on February 4th, 2022.</p> <ul> <li>Git tag: 0.13.0</li> <li>0.13.0 source tar.gz -- signature -- sha512</li> <li>0.13.0 Spark 3.2 runtime Jar</li> <li>0.13.0 Spark 3.1 runtime Jar</li> <li>0.13.0 Spark 3.0 runtime Jar</li> <li>0.13.0 Spark 2.4 runtime Jar</li> <li>0.13.0 Flink 1.14 runtime Jar</li> <li>0.13.0 Flink 1.13 runtime Jar</li> <li>0.13.0 Flink 1.12 runtime Jar</li> <li>0.13.0 Hive runtime Jar</li> </ul> <p>High-level features:</p> <ul> <li>Core</li> <li>Catalog caching now supports cache expiration through catalog property <code>cache.expiration-interval-ms</code> [#3543]</li> <li>Catalog now supports registration of Iceberg table from a given metadata file location [#3851]</li> <li>Hadoop catalog can be used with S3 and other file systems safely by using a lock manager [#3663]</li> <li>Vendor Integrations</li> <li>Google Cloud Storage (GCS) <code>FileIO</code> is supported with optimized read and write using GCS streaming transfer [#3711]</li> <li>Aliyun Object Storage Service (OSS) <code>FileIO</code> is supported [#3553]</li> <li>Any S3-compatible storage (e.g. MinIO) can now be accessed through AWS <code>S3FileIO</code> with custom endpoint and credential configurations [#3656] [#3658]</li> <li>AWS <code>S3FileIO</code> now supports server-side checksum validation [#3813]</li> <li>AWS <code>GlueCatalog</code> now displays more table information including table location, description [#3467] and columns [#3888]</li> <li>Using multiple <code>FileIO</code>s based on file path scheme is supported by configuring a <code>ResolvingFileIO</code> [#3593]</li> <li>Spark</li> <li>Spark 3.2 is supported [#3335] with merge-on-read <code>DELETE</code> [#3970]</li> <li><code>RewriteDataFiles</code> action now supports sort-based table optimization [#2829] and merge-on-read delete compaction [#3454]. The corresponding Spark call procedure <code>rewrite_data_files</code> is also supported [#3375]</li> <li>Time travel queries now use snapshot schema instead of the table's latest schema [#3722]</li> <li>Spark vectorized reads now support row-level deletes [#3557] [#3287]</li> <li><code>add_files</code> procedure now skips duplicated files by default (can be turned off with the <code>check_duplicate_files</code> flag) [#2895], skips folder without file [#2895] and partitions with <code>null</code> values [#2895] instead of throwing exception, and supports partition pruning for faster table import [#3745]</li> <li>Flink</li> <li>Flink 1.13 and 1.14 are supported [#3116] [#3434]</li> <li>Flink connector support is supported [#2666]</li> <li>Upsert write option is supported [#2863]</li> <li>Hive</li> <li>Table listing in Hive catalog can now skip non-Iceberg tables by disabling flag <code>list-all-tables</code> [#3908]</li> <li>Hive tables imported to Iceberg can now be read by <code>IcebergInputFormat</code> [#3312]</li> <li>File Formats</li> <li>ORC now supports writing delete file [#3248] [#3250] [#3366]</li> </ul> <p>Important bug fixes:</p> <ul> <li>Core</li> <li>Iceberg new data file root path is configured through <code>write.data.path</code> going forward. <code>write.folder-storage.path</code> and <code>write.object-storage.path</code> are deprecated [#3094]</li> <li>Catalog commit status is <code>UNKNOWN</code> instead of <code>FAILURE</code> when new metadata location cannot be found in snapshot history [#3717]</li> <li>Dropping table now also deletes old metadata files instead of leaving them strained [#3622]</li> <li><code>history</code> and <code>snapshots</code> metadata tables can now query tables with no current snapshot instead of returning empty [#3812]</li> <li>Vendor Integrations</li> <li>Using cloud service integrations such as AWS <code>GlueCatalog</code> and <code>S3FileIO</code> no longer fail when missing Hadoop dependencies in the execution environment [#3590]</li> <li>AWS clients are now auto-closed when related <code>FileIO</code> or <code>Catalog</code> is closed. There is no need to close the AWS clients separately [#2878]</li> <li>Spark</li> <li>For Spark >= 3.1, <code>REFRESH TABLE</code> can now be used with Spark session catalog instead of throwing exception [#3072]</li> <li>Insert overwrite mode now skips partition with 0 record instead of failing the write operation [#2895]</li> <li>Spark snapshot expiration action now supports custom <code>FileIO</code> instead of just <code>HadoopFileIO</code> [#3089]</li> <li><code>REPLACE TABLE AS SELECT</code> can now work with tables with columns that have changed partition transform. Each old partition field of the same column is converted to a void transform with a different name [#3421]</li> <li>Spark SQL filters containing binary or fixed literals can now be pushed down instead of throwing exception [#3728]</li> <li>Flink</li> <li>A <code>ValidationException</code> will be thrown if a user configures both <code>catalog-type</code> and <code>catalog-impl</code>. Previously it chose to use <code>catalog-type</code>. The new behavior brings Flink consistent with Spark and Hive [#3308]</li> <li>Changelog tables can now be queried without <code>RowData</code> serialization issues [#3240]</li> <li><code>java.sql.Time</code> data type can now be written without data overflow problem [#3740]</li> <li>Avro position delete files can now be read without encountering <code>NullPointerException</code> [#3540]</li> <li>Hive</li> <li>Hive catalog can now be initialized with a <code>null</code> Hadoop configuration instead of throwing exception [#3252]</li> <li>Table creation can now succeed instead of throwing exception when some columns do not have comments [#3531]</li> <li>File Formats</li> <li>Parquet file writing issue is fixed for string data with over 16 unparseable chars (e.g. high/low surrogates) [#3760]</li> <li>ORC vectorized read is now configured using <code>read.orc.vectorization.batch-size</code> instead of <code>read.parquet.vectorization.batch-size</code> [#3133]</li> </ul> <p>Other notable changes:</p> <ul> <li>The community has finalized the long-term strategy of Spark, Flink and Hive support. See Multi-Engine Support page for more details.</li> </ul>"},{"location":"releases/#0121","title":"0.12.1","text":"<p>Apache Iceberg 0.12.1 was released on November 8th, 2021.</p> <ul> <li>Git tag: 0.12.1</li> <li>0.12.1 source tar.gz -- signature -- sha512</li> <li>0.12.1 Spark 3.x runtime Jar</li> <li>0.12.1 Spark 2.4 runtime Jar</li> <li>0.12.1 Flink runtime Jar</li> <li>0.12.1 Hive runtime Jar</li> </ul> <p>Important bug fixes and changes:</p> <ul> <li>#3264 fixes validation failures that occurred after snapshot expiration when writing Flink CDC streams to Iceberg tables.</li> <li>#3264 fixes reading projected map columns from Parquet files written before Parquet 1.11.1.</li> <li>#3195 allows validating that commits that produce row-level deltas don't conflict with concurrently added files. Ensures users can maintain serializable isolation for update and delete operations, including merge operations.</li> <li>#3199 allows validating that commits that overwrite files don't conflict with concurrently added files. Ensures users can maintain serializable isolation for overwrite operations.</li> <li>#3135 fixes equality-deletes using <code>DATE</code>, <code>TIMESTAMP</code>, and <code>TIME</code> types.</li> <li>#3078 prevents the JDBC catalog from overwriting the <code>jdbc.user</code> property if any property called user exists in the environment.</li> <li>#3035 fixes drop namespace calls with the DyanmoDB catalog.</li> <li>#3273 fixes importing Avro files via <code>add_files</code> by correctly setting the number of records.</li> <li>#3332 fixes importing ORC files with float or double columns in <code>add_files</code>.</li> </ul> <p>A more exhaustive list of changes is available under the 0.12.1 release milestone.</p>"},{"location":"releases/#0120","title":"0.12.0","text":"<p>Apache Iceberg 0.12.0 was released on August 15, 2021. It consists of 395 commits authored by 74 contributors over a 139 day period.</p> <ul> <li>Git tag: 0.12.0</li> <li>0.12.0 source tar.gz -- signature -- sha512</li> <li>0.12.0 Spark 3.x runtime Jar</li> <li>0.12.0 Spark 2.4 runtime Jar</li> <li>0.12.0 Flink runtime Jar</li> <li>0.12.0 Hive runtime Jar</li> </ul> <p>High-level features:</p> <ul> <li>Core<ul> <li>Allow Iceberg schemas to specify one or more columns as row identifiers [#2465]. Note that this is a prerequisite for supporting upserts in Flink.</li> <li>Added JDBC [#1870] and DynamoDB [#2688] catalog implementations.</li> <li>Added predicate pushdown for partitions and files metadata tables [#2358, #2926].</li> <li>Added a new, more flexible compaction action for Spark that can support different strategies such as bin packing and sorting. [#2501, #2609].</li> <li>Added the ability to upgrade to v2 or create a v2 table using the table property format-version=2 [#2887].</li> <li>Added support for nulls in StructLike collections [#2929].</li> <li>Added <code>key_metadata</code> field to manifest lists for encryption [#2675].</li> </ul> </li> <li>Flink<ul> <li>Added support for SQL primary keys [#2410].</li> </ul> </li> <li>Hive<ul> <li>Added the ability to set the catalog at the table level in the Hive Metastore. This makes it possible to write queries that reference tables from multiple catalogs [#2129].</li> <li>As a result of [#2129], deprecated the configuration property <code>iceberg.mr.catalog</code> which was previously used to configure the Iceberg catalog in MapReduce and Hive [#2565].</li> <li>Added table-level JVM lock on commits[#2547].</li> <li>Added support for Hive's vectorized ORC reader [#2613].</li> </ul> </li> <li>Spark<ul> <li>Added <code>SET</code> and <code>DROP IDENTIFIER FIELDS</code> clauses to <code>ALTER TABLE</code> so people don't have to look up the DDL [#2560].</li> <li>Added support for <code>ALTER TABLE REPLACE PARTITION FIELD</code> DDL [#2365].</li> <li>Added support for micro-batch streaming reads for structured streaming in Spark3 [#2660].</li> <li>Improved the performance of importing a Hive table by not loading all partitions from Hive and instead pushing the partition filter to the Metastore [#2777].</li> <li>Added support for <code>UPDATE</code> statements in Spark [#2193, #2206].</li> <li>Added support for Spark 3.1 [#2512].</li> <li>Added <code>RemoveReachableFiles</code> action [#2415].</li> <li>Added <code>add_files</code> stored procedure [#2210].</li> <li>Refactored Actions API and added a new entry point.</li> <li>Added support for Hadoop configuration overrides [#2922].</li> <li>Added support for the <code>TIMESTAMP WITHOUT TIMEZONE</code> type in Spark [#2757].</li> <li>Added validation that files referenced by row-level deletes are not concurrently rewritten [#2308].</li> </ul> </li> </ul> <p>Important bug fixes:</p> <ul> <li>Core<ul> <li>Fixed string bucketing with non-BMP characters [#2849].</li> <li>Fixed Parquet dictionary filtering with fixed-length byte arrays and decimals [#2551].</li> <li>Fixed a problem with the configuration of HiveCatalog [#2550].</li> <li>Fixed partition field IDs in table replacement [#2906].</li> </ul> </li> <li>Hive<ul> <li>Enabled dropping HMS tables even if the metadata on disk gets corrupted [#2583].</li> </ul> </li> <li>Parquet<ul> <li>Fixed Parquet row group filters when types are promoted from <code>int</code> to <code>long</code> or from <code>float</code> to <code>double</code> [#2232]</li> </ul> </li> <li>Spark<ul> <li>Fixed <code>MERGE INTO</code> in Spark when used with <code>SinglePartition</code> partitioning [#2584].</li> <li>Fixed nested struct pruning in Spark [#2877].</li> <li>Fixed NaN handling for float and double metrics [#2464].</li> <li>Fixed Kryo serialization for data and delete files [#2343].</li> </ul> </li> </ul> <p>Other notable changes:</p> <ul> <li>The Iceberg Community voted to approve version 2 of the Apache Iceberg Format Specification. The differences between version 1 and 2 of the specification are documented here.</li> <li>Bugfixes and stability improvements for NessieCatalog.</li> <li>Improvements and fixes for Iceberg's Python library.</li> <li>Added a vectorized reader for Apache Arrow [#2286].</li> <li>The following Iceberg dependencies were upgraded:<ul> <li>Hive 2.3.8 [#2110].</li> <li>Avro 1.10.1 [#1648].</li> <li>Parquet 1.12.0 [#2441].</li> </ul> </li> </ul>"},{"location":"releases/#0111","title":"0.11.1","text":"<ul> <li>Git tag: 0.11.1</li> <li>0.11.1 source tar.gz -- signature -- sha512</li> <li>0.11.1 Spark 3.0 runtime Jar</li> <li>0.11.1 Spark 2.4 runtime Jar</li> <li>0.11.1 Flink runtime Jar</li> <li>0.11.1 Hive runtime Jar</li> </ul> <p>Important bug fixes:</p> <ul> <li>#2367 prohibits deleting data files when tables are dropped if GC is disabled.</li> <li>#2196 fixes data loss after compaction when large files are split into multiple parts and only some parts are combined with other files.</li> <li>#2232 fixes row group filters with promoted types in Parquet.</li> <li>#2267 avoids listing non-Iceberg tables in Glue.</li> <li>#2254 fixes predicate pushdown for Date in Hive.</li> <li>#2126 fixes writing of Date, Decimal, Time, UUID types in Hive.</li> <li>#2241 fixes vectorized ORC reads with metadata columns in Spark.</li> <li>#2154 refreshes the relation cache in DELETE and MERGE operations in Spark.</li> </ul>"},{"location":"releases/#0110","title":"0.11.0","text":"<ul> <li>Git tag: 0.11.0</li> <li>0.11.0 source tar.gz -- signature -- sha512</li> <li>0.11.0 Spark 3.0 runtime Jar</li> <li>0.11.0 Spark 2.4 runtime Jar</li> <li>0.11.0 Flink runtime Jar</li> <li>0.11.0 Hive runtime Jar</li> </ul> <p>High-level features:</p> <ul> <li>Core API now supports partition spec and sort order evolution</li> <li>Spark 3 now supports the following SQL extensions:<ul> <li>MERGE INTO (experimental)</li> <li>DELETE FROM (experimental)</li> <li>ALTER TABLE ... ADD/DROP PARTITION</li> <li>ALTER TABLE ... WRITE ORDERED BY</li> <li>Invoke stored procedures using CALL</li> </ul> </li> <li>Flink now supports streaming reads, CDC writes (experimental), and filter pushdown</li> <li>AWS module is added to support better integration with AWS, with AWS Glue catalog support and dedicated S3 FileIO implementation</li> <li>Nessie module is added to support integration with project Nessie</li> </ul> <p>Important bug fixes:</p> <ul> <li>#1981 fixes bug that date and timestamp transforms were producing incorrect values for dates and times before 1970. Before the fix, negative values were incorrectly transformed by date and timestamp transforms to 1 larger than the correct value. For example, <code>day(1969-12-31 10:00:00)</code> produced 0 instead of -1. The fix is backwards compatible, which means predicate projection can still work with the incorrectly transformed partitions written using older versions.</li> <li>#2091 fixes <code>ClassCastException</code> for type promotion <code>int</code> to <code>long</code> and <code>float</code> to <code>double</code> during Parquet vectorized read. Now Arrow vector is created by looking at Parquet file schema instead of Iceberg schema for <code>int</code> and <code>float</code> fields.</li> <li>#1998 fixes bug in <code>HiveTableOperation</code> that <code>unlock</code> is not called if new metadata cannot be deleted. Now it is guaranteed that <code>unlock</code> is always called for Hive catalog users.</li> <li>#1979 fixes table listing failure in Hadoop catalog when user does not have permission to some tables. Now the tables with no permission are ignored in listing.</li> <li>#1798 fixes scan task failure when encountering duplicate entries of data files. Spark and Flink readers can now ignore duplicated entries in data files for each scan task.</li> <li>#1785 fixes invalidation of metadata tables in <code>CachingCatalog</code>. When a table is dropped, all the metadata tables associated with it are also invalidated in the cache.</li> <li>#1960 fixes bug that ORC writer does not read metrics config and always use the default. Now customized metrics config is respected.</li> </ul> <p>Other notable changes:</p> <ul> <li>NaN counts are now supported in metadata</li> <li>Shared catalog properties are added in core library to standardize catalog level configurations</li> <li>Spark and Flink now support dynamically loading customized <code>Catalog</code> and <code>FileIO</code> implementations</li> <li>Spark 2 now supports loading tables from other catalogs, like Spark 3</li> <li>Spark 3 now supports catalog names in DataFrameReader when using Iceberg as a format</li> <li>Flink now uses the number of Iceberg read splits as its job parallelism to improve performance and save resource.</li> <li>Hive (experimental) now supports INSERT INTO, case insensitive query, projection pushdown, create DDL with schema and auto type conversion</li> <li>ORC now supports reading tinyint, smallint, char, varchar types</li> <li>Avro to Iceberg schema conversion now preserves field docs</li> </ul>"},{"location":"releases/#0100","title":"0.10.0","text":"<ul> <li>Git tag: 0.10.0</li> <li>0.10.0 source tar.gz -- signature -- sha512</li> <li>0.10.0 Spark 3.0 runtime Jar</li> <li>0.10.0 Spark 2.4 runtime Jar</li> <li>0.10.0 Flink runtime Jar</li> <li>0.10.0 Hive runtime Jar</li> </ul> <p>High-level features:</p> <ul> <li>Format v2 support for building row-level operations (<code>MERGE INTO</code>) in processing engines<ul> <li>Note: format v2 is not yet finalized and does not have a forward-compatibility guarantee</li> </ul> </li> <li>Flink integration for writing to Iceberg tables and reading from Iceberg tables (reading supports batch mode only)</li> <li>Hive integration for reading from Iceberg tables, with filter pushdown (experimental; configuration may change)</li> </ul> <p>Important bug fixes:</p> <ul> <li>#1706 fixes non-vectorized ORC reads in Spark that incorrectly skipped rows</li> <li>#1536 fixes ORC conversion of <code>notIn</code> and <code>notEqual</code> to match null values</li> <li>#1722 fixes <code>Expressions.notNull</code> returning an <code>isNull</code> predicate; API only, method was not used by processing engines</li> <li>#1736 fixes <code>IllegalArgumentException</code> in vectorized Spark reads with negative decimal values</li> <li>#1666 fixes file lengths returned by the ORC writer, using compressed size rather than uncompressed size</li> <li>#1674 removes catalog expiration in HiveCatalogs</li> <li>#1545 automatically refreshes tables in Spark when not caching table instances</li> </ul> <p>Other notable changes:</p> <ul> <li>The <code>iceberg-hive</code> module has been renamed to <code>iceberg-hive-metastore</code> to avoid confusion</li> <li>Spark 3 is based on 3.0.1 that includes the fix for SPARK-32168</li> <li>Hadoop tables will recover from version hint corruption</li> <li>Tables can be configured with a required sort order</li> <li>Data file locations can be customized with a dynamically loaded <code>LocationProvider</code></li> <li>ORC file imports can apply a name mapping for stats</li> </ul> <p>A more exhaustive list of changes is available under the 0.10.0 release milestone.</p>"},{"location":"releases/#091","title":"0.9.1","text":"<ul> <li>Git tag: 0.9.1</li> <li>0.9.1 source tar.gz -- signature -- sha512</li> <li>0.9.1 Spark 3.0 runtime Jar</li> <li>0.9.1 Spark 2.4 runtime Jar</li> </ul>"},{"location":"releases/#090","title":"0.9.0","text":"<ul> <li>Git tag: 0.9.0</li> <li>0.9.0 source tar.gz -- signature -- sha512</li> <li>0.9.0 Spark 3.0 runtime Jar</li> <li>0.9.0 Spark 2.4 runtime Jar</li> </ul>"},{"location":"releases/#080","title":"0.8.0","text":"<ul> <li>Git tag: apache-iceberg-0.8.0-incubating</li> <li>0.8.0-incubating source tar.gz -- signature -- sha512</li> <li>0.8.0-incubating Spark 2.4 runtime Jar</li> </ul>"},{"location":"releases/#070","title":"0.7.0","text":"<ul> <li>Git tag: apache-iceberg-0.7.0-incubating</li> <li>0.7.0-incubating source tar.gz -- signature -- sha512</li> <li>0.7.0-incubating Spark 2.4 runtime Jar</li> </ul>"},{"location":"roadmap/","title":"Roadmap","text":""},{"location":"roadmap/#roadmap-overview","title":"Roadmap Overview","text":"<p>This roadmap outlines projects that the Iceberg community is working on. Each high-level item links to a Github project board that tracks the current status. Related design docs will be linked on the planning boards.</p>"},{"location":"roadmap/#general","title":"General","text":"<ul> <li>Multi-table transaction support</li> <li>Views Support</li> <li>Change Data Capture (CDC) Support</li> <li>Snapshot tagging and branching</li> <li>Inline file compaction</li> <li>Delete File compaction</li> <li>Z-ordering / Space-filling curves</li> <li>Support UPSERT</li> </ul>"},{"location":"roadmap/#clients","title":"Clients","text":"<p>Python, Rust, and Go projects are pointing to their respective repositories which include their own issues as the implementations are not final.</p> <ul> <li>Add the Iceberg Python Client</li> <li>Add the Iceberg Rust Client</li> <li>Add the Iceberg Go Client</li> </ul>"},{"location":"roadmap/#spec-v2","title":"Spec V2","text":"<ul> <li>Views Spec</li> <li>DSv2 streaming improvements</li> <li>Secondary indexes</li> </ul>"},{"location":"roadmap/#spec-v3","title":"Spec V3","text":"<ul> <li>Encryption</li> <li>Relative paths</li> <li>Default field values</li> </ul>"},{"location":"security/","title":"Security","text":""},{"location":"security/#reporting-security-issues","title":"Reporting Security Issues","text":"<p>The Apache Iceberg Project uses the standard process outlined by the Apache Security Team for reporting vulnerabilities. Note that vulnerabilities should not be publicly disclosed until the project has responded.</p> <p>To report a possible security vulnerability, please email security@iceberg.apache.org.</p>"},{"location":"security/#verifying-signed-releases","title":"Verifying Signed Releases","text":"<p>Please refer to the instructions on the Release Verification page.</p>"},{"location":"spark-quickstart/","title":"Spark and Iceberg Quickstart","text":""},{"location":"spark-quickstart/#spark-and-iceberg-quickstart","title":"Spark and Iceberg Quickstart","text":"<p>This guide will get you up and running with an Iceberg and Spark environment, including sample code to highlight some powerful features. You can learn more about Iceberg's Spark runtime by checking out the Spark section.</p> <ul> <li>Docker-Compose</li> <li>Creating a table</li> <li>Writing Data to a Table</li> <li>Reading Data from a Table</li> <li>Adding A Catalog</li> <li>Next Steps</li> </ul>"},{"location":"spark-quickstart/#docker-compose","title":"Docker-Compose","text":"<p>The fastest way to get started is to use a docker-compose file that uses the tabulario/spark-iceberg image which contains a local Spark cluster with a configured Iceberg catalog. To use this, you'll need to install the Docker CLI as well as the Docker Compose CLI.</p> <p>Once you have those, save the yaml below into a file named <code>docker-compose.yml</code>:</p> <pre><code>version: \"3\"\n\nservices:\n spark-iceberg:\n image: tabulario/spark-iceberg\n container_name: spark-iceberg\n build: spark/\n networks:\n iceberg_net:\n depends_on:\n - rest\n - minio\n volumes:\n - ./warehouse:/home/iceberg/warehouse\n - ./notebooks:/home/iceberg/notebooks/notebooks\n environment:\n - AWS_ACCESS_KEY_ID=admin\n - AWS_SECRET_ACCESS_KEY=password\n - AWS_REGION=us-east-1\n ports:\n - 8888:8888\n - 8080:8080\n - 10000:10000\n - 10001:10001\n rest:\n image: tabulario/iceberg-rest\n container_name: iceberg-rest\n networks:\n iceberg_net:\n ports:\n - 8181:8181\n environment:\n - AWS_ACCESS_KEY_ID=admin\n - AWS_SECRET_ACCESS_KEY=password\n - AWS_REGION=us-east-1\n - CATALOG_WAREHOUSE=s3://warehouse/\n - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO\n - CATALOG_S3_ENDPOINT=http://minio:9000\n minio:\n image: minio/minio\n container_name: minio\n environment:\n - MINIO_ROOT_USER=admin\n - MINIO_ROOT_PASSWORD=password\n - MINIO_DOMAIN=minio\n networks:\n iceberg_net:\n aliases:\n - warehouse.minio\n ports:\n - 9001:9001\n - 9000:9000\n command: [\"server\", \"/data\", \"--console-address\", \":9001\"]\n mc:\n depends_on:\n - minio\n image: minio/mc\n container_name: mc\n networks:\n iceberg_net:\n environment:\n - AWS_ACCESS_KEY_ID=admin\n - AWS_SECRET_ACCESS_KEY=password\n - AWS_REGION=us-east-1\n entrypoint: >\n /bin/sh -c \"\n until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;\n /usr/bin/mc rm -r --force minio/warehouse;\n /usr/bin/mc mb minio/warehouse;\n /usr/bin/mc policy set public minio/warehouse;\n tail -f /dev/null\n \"\nnetworks:\n iceberg_net:\n</code></pre> <p>Next, start up the docker containers with this command: <pre><code>docker-compose up\n</code></pre></p> <p>You can then run any of the following commands to start a Spark session.</p> SparkSQLSpark-ShellPySpark <pre><code>docker exec -it spark-iceberg spark-sql\n</code></pre> <pre><code>docker exec -it spark-iceberg spark-shell\n</code></pre> <pre><code>docker exec -it spark-iceberg pyspark\n</code></pre> <p>Note</p> <p>You can also launch a notebook server by running <code>docker exec -it spark-iceberg notebook</code>. The notebook server will be available at http://localhost:8888</p>"},{"location":"spark-quickstart/#creating-a-table","title":"Creating a table","text":"<p>To create your first Iceberg table in Spark, run a <code>CREATE TABLE</code> command. Let's create a table using <code>demo.nyc.taxis</code> where <code>demo</code> is the catalog name, <code>nyc</code> is the database name, and <code>taxis</code> is the table name.</p> SparkSQLSpark-ShellPySpark <pre><code>CREATE TABLE demo.nyc.taxis\n(\n vendor_id bigint,\n trip_id bigint,\n trip_distance float,\n fare_amount double,\n store_and_fwd_flag string\n)\nPARTITIONED BY (vendor_id);\n</code></pre> <pre><code>import org.apache.spark.sql.types._\nimport org.apache.spark.sql.Row\nval schema = StructType( Array(\n StructField(\"vendor_id\", LongType,true),\n StructField(\"trip_id\", LongType,true),\n StructField(\"trip_distance\", FloatType,true),\n StructField(\"fare_amount\", DoubleType,true),\n StructField(\"store_and_fwd_flag\", StringType,true)\n))\nval df = spark.createDataFrame(spark.sparkContext.emptyRDD[Row],schema)\ndf.writeTo(\"demo.nyc.taxis\").create()\n</code></pre> <pre><code>from pyspark.sql.types import DoubleType, FloatType, LongType, StructType,StructField, StringType\nschema = StructType([\n StructField(\"vendor_id\", LongType(), True),\n StructField(\"trip_id\", LongType(), True),\n StructField(\"trip_distance\", FloatType(), True),\n StructField(\"fare_amount\", DoubleType(), True),\n StructField(\"store_and_fwd_flag\", StringType(), True)\n])\n\ndf = spark.createDataFrame([], schema)\ndf.writeTo(\"demo.nyc.taxis\").create()\n</code></pre> <p>Iceberg catalogs support the full range of SQL DDL commands, including:</p> <ul> <li><code>CREATE TABLE ... PARTITIONED BY</code></li> <li><code>CREATE TABLE ... AS SELECT</code></li> <li><code>ALTER TABLE</code></li> <li><code>DROP TABLE</code></li> </ul>"},{"location":"spark-quickstart/#writing-data-to-a-table","title":"Writing Data to a Table","text":"<p>Once your table is created, you can insert records.</p> SparkSQLSpark-ShellPySpark <pre><code>INSERT INTO demo.nyc.taxis\nVALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');\n</code></pre> <pre><code>import org.apache.spark.sql.Row\n\nval schema = spark.table(\"demo.nyc.taxis\").schema\nval data = Seq(\n Row(1: Long, 1000371: Long, 1.8f: Float, 15.32: Double, \"N\": String),\n Row(2: Long, 1000372: Long, 2.5f: Float, 22.15: Double, \"N\": String),\n Row(2: Long, 1000373: Long, 0.9f: Float, 9.01: Double, \"N\": String),\n Row(1: Long, 1000374: Long, 8.4f: Float, 42.13: Double, \"Y\": String)\n)\nval df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)\ndf.writeTo(\"demo.nyc.taxis\").append()\n</code></pre> <pre><code>schema = spark.table(\"demo.nyc.taxis\").schema\ndata = [\n (1, 1000371, 1.8, 15.32, \"N\"),\n (2, 1000372, 2.5, 22.15, \"N\"),\n (2, 1000373, 0.9, 9.01, \"N\"),\n (1, 1000374, 8.4, 42.13, \"Y\")\n ]\ndf = spark.createDataFrame(data, schema)\ndf.writeTo(\"demo.nyc.taxis\").append()\n</code></pre>"},{"location":"spark-quickstart/#reading-data-from-a-table","title":"Reading Data from a Table","text":"<p>To read a table, simply use the Iceberg table's name.</p> SparkSQLSpark-ShellPySpark <pre><code>SELECT * FROM demo.nyc.taxis;\n</code></pre> <pre><code>val df = spark.table(\"demo.nyc.taxis\").show()\n</code></pre> <pre><code>df = spark.table(\"demo.nyc.taxis\").show()\n</code></pre>"},{"location":"spark-quickstart/#adding-a-catalog","title":"Adding A Catalog","text":"<p>Iceberg has several catalog back-ends that can be used to track tables, like JDBC, Hive MetaStore and Glue. Catalogs are configured using properties under <code>spark.sql.catalog.(catalog_name)</code>. In this guide, we use JDBC, but you can follow these instructions to configure other catalog types. To learn more, check out the Catalog page in the Spark section.</p> <p>This configuration creates a path-based catalog named <code>local</code> for tables under <code>$PWD/warehouse</code> and adds support for Iceberg tables to Spark's built-in catalog.</p> CLIspark-defaults.conf <pre><code>spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3\\\n --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\\n --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \\\n --conf spark.sql.catalog.spark_catalog.type=hive \\\n --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \\\n --conf spark.sql.catalog.local.type=hadoop \\\n --conf spark.sql.catalog.local.warehouse=$PWD/warehouse \\\n --conf spark.sql.defaultCatalog=local\n</code></pre> <pre><code>spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3\nspark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions\nspark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog\nspark.sql.catalog.spark_catalog.type hive\nspark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog\nspark.sql.catalog.local.type hadoop\nspark.sql.catalog.local.warehouse $PWD/warehouse\nspark.sql.defaultCatalog local\n</code></pre> <p>Note</p> <p>If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing <code>USE local;</code></p>"},{"location":"spark-quickstart/#next-steps","title":"Next steps","text":""},{"location":"spark-quickstart/#adding-iceberg-to-spark","title":"Adding Iceberg to Spark","text":"<p>If you already have a Spark environment, you can add Iceberg, using the <code>--packages</code> option.</p> SparkSQLSpark-ShellPySpark <pre><code>spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3\n</code></pre> <pre><code>spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3\n</code></pre> <pre><code>pyspark --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3\n</code></pre> <p>Note</p> <p>If you want to include Iceberg in your Spark installation, add the Iceberg Spark runtime to Spark's <code>jars</code> folder. You can download the runtime by visiting to the Releases page.</p>"},{"location":"spark-quickstart/#learn-more","title":"Learn More","text":"<p>Now that you're up an running with Iceberg and Spark, check out the Iceberg-Spark docs to learn more!</p>"},{"location":"spec/","title":"Spec","text":""},{"location":"spec/#iceberg-table-spec","title":"Iceberg Table Spec","text":"<p>This is a specification for the Iceberg table format that is designed to manage a large, slow-changing collection of files in a distributed file system or key-value store as a table.</p>"},{"location":"spec/#format-versioning","title":"Format Versioning","text":"<p>Versions 1 and 2 of the Iceberg spec are complete and adopted by the community.</p> <p>The format version number is incremented when new features are added that will break forward-compatibility---that is, when older readers would not read newer table features correctly. Tables may continue to be written with an older version of the spec to ensure compatibility by not using features that are not yet implemented by processing engines.</p>"},{"location":"spec/#version-1-analytic-data-tables","title":"Version 1: Analytic Data Tables","text":"<p>Version 1 of the Iceberg spec defines how to manage large analytic tables using immutable file formats: Parquet, Avro, and ORC.</p> <p>All version 1 data and metadata files are valid after upgrading a table to version 2. Appendix E documents how to default version 2 fields when reading version 1 metadata.</p>"},{"location":"spec/#version-2-row-level-deletes","title":"Version 2: Row-level Deletes","text":"<p>Version 2 of the Iceberg spec adds row-level updates and deletes for analytic tables with immutable files.</p> <p>The primary change in version 2 adds delete files to encode rows that are deleted in existing data files. This version can be used to delete or replace individual rows in immutable data files without rewriting the files.</p> <p>In addition to row-level deletes, version 2 makes some requirements stricter for writers. The full set of changes are listed in Appendix E.</p>"},{"location":"spec/#goals","title":"Goals","text":"<ul> <li>Serializable isolation -- Reads will be isolated from concurrent writes and always use a committed snapshot of a table\u2019s data. Writes will support removing and adding files in a single operation and are never partially visible. Readers will not acquire locks.</li> <li>Speed -- Operations will use O(1) remote calls to plan the files for a scan and not O(n) where n grows with the size of the table, like the number of partitions or files.</li> <li>Scale -- Job planning will be handled primarily by clients and not bottleneck on a central metadata store. Metadata will include information needed for cost-based optimization.</li> <li>Evolution -- Tables will support full schema and partition spec evolution. Schema evolution supports safe column add, drop, reorder and rename, including in nested structures.</li> <li>Dependable types -- Tables will provide well-defined and dependable support for a core set of types.</li> <li>Storage separation -- Partitioning will be table configuration. Reads will be planned using predicates on data values, not partition values. Tables will support evolving partition schemes.</li> <li>Formats -- Underlying data file formats will support identical schema evolution rules and types. Both read-optimized and write-optimized formats will be available.</li> </ul>"},{"location":"spec/#overview","title":"Overview","text":"<p>This table format tracks individual data files in a table instead of directories. This allows writers to create data files in-place and only adds files to the table in an explicit commit.</p> <p>Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table.</p> <p>Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, and its metrics. The data in a snapshot is the union of all files in its manifests. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data files with any subset of a table and are not associated with partitions.</p> <p>The manifests that make up a snapshot are stored in a manifest list file. Each manifest list stores metadata about manifests, including partition stats and data file counts. These stats are used to avoid reading manifests that are not required for an operation.</p>"},{"location":"spec/#optimistic-concurrency","title":"Optimistic Concurrency","text":"<p>An atomic swap of one table metadata file for another provides the basis for serializable isolation. Readers use the snapshot that was current when they load the table metadata and are not affected by changes until they refresh and pick up a new metadata location.</p> <p>Writers create table metadata files optimistically, assuming that the current version will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the table\u2019s metadata file pointer from the base version to the new version.</p> <p>If the snapshot on which an update is based is no longer current, the writer must retry the update based on the new current version. Some operations support retry by re-applying metadata changes and committing, under well-defined conditions. For example, a change that rewrites files can be applied to a new table snapshot if all of the rewritten files are still in the table.</p> <p>The conditions required by a write to successfully commit determines the isolation level. Writers can select what to validate and can make different isolation guarantees.</p>"},{"location":"spec/#sequence-numbers","title":"Sequence Numbers","text":"<p>The relative age of data and delete files relies on a sequence number that is assigned to every successful commit. When a snapshot is created for a commit, it is optimistically assigned the next sequence number, and it is written into the snapshot's metadata. If the commit fails and must be retried, the sequence number is reassigned and written into new snapshot metadata.</p> <p>All manifests, data files, and delete files created for a snapshot inherit the snapshot's sequence number. Manifest file metadata in the manifest list stores a manifest's sequence number. New data and metadata file entries are written with <code>null</code> in place of a sequence number, which is replaced with the manifest's sequence number at read time. When a data or delete file is written to a new manifest (as \"existing\"), the inherited sequence number is written to ensure it does not change after it is first inherited.</p> <p>Inheriting the sequence number from manifest metadata allows writing a new manifest once and reusing it in commit retries. To change a sequence number for a retry, only the manifest list must be rewritten -- which would be rewritten anyway with the latest set of manifests.</p>"},{"location":"spec/#row-level-deletes","title":"Row-level Deletes","text":"<p>Row-level deletes are stored in delete files.</p> <p>There are two ways to encode a row-level delete:</p> <ul> <li>Position deletes mark a row deleted by data file path and the row position in the data file</li> <li>Equality deletes mark a row deleted by one or more column values, like <code>id = 5</code></li> </ul> <p>Like data files, delete files are tracked by partition. In general, a delete file must be applied to older data files with the same partition; see Scan Planning for details. Column metrics can be used to determine whether a delete file's rows overlap the contents of a data file or a scan range.</p>"},{"location":"spec/#file-system-operations","title":"File System Operations","text":"<p>Iceberg only requires that file systems support the following operations:</p> <ul> <li>In-place write -- Files are not moved or altered once they are written.</li> <li>Seekable reads -- Data file formats require seek support.</li> <li>Deletes -- Tables delete files that are no longer used.</li> </ul> <p>These requirements are compatible with object stores, like S3.</p> <p>Tables do not require random-access writes. Once written, data and metadata files are immutable until they are deleted.</p> <p>Tables do not require rename, except for tables that use atomic rename to implement the commit operation for new metadata files.</p>"},{"location":"spec/#specification","title":"Specification","text":""},{"location":"spec/#terms","title":"Terms","text":"<ul> <li>Schema -- Names and types of fields in a table.</li> <li>Partition spec -- A definition of how partition values are derived from data fields.</li> <li>Snapshot -- The state of a table at some point in time, including the set of all data files.</li> <li>Manifest list -- A file that lists manifest files; one per snapshot.</li> <li>Manifest -- A file that lists data or delete files; a subset of a snapshot.</li> <li>Data file -- A file that contains rows of a table.</li> <li>Delete file -- A file that encodes rows of a table that are deleted by position or data values.</li> </ul>"},{"location":"spec/#writer-requirements","title":"Writer requirements","text":"<p>Some tables in this spec have columns that specify requirements for v1 and v2 tables. These requirements are intended for writers when adding metadata files to a table with the given version.</p> Requirement Write behavior (blank) The field should be omitted optional The field can be written required The field must be written <p>Readers should be more permissive because v1 metadata files are allowed in v2 tables so that tables can be upgraded to v2 without rewriting the metadata tree. For manifest list and manifest files, this table shows the expected v2 read behavior:</p> v1 v2 v2 read behavior optional Read the field as optional required Read the field as optional; it may be missing in v1 files optional Ignore the field optional optional Read the field as optional optional required Read the field as optional; it may be missing in v1 files required Ignore the field required optional Read the field as optional required required Fill in a default or throw an exception if the field is missing <p>Readers may be more strict for metadata JSON files because the JSON files are not reused and will always match the table version. Required v2 fields that were not present in v1 or optional in v1 may be handled as required fields. For example, a v2 table that is missing <code>last-sequence-number</code> can throw an exception.</p>"},{"location":"spec/#schemas-and-data-types","title":"Schemas and Data Types","text":"<p>A table's schema is a list of named columns. All data types are either primitives or nested types, which are maps, lists, or structs. A table schema is also a struct type.</p> <p>For the representations of these types in Avro, ORC, and Parquet file formats, see Appendix A.</p>"},{"location":"spec/#nested-types","title":"Nested Types","text":"<p>A <code>struct</code> is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. Fields may have an optional comment or doc string. Fields can have default values.</p> <p>A <code>list</code> is a collection of values with some element type. The element field has an integer id that is unique in the table schema. Elements can be either optional or required. Element types may be any type.</p> <p>A <code>map</code> is a collection of key-value pairs with a key type and a value type. Both the key field and value field each have an integer id that is unique in the table schema. Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types.</p>"},{"location":"spec/#primitive-types","title":"Primitive Types","text":"Primitive type Description Requirements <code>boolean</code> True or false <code>int</code> 32-bit signed integers Can promote to <code>long</code> <code>long</code> 64-bit signed integers <code>float</code> 32-bit IEEE 754 floating point Can promote to double <code>double</code> 64-bit IEEE 754 floating point <code>decimal(P,S)</code> Fixed-point decimal; precision P, scale S Scale is fixed [1], precision must be 38 or less <code>date</code> Calendar date without timezone or time <code>time</code> Time of day without date, timezone Microsecond precision [2] <code>timestamp</code> Timestamp without timezone Microsecond precision [2] <code>timestamptz</code> Timestamp with timezone Stored as UTC [2] <code>string</code> Arbitrary-length character sequences Encoded with UTF-8 [3] <code>uuid</code> Universally unique identifiers Should use 16-byte fixed <code>fixed(L)</code> Fixed-length byte array of length L <code>binary</code> Arbitrary-length byte array <p>Notes:</p> <ol> <li>Decimal scale is fixed and cannot be changed by schema evolution. Precision can only be widened.</li> <li>All time and timestamp values are stored with microsecond precision.<ul> <li>Timestamps with time zone represent a point in time: values are stored as UTC and do not retain a source time zone (<code>2017-11-16 17:10:34 PST</code> is stored/retrieved as <code>2017-11-17 01:10:34 UTC</code> and these values are considered identical).</li> <li>Timestamps without time zone represent a date and time of day regardless of zone: the time value is independent of zone adjustments (<code>2017-11-16 17:10:34</code> is always retrieved as <code>2017-11-16 17:10:34</code>). Timestamp values are stored as a long that encodes microseconds from the unix epoch.</li> </ul> </li> <li>Character strings must be stored as UTF-8 encoded byte arrays.</li> </ol> <p>For details on how to serialize a schema to JSON, see Appendix C.</p>"},{"location":"spec/#default-values","title":"Default values","text":"<p>Default values can be tracked for struct fields (both nested structs and the top-level schema's struct). There can be two defaults with a field: - <code>initial-default</code> is used to populate the field's value for all records that were written before the field was added to the schema - <code>write-default</code> is used to populate the field's value for any records written after the field was added to the schema, if the writer does not supply the field's value</p> <p>The <code>initial-default</code> is set only when a field is added to an existing schema. The <code>write-default</code> is initially set to the same value as <code>initial-default</code> and can be changed through schema evolution. If either default is not set for an optional field, then the default value is null for compatibility with older spec versions.</p> <p>The <code>initial-default</code> and <code>write-default</code> produce SQL default value behavior, without rewriting data files. SQL default value behavior when a field is added handles all existing rows as though the rows were written with the new field's default value. Default value changes may only affect future records and all known fields are written into data files. Omitting a known field when writing a data file is never allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail.</p> <p>Default values are attributes of fields in schemas and serialized with fields in the JSON format. See Appendix C.</p>"},{"location":"spec/#schema-evolution","title":"Schema Evolution","text":"<p>Schemas may be evolved by type promotion or adding, deleting, renaming, or reordering fields in structs (both nested structs and the top-level schema\u2019s struct).</p> <p>Evolution applies changes to the table's current schema to produce a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema.</p> <p>Valid type promotions are:</p> <ul> <li><code>int</code> to <code>long</code></li> <li><code>float</code> to <code>double</code></li> <li><code>decimal(P, S)</code> to <code>decimal(P', S)</code> if <code>P' > P</code> -- widen the precision of decimal types.</li> </ul> <p>Any struct, including a top-level schema, can evolve through deleting fields, adding new fields, renaming existing fields, reordering existing fields, or promoting a primitive using the valid type promotions. Adding a new field assigns a new ID for that field and for any nested fields. Renaming an existing field must change the name, but not the field ID. Deleting a field removes it from the current schema. Field deletion cannot be rolled back unless the field was nullable or if the current snapshot has not changed.</p> <p>Grouping a subset of a struct\u2019s fields into a nested struct is not allowed, nor is moving fields from a nested struct into its immediate parent struct (<code>struct<a, b, c> \u2194 struct<a, struct<b, c>></code>). Evolving primitive types to structs is not allowed, nor is evolving a single-field struct to a primitive (<code>map<string, int> \u2194 map<string, struct<int>></code>).</p> <p>Struct evolution requires the following rules for default values: * The <code>initial-default</code> must be set when a field is added and cannot change * The <code>write-default</code> must be set when a field is added and may change * When a required field is added, both defaults must be set to a non-null value * When an optional field is added, the defaults may be null and should be explicitly set * When a new field is added to a struct with a default value, updating the struct's default is optional * If a field value is missing from a struct's <code>initial-default</code>, the field's <code>initial-default</code> must be used for the field * If a field value is missing from a struct's <code>write-default</code>, the field's <code>write-default</code> must be used for the field</p>"},{"location":"spec/#column-projection","title":"Column Projection","text":"<p>Columns in Iceberg data files are selected by field id. The table schema's column names and order may change after a data file is written, and projection must be done using field ids. If a field id is missing from a data file, its value for each row should be <code>null</code>.</p> <p>For example, a file may be written with schema <code>1: a int, 2: b string, 3: c double</code> and read using projection schema <code>3: measurement, 2: name, 4: a</code>. This must select file columns <code>c</code> (renamed to <code>measurement</code>), <code>b</code> (now called <code>name</code>), and a column of <code>null</code> values called <code>a</code>; in that order.</p> <p>Tables may also define a property <code>schema.name-mapping.default</code> with a JSON name mapping containing a list of field mapping objects. These mappings provide fallback field ids to be used when a data file does not contain field id information. Each object should contain</p> <ul> <li><code>names</code>: A required list of 0 or more names for a field. </li> <li><code>field-id</code>: An optional Iceberg field ID used when a field's name is present in <code>names</code></li> <li><code>fields</code>: An optional list of field mappings for child field of structs, maps, and lists.</li> </ul> <p>Field mapping fields are constrained by the following rules:</p> <ul> <li>A name may contain <code>.</code> but this refers to a literal name, not a nested field. For example, <code>a.b</code> refers to a field named <code>a.b</code>, not child field <code>b</code> of field <code>a</code>. </li> <li>Each child field should be defined with their own field mapping under <code>fields</code>. </li> <li>Multiple values for <code>names</code> may be mapped to a single field ID to support cases where a field may have different names in different data files. For example, all Avro field aliases should be listed in <code>names</code>.</li> <li>Fields which exist only in the Iceberg schema and not in imported data files may use an empty <code>names</code> list.</li> <li>Fields that exist in imported files but not in the Iceberg schema may omit <code>field-id</code>.</li> <li>List types should contain a mapping in <code>fields</code> for <code>element</code>. </li> <li>Map types should contain mappings in <code>fields</code> for <code>key</code> and <code>value</code>. </li> <li>Struct types should contain mappings in <code>fields</code> for their child fields.</li> </ul> <p>For details on serialization, see Appendix C.</p>"},{"location":"spec/#identifier-field-ids","title":"Identifier Field IDs","text":"<p>A schema can optionally track the set of primitive fields that identify rows in a table, using the property <code>identifier-field-ids</code> (see JSON encoding in Appendix C).</p> <p>Two rows are the \"same\"---that is, the rows represent the same entity---if the identifier fields are equal. However, uniqueness of rows by this identifier is not guaranteed or required by Iceberg and it is the responsibility of processing engines or data providers to enforce.</p> <p>Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers.</p>"},{"location":"spec/#reserved-field-ids","title":"Reserved Field IDs","text":"<p>Iceberg tables must not use field ids greater than 2147483447 (<code>Integer.MAX_VALUE - 200</code>). This id range is reserved for metadata columns that can be used in user data schemas, like the <code>_file</code> column that holds the file path in which a row was stored.</p> <p>The set of metadata columns is:</p> Field id, name Type Description <code>2147483646 _file</code> <code>string</code> Path of the file in which a row is stored <code>2147483645 _pos</code> <code>long</code> Ordinal position of a row in the source data file <code>2147483644 _deleted</code> <code>boolean</code> Whether the row has been deleted <code>2147483643 _spec_id</code> <code>int</code> Spec ID used to track the file containing a row <code>2147483642 _partition</code> <code>struct</code> Partition to which a row belongs <code>2147483546 file_path</code> <code>string</code> Path of a file, used in position-based delete files <code>2147483545 pos</code> <code>long</code> Ordinal position of a row, used in position-based delete files <code>2147483544 row</code> <code>struct<...></code> Deleted row values, used in position-based delete files"},{"location":"spec/#partitioning","title":"Partitioning","text":"<p>Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan\u2019s filter predicate. Partition values for a data file must be the same for all records stored in the data file. (Manifests store data files from any partition, as long as the partition spec is the same for the data files.)</p> <p>Tables are configured with a partition spec that defines how to produce a tuple of partition values from a record. A partition spec has a list of fields that consist of:</p> <ul> <li>A source column id from the table\u2019s schema</li> <li>A partition field id that is used to identify a partition field and is unique within a partition spec. In v2 table metadata, it is unique across all partition specs.</li> <li>A transform that is applied to the source column to produce a partition value</li> <li>A partition name</li> </ul> <p>The source column, selected by id, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct. For details on how to serialize a partition spec to JSON, see Appendix C.</p> <p>Partition specs capture the transform from table data to partition values. This is used to transform predicates to partition predicates, in addition to transforming data values. Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates. This simplifies queries because users don\u2019t have to supply both logical predicates and partition predicates. For more information, see Scan Planning below.</p>"},{"location":"spec/#partition-transforms","title":"Partition Transforms","text":"Transform name Description Source types Result type <code>identity</code> Source value, unmodified Any Source type <code>bucket[N]</code> Hash of value, mod <code>N</code> (see below) <code>int</code>, <code>long</code>, <code>decimal</code>, <code>date</code>, <code>time</code>, <code>timestamp</code>, <code>timestamptz</code>, <code>string</code>, <code>uuid</code>, <code>fixed</code>, <code>binary</code> <code>int</code> <code>truncate[W]</code> Value truncated to width <code>W</code> (see below) <code>int</code>, <code>long</code>, <code>decimal</code>, <code>string</code> Source type <code>year</code> Extract a date or timestamp year, as years from 1970 <code>date</code>, <code>timestamp</code>, <code>timestamptz</code> <code>int</code> <code>month</code> Extract a date or timestamp month, as months from 1970-01-01 <code>date</code>, <code>timestamp</code>, <code>timestamptz</code> <code>int</code> <code>day</code> Extract a date or timestamp day, as days from 1970-01-01 <code>date</code>, <code>timestamp</code>, <code>timestamptz</code> <code>int</code> <code>hour</code> Extract a timestamp hour, as hours from 1970-01-01 00:00:00 <code>timestamp</code>, <code>timestamptz</code> <code>int</code> <code>void</code> Always produces <code>null</code> Any Source type or <code>int</code> <p>All transforms must return <code>null</code> for a <code>null</code> input value.</p> <p>The <code>void</code> transform may be used to replace the transform in an existing partition field so that the field is effectively dropped in v1 tables. See partition evolution below.</p>"},{"location":"spec/#bucket-transform-details","title":"Bucket Transform Details","text":"<p>Bucket partition transforms use a 32-bit hash of the source value. The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0.</p> <p>Transforms are parameterized by a number of buckets [1], <code>N</code>. The hash mod <code>N</code> must produce a positive value by first discarding the sign bit of the hash value. In pseudo-code, the function is:</p> <pre><code> def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N\n</code></pre> <p>Notes:</p> <ol> <li>Changing the number of buckets as a table grows is possible by evolving the partition spec.</li> </ol> <p>For hash function details by type, see Appendix B.</p>"},{"location":"spec/#truncate-transform-details","title":"Truncate Transform Details","text":"Type Config Truncate specification Examples <code>int</code> <code>W</code>, width <code>v - (v % W)</code> remainders must be positive [1] <code>W=10</code>: <code>1</code> \uffeb <code>0</code>, <code>-1</code> \uffeb <code>-10</code> <code>long</code> <code>W</code>, width <code>v - (v % W)</code> remainders must be positive [1] <code>W=10</code>: <code>1</code> \uffeb <code>0</code>, <code>-1</code> \uffeb <code>-10</code> <code>decimal</code> <code>W</code>, width (no scale) <code>scaled_W = decimal(W, scale(v))</code> <code>v - (v % scaled_W)</code> [1, 2] <code>W=50</code>, <code>s=2</code>: <code>10.65</code> \uffeb <code>10.50</code> <code>string</code> <code>L</code>, length Substring of length <code>L</code>: <code>v.substring(0, L)</code> [3] <code>L=3</code>: <code>iceberg</code> \uffeb <code>ice</code> <p>Notes:</p> <ol> <li>The remainder, <code>v % W</code>, must be positive. For languages where <code>%</code> can produce negative values, the correct truncate function is: <code>v - (((v % W) + W) % W)</code></li> <li>The width, <code>W</code>, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.</li> <li>Strings are truncated to a valid UTF-8 string with no more than <code>L</code> code points.</li> </ol>"},{"location":"spec/#partition-evolution","title":"Partition Evolution","text":"<p>Table partitioning can be evolved by adding, removing, renaming, or reordering partition spec fields.</p> <p>Changing a partition spec produces a new spec identified by a unique spec ID that is added to the table's list of partition specs and may be set as the table's default spec.</p> <p>When evolving a spec, changes should not cause partition field IDs to change because the partition field IDs are used as the partition tuple field IDs in manifest files.</p> <p>In v2, partition field IDs must be explicitly tracked for each partition field. New IDs are assigned based on the last assigned partition ID in table metadata.</p> <p>In v1, partition field IDs were not tracked, but were assigned sequentially starting at 1000 in the reference implementation. This assignment caused problems when reading metadata tables based on manifest files from multiple specs because partition fields with the same ID may contain different data types. For compatibility with old versions, the following rules are recommended for partition evolution in v1 tables:</p> <ol> <li>Do not reorder partition fields</li> <li>Do not drop partition fields; instead replace the field's transform with the <code>void</code> transform</li> <li>Only add partition fields at the end of the previous partition spec</li> </ol>"},{"location":"spec/#sorting","title":"Sorting","text":"<p>Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a sort order.</p> <p>A sort order is defined by a sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:</p> <ul> <li>A source column id from the table's schema</li> <li>A transform that is used to produce values to be sorted on from the source column. This is the same transform as described in partition transforms.</li> <li>A sort direction, that can only be either <code>asc</code> or <code>desc</code></li> <li>A null order that describes the order of null values when sorted. Can only be either <code>nulls-first</code> or <code>nulls-last</code></li> </ul> <p>Order id <code>0</code> is reserved for the unsorted order. </p> <p>Sorting floating-point numbers should produce the following behavior: <code>-NaN</code> < <code>-Infinity</code> < <code>-value</code> < <code>-0</code> < <code>0</code> < <code>value</code> < <code>Infinity</code> < <code>NaN</code>. This aligns with the implementation of Java floating-point types comparisons. </p> <p>A data or delete file is associated with a sort order by the sort order's id within a manifest. Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes.</p>"},{"location":"spec/#manifests","title":"Manifests","text":"<p>A manifest is an immutable Avro file that lists data files or delete files, along with each file\u2019s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a snapshot, which tracks all of the files in a table at some point in time. Manifests are tracked by a manifest list for each table snapshot.</p> <p>A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.</p> <p>A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.</p> <p>A manifest stores files for a single partition spec. When a table\u2019s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file\u2019s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.</p> <p>A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:</p> v1 v2 Key Value required required <code>schema</code> JSON representation of the table schema at the time the manifest was written optional required <code>schema-id</code> ID of the schema used to write the manifest as a string required required <code>partition-spec</code> JSON fields representation of the partition spec used to write the manifest optional required <code>partition-spec-id</code> ID of the partition spec used to write the manifest as a string optional required <code>format-version</code> Table format version number of the manifest as a string required <code>content</code> Type of content files tracked by the manifest: \"data\" or \"deletes\" <p>The schema of a manifest file is a struct called <code>manifest_entry</code> with the following fields:</p> v1 v2 Field id, name Type Description required required <code>0 status</code> <code>int</code> with meaning: <code>0: EXISTING</code> <code>1: ADDED</code> <code>2: DELETED</code> Used to track additions and deletions. Deletes are informational only and not used in scans. required optional <code>1 snapshot_id</code> <code>long</code> Snapshot id where the file was added, or deleted if status is 2. Inherited when null. optional <code>3 sequence_number</code> <code>long</code> Data sequence number of the file. Inherited when null and status is 1 (added). optional <code>4 file_sequence_number</code> <code>long</code> File sequence number indicating when the file was added. Inherited when null and status is 1 (added). required required <code>2 data_file</code> <code>data_file</code> <code>struct</code> (see below) File path, partition tuple, metrics, ... <p><code>data_file</code> is a struct with the following fields:</p> v1 v2 Field id, name Type Description required <code>134 content</code> <code>int</code> with meaning: <code>0: DATA</code>, <code>1: POSITION DELETES</code>, <code>2: EQUALITY DELETES</code> Type of content stored by the data file: data, equality deletes, or position deletes (all v1 files are data files) required required <code>100 file_path</code> <code>string</code> Full URI for the file with FS scheme required required <code>101 file_format</code> <code>string</code> String file format name, avro, orc or parquet required required <code>102 partition</code> <code>struct<...></code> Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids required required <code>103 record_count</code> <code>long</code> Number of records in this file required required <code>104 file_size_in_bytes</code> <code>long</code> Total file size in bytes required ~~<code>105 block_size_in_bytes</code>~~ <code>long</code> Deprecated. Always write a default in v1. Do not write in v2. optional ~~<code>106 file_ordinal</code>~~ <code>int</code> Deprecated. Do not write. optional ~~<code>107 sort_columns</code>~~ <code>list<112: int></code> Deprecated. Do not write. optional optional <code>108 column_sizes</code> <code>map<117: int, 118: long></code> Map from column id to the total size on disk of all regions that store the column. Does not include bytes necessary to read other columns, like footers. Leave null for row-oriented formats (Avro) optional optional <code>109 value_counts</code> <code>map<119: int, 120: long></code> Map from column id to number of values in the column (including null and NaN values) optional optional <code>110 null_value_counts</code> <code>map<121: int, 122: long></code> Map from column id to number of null values in the column optional optional <code>137 nan_value_counts</code> <code>map<138: int, 139: long></code> Map from column id to number of NaN values in the column optional optional <code>111 distinct_counts</code> <code>map<123: int, 124: long></code> Map from column id to number of distinct values in the column; distinct counts must be derived using values in the file by counting or using sketches, but not using methods like merging existing distinct counts optional optional <code>125 lower_bounds</code> <code>map<126: int, 127: binary></code> Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] optional optional <code>128 upper_bounds</code> <code>map<129: int, 130: binary></code> Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] optional optional <code>131 key_metadata</code> <code>binary</code> Implementation-specific key metadata for encryption optional optional <code>132 split_offsets</code> <code>list<133: long></code> Split offsets for the data file. For example, all row group offsets in a Parquet file. Must be sorted ascending optional <code>135 equality_ids</code> <code>list<136: int></code> Field ids used to determine row equality in equality delete files. Required when <code>content=2</code> and should be null otherwise. Fields with ids listed in this column must be present in the delete file optional optional <code>140 sort_order_id</code> <code>int</code> ID representing sort order for this file [3]. <p>Notes:</p> <ol> <li>Single-value serialization for lower and upper bounds is detailed in Appendix D.</li> <li>For <code>float</code> and <code>double</code>, the value <code>-0.0</code> must precede <code>+0.0</code>, as in the IEEE 754 <code>totalOrder</code> predicate. NaNs are not permitted as lower or upper bounds.</li> <li>If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files should be written with a non-null order id. Position deletes are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files.</li> <li>The following field ids are reserved on <code>data_file</code>: 141.</li> </ol> <p>The <code>partition</code> struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct's field ids must match the ids from the partition spec.</p> <p>The column metrics maps are used when filtering to select both data and delete files. For delete files, the metrics must store bounds and counts for all deleted rows, or must be omitted. Storing metrics for deleted rows ensures that the values can be used during job planning to find delete files that must be merged during a scan.</p>"},{"location":"spec/#manifest-entry-fields","title":"Manifest Entry Fields","text":"<p>The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The <code>data_file</code> struct is nested inside of the manifest entry so that it can be easily passed to job planning without the manifest entry fields.</p> <p>When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).</p> <p>When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].</p> <p>Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when <code>null</code>. That is, if the field is <code>null</code> for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list. The <code>sequence_number</code> field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file. The <code>file_sequence_number</code> field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number. The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.</p> <p>Notes:</p> <ol> <li>Technically, data files can be deleted when the last snapshot that contains the file as \u201clive\u201d data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.</li> <li>Manifest list files are required in v2, so that the <code>sequence_number</code> and <code>snapshot_id</code> to inherit are always available.</li> </ol>"},{"location":"spec/#sequence-number-inheritance","title":"Sequence Number Inheritance","text":"<p>Manifests track the sequence number when a data or delete file was added to the table.</p> <p>When adding a new file, its data and file sequence numbers are set to <code>null</code> because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing <code>null</code> with the manifest's sequence number from the manifest list. It is also possible to add a new file with data that logically belongs to an older sequence number. In that case, the data sequence number must be provided explicitly and not inherited. However, the file sequence number must be always assigned when the snapshot is successfully committed.</p> <p>When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time.</p> <p>Inheriting sequence numbers through the metadata tree allows writing a new manifest without a known sequence number, so that a manifest can be written once and reused in commit retries. To change a sequence number for a retry, only the manifest list must be rewritten.</p> <p>When reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0.</p>"},{"location":"spec/#snapshots","title":"Snapshots","text":"<p>A snapshot consists of the following fields:</p> v1 v2 Field Description required required <code>snapshot-id</code> A unique long ID optional optional <code>parent-snapshot-id</code> The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent required <code>sequence-number</code> A monotonically increasing long that tracks the order of changes to a table required required <code>timestamp-ms</code> A timestamp when the snapshot was created, used for garbage collection and table inspection optional required <code>manifest-list</code> The location of a manifest list for this snapshot that tracks manifest files with additional metadata optional <code>manifests</code> A list of manifest file locations. Must be omitted if <code>manifest-list</code> is present optional required <code>summary</code> A string map that summarizes the snapshot changes, including <code>operation</code> (see below) optional optional <code>schema-id</code> ID of the table's current schema when the snapshot was created <p>The snapshot summary's <code>operation</code> field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible <code>operation</code> values are:</p> <ul> <li><code>append</code> -- Only data files were added and no files were removed.</li> <li><code>replace</code> -- Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files.</li> <li><code>overwrite</code> -- Data and delete files were added and removed in a logical overwrite operation.</li> <li><code>delete</code> -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows.</li> </ul> <p>Data and delete files for a snapshot can be stored in more than one manifest. This enables:</p> <ul> <li>Appends can add a new manifest to minimize the amount of data written, instead of adding new records by rewriting and appending to an existing manifest. (This is called a \u201cfast append\u201d.)</li> <li>Tables can use multiple partition specs. A table\u2019s partition configuration can evolve if, for example, its data volume changes. Each manifest uses a single partition spec, and queries do not need to change because partition filters are derived from data predicates.</li> <li>Large tables can be split across multiple manifests so that implementations can parallelize job planning or reduce the cost of rewriting a manifest.</li> </ul> <p>Manifests for a snapshot are tracked by a manifest list.</p> <p>Valid snapshots are stored as a list in table metadata. For serialization, see Appendix C.</p>"},{"location":"spec/#manifest-lists","title":"Manifest Lists","text":"<p>Snapshots are embedded in table metadata, but the list of manifests for a snapshot are stored in a separate manifest list file.</p> <p>A new manifest list is written for each attempt to commit a snapshot because the list of manifests always changes to produce a new snapshot. When a manifest list is written, the (optimistic) sequence number of the snapshot is written for all new manifest files tracked by the list.</p> <p>A manifest list includes summary metadata that can be used to avoid scanning all of the manifests in a snapshot when planning a table scan. This includes the number of added, existing, and deleted files, and a summary of values for each field of the partition spec used to write the manifest.</p> <p>A manifest list is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.</p> <p>Manifest list files store <code>manifest_file</code>, a struct with the following fields:</p> v1 v2 Field id, name Type Description required required <code>500 manifest_path</code> <code>string</code> Location of the manifest file required required <code>501 manifest_length</code> <code>long</code> Length of the manifest file in bytes required required <code>502 partition_spec_id</code> <code>int</code> ID of a partition spec used to write the manifest; must be listed in table metadata <code>partition-specs</code> required <code>517 content</code> <code>int</code> with meaning: <code>0: data</code>, <code>1: deletes</code> The type of files tracked by the manifest, either data or delete files; 0 for all v1 manifests required <code>515 sequence_number</code> <code>long</code> The sequence number when the manifest was added to the table; use 0 when reading v1 manifest lists required <code>516 min_sequence_number</code> <code>long</code> The minimum data sequence number of all live data or delete files in the manifest; use 0 when reading v1 manifest lists required required <code>503 added_snapshot_id</code> <code>long</code> ID of the snapshot where the manifest file was added optional required <code>504 added_files_count</code> <code>int</code> Number of entries in the manifest that have status <code>ADDED</code> (1), when <code>null</code> this is assumed to be non-zero optional required <code>505 existing_files_count</code> <code>int</code> Number of entries in the manifest that have status <code>EXISTING</code> (0), when <code>null</code> this is assumed to be non-zero optional required <code>506 deleted_files_count</code> <code>int</code> Number of entries in the manifest that have status <code>DELETED</code> (2), when <code>null</code> this is assumed to be non-zero optional required <code>512 added_rows_count</code> <code>long</code> Number of rows in all of files in the manifest that have status <code>ADDED</code>, when <code>null</code> this is assumed to be non-zero optional required <code>513 existing_rows_count</code> <code>long</code> Number of rows in all of files in the manifest that have status <code>EXISTING</code>, when <code>null</code> this is assumed to be non-zero optional required <code>514 deleted_rows_count</code> <code>long</code> Number of rows in all of files in the manifest that have status <code>DELETED</code>, when <code>null</code> this is assumed to be non-zero optional optional <code>507 partitions</code> <code>list<508: field_summary></code> (see below) A list of field summaries for each partition field in the spec. Each field in the list corresponds to a field in the manifest file\u2019s partition spec. optional optional <code>519 key_metadata</code> <code>binary</code> Implementation-specific key metadata for encryption <p><code>field_summary</code> is a struct with the following fields:</p> v1 v2 Field id, name Type Description required required <code>509 contains_null</code> <code>boolean</code> Whether the manifest contains at least one partition with a null value for the field optional optional <code>518 contains_nan</code> <code>boolean</code> Whether the manifest contains at least one partition with a NaN value for the field optional optional <code>510 lower_bound</code> <code>bytes</code> [1] Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2] optional optional <code>511 upper_bound</code> <code>bytes</code> [1] Upper bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2] <p>Notes:</p> <ol> <li>Lower and upper bounds are serialized to bytes using the single-object serialization in Appendix D. The type of used to encode the value is the type of the partition field data.</li> <li>If -0.0 is a value of the partition field, the <code>lower_bound</code> must not be +0.0, and if +0.0 is a value of the partition field, the <code>upper_bound</code> must not be -0.0.</li> </ol>"},{"location":"spec/#scan-planning","title":"Scan Planning","text":"<p>Scans are planned by reading the manifest files for the current snapshot. Deleted entries in data and delete manifests (those marked with status \"DELETED\") are not used in a scan.</p> <p>Manifests that contain no matching files, determined using either file counts or partition summaries, may be skipped.</p> <p>For each manifest, scan predicates, which filter data rows, are converted to partition predicates, which filter data and delete files. These partition predicates are used to select the data and delete files in the manifest. This conversion uses the partition spec used to write the manifest file.</p> <p>Scan predicates are converted to partition predicates using an inclusive projection: if a scan predicate matches a row, then the partition predicate must match that row\u2019s partition. This is called inclusive [1] because rows that do not match the scan predicate may be included in the scan by the partition predicate.</p> <p>For example, an <code>events</code> table with a timestamp column named <code>ts</code> that is partitioned by <code>ts_day=day(ts)</code> is queried by users with ranges over the timestamp column: <code>ts > X</code>. The inclusive projection is <code>ts_day >= day(X)</code>, which is used to select files that may have matching rows. Note that, in most cases, timestamps just before <code>X</code> will be included in the scan because the file contains rows that match the predicate and rows that do not match the predicate.</p> <p>Scan predicates are also used to filter data and delete files using column bounds and counts that are stored by field id in manifests. The same filter logic can be used for both data and delete files because both store metrics of the rows either inserted or deleted. If metrics show that a delete file has no rows that match a scan predicate, it may be ignored just as a data file would be ignored [2].</p> <p>Data files that match the query filter must be read by the scan. </p> <p>Note that for any snapshot, all file paths marked with \"ADDED\" or \"EXISTING\" may appear at most once across all manifest files in the snapshot. If a file path appears more than once, the results of the scan are undefined. Reader implementations may raise an error in this case, but are not required to do so.</p> <p>Delete files that match the query filter must be applied to data files at read time, limited by the scope of the delete file using the following rules.</p> <ul> <li>A position delete file must be applied to a data file when all of the following are true:<ul> <li>The data file's data sequence number is less than or equal to the delete file's data sequence number</li> <li>The data file's partition (both spec and partition values) is equal to the delete file's partition</li> </ul> </li> <li>An equality delete file must be applied to a data file when all of the following are true:<ul> <li>The data file's data sequence number is strictly less than the delete's data sequence number</li> <li>The data file's partition (both spec and partition values) is equal to the delete file's partition or the delete file's partition spec is unpartitioned</li> </ul> </li> </ul> <p>In general, deletes are applied only to data files that are older and in the same partition, except for two special cases:</p> <ul> <li>Equality delete files stored with an unpartitioned spec are applied as global deletes. Otherwise, delete files do not apply to files in other partitions.</li> <li>Position delete files must be applied to data files from the same commit, when the data and delete file data sequence numbers are equal. This allows deleting rows that were added in the same commit.</li> </ul> <p>Notes:</p> <ol> <li>An alternative, strict projection, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan.</li> <li>For example, if <code>file_a</code> has rows with <code>id</code> between 1 and 10 and a delete file contains rows with <code>id</code> between 1 and 4, a scan for <code>id = 9</code> may ignore the delete file because none of the deletes can match a row that will be selected.</li> </ol>"},{"location":"spec/#snapshot-reference","title":"Snapshot Reference","text":"<p>Iceberg tables keep track of branches and tags using snapshot references. Tags are labels for individual snapshots. Branches are mutable named references that can be updated by committing a new snapshot as the branch's referenced snapshot using the Commit Conflict Resolution and Retry procedures.</p> <p>The snapshot reference object records all the information of a reference including snapshot ID, reference type and Snapshot Retention Policy.</p> v1 v2 Field name Type Description required required <code>snapshot-id</code> <code>long</code> A reference's snapshot ID. The tagged snapshot or latest snapshot of a branch. required required <code>type</code> <code>string</code> Type of the reference, <code>tag</code> or <code>branch</code> optional optional <code>min-snapshots-to-keep</code> <code>int</code> For <code>branch</code> type only, a positive number for the minimum number of snapshots to keep in a branch while expiring snapshots. Defaults to table property <code>history.expire.min-snapshots-to-keep</code>. optional optional <code>max-snapshot-age-ms</code> <code>long</code> For <code>branch</code> type only, a positive number for the max age of snapshots to keep when expiring, including the latest snapshot. Defaults to table property <code>history.expire.max-snapshot-age-ms</code>. optional optional <code>max-ref-age-ms</code> <code>long</code> For snapshot references except the <code>main</code> branch, a positive number for the max age of the snapshot reference to keep while expiring snapshots. Defaults to table property <code>history.expire.max-ref-age-ms</code>. The <code>main</code> branch never expires. <p>Valid snapshot references are stored as the values of the <code>refs</code> map in table metadata. For serialization, see Appendix C.</p>"},{"location":"spec/#snapshot-retention-policy","title":"Snapshot Retention Policy","text":"<p>Table snapshots expire and are removed from metadata to allow removed or replaced data files to be physically deleted. The snapshot expiration procedure removes snapshots from table metadata and applies the table's retention policy. Retention policy can be configured both globally and on snapshot reference through properties <code>min-snapshots-to-keep</code>, <code>max-snapshot-age-ms</code> and <code>max-ref-age-ms</code>.</p> <p>When expiring snapshots, retention policies in table and snapshot references are evaluated in the following way:</p> <ol> <li>Start with an empty set of snapshots to retain</li> <li>Remove any refs (other than main) where the referenced snapshot is older than <code>max-ref-age-ms</code></li> <li>For each branch and tag, add the referenced snapshot to the retained set</li> <li>For each branch, add its ancestors to the retained set until:<ol> <li>The snapshot is older than <code>max-snapshot-age-ms</code>, AND</li> <li>The snapshot is not one of the first <code>min-snapshots-to-keep</code> in the branch (including the branch's referenced snapshot)</li> </ol> </li> <li>Expire any snapshot not in the set of snapshots to retain.</li> </ol>"},{"location":"spec/#table-metadata","title":"Table Metadata","text":"<p>Table metadata is stored as JSON. Each table metadata change creates a new table metadata file that is committed by an atomic operation. This operation is used to ensure that a new version of table metadata replaces the version on which it was based. This produces a linear history of table versions and ensures that concurrent writes are not lost.</p> <p>The atomic operation used to commit metadata depends on how tables are tracked and is not standardized by this spec. See the sections below for examples.</p>"},{"location":"spec/#table-metadata-fields","title":"Table Metadata Fields","text":"<p>Table metadata consists of the following fields:</p> v1 v2 Field Description required required <code>format-version</code> An integer version number for the format. Currently, this can be 1 or 2 based on the spec. Implementations must throw an exception if a table's version is higher than the supported version. optional required <code>table-uuid</code> A UUID that identifies the table, generated when the table is created. Implementations must throw an exception if a table's UUID does not match the expected UUID after refreshing metadata. required required <code>location</code> The table's base location. This is used by writers to determine where to store data files, manifest files, and table metadata files. required <code>last-sequence-number</code> The table's highest assigned sequence number, a monotonically increasing long that tracks the order of snapshots in a table. required required <code>last-updated-ms</code> Timestamp in milliseconds from the unix epoch when the table was last updated. Each table metadata file should update this field just before writing. required required <code>last-column-id</code> An integer; the highest assigned column ID for the table. This is used to ensure columns are always assigned an unused ID when evolving schemas. required <code>schema</code> The table\u2019s current schema. (Deprecated: use <code>schemas</code> and <code>current-schema-id</code> instead) optional required <code>schemas</code> A list of schemas, stored as objects with <code>schema-id</code>. optional required <code>current-schema-id</code> ID of the table's current schema. required <code>partition-spec</code> The table\u2019s current partition spec, stored as only fields. Note that this is used by writers to partition data, but is not used when reading because reads use the specs stored in manifest files. (Deprecated: use <code>partition-specs</code> and <code>default-spec-id</code> instead) optional required <code>partition-specs</code> A list of partition specs, stored as full partition spec objects. optional required <code>default-spec-id</code> ID of the \"current\" spec that writers should use by default. optional required <code>last-partition-id</code> An integer; the highest assigned partition field ID across all partition specs for the table. This is used to ensure partition fields are always assigned an unused ID when evolving specs. optional optional <code>properties</code> A string to string map of table properties. This is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata. For example, <code>commit.retry.num-retries</code> is used to control the number of commit retries. optional optional <code>current-snapshot-id</code> <code>long</code> ID of the current table snapshot; must be the same as the current ID of the <code>main</code> branch in <code>refs</code>. optional optional <code>snapshots</code> A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. optional optional <code>snapshot-log</code> A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. optional optional <code>metadata-log</code> A list (optional) of timestamp and metadata file location pairs that encodes changes to the previous metadata files for the table. Each time a new metadata file is created, a new entry of the previous metadata file location should be added to the list. Tables can be configured to remove oldest metadata log entries and keep a fixed-size log of the most recent entries after a commit. optional required <code>sort-orders</code> A list of sort orders, stored as full sort order objects. optional required <code>default-sort-order-id</code> Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. optional <code>refs</code> A map of snapshot references. The map keys are the unique snapshot reference names in the table, and the map values are snapshot reference objects. There is always a <code>main</code> branch reference pointing to the <code>current-snapshot-id</code> even if the <code>refs</code> map is null. optional optional <code>statistics</code> A list (optional) of table statistics. <p>For serialization details, see Appendix C.</p>"},{"location":"spec/#table-statistics","title":"Table statistics","text":"<p>Table statistics files are valid Puffin files. Statistics are informational. A reader can choose to ignore statistics information. Statistics support is not required to read the table correctly. A table can contain many statistics files associated with different table snapshots.</p> <p>Statistics files metadata within <code>statistics</code> table metadata field is a struct with the following fields:</p> v1 v2 Field name Type Description required required <code>snapshot-id</code> <code>string</code> ID of the Iceberg table's snapshot the statistics file is associated with. required required <code>statistics-path</code> <code>string</code> Path of the statistics file. See Puffin file format. required required <code>file-size-in-bytes</code> <code>long</code> Size of the statistics file. required required <code>file-footer-size-in-bytes</code> <code>long</code> Total size of the statistics file's footer (not the footer payload size). See Puffin file format for footer definition. optional optional <code>key-metadata</code> Base64-encoded implementation-specific key metadata for encryption. required required <code>blob-metadata</code> <code>list<blob metadata></code> (see below) A list of the blob metadata for statistics contained in the file with structure described below. <p>Blob metadata is a struct with the following fields:</p> v1 v2 Field name Type Description required required <code>type</code> <code>string</code> Type of the blob. Matches Blob type in the Puffin file. required required <code>snapshot-id</code> <code>long</code> ID of the Iceberg table's snapshot the blob was computed from. required required <code>sequence-number</code> <code>long</code> Sequence number of the Iceberg table's snapshot the blob was computed from. required required <code>fields</code> <code>list<integer></code> Ordered list of fields, given by field ID, on which the statistic was calculated. optional optional <code>properties</code> <code>map<string, string></code> Additional properties associated with the statistic. Subset of Blob properties in the Puffin file."},{"location":"spec/#commit-conflict-resolution-and-retry","title":"Commit Conflict Resolution and Retry","text":"<p>When two commits happen at the same time and are based on the same version, only one commit will succeed. In most cases, the failed commit can be applied to the new current version of table metadata and retried. Updates verify the conditions under which they can be applied to a new version and retry if those conditions are met.</p> <ul> <li>Append operations have no requirements and can always be applied.</li> <li>Replace operations must verify that the files that will be deleted are still in the table. Examples of replace operations include format changes (replace an Avro file with a Parquet file) and compactions (several files are replaced with a single file that contains the same rows).</li> <li>Delete operations must verify that specific files to delete are still in the table. Delete operations based on expressions can always be applied (e.g., where timestamp < X).</li> <li>Table schema updates and partition spec changes must validate that the schema has not changed between the base version and the current version.</li> </ul>"},{"location":"spec/#file-system-tables","title":"File System Tables","text":"<p>An atomic swap can be implemented using atomic rename in file systems that support it, like HDFS or most local file systems [1].</p> <p>Each version of table metadata is stored in a metadata folder under the table\u2019s base location using a file naming scheme that includes a version number, <code>V</code>: <code>v<V>.metadata.json</code>. To commit a new metadata version, <code>V+1</code>, the writer performs the following steps:</p> <ol> <li>Read the current table metadata version <code>V</code>.</li> <li>Create new table metadata based on version <code>V</code>.</li> <li>Write the new table metadata to a unique file: <code><random-uuid>.metadata.json</code>.</li> <li>Rename the unique file to the well-known file for version <code>V</code>: <code>v<V+1>.metadata.json</code>.<ol> <li>If the rename succeeds, the commit succeeded and <code>V+1</code> is the table\u2019s current version</li> <li>If the rename fails, go back to step 1.</li> </ol> </li> </ol> <p>Notes:</p> <ol> <li>The file system table scheme is implemented in HadoopTableOperations.</li> </ol>"},{"location":"spec/#metastore-tables","title":"Metastore Tables","text":"<p>The atomic swap needed to commit new versions of table metadata can be implemented by storing a pointer in a metastore or database that is updated with a check-and-put operation [1]. The check-and-put validates that the version of the table that a write is based on is still current and then makes the new metadata from the write the current version.</p> <p>Each version of table metadata is stored in a metadata folder under the table\u2019s base location using a naming scheme that includes a version and UUID: <code><V>-<random-uuid>.metadata.json</code>. To commit a new metadata version, <code>V+1</code>, the writer performs the following steps:</p> <ol> <li>Create a new table metadata file based on the current metadata.</li> <li>Write the new table metadata to a unique file: <code><V+1>-<random-uuid>.metadata.json</code>.</li> <li>Request that the metastore swap the table\u2019s metadata pointer from the location of <code>V</code> to the location of <code>V+1</code>.<ol> <li>If the swap succeeds, the commit succeeded. <code>V</code> was still the latest metadata version and the metadata file for <code>V+1</code> is now the current metadata.</li> <li>If the swap fails, another writer has already created <code>V+1</code>. The current writer goes back to step 1.</li> </ol> </li> </ol> <p>Notes:</p> <ol> <li>The metastore table scheme is partly implemented in BaseMetastoreTableOperations.</li> </ol>"},{"location":"spec/#delete-formats","title":"Delete Formats","text":"<p>This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are not supported in v1.</p> <p>Row-level delete files are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection. It is recommended that delete files are written using the table's default file format.</p> <p>Row-level delete files are tracked by manifests, like data files. A separate set of manifests is used for delete files, but the manifest schemas are identical.</p> <p>Both position and equality deletes allow encoding deleted row values with a delete. This can be used to reconstruct a stream of changes to a table.</p>"},{"location":"spec/#position-delete-files","title":"Position Delete Files","text":"<p>Position-based delete files identify deleted rows by file and position in one or more data files, and may optionally contain the deleted row.</p> <p>A data row is deleted if there is an entry in a position delete file for the row's file and position in the data file, starting at 0.</p> <p>Position-based delete files store <code>file_position_delete</code>, a struct with the following fields:</p> Field id, name Type Description <code>2147483546 file_path</code> <code>string</code> Full URI of a data file with FS scheme. This must match the <code>file_path</code> of the target data file in a manifest entry <code>2147483545 pos</code> <code>long</code> Ordinal position of a deleted row in the target data file identified by <code>file_path</code>, starting at <code>0</code> <code>2147483544 row</code> <code>required struct<...></code> [1] Deleted row values. Omit the column when not storing deleted rows. <ol> <li>When present in the delete file, <code>row</code> is required because all delete entries must include the row values.</li> </ol> <p>When the deleted row column is present, its schema may be any subset of the table schema and must use field ids matching the table.</p> <p>To ensure the accuracy of statistics, all delete entries must include row values, or the column must be omitted (this is why the column type is <code>required</code>).</p> <p>The rows in the delete file must be sorted by <code>file_path</code> then <code>pos</code> to optimize filtering rows while scanning. </p> <ul> <li>Sorting by <code>file_path</code> allows filter pushdown by file in columnar storage formats.</li> <li>Sorting by <code>pos</code> allows filtering rows while scanning, to avoid keeping deletes in memory.</li> </ul>"},{"location":"spec/#equality-delete-files","title":"Equality Delete Files","text":"<p>Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.</p> <p>Equality delete files store any subset of a table's columns and use the table's field ids. The delete columns are the columns of the delete file used to match data rows. Delete columns are identified by id in the delete file metadata column <code>equality_ids</code>. Float and double columns cannot be used as delete columns in equality delete files.</p> <p>A data row is deleted if its values are equal to all delete columns for any row in an equality delete file that applies to the row's data file (see <code>Scan Planning</code>).</p> <p>Each row of the delete file produces one equality predicate that matches any row where the delete columns are equal. Multiple columns can be thought of as an <code>AND</code> of equality predicates. A <code>null</code> value in a delete column matches a row if the row's value is <code>null</code>, equivalent to <code>col IS NULL</code>.</p> <p>For example, a table with the following data:</p> <pre><code> 1: id | 2: category | 3: name\n-------|-------------|---------\n 1 | marsupial | Koala\n 2 | toy | Teddy\n 3 | NULL | Grizzly\n 4 | NULL | Polar\n</code></pre> <p>The delete <code>id = 3</code> could be written as either of the following equality delete files:</p> <pre><code>equality_ids=[1]\n\n 1: id\n-------\n 3\n</code></pre> <pre><code>equality_ids=[1]\n\n 1: id | 2: category | 3: name\n-------|-------------|---------\n 3 | NULL | Grizzly\n</code></pre> <p>The delete <code>id = 4 AND category IS NULL</code> could be written as the following equality delete file:</p> <pre><code>equality_ids=[1, 2]\n\n 1: id | 2: category | 3: name\n-------|-------------|---------\n 4 | NULL | Polar\n</code></pre> <p>If a delete column in an equality delete file is later dropped from the table, it must still be used when applying the equality deletes. If a column was added to a table and later used as a delete column in an equality delete file, the column value is read for older data files using normal projection rules (defaults to <code>null</code>).</p>"},{"location":"spec/#delete-file-stats","title":"Delete File Stats","text":"<p>Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted.</p>"},{"location":"spec/#appendix-a-format-specific-requirements","title":"Appendix A: Format-specific Requirements","text":""},{"location":"spec/#avro","title":"Avro","text":"<p>Data Type Mappings</p> <p>Values should be stored in Avro using the Avro types and logical type annotations in the table below.</p> <p>Optional fields, array elements, and map values must be wrapped in an Avro <code>union</code> with <code>null</code>. This is the only union type allowed in Iceberg data files.</p> <p>Optional fields must always set the Avro field default value to null.</p> <p>Maps with non-string keys must use an array representation with the <code>map</code> logical type. The array representation or Avro\u2019s map type may be used for maps with string keys.</p> Type Avro type Notes <code>boolean</code> <code>boolean</code> <code>int</code> <code>int</code> <code>long</code> <code>long</code> <code>float</code> <code>float</code> <code>double</code> <code>double</code> <code>decimal(P,S)</code> <code>{ \"type\": \"fixed\",</code> <code>\"size\": minBytesRequired(P),</code> <code>\"logicalType\": \"decimal\",</code> <code>\"precision\": P,</code> <code>\"scale\": S }</code> Stored as fixed using the minimum number of bytes for the given precision. <code>date</code> <code>{ \"type\": \"int\",</code> <code>\"logicalType\": \"date\" }</code> Stores days from the 1970-01-01. <code>time</code> <code>{ \"type\": \"long\",</code> <code>\"logicalType\": \"time-micros\" }</code> Stores microseconds from midnight. <code>timestamp</code> <code>{ \"type\": \"long\",</code> <code>\"logicalType\": \"timestamp-micros\",</code> <code>\"adjust-to-utc\": false }</code> Stores microseconds from 1970-01-01 00:00:00.000000. <code>timestamptz</code> <code>{ \"type\": \"long\",</code> <code>\"logicalType\": \"timestamp-micros\",</code> <code>\"adjust-to-utc\": true }</code> Stores microseconds from 1970-01-01 00:00:00.000000 UTC. <code>string</code> <code>string</code> <code>uuid</code> <code>{ \"type\": \"fixed\",</code> <code>\"size\": 16,</code> <code>\"logicalType\": \"uuid\" }</code> <code>fixed(L)</code> <code>{ \"type\": \"fixed\",</code> <code>\"size\": L }</code> <code>binary</code> <code>bytes</code> <code>struct</code> <code>record</code> <code>list</code> <code>array</code> <code>map</code> <code>array</code> of key-value records, or <code>map</code> when keys are strings (optional). Array storage must use logical type name <code>map</code> and must store elements that are 2-field records. The first field is a non-null key and the second field is the value. <p>Field IDs</p> <p>Iceberg struct, list, and map types identify nested types by ID. When writing data to Avro files, these IDs must be stored in the Avro schema to support ID-based column pruning.</p> <p>IDs are stored as JSON integers in the following locations:</p> ID Avro schema location Property Example Struct field Record field object <code>field-id</code> <code>{ \"type\": \"record\", ...</code> <code>\"fields\": [</code> <code>{ \"name\": \"l\",</code> <code>\"type\": [\"null\", \"long\"],</code> <code>\"default\": null,</code> <code>\"field-id\": 8 }</code> <code>] }</code> List element Array schema object <code>element-id</code> <code>{ \"type\": \"array\",</code> <code>\"items\": \"int\",</code> <code>\"element-id\": 9 }</code> String map key Map schema object <code>key-id</code> <code>{ \"type\": \"map\",</code> <code>\"values\": \"int\",</code> <code>\"key-id\": 10,</code> <code>\"value-id\": 11 }</code> String map value Map schema object <code>value-id</code> Map key, value Key, value fields in the element record. <code>field-id</code> <code>{ \"type\": \"array\",</code> <code>\"logicalType\": \"map\",</code> <code>\"items\": {</code> <code>\"type\": \"record\",</code> <code>\"name\": \"k12_v13\",</code> <code>\"fields\": [</code> <code>{ \"name\": \"key\",</code> <code>\"type\": \"int\",</code> <code>\"field-id\": 12 },</code> <code>{ \"name\": \"value\",</code> <code>\"type\": \"string\",</code> <code>\"field-id\": 13 }</code> <code>] } }</code> <p>Note that the string map case is for maps where the key type is a string. Using Avro\u2019s map type in this case is optional. Maps with string keys may be stored as arrays.</p>"},{"location":"spec/#parquet","title":"Parquet","text":"<p>Data Type Mappings</p> <p>Values should be stored in Parquet using the types and logical type annotations in the table below. Column IDs are required.</p> <p>Lists must use the 3-level representation.</p> Type Parquet physical type Logical type Notes <code>boolean</code> <code>boolean</code> <code>int</code> <code>int</code> <code>long</code> <code>long</code> <code>float</code> <code>float</code> <code>double</code> <code>double</code> <code>decimal(P,S)</code> <code>P <= 9</code>: <code>int32</code>,<code>P <= 18</code>: <code>int64</code>,<code>fixed</code> otherwise <code>DECIMAL(P,S)</code> Fixed must use the minimum number of bytes that can store <code>P</code>. <code>date</code> <code>int32</code> <code>DATE</code> Stores days from the 1970-01-01. <code>time</code> <code>int64</code> <code>TIME_MICROS</code> with <code>adjustToUtc=false</code> Stores microseconds from midnight. <code>timestamp</code> <code>int64</code> <code>TIMESTAMP_MICROS</code> with <code>adjustToUtc=false</code> Stores microseconds from 1970-01-01 00:00:00.000000. <code>timestamptz</code> <code>int64</code> <code>TIMESTAMP_MICROS</code> with <code>adjustToUtc=true</code> Stores microseconds from 1970-01-01 00:00:00.000000 UTC. <code>string</code> <code>binary</code> <code>UTF8</code> Encoding must be UTF-8. <code>uuid</code> <code>fixed_len_byte_array[16]</code> <code>UUID</code> <code>fixed(L)</code> <code>fixed_len_byte_array[L]</code> <code>binary</code> <code>binary</code> <code>struct</code> <code>group</code> <code>list</code> <code>3-level list</code> <code>LIST</code> See Parquet docs for 3-level representation. <code>map</code> <code>3-level map</code> <code>MAP</code> See Parquet docs for 3-level representation."},{"location":"spec/#orc","title":"ORC","text":"<p>Data Type Mappings</p> Type ORC type ORC type attributes Notes <code>boolean</code> <code>boolean</code> <code>int</code> <code>int</code> ORC <code>tinyint</code> and <code>smallint</code> would also map to <code>int</code>. <code>long</code> <code>long</code> <code>float</code> <code>float</code> <code>double</code> <code>double</code> <code>decimal(P,S)</code> <code>decimal</code> <code>date</code> <code>date</code> <code>time</code> <code>long</code> <code>iceberg.long-type</code>=<code>TIME</code> Stores microseconds from midnight. <code>timestamp</code> <code>timestamp</code> [1] <code>timestamptz</code> <code>timestamp_instant</code> [1] <code>string</code> <code>string</code> ORC <code>varchar</code> and <code>char</code> would also map to <code>string</code>. <code>uuid</code> <code>binary</code> <code>iceberg.binary-type</code>=<code>UUID</code> <code>fixed(L)</code> <code>binary</code> <code>iceberg.binary-type</code>=<code>FIXED</code> & <code>iceberg.length</code>=<code>L</code> The length would not be checked by the ORC reader and should be checked by the adapter. <code>binary</code> <code>binary</code> <code>struct</code> <code>struct</code> <code>list</code> <code>array</code> <code>map</code> <code>map</code> <p>Notes:</p> <ol> <li>ORC's TimestampColumnVector consists of a time field (milliseconds since epoch) and a nanos field (nanoseconds within the second). Hence the milliseconds within the second are reported twice; once in the time field and again in the nanos field. The read adapter should only use milliseconds within the second from one of these fields. The write adapter should also report milliseconds within the second twice; once in the time field and again in the nanos field. ORC writer is expected to correctly consider millis information from one of the fields. More details at https://issues.apache.org/jira/browse/ORC-546</li> </ol> <p>One of the interesting challenges with this is how to map Iceberg\u2019s schema evolution (id based) on to ORC\u2019s (name based). In theory, we could use Iceberg\u2019s column ids as the column and field names, but that would be inconvenient.</p> <p>The column IDs must be stored in ORC type attributes using the key <code>iceberg.id</code>, and <code>iceberg.required</code> to store <code>\"true\"</code> if the Iceberg column is required, otherwise it will be optional.</p> <p>Iceberg would build the desired reader schema with their schema evolution rules and pass that down to the ORC reader, which would then use its schema evolution to map that to the writer\u2019s schema. Basically, Iceberg would need to change the names of columns and fields to get the desired mapping.</p> Iceberg writer ORC writer Iceberg reader ORC reader <code>struct<a (1): int, b (2): string></code> <code>struct<a: int, b: string></code> <code>struct<a (2): string, c (3): date></code> <code>struct<b: string, c: date></code> <code>struct<a (1): struct<b (2): string, c (3): date>></code> <code>struct<a: struct<b:string, c:date>></code> <code>struct<aa (1): struct<cc (3): date, bb (2): string>></code> <code>struct<a: struct<c:date, b:string>></code>"},{"location":"spec/#appendix-b-32-bit-hash-requirements","title":"Appendix B: 32-bit Hash Requirements","text":"<p>The 32-bit hash implementation is 32-bit Murmur3 hash, x86 variant, seeded with 0.</p> Primitive type Hash specification Test value <code>int</code> <code>hashLong(long(v))</code> [1] <code>34</code> \uffeb <code>2017239379</code> <code>long</code> <code>hashBytes(littleEndianBytes(v))</code> <code>34L</code> \uffeb <code>2017239379</code> <code>decimal(P,S)</code> <code>hashBytes(minBigEndian(unscaled(v)))</code>[2] <code>14.20</code> \uffeb <code>-500754589</code> <code>date</code> <code>hashInt(daysFromUnixEpoch(v))</code> <code>2017-11-16</code> \uffeb <code>-653330422</code> <code>time</code> <code>hashLong(microsecsFromMidnight(v))</code> <code>22:31:08</code> \uffeb <code>-662762989</code> <code>timestamp</code> <code>hashLong(microsecsFromUnixEpoch(v))</code> <code>2017-11-16T22:31:08</code> \uffeb <code>-2047944441</code> <code>timestamptz</code> <code>hashLong(microsecsFromUnixEpoch(v))</code> <code>2017-11-16T14:31:08-08:00</code>\uffeb <code>-2047944441</code> <code>string</code> <code>hashBytes(utf8Bytes(v))</code> <code>iceberg</code> \uffeb <code>1210000089</code> <code>uuid</code> <code>hashBytes(uuidBytes(v))</code> [3] <code>f79c3e09-677c-4bbd-a479-3f349cb785e7</code> \uffeb <code>1488055340</code> <code>fixed(L)</code> <code>hashBytes(v)</code> <code>00 01 02 03</code> \uffeb <code>-188683207</code> <code>binary</code> <code>hashBytes(v)</code> <code>00 01 02 03</code> \uffeb <code>-188683207</code> <p>The types below are not currently valid for bucketing, and so are not hashed. However, if that changes and a hash value is needed, the following table shall apply:</p> Primitive type Hash specification Test value <code>boolean</code> <code>false: hashInt(0)</code>, <code>true: hashInt(1)</code> <code>true</code> \uffeb <code>1392991556</code> <code>float</code> <code>hashLong(doubleToLongBits(double(v))</code> [4] <code>1.0F</code> \uffeb <code>-142385009</code>, <code>0.0F</code> \uffeb <code>1669671676</code>, <code>-0.0F</code> \uffeb <code>1669671676</code> <code>double</code> <code>hashLong(doubleToLongBits(v))</code> [4] <code>1.0D</code> \uffeb <code>-142385009</code>, <code>0.0D</code> \uffeb <code>1669671676</code>, <code>-0.0D</code> \uffeb <code>1669671676</code> <p>Notes:</p> <ol> <li>Integer and long hash results must be identical for all integer values. This ensures that schema evolution does not change bucket partition values if integer types are promoted.</li> <li>Decimal values are hashed using the minimum number of bytes required to hold the unscaled value as a two\u2019s complement big-endian; this representation does not include padding bytes required for storage in a fixed-length array. Hash results are not dependent on decimal scale, which is part of the type, not the data value.</li> <li>UUIDs are encoded using big endian. The test UUID for the example above is: <code>f79c3e09-677c-4bbd-a479-3f349cb785e7</code>. This UUID encoded as a byte array is: <code>F7 9C 3E 09 67 7C 4B BD A4 79 3F 34 9C B7 85 E7</code></li> <li><code>doubleToLongBits</code> must give the IEEE 754 compliant bit representation of the double value. All <code>NaN</code> bit patterns must be canonicalized to <code>0x7ff8000000000000L</code>. Negative zero (<code>-0.0</code>) must be canonicalized to positive zero (<code>0.0</code>). Float hash values are the result of hashing the float cast to double to ensure that schema evolution does not change hash values if float types are promoted.</li> </ol>"},{"location":"spec/#appendix-c-json-serialization","title":"Appendix C: JSON serialization","text":""},{"location":"spec/#schemas","title":"Schemas","text":"<p>Schemas are serialized as a JSON object with the same fields as a struct in the table below, and the following additional fields:</p> v1 v2 Field JSON representation Example optional required <code>schema-id</code> <code>JSON int</code> <code>0</code> optional optional <code>identifier-field-ids</code> <code>JSON list of ints</code> <code>[1, 2]</code> <p>Types are serialized according to this table:</p> Type JSON representation Example <code>boolean</code> <code>JSON string: \"boolean\"</code> <code>\"boolean\"</code> <code>int</code> <code>JSON string: \"int\"</code> <code>\"int\"</code> <code>long</code> <code>JSON string: \"long\"</code> <code>\"long\"</code> <code>float</code> <code>JSON string: \"float\"</code> <code>\"float\"</code> <code>double</code> <code>JSON string: \"double\"</code> <code>\"double\"</code> <code>date</code> <code>JSON string: \"date\"</code> <code>\"date\"</code> <code>time</code> <code>JSON string: \"time\"</code> <code>\"time\"</code> <code>timestamp without zone</code> <code>JSON string: \"timestamp\"</code> <code>\"timestamp\"</code> <code>timestamp with zone</code> <code>JSON string: \"timestamptz\"</code> <code>\"timestamptz\"</code> <code>string</code> <code>JSON string: \"string\"</code> <code>\"string\"</code> <code>uuid</code> <code>JSON string: \"uuid\"</code> <code>\"uuid\"</code> <code>fixed(L)</code> <code>JSON string: \"fixed[<L>]\"</code> <code>\"fixed[16]\"</code> <code>binary</code> <code>JSON string: \"binary\"</code> <code>\"binary\"</code> <code>decimal(P, S)</code> <code>JSON string: \"decimal(<P>,<S>)\"</code> <code>\"decimal(9,2)\"</code>,<code>\"decimal(9, 2)\"</code> <code>struct</code> <code>JSON object: {</code> <code>\"type\": \"struct\",</code> <code>\"fields\": [ {</code> <code>\"id\": <field id int>,</code> <code>\"name\": <name string>,</code> <code>\"required\": <boolean>,</code> <code>\"type\": <type JSON>,</code> <code>\"doc\": <comment string>,</code> <code>\"initial-default\": <JSON encoding of default value>,</code> <code>\"write-default\": <JSON encoding of default value></code> <code>}, ...</code> <code>] }</code> <code>{</code> <code>\"type\": \"struct\",</code> <code>\"fields\": [ {</code> <code>\"id\": 1,</code> <code>\"name\": \"id\",</code> <code>\"required\": true,</code> <code>\"type\": \"uuid\",</code> <code>\"initial-default\": \"0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb\",</code> <code>\"write-default\": \"ec5911be-b0a7-458c-8438-c9a3e53cffae\"</code> <code>}, {</code> <code>\"id\": 2,</code> <code>\"name\": \"data\",</code> <code>\"required\": false,</code> <code>\"type\": {</code> <code>\"type\": \"list\",</code> <code>...</code> <code>}</code> <code>} ]</code><code>}</code> <code>list</code> <code>JSON object: {</code> <code>\"type\": \"list\",</code> <code>\"element-id\": <id int>,</code> <code>\"element-required\": <bool></code> <code>\"element\": <type JSON></code><code>}</code> <code>{</code> <code>\"type\": \"list\",</code> <code>\"element-id\": 3,</code> <code>\"element-required\": true,</code> <code>\"element\": \"string\"</code><code>}</code> <code>map</code> <code>JSON object: {</code> <code>\"type\": \"map\",</code> <code>\"key-id\": <key id int>,</code> <code>\"key\": <type JSON>,</code> <code>\"value-id\": <val id int>,</code> <code>\"value-required\": <bool></code> <code>\"value\": <type JSON></code><code>}</code> <code>{</code> <code>\"type\": \"map\",</code> <code>\"key-id\": 4,</code> <code>\"key\": \"string\",</code> <code>\"value-id\": 5,</code> <code>\"value-required\": false,</code> <code>\"value\": \"double\"</code><code>}</code> <p>Note that default values are serialized using the JSON single-value serialization in Appendix D.</p>"},{"location":"spec/#partition-specs","title":"Partition Specs","text":"<p>Partition specs are serialized as a JSON object with the following fields:</p> Field JSON representation Example <code>spec-id</code> <code>JSON int</code> <code>0</code> <code>fields</code> <code>JSON list: [</code> <code><partition field JSON>,</code> <code>...</code><code>]</code> <code>[ {</code> <code>\"source-id\": 4,</code> <code>\"field-id\": 1000,</code> <code>\"name\": \"ts_day\",</code> <code>\"transform\": \"day\"</code><code>}, {</code> <code>\"source-id\": 1,</code> <code>\"field-id\": 1001,</code> <code>\"name\": \"id_bucket\",</code> <code>\"transform\": \"bucket[16]\"</code><code>} ]</code> <p>Each partition field in the fields list is stored as an object. See the table for more detail:</p> Transform or Field JSON representation Example <code>identity</code> <code>JSON string: \"identity\"</code> <code>\"identity\"</code> <code>bucket[N]</code> <code>JSON string: \"bucket[<N>]\"</code> <code>\"bucket[16]\"</code> <code>truncate[W]</code> <code>JSON string: \"truncate[<W>]\"</code> <code>\"truncate[20]\"</code> <code>year</code> <code>JSON string: \"year\"</code> <code>\"year\"</code> <code>month</code> <code>JSON string: \"month\"</code> <code>\"month\"</code> <code>day</code> <code>JSON string: \"day\"</code> <code>\"day\"</code> <code>hour</code> <code>JSON string: \"hour\"</code> <code>\"hour\"</code> <code>Partition Field</code> <code>JSON object: {</code> <code>\"source-id\": <id int>,</code> <code>\"field-id\": <field id int>,</code> <code>\"name\": <name string>,</code> <code>\"transform\": <transform JSON></code><code>}</code> <code>{</code> <code>\"source-id\": 1,</code> <code>\"field-id\": 1000,</code> <code>\"name\": \"id_bucket\",</code> <code>\"transform\": \"bucket[16]\"</code><code>}</code> <p>In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated <code>partition-spec</code> field in table metadata. The object format should be used unless otherwise noted in this spec.</p> <p>The <code>field-id</code> property was added for each partition field in v2. In v1, the reference implementation assigned field ids sequentially in each spec starting at 1,000. See Partition Evolution for more details.</p>"},{"location":"spec/#sort-orders","title":"Sort Orders","text":"<p>Sort orders are serialized as a list of JSON object, each of which contains the following fields:</p> Field JSON representation Example <code>order-id</code> <code>JSON int</code> <code>1</code> <code>fields</code> <code>JSON list: [</code> <code><sort field JSON>,</code> <code>...</code><code>]</code> <code>[ {</code> <code>\"transform\": \"identity\",</code> <code>\"source-id\": 2,</code> <code>\"direction\": \"asc\",</code> <code>\"null-order\": \"nulls-first\"</code> <code>}, {</code> <code>\"transform\": \"bucket[4]\",</code> <code>\"source-id\": 3,</code> <code>\"direction\": \"desc\",</code> <code>\"null-order\": \"nulls-last\"</code><code>} ]</code> <p>Each sort field in the fields list is stored as an object with the following properties:</p> Field JSON representation Example <code>Sort Field</code> <code>JSON object: {</code> <code>\"transform\": <transform JSON>,</code> <code>\"source-id\": <source id int>,</code> <code>\"direction\": <direction string>,</code> <code>\"null-order\": <null-order string></code><code>}</code> <code>{</code> <code>\"transform\": \"bucket[4]\",</code> <code>\"source-id\": 3,</code> <code>\"direction\": \"desc\",</code> <code>\"null-order\": \"nulls-last\"</code><code>}</code> <p>The following table describes the possible values for the some of the field within sort field: </p> Field JSON representation Possible values <code>direction</code> <code>JSON string</code> <code>\"asc\", \"desc\"</code> <code>null-order</code> <code>JSON string</code> <code>\"nulls-first\", \"nulls-last\"</code>"},{"location":"spec/#table-metadata-and-snapshots","title":"Table Metadata and Snapshots","text":"<p>Table metadata is serialized as a JSON object according to the following table. Snapshots are not serialized separately. Instead, they are stored in the table metadata JSON.</p> Metadata field JSON representation Example <code>format-version</code> <code>JSON int</code> <code>1</code> <code>table-uuid</code> <code>JSON string</code> <code>\"fb072c92-a02b-11e9-ae9c-1bb7bc9eca94\"</code> <code>location</code> <code>JSON string</code> <code>\"s3://b/wh/data.db/table\"</code> <code>last-updated-ms</code> <code>JSON long</code> <code>1515100955770</code> <code>last-column-id</code> <code>JSON int</code> <code>22</code> <code>schema</code> <code>JSON schema (object)</code> <code>See above, read schemas instead</code> <code>schemas</code> <code>JSON schemas (list of objects)</code> <code>See above</code> <code>current-schema-id</code> <code>JSON int</code> <code>0</code> <code>partition-spec</code> <code>JSON partition fields (list)</code> <code>See above, read partition-specs instead</code> <code>partition-specs</code> <code>JSON partition specs (list of objects)</code> <code>See above</code> <code>default-spec-id</code> <code>JSON int</code> <code>0</code> <code>last-partition-id</code> <code>JSON int</code> <code>1000</code> <code>properties</code> <code>JSON object: {</code> <code>\"<key>\": \"<val>\",</code> <code>...</code><code>}</code> <code>{</code> <code>\"write.format.default\": \"avro\",</code> <code>\"commit.retry.num-retries\": \"4\"</code><code>}</code> <code>current-snapshot-id</code> <code>JSON long</code> <code>3051729675574597004</code> <code>snapshots</code> <code>JSON list of objects: [ {</code> <code>\"snapshot-id\": <id>,</code> <code>\"timestamp-ms\": <timestamp-in-ms>,</code> <code>\"summary\": {</code> <code>\"operation\": <operation>,</code> <code>... },</code> <code>\"manifest-list\": \"<location>\",</code> <code>\"schema-id\": \"<id>\"</code> <code>},</code> <code>...</code><code>]</code> <code>[ {</code> <code>\"snapshot-id\": 3051729675574597004,</code> <code>\"timestamp-ms\": 1515100955770,</code> <code>\"summary\": {</code> <code>\"operation\": \"append\"</code> <code>},</code> <code>\"manifest-list\": \"s3://b/wh/.../s1.avro\"</code> <code>\"schema-id\": 0</code><code>} ]</code> <code>snapshot-log</code> <code>JSON list of objects: [</code> <code>{</code> <code>\"snapshot-id\": ,</code> <code>\"timestamp-ms\":</code> <code>},</code> <code>...</code><code>]</code> <code>[ {</code> <code>\"snapshot-id\": 30517296...,</code> <code>\"timestamp-ms\": 1515100...</code><code>} ]</code> <code>metadata-log</code> <code>JSON list of objects: [</code> <code>{</code> <code>\"metadata-file\": ,</code> <code>\"timestamp-ms\":</code> <code>},</code> <code>...</code><code>]</code> <code>[ {</code> <code>\"metadata-file\": \"s3://bucket/.../v1.json\",</code> <code>\"timestamp-ms\": 1515100...</code><code>} ]</code> <code>sort-orders</code> <code>JSON sort orders (list of sort field object)</code> <code>See above</code> <code>default-sort-order-id</code> <code>JSON int</code> <code>0</code> <code>refs</code> <code>JSON map with string key and object value:</code><code>{</code> <code>\"<name>\": {</code> <code>\"snapshot-id\": <id>,</code> <code>\"type\": <type>,</code> <code>\"max-ref-age-ms\": <long>,</code> <code>...</code> <code>}</code> <code>...</code><code>}</code> <code>{</code> <code>\"test\": {</code> <code>\"snapshot-id\": 123456789000,</code> <code>\"type\": \"tag\",</code> <code>\"max-ref-age-ms\": 10000000</code> <code>}</code><code>}</code>"},{"location":"spec/#name-mapping-serialization","title":"Name Mapping Serialization","text":"<p>Name mapping is serialized as a list of field mapping JSON Objects which are serialized as follows</p> Field mapping field JSON representation Example <code>names</code> <code>JSON list of strings</code> <code>[\"latitude\", \"lat\"]</code> <code>field_id</code> <code>JSON int</code> <code>1</code> <code>fields</code> <code>JSON field mappings (list of objects)</code> <code>[{</code> <code>\"field-id\": 4,</code> <code>\"names\": [\"latitude\", \"lat\"]</code><code>}, {</code> <code>\"field-id\": 5,</code> <code>\"names\": [\"longitude\", \"long\"]</code><code>}]</code> <p>Example <pre><code>[ { \"field-id\": 1, \"names\": [\"id\", \"record_id\"] },\n { \"field-id\": 2, \"names\": [\"data\"] },\n { \"field-id\": 3, \"names\": [\"location\"], \"fields\": [\n { \"field-id\": 4, \"names\": [\"latitude\", \"lat\"] },\n { \"field-id\": 5, \"names\": [\"longitude\", \"long\"] }\n ] } ]\n</code></pre></p>"},{"location":"spec/#content-file-data-and-delete-serialization","title":"Content File (Data and Delete) Serialization","text":"<p>Content file (data or delete) is serialized as a JSON object according to the following table.</p> Metadata field JSON representation Example <code>spec-id</code> <code>JSON int</code> <code>1</code> <code>content</code> <code>JSON string</code> <code>DATA</code>, <code>POSITION_DELETES</code>, <code>EQUALITY_DELETES</code> <code>file-path</code> <code>JSON string</code> <code>\"s3://b/wh/data.db/table\"</code> <code>file-format</code> <code>JSON string</code> <code>AVRO</code>, <code>ORC</code>, <code>PARQUET</code> <code>partition</code> <code>JSON object: Partition data tuple using partition field ids for the struct field ids</code> <code>{\"1000\":1}</code> <code>record-count</code> <code>JSON long</code> <code>1</code> <code>file-size-in-bytes</code> <code>JSON long</code> <code>1024</code> <code>column-sizes</code> <code>JSON object: Map from column id to the total size on disk of all regions that store the column.</code> <code>{\"keys\":[3,4],\"values\":[100,200]}</code> <code>value-counts</code> <code>JSON object: Map from column id to number of values in the column (including null and NaN values)</code> <code>{\"keys\":[3,4],\"values\":[90,180]}</code> <code>null-value-counts</code> <code>JSON object: Map from column id to number of null values in the column</code> <code>{\"keys\":[3,4],\"values\":[10,20]}</code> <code>nan-value-counts</code> <code>JSON object: Map from column id to number of NaN values in the column</code> <code>{\"keys\":[3,4],\"values\":[0,0]}</code> <code>lower-bounds</code> <code>JSON object: Map from column id to lower bound binary in the column serialized as hexadecimal string</code> <code>{\"keys\":[3,4],\"values\":[\"01000000\",\"02000000\"]}</code> <code>upper-bounds</code> <code>JSON object: Map from column id to upper bound binary in the column serialized as hexadecimal string</code> <code>{\"keys\":[3,4],\"values\":[\"05000000\",\"0A000000\"]}</code> <code>key-metadata</code> <code>JSON string: Encryption key metadata binary serialized as hexadecimal string</code> <code>00000000000000000000000000000000</code> <code>split-offsets</code> <code>JSON list of long: Split offsets for the data file</code> <code>[128,256]</code> <code>equality-ids</code> <code>JSON list of int: Field ids used to determine row equality in equality delete files</code> <code>[1]</code> <code>sort-order-id</code> <code>JSON int</code> <code>1</code>"},{"location":"spec/#file-scan-task-serialization","title":"File Scan Task Serialization","text":"<p>File scan task is serialized as a JSON object according to the following table.</p> Metadata field JSON representation Example <code>schema</code> <code>JSON object</code> <code>See above, read schemas instead</code> <code>spec</code> <code>JSON object</code> <code>See above, read partition specs instead</code> <code>data-file</code> <code>JSON object</code> <code>See above, read content file instead</code> <code>delete-files</code> <code>JSON list of objects</code> <code>See above, read content file instead</code> <code>residual-filter</code> <code>JSON object: residual filter expression</code> <code>{\"type\":\"eq\",\"term\":\"id\",\"value\":1}</code>"},{"location":"spec/#appendix-d-single-value-serialization","title":"Appendix D: Single-value serialization","text":""},{"location":"spec/#binary-single-value-serialization","title":"Binary single-value serialization","text":"<p>This serialization scheme is for storing single values as individual binary values in the lower and upper bounds maps of manifest files.</p> Type Binary serialization <code>boolean</code> <code>0x00</code> for false, non-zero byte for true <code>int</code> Stored as 4-byte little-endian <code>long</code> Stored as 8-byte little-endian <code>float</code> Stored as 4-byte little-endian <code>double</code> Stored as 8-byte little-endian <code>date</code> Stores days from the 1970-01-01 in an 4-byte little-endian int <code>time</code> Stores microseconds from midnight in an 8-byte little-endian long <code>timestamp without zone</code> Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long <code>timestamp with zone</code> Stores microseconds from 1970-01-01 00:00:00.000000 UTC in an 8-byte little-endian long <code>string</code> UTF-8 bytes (without length) <code>uuid</code> 16-byte big-endian value, see example in Appendix B <code>fixed(L)</code> Binary value <code>binary</code> Binary value (without length) <code>decimal(P, S)</code> Stores unscaled value as two\u2019s-complement big-endian binary, using the minimum number of bytes for the value <code>struct</code> Not supported <code>list</code> Not supported <code>map</code> Not supported"},{"location":"spec/#json-single-value-serialization","title":"JSON single-value serialization","text":"<p>Single values are serialized as JSON by type according to the following table:</p> Type JSON representation Example Description <code>boolean</code> <code>JSON boolean</code> <code>true</code> <code>int</code> <code>JSON int</code> <code>34</code> <code>long</code> <code>JSON long</code> <code>34</code> <code>float</code> <code>JSON number</code> <code>1.0</code> <code>double</code> <code>JSON number</code> <code>1.0</code> <code>decimal(P,S)</code> <code>JSON string</code> <code>\"14.20\"</code>, <code>\"2E+20\"</code> Stores the string representation of the decimal value, specifically, for values with a positive scale, the number of digits to the right of the decimal point is used to indicate scale, for values with a negative scale, the scientific notation is used and the exponent must equal the negated scale <code>date</code> <code>JSON string</code> <code>\"2017-11-16\"</code> Stores ISO-8601 standard date <code>time</code> <code>JSON string</code> <code>\"22:31:08.123456\"</code> Stores ISO-8601 standard time with microsecond precision <code>timestamp</code> <code>JSON string</code> <code>\"2017-11-16T22:31:08.123456\"</code> Stores ISO-8601 standard timestamp with microsecond precision; must not include a zone offset <code>timestamptz</code> <code>JSON string</code> <code>\"2017-11-16T22:31:08.123456+00:00\"</code> Stores ISO-8601 standard timestamp with microsecond precision; must include a zone offset and it must be '+00:00' <code>string</code> <code>JSON string</code> <code>\"iceberg\"</code> <code>uuid</code> <code>JSON string</code> <code>\"f79c3e09-677c-4bbd-a479-3f349cb785e7\"</code> Stores the lowercase uuid string <code>fixed(L)</code> <code>JSON string</code> <code>\"000102ff\"</code> Stored as a hexadecimal string <code>binary</code> <code>JSON string</code> <code>\"000102ff\"</code> Stored as a hexadecimal string <code>struct</code> <code>JSON object by field ID</code> <code>{\"1\": 1, \"2\": \"bar\"}</code> Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format <code>list</code> <code>JSON array of values</code> <code>[1, 2, 3]</code> Stores a JSON array of values that are serialized using this JSON single-value format <code>map</code> <code>JSON object of key and value arrays</code> <code>{ \"keys\": [\"a\", \"b\"], \"values\": [1, 2] }</code> Stores arrays of keys and values; individual keys and values are serialized using this JSON single-value format"},{"location":"spec/#appendix-e-format-version-changes","title":"Appendix E: Format version changes","text":""},{"location":"spec/#version-3","title":"Version 3","text":"<p>Default values are added to struct fields in v3. * The <code>write-default</code> is a forward-compatible change because it is only used at write time. Old writers will fail because the field is missing. * Tables with <code>initial-default</code> will be read correctly by older readers if <code>initial-default</code> is always null for optional fields. Otherwise, old readers will default optional columns with null. Old readers will fail to read required fields which are populated by <code>initial-default</code> because that default is not supported.</p>"},{"location":"spec/#version-2","title":"Version 2","text":"<p>Writing v1 metadata:</p> <ul> <li>Table metadata field <code>last-sequence-number</code> should not be written</li> <li>Snapshot field <code>sequence-number</code> should not be written</li> <li>Manifest list field <code>sequence-number</code> should not be written</li> <li>Manifest list field <code>min-sequence-number</code> should not be written</li> <li>Manifest list field <code>content</code> must be 0 (data) or omitted</li> <li>Manifest entry field <code>sequence_number</code> should not be written</li> <li>Manifest entry field <code>file_sequence_number</code> should not be written</li> <li>Data file field <code>content</code> must be 0 (data) or omitted</li> </ul> <p>Reading v1 metadata for v2:</p> <ul> <li>Table metadata field <code>last-sequence-number</code> must default to 0</li> <li>Snapshot field <code>sequence-number</code> must default to 0</li> <li>Manifest list field <code>sequence-number</code> must default to 0</li> <li>Manifest list field <code>min-sequence-number</code> must default to 0</li> <li>Manifest list field <code>content</code> must default to 0 (data)</li> <li>Manifest entry field <code>sequence_number</code> must default to 0</li> <li>Manifest entry field <code>file_sequence_number</code> must default to 0</li> <li>Data file field <code>content</code> must default to 0 (data)</li> </ul> <p>Writing v2 metadata:</p> <ul> <li>Table metadata JSON:<ul> <li><code>last-sequence-number</code> was added and is required; default to 0 when reading v1 metadata</li> <li><code>table-uuid</code> is now required</li> <li><code>current-schema-id</code> is now required</li> <li><code>schemas</code> is now required</li> <li><code>partition-specs</code> is now required</li> <li><code>default-spec-id</code> is now required</li> <li><code>last-partition-id</code> is now required</li> <li><code>sort-orders</code> is now required</li> <li><code>default-sort-order-id</code> is now required</li> <li><code>schema</code> is no longer required and should be omitted; use <code>schemas</code> and <code>current-schema-id</code> instead</li> <li><code>partition-spec</code> is no longer required and should be omitted; use <code>partition-specs</code> and <code>default-spec-id</code> instead</li> </ul> </li> <li>Snapshot JSON:<ul> <li><code>sequence-number</code> was added and is required; default to 0 when reading v1 metadata</li> <li><code>manifest-list</code> is now required</li> <li><code>manifests</code> is no longer required and should be omitted; always use <code>manifest-list</code> instead</li> </ul> </li> <li>Manifest list <code>manifest_file</code>:<ul> <li><code>content</code> was added and is required; 0=data, 1=deletes; default to 0 when reading v1 manifest lists</li> <li><code>sequence_number</code> was added and is required</li> <li><code>min_sequence_number</code> was added and is required</li> <li><code>added_files_count</code> is now required</li> <li><code>existing_files_count</code> is now required</li> <li><code>deleted_files_count</code> is now required</li> <li><code>added_rows_count</code> is now required</li> <li><code>existing_rows_count</code> is now required</li> <li><code>deleted_rows_count</code> is now required</li> </ul> </li> <li>Manifest key-value metadata:<ul> <li><code>schema-id</code> is now required</li> <li><code>partition-spec-id</code> is now required</li> <li><code>format-version</code> is now required</li> <li><code>content</code> was added and is required (must be \"data\" or \"deletes\")</li> </ul> </li> <li>Manifest <code>manifest_entry</code>:<ul> <li><code>snapshot_id</code> is now optional to support inheritance</li> <li><code>sequence_number</code> was added and is optional, to support inheritance</li> <li><code>file_sequence_number</code> was added and is optional, to support inheritance</li> </ul> </li> <li>Manifest <code>data_file</code>:<ul> <li><code>content</code> was added and is required; 0=data, 1=position deletes, 2=equality deletes; default to 0 when reading v1 manifests</li> <li><code>equality_ids</code> was added, to be used for equality deletes only</li> <li><code>block_size_in_bytes</code> was removed (breaks v1 reader compatibility)</li> <li><code>file_ordinal</code> was removed</li> <li><code>sort_columns</code> was removed</li> </ul> </li> </ul> <p>Note that these requirements apply when writing data to a v2 table. Tables that are upgraded from v1 may contain metadata that does not follow these requirements. Implementations should remain backward-compatible with v1 metadata requirements.</p>"},{"location":"talks/","title":"Talks","text":""},{"location":"talks/#iceberg-talks","title":"Iceberg Talks","text":"<p>Here is a list of talks and other videos related to Iceberg.</p>"},{"location":"talks/#eliminating-shuffles-in-delete-update-merge","title":"Eliminating Shuffles in DELETE, UPDATE, MERGE","text":"<p>Date: July 27, 2023, Authors: Anton Okolnychyi, Chao Sun</p>"},{"location":"talks/#write-distribution-modes-in-apache-iceberg","title":"Write Distribution Modes in Apache Iceberg","text":"<p>Date: March 15, 2023, Author: Russell Spitzer</p>"},{"location":"talks/#technical-evolution-of-apache-iceberg","title":"Technical Evolution of Apache Iceberg","text":"<p>Date: March 15, 2023, Author: Anton Okolnychyi</p>"},{"location":"talks/#icebergs-best-secret-exploring-metadata-tables","title":"Iceberg's Best Secret Exploring Metadata Tables","text":"<p>Date: January 12, 2023, Author: Szehon Ho</p>"},{"location":"talks/#data-architecture-in-2022","title":"Data architecture in 2022","text":"<p>Date: May 5, 2022, Authors: Ryan Blue</p>"},{"location":"talks/#why-you-shouldnt-care-about-iceberg-tabular","title":"Why You Shouldn\u2019t Care About Iceberg | Tabular","text":"<p>Date: March 24, 2022, Authors: Ryan Blue</p>"},{"location":"talks/#managing-data-files-in-apache-iceberg","title":"Managing Data Files in Apache Iceberg","text":"<p>Date: March 2, 2022, Author: Russell Spitzer</p>"},{"location":"talks/#tuning-row-level-operations-in-apache-iceberg","title":"Tuning Row-Level Operations in Apache Iceberg","text":"<p>Date: March 2, 2022, Author: Anton Okolnychyi</p>"},{"location":"talks/#multi-dimensional-clustering-with-z-ordering","title":"Multi Dimensional Clustering with Z Ordering","text":"<p>Date: December 6, 2021, Author: Russell Spitzer</p>"},{"location":"talks/#expert-roundtable-the-future-of-metadata-after-hive-metastore","title":"Expert Roundtable: The Future of Metadata After Hive Metastore","text":"<p>Date: November 15, 2021, Authors: Lior Ebel, Seshu Adunuthula, Ryan Blue & Oz Katz</p>"},{"location":"talks/#presto-and-apache-iceberg-building-out-modern-open-data-lakes","title":"Presto and Apache Iceberg: Building out Modern Open Data Lakes","text":"<p>Date: November 10, 2021, Authors: Daniel Weeks, Chunxu Tang</p>"},{"location":"talks/#iceberg-case-studies","title":"Iceberg Case Studies","text":"<p>Date: September 29, 2021, Authors: Ryan Blue</p>"},{"location":"talks/#deep-dive-into-iceberg-sql-extensions","title":"Deep Dive into Iceberg SQL Extensions","text":"<p>Date: July 13, 2021, Author: Anton Okolnychyi</p>"},{"location":"talks/#building-efficient-and-reliable-data-lakes-with-apache-iceberg","title":"Building efficient and reliable data lakes with Apache Iceberg","text":"<p>Date: October 21, 2020, Authors: Anton Okolnychyi, Vishwa Lakkundi</p>"},{"location":"talks/#spark-and-iceberg-at-apples-scale-leveraging-differential-files-for-efficient-upserts-and-deletes","title":"Spark and Iceberg at Apple's Scale - Leveraging differential files for efficient upserts and deletes","text":"<p>Date: October 21, 2020, Authors: Anton Okolnychyi, Vishwa Lakkundi</p>"},{"location":"talks/#apache-iceberg-a-table-format-for-huge-analytic-datasets","title":"Apache Iceberg - A Table Format for Huge Analytic Datasets","text":"<p>Date: October 21, 2020, Author: Ryan Blue </p>"},{"location":"terms/","title":"Terms","text":""},{"location":"terms/#terms","title":"Terms","text":""},{"location":"terms/#snapshot","title":"Snapshot","text":"<p>A snapshot is the state of a table at some time.</p> <p>Each snapshot lists all of the data files that make up the table's contents at the time of the snapshot. Data files are stored across multiple manifest files, and the manifests for a snapshot are listed in a single manifest list file.</p>"},{"location":"terms/#manifest-list","title":"Manifest list","text":"<p>A manifest list is a metadata file that lists the manifests that make up a table snapshot.</p> <p>Each manifest file in the manifest list is stored with information about its contents, like partition value ranges, used to speed up metadata operations.</p>"},{"location":"terms/#manifest-file","title":"Manifest file","text":"<p>A manifest file is a metadata file that lists a subset of data files that make up a snapshot.</p> <p>Each data file in a manifest is stored with a partition tuple, column-level stats, and summary information used to prune splits during scan planning.</p>"},{"location":"terms/#partition-spec","title":"Partition spec","text":"<p>A partition spec is a description of how to partition data in a table.</p> <p>A spec consists of a list of source columns and transforms. A transform produces a partition value from a source value. For example, <code>date(ts)</code> produces the date associated with a timestamp column named <code>ts</code>.</p>"},{"location":"terms/#partition-tuple","title":"Partition tuple","text":"<p>A partition tuple is a tuple or struct of partition data stored with each data file.</p> <p>All values in a partition tuple are the same for all rows stored in a data file. Partition tuples are produced by transforming values from row data using a partition spec.</p> <p>Iceberg stores partition values unmodified, unlike Hive tables that convert values to and from strings in file system paths and keys.</p>"},{"location":"terms/#snapshot-log-history-table","title":"Snapshot log (history table)","text":"<p>The snapshot log is a metadata log of how the table's current snapshot has changed over time.</p> <p>The log is a list of timestamp and ID pairs: when the current snapshot changed and the snapshot ID the current snapshot was changed to.</p> <p>The snapshot log is stored in table metadata as <code>snapshot-log</code>.</p>"},{"location":"vendors/","title":"Vendors","text":""},{"location":"vendors/#vendors-supporting-iceberg-tables","title":"Vendors Supporting Iceberg Tables","text":"<p>This page contains some of the vendors who are shipping and supporting Apache Iceberg in their products</p>"},{"location":"vendors/#celerdata","title":"CelerData","text":"<p>CelerData provides commercial offerings for StarRocks, a distributed MPP SQL engine for enterprise analytics on Iceberg. With its fully vectorized technology, local caching, and intelligent materialized view, StarRocks delivers sub-second query latency for both batch and real-time analytics. CelerData offers both an enterprise deployment and a cloud service to help customers use StarRocks more smoothly. Learn more about how to query Iceberg with StarRocks here.</p>"},{"location":"vendors/#clickhouse","title":"ClickHouse","text":"<p>ClickHouse is a column-oriented database that enables its users to generate powerful analytics, using SQL queries, in real-time. ClickHouse integrates well with Iceberg and offers two options to work with it: 1. Via Iceberg table function: Provides a read-only table-like interface to Apache Iceberg tables in Amazon S3. 2. Via the Iceberg table engine: An engine that provides a read-only integration with existing Apache Iceberg tables in Amazon S3.</p>"},{"location":"vendors/#cloudera","title":"Cloudera","text":"<p>Cloudera Data Platform integrates Apache Iceberg to the following components: * Apache Hive, Apache Impala, and Apache Spark to query Apache Iceberg tables * Cloudera Data Warehouse service providing access to Apache Iceberg tables through Apache Hive and Apache Impala * Cloudera Data Engineering service providing access to Apache Iceberg tables through Apache Spark * The CDP Shared Data Experience (SDX) provides compliance and self-service data access for Apache Iceberg tables * Hive metastore, which plays a lightweight role in providing the Iceberg Catalog * Data Visualization to visualize data stored in Apache Iceberg</p> <p>https://docs.cloudera.com/cdp-public-cloud/cloud/cdp-iceberg/topics/iceberg-in-cdp.html</p>"},{"location":"vendors/#dremio","title":"Dremio","text":"<p>With Dremio, an organization can easily build and manage a data lakehouse in which data is stored in open formats like Apache Iceberg and can be processed with Dremio\u2019s interactive SQL query engine and non-Dremio processing engines. Dremio Cloud provides these capabilities in a fully managed offering.</p> <ul> <li>Dremio Sonar is a lakehouse query engine that provides interactive performance and DML on Apache Iceberg, as well as other formats and data sources.</li> <li>Dremio Arctic is a lakehouse catalog and optimization service for Apache Iceberg. Arctic automatically optimizes tables in the background to ensure high-performance access for any engine. Arctic also simplifies experimentation, data engineering, and data governance by providing Git concepts like branches and tags on Apache Iceberg tables.</li> </ul>"},{"location":"vendors/#iomete","title":"IOMETE","text":"<p>IOMETE is a fully-managed ready to use, batteries included Data Platform. IOMETE optimizes clustering, compaction, and access control to Apache Iceberg tables. Customer data remains on customer's account to prevent vendor lock-in. The core of IOMETE platform is a serverless Lakehouse that leverages Apache Iceberg as its core table format. IOMETE platform also includes Serverless Spark, an SQL Editor, A Data Catalog, and granular data access control. IOMETE supports Hybrid-multi-cloud setups. </p>"},{"location":"vendors/#puppygraph","title":"PuppyGraph","text":"<p>PuppyGraph is a cloud-native graph analytics engine that enables users to query one or more relational data stores as a unified graph model. This eliminates the overhead of deploying and maintaining a siloed graph database system, with no ETL required. PuppyGraph\u2019s native Apache Iceberg integration adds native graph capabilities to your existing data lake in an easy and performant way.</p>"},{"location":"vendors/#snowflake","title":"Snowflake","text":"<p>Snowflake is a single, cross-cloud platform that enables every organization to mobilize their data with Snowflake\u2019s Data Cloud. Snowflake supports Apache Iceberg by offering native support for Iceberg Tables for full DML as well as connectors to External Tables for read-only access.</p>"},{"location":"vendors/#starburst","title":"Starburst","text":"<p>Starburst is a commercial offering for the Trino query engine. Trino is a distributed MPP SQL query engine that can query data in Iceberg at interactive speeds. Trino also enables you to join Iceberg tables with an array of other systems. Starburst offers both an enterprise deployment and a fully managed service to make managing and scaling Trino a flawless experience. Starburst also provides customer support and houses many of the original contributors to the open-source project that know Trino best. Learn more about the Starburst Iceberg connector.</p>"},{"location":"vendors/#tabular","title":"Tabular","text":"<p>Tabular is a managed warehouse and automation platform. Tabular offers a central store for analytic data that can be used with any query engine or processing framework that supports Iceberg. Tabular warehouses add role-based access control and automatic optimization, clustering, and compaction to Iceberg tables.</p>"},{"location":"view-spec/","title":"View Spec","text":""},{"location":"view-spec/#iceberg-view-spec","title":"Iceberg View Spec","text":""},{"location":"view-spec/#background-and-motivation","title":"Background and Motivation","text":"<p>Most compute engines (e.g. Trino and Apache Spark) support views. A view is a logical table that can be referenced by future queries. Views do not contain any data. Instead, the query stored by the view is executed every time the view is referenced by another query.</p> <p>Each compute engine stores the metadata of the view in its proprietary format in the metastore of choice. Thus, views created from one engine can not be read or altered easily from another engine even when engines share the metastore as well as the storage system. This document standardizes the view metadata for ease of sharing the views across engines.</p>"},{"location":"view-spec/#goals","title":"Goals","text":"<ul> <li>A common metadata format for view metadata, similar to how Iceberg supports a common table format for tables.</li> </ul>"},{"location":"view-spec/#overview","title":"Overview","text":"<p>View metadata storage mirrors how Iceberg table metadata is stored and retrieved. View metadata is maintained in metadata files. All changes to view state create a new view metadata file and completely replace the old metadata using an atomic swap. Like Iceberg tables, this atomic swap is delegated to the metastore that tracks tables and/or views by name. The view metadata file tracks the view schema, custom properties, current and past versions, as well as other metadata.</p> <p>Each metadata file is self-sufficient. It contains the history of the last few versions of the view and can be used to roll back the view to a previous version.</p>"},{"location":"view-spec/#metadata-location","title":"Metadata Location","text":"<p>An atomic swap of one view metadata file for another provides the basis for making atomic changes. Readers use the version of the view that was current when they loaded the view metadata and are not affected by changes until they refresh and pick up a new metadata location.</p> <p>Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location.</p>"},{"location":"view-spec/#specification","title":"Specification","text":""},{"location":"view-spec/#terms","title":"Terms","text":"<ul> <li>Schema -- Names and types of fields in a view.</li> <li>Version -- The state of a view at some point in time.</li> </ul>"},{"location":"view-spec/#view-metadata","title":"View Metadata","text":"<p>The view version metadata file has the following fields:</p> Requirement Field name Description required <code>view-uuid</code> A UUID that identifies the view, generated when the view is created. Implementations must throw an exception if a view's UUID does not match the expected UUID after refreshing metadata required <code>format-version</code> An integer version number for the view format; must be 1 required <code>location</code> The view's base location; used to create metadata file locations required <code>schemas</code> A list of known schemas required <code>current-version-id</code> ID of the current version of the view (<code>version-id</code>) required <code>versions</code> A list of known versions of the view [1] required <code>version-log</code> A list of version log entries with the timestamp and <code>version-id</code> for every change to <code>current-version-id</code> optional <code>properties</code> A string to string map of view properties [2] <p>Notes: 1. The number of versions to retain is controlled by the table property: <code>version.history.num-entries</code>. 2. Properties are used for metadata such as <code>comment</code> and for settings that affect view maintenance. This is not intended to be used for arbitrary metadata.</p>"},{"location":"view-spec/#versions","title":"Versions","text":"<p>Each version in <code>versions</code> is a struct with the following fields:</p> Requirement Field name Description required <code>version-id</code> ID for the version required <code>schema-id</code> ID of the schema for the view version required <code>timestamp-ms</code> Timestamp when the version was created (ms from epoch) required <code>summary</code> A string to string map of summary metadata about the version required <code>representations</code> A list of representations for the view definition optional <code>default-catalog</code> Catalog name to use when a reference in the SELECT does not contain a catalog required <code>default-namespace</code> Namespace to use when a reference in the SELECT is a single identifier <p>When <code>default-catalog</code> is <code>null</code> or not set, the catalog in which the view is stored must be used as the default catalog.</p>"},{"location":"view-spec/#summary","title":"Summary","text":"<p>Summary is a string to string map of metadata about a view version. Common metadata keys are documented here.</p> Requirement Key Value required <code>operation</code> Operation that caused this metadata to be created; must be <code>create</code> or <code>replace</code> optional <code>engine-name</code> Name of the engine that created the view version optional <code>engine-version</code> Version of the engine that created the view version"},{"location":"view-spec/#representations","title":"Representations","text":"<p>View definitions can be represented in multiple ways. Representations are documented ways to express a view definition.</p> <p>A view version can have more than one representation. All representations for a version must express the same underlying definition. Engines are free to choose the representation to use.</p> <p>View versions are immutable. Once a version is created, it cannot be changed. This means that representations for a version cannot be changed. If a view definition changes (or new representations are to be added), a new version must be created.</p> <p>Each representation is an object with at least one common field, <code>type</code>, that is one of the following: * <code>sql</code>: a SQL SELECT statement that defines the view</p> <p>Representations further define metadata for each type.</p>"},{"location":"view-spec/#sql-representation","title":"SQL representation","text":"<p>The SQL representation stores the view definition as a SQL SELECT, with metadata such as the SQL dialect.</p> <p>A view version can have multiple SQL representations of different dialects, but only one SQL representation per dialect.</p> Requirement Field name Type Description required <code>type</code> <code>string</code> Must be <code>sql</code> required <code>sql</code> <code>string</code> A SQL SELECT statement required <code>dialect</code> <code>string</code> The dialect of the <code>sql</code> SELECT statement (e.g., \"trino\" or \"spark\") <p>For example:</p> <p><pre><code>USE prod.default\n</code></pre> <pre><code>CREATE OR REPLACE VIEW event_agg (\n event_count COMMENT 'Count of events',\n event_date) AS\nSELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2\n</code></pre></p> <p>This create statement would produce the following <code>sql</code> representation metadata:</p> Field name Value <code>type</code> <code>\"sql\"</code> <code>sql</code> <code>\"SELECT\\n COUNT(1), CAST(event_ts AS DATE)\\nFROM events\\nGROUP BY 2\"</code> <code>dialect</code> <code>\"spark\"</code> <p>If a create statement does not include column names or comments before <code>AS</code>, the fields should be omitted.</p> <p>The <code>event_count</code> (with the <code>Count of events</code> comment) and <code>event_date</code> field aliases must be part of the view version's <code>schema</code>.</p>"},{"location":"view-spec/#version-log","title":"Version log","text":"<p>The version log tracks changes to the view's current version. This is the view's history and allows reconstructing what version of the view would have been used at some point in time.</p> <p>Note that this is not the version's creation time, which is stored in each version's metadata. A version can appear multiple times in the version log, indicating that the view definition was rolled back.</p> <p>Each entry in <code>version-log</code> is a struct with the following fields:</p> Requirement Field name Description required <code>timestamp-ms</code> Timestamp when the view's <code>current-version-id</code> was updated (ms from epoch) required <code>version-id</code> ID that <code>current-version-id</code> was set to"},{"location":"view-spec/#appendix-a-an-example","title":"Appendix A: An Example","text":"<p>The JSON metadata file format is described using an example below.</p> <p>Imagine the following sequence of operations:</p> <p><pre><code>USE prod.default\n</code></pre> <pre><code>CREATE OR REPLACE VIEW event_agg (\n event_count COMMENT 'Count of events',\n event_date)\nCOMMENT 'Daily event counts'\nAS\nSELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2\n</code></pre></p> <p>The metadata JSON file created looks as follows.</p> <p>The path is intentionally similar to the path for Iceberg tables and uses a <code>metadata</code> directory.</p> <p><pre><code>s3://bucket/warehouse/default.db/event_agg/metadata/00001-(uuid).metadata.json\n</code></pre> <pre><code>{\n \"view-uuid\": \"fa6506c3-7681-40c8-86dc-e36561f83385\",\n \"format-version\" : 1,\n \"location\" : \"s3://bucket/warehouse/default.db/event_agg\",\n \"current-version-id\" : 1,\n \"properties\" : {\n \"comment\" : \"Daily event counts\"\n },\n \"versions\" : [ {\n \"version-id\" : 1,\n \"timestamp-ms\" : 1573518431292,\n \"schema-id\" : 1,\n \"default-catalog\" : \"prod\",\n \"default-namespace\" : [ \"default\" ],\n \"summary\" : {\n \"operation\" : \"create\",\n \"engine-name\" : \"Spark\",\n \"engineVersion\" : \"3.3.2\"\n },\n \"representations\" : [ {\n \"type\" : \"sql\",\n \"sql\" : \"SELECT\\n COUNT(1), CAST(event_ts AS DATE)\\nFROM events\\nGROUP BY 2\",\n \"dialect\" : \"spark\"\n } ]\n } ],\n \"schemas\": [ {\n \"schema-id\": 1,\n \"type\" : \"struct\",\n \"fields\" : [ {\n \"id\" : 1,\n \"name\" : \"event_count\",\n \"required\" : false,\n \"type\" : \"int\",\n \"doc\" : \"Count of events\"\n }, {\n \"id\" : 2,\n \"name\" : \"event_date\",\n \"required\" : false,\n \"type\" : \"date\"\n } ]\n } ],\n \"version-log\" : [ {\n \"timestamp-ms\" : 1573518431292,\n \"version-id\" : 1\n } ]\n}\n</code></pre></p> <p>Each change creates a new metadata JSON file.</p> <pre><code>USE prod.other_db;\nCREATE OR REPLACE VIEW default.event_agg (\n event_count,\n event_date)\nAS\nSELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM prod.default.events\nGROUP BY 2\n</code></pre> <p>Updating the view produces a new metadata file that completely replaces the old:</p> <p><pre><code>s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json\n</code></pre> <pre><code>{\n \"view-uuid\": \"fa6506c3-7681-40c8-86dc-e36561f83385\",\n \"format-version\" : 1,\n \"location\" : \"s3://bucket/warehouse/default.db/event_agg\",\n \"current-version-id\" : 1,\n \"properties\" : {\n \"comment\" : \"Daily event counts\"\n },\n \"versions\" : [ {\n \"version-id\" : 1,\n \"timestamp-ms\" : 1573518431292,\n \"schema-id\" : 1,\n \"default-catalog\" : \"prod\",\n \"default-namespace\" : [ \"default\" ],\n \"summary\" : {\n \"operation\" : \"create\",\n \"engine-name\" : \"Spark\",\n \"engineVersion\" : \"3.3.2\"\n },\n \"representations\" : [ {\n \"type\" : \"sql\",\n \"sql\" : \"SELECT\\n COUNT(1), CAST(event_ts AS DATE)\\nFROM events\\nGROUP BY 2\",\n \"dialect\" : \"spark\"\n } ]\n }, {\n \"version-id\" : 2,\n \"timestamp-ms\" : 1573518981593,\n \"schema-id\" : 1,\n \"default-catalog\" : \"prod\",\n \"default-namespace\" : [ \"default\" ],\n \"summary\" : {\n \"operation\" : \"create\",\n \"engine-name\" : \"Spark\",\n \"engineVersion\" : \"3.3.2\"\n },\n \"representations\" : [ {\n \"type\" : \"sql\",\n \"sql\" : \"SELECT\\n COUNT(1), CAST(event_ts AS DATE)\\nFROM prod.default.events\\nGROUP BY 2\",\n \"dialect\" : \"spark\"\n } ]\n } ],\n \"schemas\": [ {\n \"schema-id\": 1,\n \"type\" : \"struct\",\n \"fields\" : [ {\n \"id\" : 1,\n \"name\" : \"event_count\",\n \"required\" : false,\n \"type\" : \"int\",\n \"doc\" : \"Count of events\"\n }, {\n \"id\" : 2,\n \"name\" : \"event_date\",\n \"required\" : false,\n \"type\" : \"date\"\n } ]\n } ],\n \"version-log\" : [ {\n \"timestamp-ms\" : 1573518431292,\n \"version-id\" : 1\n }, {\n \"timestamp-ms\" : 1573518981593,\n \"version-id\" : 2\n } ]\n}\n</code></pre></p>"}]} |