Apache Iceberg is an open standard for huge analytic tables that can be used by any processing engine. The community continuously improves Iceberg core library components to enable integrations with different compute engines that power analytics, business intelligence, machine learning, etc. Connectors for Spark, Flink and Hive are maintained in the main Iceberg repository.
Processing engine connectors maintained in the iceberg repository are built for multiple versions.
For Spark and Flink, each new version that introduces backwards incompatible upgrade has its dedicated integration codebase and release artifacts. For example, the code for Iceberg Spark 3.1 integration is under /spark/v3.1
and the code for Iceberg Spark 3.2 integration is under /spark/v3.2
. Different artifacts (iceberg-spark-3.1_2.12
and iceberg-spark-3.2_2.12
) are released for users to consume. By doing this, changes across versions are isolated. New features in Iceberg could be developed against the latest features of an engine without breaking support of old APIs in past engine versions.
For Hive, Hive 2 uses the iceberg-mr
package for Iceberg integration, and Hive 3 requires an additional dependency of the iceberg-hive3
package.
Iceberg provides a runtime connector jar for each supported version of Spark, Flink and Hive. When using Iceberg with these engines, the runtime jar is the only addition to the classpath needed in addition to vendor dependencies. For example, to use Iceberg with Spark 3.2 and AWS integrations, iceberg-spark-runtime-3.2_2.12
and AWS SDK dependencies are needed for the Spark installation.
Spark and Flink provide different runtime jars for each supported engine version. Hive 2 and Hive 3 currently share the same runtime jar. The runtime jar names and latest version download links are listed in the tables below.
Each engine version undergoes the following lifecycle stages:
3.0.0-preview
), or the engine does not yet have full feature compatibility compared to old versions yet. This stage allows Iceberg to release an engine version support without the need to wait for feature parity, shortening the release time.Version | Lifecycle Stage | Initial Iceberg Support | Latest Iceberg Support | Latest Runtime Jar |
---|---|---|---|---|
2.4 | End of Life | 0.7.0-incubating | 1.2.1 | iceberg-spark-runtime-2.4 |
3.0 | End of Life | 0.9.0 | 1.0.0 | iceberg-spark-runtime-3.0_2.12 |
3.1 | Deprecated | 0.12.0 | {{% icebergVersion %}} | [iceberg-spark-runtime-3.1_2.12](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.1_2.12/{{% icebergVersion %}}/iceberg-spark-runtime-3.1_2.12-{{% icebergVersion %}}.jar) [1] |
3.2 | Maintained | 0.13.0 | {{% icebergVersion %}} | [iceberg-spark-runtime-3.2_2.12](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.2_2.12/{{% icebergVersion %}}/iceberg-spark-runtime-3.2_2.12-{{% icebergVersion %}}.jar) |
3.3 | Maintained | 0.14.0 | {{% icebergVersion %}} | [iceberg-spark-runtime-3.3_2.12](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/{{% icebergVersion %}}/iceberg-spark-runtime-3.3_2.12-{{% icebergVersion %}}.jar) |
3.4 | Maintained | 1.3.0 | {{% icebergVersion %}} | [iceberg-spark-runtime-3.4_2.12](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/{{% icebergVersion %}}/iceberg-spark-runtime-3.3_2.12-{{% icebergVersion %}}.jar) |
iceberg-spark3-runtime
with Spark 3.0 before Iceberg 0.13.0Based on the guideline of the Flink community, only the latest 2 minor versions are actively maintained. Users should continuously upgrade their Flink version to stay up-to-date.
Version | Lifecycle Stage | Initial Iceberg Support | Latest Iceberg Support | Latest Runtime Jar |
---|---|---|---|---|
1.11 | End of Life | 0.9.0 | 0.12.1 | iceberg-flink-runtime |
1.12 | End of Life | 0.12.0 | 0.13.1 | iceberg-flink-runtime-1.12 [3] |
1.13 | End of Life | 0.13.0 | 1.0.0 | iceberg-flink-runtime-1.13 |
1.14 | End of Life | 0.13.0 | 1.2.0 | iceberg-flink-runtime-1.14 |
1.15 | Deprecated | 0.14.0 | {{% icebergVersion %}} | [iceberg-flink-runtime-1.15](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-flink-runtime-1.15/{{% icebergVersion %}}/iceberg-flink-runtime-1.15-{{% icebergVersion %}}.jar) |
1.16 | Maintained | 1.1.0 | {{% icebergVersion %}} | [iceberg-flink-runtime-1.16](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-flink-runtime-1.16/{{% icebergVersion %}}/iceberg-flink-runtime-1.16-{{% icebergVersion %}}.jar) |
1.17 | Maintained | 1.3.0 | {{% icebergVersion %}} | [iceberg-flink-runtime-1.17](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-flink-runtime-1.17/{{% icebergVersion %}}/iceberg-flink-runtime-1.17-{{% icebergVersion %}}.jar) |
iceberg-flink-runtime
with Flink 1.11 before Iceberg 0.13.0Version | Recommended minor version | Lifecycle Stage | Initial Iceberg Support | Latest Iceberg Support | Latest Runtime Jar |
---|---|---|---|---|---|
2 | 2.3.8 | Maintained | 0.8.0-incubating | {{% icebergVersion %}} | [iceberg-hive-runtime](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-hive-runtime/{{% icebergVersion %}}/iceberg-hive-runtime-{{% icebergVersion %}}.jar) |
3 | 3.1.2 | Maintained | 0.10.0 | {{% icebergVersion %}} | [iceberg-hive-runtime](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-hive-runtime/{{% icebergVersion %}}/iceberg-hive-runtime-{{% icebergVersion %}}.jar) |
Iceberg recommends the following for developers who are maintaining existing engine versions:
Iceberg recommends new engines to build support by importing the Iceberg libraries to the engine's project. This allows the Iceberg support to evolve with the engine. Projects such as Trino and Presto are good examples of such support strategy.
In this approach, an Iceberg version upgrade is needed for an engine to consume new Iceberg features. To facilitate engine development against unreleased Iceberg features, a daily snapshot is published in the Apache snapshot repository.
If bringing an engine directly to the Iceberg main repository is needed, please raise a discussion thread in the Iceberg community.