| <?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Introduction on Apache Iceberg</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/</link><description>Recent content in Introduction on Apache Iceberg</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><atom:link href="https://iceberg.apache.org/docs/UpdateSlackLink/index.xml" rel="self" type="application/rss+xml"/><item><title>Getting Started</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/getting-started/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/getting-started/</guid><description>Getting Started The latest version of Iceberg is 1.2.1. |
| Spark is currently the most feature-rich compute engine for Iceberg operations. We recommend you to get started with Spark to understand Iceberg concepts and features with examples. You can also view documentations of using Iceberg with other compute engine under the Multi-Engine Support page. |
| Using Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:</description></item><item><title>Hive</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/hive/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/hive/</guid><description>Hive Iceberg supports reading and writing Iceberg tables through Hive by using a StorageHandler. |
| Feature support Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features: |
| Creating a table Dropping a table Reading a table Inserting into a table (INSERT INTO) DML operations work only with MapReduce execution engine. With Hive version 4.0.0-alpha-2 and above, the Iceberg integration when using HiveCatalog supports the following additional features: |
| Altering a table with expiring snapshots.</description></item><item><title>AWS</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/aws/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/aws/</guid><description>Iceberg AWS Integrations Iceberg provides integration with different AWS services through the iceberg-aws module. This section describes how to use Iceberg with AWS. |
| Enabling AWS Integration The iceberg-aws module is bundled with Spark and Flink engine runtimes for all versions from 0.11.0 onwards. However, the AWS clients are not bundled so that you can use the same client version as your application. You will need to provide the AWS v2 SDK because that is what Iceberg depends on.</description></item><item><title>Branching and Tagging</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/branching/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/branching/</guid><description>Branching and Tagging Overview Iceberg table metadata maintains a log of snapshots which represent the changes applied to a table. Snapshots are fundamental in Iceberg as they are the basis for reader isolation and time travel queries. For controlling metadata size and storage costs, Iceberg provides snapshot lifecycle management procedures such as expire_snapshots for removing unused snapshots and no longer neccessary data files based on table snapshot retention properties.</description></item><item><title>Configuration</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/configuration/</guid><description>Configuration Table properties Iceberg tables support table properties to configure table behavior, like the default split size for readers. |
| Read properties Property Default Description read.split.target-size 134217728 (128 MB) Target size when combining data input splits read.split.metadata-target-size 33554432 (32 MB) Target size when combining metadata input splits read.split.planning-lookback 10 Number of bins to consider when combining input splits read.split.open-file-cost 4194304 (4 MB) The estimated cost to open a file, used as a minimum weight when combining splits.</description></item><item><title>Configuration</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-configuration/</guid><description>Spark Configuration Catalogs Spark adds an API to plug in table catalogs that are used to load, create, and manage Iceberg tables. Spark catalogs are configured by setting Spark properties under spark.sql.catalog. |
| This creates an Iceberg catalog named hive_prod that loads tables from a Hive metastore: |
| spark.sql.catalog.hive_prod = org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.hive_prod.type = hive spark.sql.catalog.hive_prod.uri = thrift://metastore-host:port # omit uri to use the same URI as Spark: hive.metastore.uris in hive-site.xml Iceberg also supports a directory-based catalog in HDFS that can be configured using type=hadoop:</description></item><item><title>DDL</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-ddl/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-ddl/</guid><description>Spark DDL To use Iceberg in Spark, first configure Spark catalogs. Iceberg uses Apache Spark&rsquo;s DataSourceV2 API for data source and catalog implementations. |
| CREATE TABLE Spark 3 can create tables in any Iceberg catalog with the clause USING iceberg: |
| CREATE TABLE prod.db.sample ( id bigint COMMENT &#39;unique id&#39;, data string) USING iceberg Iceberg will convert the column type in Spark to corresponding Iceberg type. Please check the section of type compatibility on creating table for details.</description></item><item><title>Dell</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/dell/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/dell/</guid><description>Iceberg Dell Integration Dell ECS Integration Iceberg can be used with Dell&rsquo;s Enterprise Object Storage (ECS) by using the ECS catalog since 0.15.0. |
| See Dell ECS for more information on Dell ECS. |
| Parameters When using Dell ECS with Iceberg, these configuration parameters are required: |
| Name Description ecs.s3.endpoint ECS S3 service endpoint ecs.s3.access-key-id ECS Username ecs.s3.secret-access-key S3 Secret Key warehouse The location of data and metadata The warehouse should use the following formats:</description></item><item><title>Delta Lake Migration</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/delta-lake-migration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/delta-lake-migration/</guid><description>Delta Lake Table Migration Delta Lake is a table format that supports Parquet file format and provides time travel and versioning features. When migrating data from Delta Lake to Iceberg, it is common to migrate all snapshots to maintain the history of the data. |
| Currently, Iceberg supports the Snapshot Table action for migrating from Delta Lake to Iceberg tables. Since Delta Lake tables maintain transactions, all available transactions will be committed to the new Iceberg table as transactions in order.</description></item><item><title>Evolution</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/evolution/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/evolution/</guid><description>Evolution Iceberg supports in-place table evolution. You can evolve a table schema just like SQL &ndash; even in nested structures &ndash; or change partition layout when data volume changes. Iceberg does not require costly distractions, like rewriting table data or migrating to a new table. |
| For example, Hive table partitioning cannot change so moving from a daily partition layout to an hourly partition layout requires a new table. And because queries are dependent on partitions, queries must be rewritten for the new table.</description></item><item><title>Flink Actions</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-actions/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-actions/</guid><description>Rewrite files action. Iceberg provides API to rewrite small files into large files by submitting Flink batch jobs. The behavior of this Flink action is the same as Spark&rsquo;s rewriteDataFiles. |
| import org.apache.iceberg.flink.actions.Actions; TableLoader tableLoader = TableLoader.fromHadoopTable(&#34;hdfs://nn:8020/warehouse/path&#34;); Table table = tableLoader.loadTable(); RewriteDataFilesActionResult result = Actions.forTable(table) .rewriteDataFiles() .execute(); For more details of the rewrite files action, please refer to RewriteDataFilesAction</description></item><item><title>Flink Configuration</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-configuration/</guid><description>Flink Configuration Catalog Configuration A catalog is created and named by executing the following query (replace &lt;catalog_name&gt; with your catalog name and &lt;config_key&gt;=&lt;config_value&gt; with catalog implementation config): |
| CREATE CATALOG &lt;catalog_name&gt; WITH ( &#39;type&#39;=&#39;iceberg&#39;, `&lt;config_key&gt;`=`&lt;config_value&gt;` ); The following properties can be set globally and are not limited to a specific catalog implementation: |
| Property Required Values Description type ✔️ iceberg Must be iceberg. catalog-type hive, hadoop or rest hive, hadoop or rest for built-in catalogs, or left unset for custom catalog implementations using catalog-impl.</description></item><item><title>Flink Connector</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-connector/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-connector/</guid><description>Flink Connector Apache Flink supports creating Iceberg table directly without creating the explicit Flink catalog in Flink SQL. That means we can just create an iceberg table by specifying 'connector'='iceberg' table option in Flink SQL which is similar to usage in the Flink official document. |
| In Flink, the SQL CREATE TABLE test (..) WITH ('connector'='iceberg', ...) will create a Flink table in current Flink catalog (use GenericInMemoryCatalog by default), which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current Flink catalog.</description></item><item><title>Flink DDL</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-ddl/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-ddl/</guid><description>DDL commands CREATE Catalog Hive catalog This creates an Iceberg catalog named hive_catalog that can be configured using 'catalog-type'='hive', which loads tables from Hive metastore: |
| CREATE CATALOG hive_catalog WITH ( &#39;type&#39;=&#39;iceberg&#39;, &#39;catalog-type&#39;=&#39;hive&#39;, &#39;uri&#39;=&#39;thrift://localhost:9083&#39;, &#39;clients&#39;=&#39;5&#39;, &#39;property-version&#39;=&#39;1&#39;, &#39;warehouse&#39;=&#39;hdfs://nn:8020/warehouse/path&#39; ); The following properties can be set if using the Hive catalog: |
| uri: The Hive metastore&rsquo;s thrift URI. (Required) clients: The Hive metastore client pool size, default value is 2. (Optional) warehouse: The Hive warehouse location, users should specify this path if neither set the hive-conf-dir to specify a location containing a hive-site.</description></item><item><title>Flink Getting Started</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink/</guid><description>Flink Apache Iceberg supports both Apache Flink&rsquo;s DataStream API and Table API. See the Multi-Engine Support#apache-flink page for the integration of Apache Flink. |
| Feature support Flink Notes SQL create catalog ✔️ SQL create database ✔️ SQL create table ✔️ SQL create table like ✔️ SQL alter table ✔️ Only support altering table properties, column and partition changes are not supported SQL drop_table ✔️ SQL select ✔️ Support both streaming and batch mode SQL insert into ✔️ ️ Support both streaming and batch mode SQL insert overwrite ✔️ ️ DataStream read ✔️ ️ DataStream append ✔️ ️ DataStream overwrite ✔️ ️ Metadata tables ✔️ Rewrite files action ✔️ ️ Preparation when using Flink SQL Client To create Iceberg table in Flink, it is recommended to use Flink SQL Client as it&rsquo;s easier for users to understand the concepts.</description></item><item><title>Flink Queries</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-queries/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-queries/</guid><description>Flink Queries Iceberg support streaming and batch read With Apache Flink&rsquo;s DataStream API and Table API. |
| Reading with SQL Iceberg support both streaming and batch read in Flink. Execute the following sql command to switch execution mode from streaming to batch, and vice versa: |
| -- Execute the flink job in streaming mode for current session context SET execution.runtime-mode = streaming; -- Execute the flink job in batch mode for current session context SET execution.</description></item><item><title>Flink Writes</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/flink-writes/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/flink-writes/</guid><description>Flink Writes Iceberg support batch and streaming writes With Apache Flink&rsquo;s DataStream API and Table API. |
| Writing with SQL Iceberg support both INSERT INTO and INSERT OVERWRITE. |
| INSERT INTO To append new data to a table with a Flink streaming job, use INSERT INTO: |
| INSERT INTO `hive_catalog`.`default`.`sample` VALUES (1, &#39;a&#39;); INSERT INTO `hive_catalog`.`default`.`sample` SELECT id, data from other_kafka_table; INSERT OVERWRITE To replace data in the table with the result of a query, use INSERT OVERWRITE in batch job (flink streaming job does not support INSERT OVERWRITE).</description></item><item><title>Hive Migration</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/hive-migration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/hive-migration/</guid><description>Hive Table Migration Apache Hive supports ORC, Parquet, and Avro file formats that could be migrated to Iceberg. When migrating data to an Iceberg table, which provides versioning and transactional updates, only the most recent data files need to be migrated. |
| Iceberg supports all three migration actions: Snapshot Table, Migrate Table, and Add Files for migrating from Hive tables to Iceberg tables. Since Hive tables do not maintain snapshots, the migration process essentially involves creating a new Iceberg table with the existing schema and committing all data files across all partitions to the new Iceberg table.</description></item><item><title>Java API</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/api/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/api/</guid><description>Iceberg Java API Tables The main purpose of the Iceberg API is to manage table metadata, like schema, partition spec, metadata, and data files that store table data. |
| Table metadata and operations are accessed through the Table interface. This interface will return table information. |
| Table metadata The Table interface provides access to the table metadata: |
| schema returns the current table schema spec returns the current table partition spec properties returns a map of key-value properties currentSnapshot returns the current table snapshot snapshots returns all valid snapshots for the table snapshot(id) returns a specific snapshot by ID location returns the table&rsquo;s base location Tables also provide refresh to update the table to the latest version, and expose helpers:</description></item><item><title>Java Custom Catalog</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/custom-catalog/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/custom-catalog/</guid><description>Custom Catalog It&rsquo;s possible to read an iceberg table either from an hdfs path or from a hive table. It&rsquo;s also possible to use a custom metastore in place of hive. The steps to do that are as follows. |
| Custom TableOperations Custom Catalog Custom FileIO Custom LocationProvider Custom IcebergSource Custom table operations implementation Extend BaseMetastoreTableOperations to provide implementation on how to read and write metadata |
| Example: |
| class CustomTableOperations extends BaseMetastoreTableOperations { private String dbName; private String tableName; private Configuration conf; private FileIO fileIO; protected CustomTableOperations(Configuration conf, String dbName, String tableName) { this.</description></item><item><title>Java Quickstart</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/java-api-quickstart/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/java-api-quickstart/</guid><description>Java API Quickstart Create a table Tables are created using either a Catalog or an implementation of the Tables interface. |
| Using a Hive catalog The Hive catalog connects to a Hive metastore to keep track of Iceberg tables. You can initialize a Hive catalog with a name and some properties. (see: Catalog properties) |
| Note: Currently, setConf is always required for hive catalogs, but this will change in the future.</description></item><item><title>JDBC</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/jdbc/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/jdbc/</guid><description>Iceberg JDBC Integration JDBC Catalog Iceberg supports using a table in a relational database to manage Iceberg tables through JDBC. The database that JDBC connects to must support atomic transaction to allow the JDBC catalog implementation to properly support atomic Iceberg table commits and read serializable isolation. |
| Configurations Because each database and database service provider might require different configurations, the JDBC catalog allows arbitrary configurations through: |
| Property Default Description uri the JDBC connection string jdbc.</description></item><item><title>Maintenance</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/maintenance/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/maintenance/</guid><description>Maintenance Maintenance operations require the Table instance. Please refer Java API quickstart page to refer how to load an existing table. Recommended Maintenance Expire Snapshots Each write to an Iceberg table creates a new snapshot, or version, of a table. Snapshots can be used for time-travel queries, or the table can be rolled back to any valid snapshot. |
| Snapshots accumulate until they are expired by the expireSnapshots operation. Regularly expiring snapshots is recommended to delete data files that are no longer needed, and to keep the size of table metadata small.</description></item><item><title>Nessie</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/nessie/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/nessie/</guid><description>Iceberg Nessie Integration Iceberg provides integration with Nessie through the iceberg-nessie module. This section describes how to use Iceberg with Nessie. Nessie provides several key features on top of Iceberg: |
| multi-table transactions git-like operations (eg branches, tags, commits) hive-like metastore capabilities See Project Nessie for more information on Nessie. Nessie requires a server to run, see Getting Started to start a Nessie server. |
| Enabling Nessie Catalog The iceberg-nessie module is bundled with Spark and Flink runtimes for all versions from 0.</description></item><item><title>Overview</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/table-migration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/table-migration/</guid><description>Table Migration Apache Iceberg supports converting existing tables in other formats to Iceberg tables. This section introduces the general concept of table migration, its approaches, and existing implementations in Iceberg. |
| Migration Approaches There are two methods for executing table migration: full data migration and in-place metadata migration. |
| Full data migration involves copying all data files from the source table to the new Iceberg table. This method makes the new table fully isolated from the source table, but is slower and doubles the space.</description></item><item><title>Partitioning</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/partitioning/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/partitioning/</guid><description>Partitioning What is partitioning? Partitioning is a way to make queries faster by grouping similar rows together when writing. |
| For example, queries for log entries from a logs table would usually include a time range, like this query for logs between 10 and 12 AM: |
| SELECT level, message FROM logs WHERE event_time BETWEEN &#39;2018-12-01 10:00:00&#39; AND &#39;2018-12-01 12:00:00&#39; Configuring the logs table to partition by the date of event_time will group log events into files with the same event date.</description></item><item><title>Performance</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/performance/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/performance/</guid><description>Performance Iceberg is designed for huge tables and is used in production where a single table can contain tens of petabytes of data. Even multi-petabyte tables can be read from a single node, without needing a distributed SQL engine to sift through table metadata. Scan planning Scan planning is the process of finding the files in a table that are needed for a query. |
| Planning in an Iceberg table fits on a single node because Iceberg&rsquo;s metadata can be used to prune metadata files that aren&rsquo;t needed, in addition to filtering data files that don&rsquo;t contain matching data.</description></item><item><title>Procedures</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-procedures/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-procedures/</guid><description>Spark Procedures To use Iceberg in Spark, first configure Spark catalogs. Stored procedures are only available when using Iceberg SQL extensions in Spark 3. |
| Usage Procedures can be used from any configured Iceberg catalog with CALL. All procedures are in the namespace system. |
| CALL supports passing arguments by name (recommended) or by position. Mixing position and named arguments is not supported. |
| Named arguments All procedure arguments are named. When passing arguments by name, arguments can be in any order and any optional argument can be omitted.</description></item><item><title>Queries</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-queries/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-queries/</guid><description>Spark Queries To use Iceberg in Spark, first configure Spark catalogs. Iceberg uses Apache Spark&rsquo;s DataSourceV2 API for data source and catalog implementations. |
| Querying with SQL In Spark 3, tables use identifiers that include a catalog name. |
| SELECT * FROM prod.db.table; -- catalog: prod, namespace: db, table: table Metadata tables, like history and snapshots, can use the Iceberg table name as a namespace. |
| For example, to read from the files metadata table for prod.</description></item><item><title>Reliability</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/reliability/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/reliability/</guid><description>Reliability Iceberg was designed to solve correctness problems that affect Hive tables running in S3. |
| Hive tables track data files using both a central metastore for partitions and a file system for individual files. This makes atomic changes to a table&rsquo;s contents impossible, and eventually consistent stores like S3 may return incorrect results due to the use of listing files to reconstruct the state of a table. It also requires job planning to make many slow listing calls: O(n) with the number of partitions.</description></item><item><title>Schemas</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/schemas/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/schemas/</guid><description>Schemas Iceberg tables support the following types: |
| Type Description Notes boolean True or false int 32-bit signed integers Can promote to long long 64-bit signed integers float 32-bit IEEE 754 floating point Can promote to double double 64-bit IEEE 754 floating point decimal(P,S) Fixed-point decimal; precision P, scale S Scale is fixed and precision must be 38 or less date Calendar date without timezone or time time Time of day without date, timezone Stored as microseconds timestamp Timestamp without timezone Stored as microseconds timestamptz Timestamp with timezone Stored as microseconds string Arbitrary-length character sequences Encoded with UTF-8 fixed(L) Fixed-length byte array of length L binary Arbitrary-length byte array struct&lt;.</description></item><item><title>Structured Streaming</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-structured-streaming/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-structured-streaming/</guid><description>Spark Structured Streaming Iceberg uses Apache Spark&rsquo;s DataSourceV2 API for data source and catalog implementations. Spark DSv2 is an evolving API with different levels of support in Spark versions. |
| As of Spark 3, DataFrame reads and writes are supported. |
| Streaming Reads Iceberg supports processing incremental data in spark structured streaming jobs which starts from a historical timestamp: |
| val df = spark.readStream .format(&#34;iceberg&#34;) .option(&#34;stream-from-timestamp&#34;, Long.toString(streamStartTimestamp)) .load(&#34;database.table_name&#34;) Iceberg only supports reading data from append snapshots.</description></item><item><title>Writes</title><link>https://iceberg.apache.org/docs/UpdateSlackLink/spark-writes/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/UpdateSlackLink/spark-writes/</guid><description>Spark Writes To use Iceberg in Spark, first configure Spark catalogs. |
| Some plans are only available when using Iceberg SQL extensions in Spark 3. |
| Iceberg uses Apache Spark&rsquo;s DataSourceV2 API for data source and catalog implementations. Spark DSv2 is an evolving API with different levels of support in Spark versions: |
| Feature support Spark 3 Notes SQL insert into ✔️ SQL merge into ✔️ ⚠ Requires Iceberg Spark extensions SQL insert overwrite ✔️ SQL delete from ✔️ ⚠ Row-level delete requires Spark extensions SQL update ✔️ ⚠ Requires Iceberg Spark extensions DataFrame append ✔️ DataFrame overwrite ✔️ DataFrame CTAS and RTAS ✔️ Writing with SQL Spark 3 supports SQL INSERT INTO, MERGE INTO, and INSERT OVERWRITE, as well as the new DataFrameWriterV2 API.</description></item></channel></rss> |
