This is a specification for Puffin, a file format designed to store information such as indexes and statistics about data managed in an Iceberg table that cannot be stored directly within the Iceberg manifest. A Puffin file contains arbitrary pieces of information (here called “blobs”), along with metadata necessary to interpret them. The blobs supported by Iceberg are documented at Blob types.
A file conforming to the Puffin file format specification should have the structure as described below.
Currently, there is a single version of the Puffin file format, described below.
The Puffin file has the following structure
Magic Blob₁ Blob₂ ... Blobₙ Footer
where
Magic is four bytes 0x50, 0x46, 0x41, 0x31 (short for: Puffin Fratercula arctica, version 1),Blobᵢ is i-th blob contained in the file, to be interpreted by application according to the footer,Footer is defined below.Footer has the following structure
Magic FooterPayload FooterPayloadSize Flags Magic
where
Magic: four bytes, same as at the beginning of the fileFooterPayload: optionally compressed, UTF-8 encoded JSON payload describing the blobs in the file, with the structure described belowFooterPayloadSize: a length in bytes of the FooterPayload (after compression, if compressed), stored as 4 byte integerFlags: 4 bytes for boolean flagsFooterPayload is compressedA 4 byte integer is always signed, in a two's complement representation, stored little-endian.
Footer payload bytes is either uncompressed or LZ4-compressed (as a single LZ4 compression frame with content size present), UTF-8 encoded JSON payload representing a single FileMetadata object.
FileMetadata has the following fields
| Field Name | Field Type | Required | Description |
|---|---|---|---|
| blobs | list of BlobMetadata objects | yes | |
| properties | JSON object with string property values | no | storage for arbitrary meta-information, like writer identification/version. See Common properties for properties that are recommended to be set by a writer. |
BlobMetadata has the following fields
| Field Name | Field Type | Required | Description |
|---|---|---|---|
| type | JSON string | yes | See Blob types |
| fields | JSON list of ints | yes | List of field IDs the blob was computed for; the order of items is used to compute sketches stored in the blob. |
| snapshot-id | JSON long | yes | ID of the Iceberg table's snapshot the blob was computed from. |
| sequence-number | JSON long | yes | Sequence number of the Iceberg table's snapshot the blob was computed from. |
| offset | JSON long | yes | The offset in the file where the blob contents start |
| length | JSON long | yes | The length of the blob stored in the file (after compression, if compressed) |
| compression-codec | JSON string | no | See Compression codecs. If omitted, the data is assumed to be uncompressed. |
| properties | JSON object with string property values | no | storage for arbitrary meta-information about the blob |
The blobs can be of a type listed below
apache-datasketches-theta-v1 blob typeA serialized form of a “compact” Theta sketch produced by the Apache DataSketches library. The sketch is obtained by constructing Alpha family sketch with default seed, and feeding it with individual distinct values converted to bytes using Iceberg's single-value serialization.
The blob metadata for this blob may include following properties:
ndv: estimate of number of distinct values, derived from the sketch.The data can also be uncompressed. If it is compressed the codec should be one of codecs listed below. For maximal interoperability, other codecs are not supported.
| Codec name | Description |
|---|---|
| lz4 | Single LZ4 compression frame, with content size present |
| zstd | Single Zstandard compression frame, with content size present |
| __ |
When writing a Puffin file it is recommended to set the following fields in the FileMetadata's properties field.
created-by - human-readable identification of the application writing the file, along with its version. Example “Trino version 381”.