docs/specification/row_format_spec.md - fory-site - Git at Google

 ---
 title: Row Format
 sidebar_position: 2
 id: row_format_spec
 license: |
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
 ---

 ## Overview

 Apache Fory Row Format is a cache-friendly, random-access binary format designed for high-performance data processing. Unlike traditional serialization formats that require full deserialization, the row format enables:

 - **Random Field Access**: Read individual fields without deserializing the entire row
 - **Zero-Copy Operations**: Direct memory access without data transformation
 - **Cache-Friendly Layout**: Optimized memory layout for CPU cache efficiency
 - **Cross-Language Support**: Consistent binary format across Java, C++, and Python

 Fory provides two row format variants:

 | Format          | Languages         | Use Case                       |
 | --------------- | ----------------- | ------------------------------ |
 | Standard Format | Java, C++, Python | Cross-language compatibility   |
 | Compact Format  | Java only         | Space efficiency, smaller rows |

 ## Format Comparison

 | Feature              | Standard Format               | Compact Format                      |
 | -------------------- | ----------------------------- | ----------------------------------- |
 | Field Slot Size      | Fixed 8 bytes                 | Natural width (1, 2, 4, or 8 bytes) |
 | Null Bitmap Size     | 8-byte aligned                | Byte-aligned, can borrow padding    |
 | Null Bitmap Position | Before field slots            | After field slots (at end)          |
 | Fixed-Size Structs   | Variable region (offset+size) | Inline in fixed region              |
 | Field Ordering       | Schema-defined order          | Sorted by alignment                 |
 | All Non-Nullable     | Bitmap still present          | Bitmap skipped entirely             |
 | Alignment            | Strict 8-byte                 | Relaxed (2, 4, or 8-byte)           |

 ---

 ## Standard Row Format

 The standard format prioritizes cross-language compatibility and simplicity with uniform 8-byte field slots.

 ### Design Principles

 1. **8-Byte Alignment**: All major structures are aligned to 8-byte boundaries for optimal memory access
 2. **Fixed-Width Field Slots**: Every field uses an 8-byte slot for uniform offset calculation
 3. **Null Bitmap**: Compact null tracking using bit vectors
 4. **Relative Offsets**: Variable-length data uses relative offsets for sub-buffer navigation

 ### Row Binary Layout

 A row stores structured data with the following layout:

 ```
 +----------------+------------------+------------------+-----+------------------+------------------+
 |  Null Bitmap   |  Field 0 Slot    |  Field 1 Slot    | ... |  Field N-1 Slot  |  Variable Data   |
 +----------------+------------------+------------------+-----+------------------+------------------+
 |  B bytes       |     8 bytes      |     8 bytes      |     |     8 bytes      |  Variable size   |
 ```

 #### Null Bitmap

 The null bitmap tracks which fields contain null values:

 - **Size**: `((num_fields + 63) / 64) * 8` bytes (rounded up to nearest 8-byte word)
 - **Encoding**: Each bit corresponds to a field index
   - Bit value `1` = field is null
   - Bit value `0` = field is not null
 - **Bit Order**: Bit 0 of the first byte corresponds to field 0

 **Example**: For 10 fields, bitmap size = `((10 + 63) / 64) * 8 = 8` bytes

 #### Field Slots

 Each field occupies a fixed 8-byte slot regardless of its actual data type:

 - **Slot Offset**: `bitmap_size + field_index * 8`
 - **Total Fixed Region**: `bitmap_size + num_fields * 8` bytes

 **Field Slot Contents by Type**:

 | Type Category  | Slot Contents                       |
 | -------------- | ----------------------------------- |
 | Fixed-width    | Value stored directly (zero-padded) |
 | Variable-width | Offset + Size encoding (see below)  |

 #### Variable-Width Data Encoding

 Variable-length fields (strings, arrays, maps, nested structs) store an offset-size pair in their slot:

 ```
 +---------------------------+---------------------------+
 |    Relative Offset        |         Size              |
 |       (32 bits)           |       (32 bits)           |
 +---------------------------+---------------------------+
 |<-------------- 64-bit field slot value -------------->|
 ```

 - **Relative Offset** (upper 32 bits): Offset from the row's base address
 - **Size** (lower 32 bits): Size of the variable-width data in bytes

 **Encoding**:

 ```
 offset_and_size = (relative_offset << 32) | size
 ```

 **Decoding**:

 ```
 relative_offset = (offset_and_size >> 32) & 0xFFFFFFFF
 size = offset_and_size & 0xFFFFFFFF
 ```

 #### Variable Data Region

 Variable-length data is stored after the fixed region:

 - Data is written sequentially as fields are set
 - Each variable-length value is padded to 8-byte alignment
 - Padding bytes are zeroed for deterministic output

 ### Array Binary Layout

 Arrays store homogeneous sequences of elements:

 ```
 +------------------+------------------+------------------+
 |  Element Count   |   Null Bitmap    |   Element Data   |
 +------------------+------------------+------------------+
 |     8 bytes      |     B bytes      |   Variable size  |
 ```

 #### Array Header

 | Field         | Size                            | Description                 |
 | ------------- | ------------------------------- | --------------------------- |
 | Element Count | 8 bytes                         | Number of elements (uint64) |
 | Null Bitmap   | `((count + 63) / 64) * 8` bytes | Per-element null flags      |

 **Header Size**: `8 + ((num_elements + 63) / 64) * 8` bytes

 #### Array Element Data

 Elements are stored contiguously after the header:

 - **Fixed-width elements**: Stored with their natural width (1, 2, 4, or 8 bytes)
 - **Variable-width elements**: Stored as 8-byte offset+size pairs

 **Element Offset**: `header_size + element_index * element_size`

 **Data Region Size**: Rounded up to nearest 8-byte boundary

 #### Array Element Sizes

 | Element Type     | Element Size          |
 | ---------------- | --------------------- |
 | bool             | 1 byte                |
 | int8             | 1 byte                |
 | int16            | 2 bytes               |
 | int32            | 4 bytes               |
 | int64            | 8 bytes               |
 | float32          | 4 bytes               |
 | float64          | 8 bytes               |
 | string/binary    | 8 bytes (offset+size) |
 | array/map/struct | 8 bytes (offset+size) |

 ### Map Binary Layout

 Maps store key-value pairs as two separate arrays:

 ```
 +------------------+------------------+------------------+
 |  Keys Array Size |   Keys Array     |   Values Array   |
 +------------------+------------------+------------------+
 |     8 bytes      |   Variable size  |   Variable size  |
 ```

 #### Map Structure

 | Field           | Size     | Description                       |
 | --------------- | -------- | --------------------------------- |
 | Keys Array Size | 8 bytes  | Total size of keys array in bytes |
 | Keys Array      | Variable | Full array structure for keys     |
 | Values Array    | Variable | Full array structure for values   |

 **Keys Array Offset**: `base_offset + 8`
 **Values Array Offset**: `base_offset + 8 + keys_array_size`

 Both keys and values arrays follow the standard array binary layout.

 ### Nested Struct Layout

 Nested structs are stored as complete row structures within the variable data region:

 1. Parent field slot contains offset+size pointing to nested row
 2. Nested row has its own null bitmap and field slots
 3. Supports arbitrary nesting depth

 ```
 Parent Row:
 +----------------+------------------+------------------+
 |  Null Bitmap   |  ... Slots ...   |  Nested Row Data |
 +----------------+------------------+------------------+
                         |                    ^
                         |  offset+size       |
                         +------------------->+

 Nested Row:
 +----------------+------------------+------------------+
 |  Null Bitmap   |  Field Slots     |  Variable Data   |
 +----------------+------------------+------------------+
 ```

 ---

 ## Compact Row Format (Java Only)

 The compact format provides space-efficient encoding with additional optimizations. It is currently implemented in Java only.

 > **Note**: The compact format is still under development and may not be stable yet.

 ### Design Principles

 1. **Natural Width Storage**: Fixed-size fields use their natural byte width instead of 8 bytes
 2. **Alignment-Based Field Sorting**: Fields are sorted by alignment requirements to minimize padding
 3. **Conditional Null Bitmap**: Null bitmap is omitted when all fields are non-nullable
 4. **Inline Fixed-Size Structs**: Nested structs with all fixed-size fields are stored inline

 ### Compact Row Binary Layout

 ```
 +------------------+------------------+-----+------------------+----------------+------------------+
 |  Field 0 Value   |  Field 1 Value   | ... |  Field N-1 Value | Null Bitmap    |  Variable Data   |
 +------------------+------------------+-----+------------------+----------------+------------------+
 |  W0 bytes        |  W1 bytes        |     |  WN-1 bytes      | B bytes (opt)  |  Variable size   |
 ```

 #### Key Differences from Standard Format

 1. **Field Slot Sizes**: Each field uses its natural width (Wi = type width or 8 for variable)
 2. **Null Bitmap Position**: Placed after field slots, can borrow alignment padding
 3. **Field Order**: Fields are sorted by alignment (8-byte → 4-byte → 2-byte → 1-byte → variable)
 4. **Conditional Bitmap**: Skipped entirely if all fields are non-nullable

 #### Null Bitmap (Compact)

 - **Size**: `(num_nullable_fields + 7) / 8` bytes (byte-aligned, not 8-byte aligned)
 - **Skipped**: When all fields are primitive/non-nullable
 - **Position**: After all fixed-size field slots, can use alignment padding space

 #### Field Sorting Algorithm

 Fields are sorted to minimize padding and optimize alignment:

 ```
 Priority order (highest to lowest):
 1. Fields with 8-byte alignment (int64, float64, variable-width)
 2. Fields with 4-byte alignment (int32, float32)
 3. Fields with 2-byte alignment (int16)
 4. Fields with 1-byte alignment (int8, bool)
 ```

 Within each alignment group, larger fields come first.

 #### Fixed-Size Struct Inlining

 Nested structs with all fixed-size fields are stored inline in the parent row:

 **Standard Format** (nested struct with 2 int32 fields):

 ```
 Parent slot: [offset (4 bytes) | size (4 bytes)]  → Points to nested row (8+ bytes elsewhere)
 ```

 **Compact Format** (same nested struct):

 ```
 Parent slot: [int32 field 0 | int32 field 1]  → 8 bytes total, inline
 ```

 This eliminates the offset+size indirection for fixed-size nested structures.

 #### Fixed-Width Calculation

 A field's fixed width is determined recursively:

 - **Primitive types**: Natural byte width (1, 2, 4, or 8)
 - **Struct types**: Sum of all child fixed widths (if all children are fixed-width)
 - **Variable types** (string, array, map): Returns -1 (uses 8-byte offset+size slot)

 ```
 fixed_width(field) =
   if primitive: type_width
   if struct and all_children_fixed: header_bytes + sum(fixed_width(child) for each child)
   else: -1 (variable, uses 8-byte slot)
 ```

 ### Compact Array Binary Layout

 ```
 +------------------+------------------+------------------+
 |  Element Count   |   Null Bitmap    |   Element Data   |
 +------------------+------------------+------------------+
 |     4 bytes      | B bytes (opt)    |   Variable size  |
 ```

 #### Compact Array Header

 | Field         | Size                          | Description                |
 | ------------- | ----------------------------- | -------------------------- |
 | Element Count | 4 bytes                       | Number of elements (int32) |
 | Null Bitmap   | `(count + 7) / 8` bytes (opt) | Per-element null flags     |

 **Header Size Calculation**:

 ```
 header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)

 // Round to 8-byte boundary only if element width is 8-byte aligned
 if (fixed_width % 8 == 0):
     header_size = round_to_8_bytes(header_size)
 ```

 #### Key Differences from Standard Array

 1. **Element Count**: 4 bytes instead of 8 bytes
 2. **Null Bitmap**: Byte-aligned, skipped if elements are non-nullable
 3. **Fixed-Size Structs**: Inline storage for fixed-width struct elements

 ---

 ## Common Specifications

 The following specifications apply to both standard and compact formats.

 ### Type Encoding

 #### Primitive Types

 | Type    | Width   | Encoding                        |
 | ------- | ------- | ------------------------------- |
 | bool    | 1 byte  | `0x00` (false) or `0x01` (true) |
 | int8    | 1 byte  | Two's complement                |
 | int16   | 2 bytes | Two's complement, little-endian |
 | int32   | 4 bytes | Two's complement, little-endian |
 | int64   | 8 bytes | Two's complement, little-endian |
 | float32 | 4 bytes | IEEE 754 single precision       |
 | float64 | 8 bytes | IEEE 754 double precision       |

 #### Temporal Types

 | Type      | Width   | Encoding                              |
 | --------- | ------- | ------------------------------------- |
 | timestamp | 8 bytes | Microseconds since Unix epoch (int64) |
 | date32    | 4 bytes | Days since Unix epoch (int32)         |
 | duration  | 8 bytes | Duration in microseconds (int64)      |

 #### String and Binary

 - **Encoding**: UTF-8 for strings, raw bytes for binary
 - **Storage**: Offset+size pair in field slot, data in variable region
 - **Padding**: Data padded to 8-byte alignment (standard) or natural alignment (compact)

 ### Null Handling

 #### Row Null Handling

 - Null fields have their corresponding bit set to 1 in the null bitmap
 - Field slot contents are undefined for null fields (standard) or zeroed (compact)
 - Reading a null field returns a null/empty value indicator

 #### Array Null Handling

 - Null elements have their corresponding bit set to 1 in the array's null bitmap
 - Element data is undefined for null elements
 - Compact format: Bitmap skipped if elements are non-nullable

 #### Variable-Width Null Semantics

 When reading variable-width data from a null field:

 - Returns size of -1 or equivalent null indicator
 - No data access is performed

 ### Alignment and Padding

 #### Standard Format Alignment

 1. **Null Bitmap**: Size rounded to 8-byte boundary
 2. **Field Slots**: Always 8 bytes each
 3. **Variable Data**: Each value padded to 8-byte boundary
 4. **Array Data**: Total data region padded to 8-byte boundary

 #### Compact Format Alignment

 1. **Field Slots**: Natural width (1, 2, 4, or 8 bytes)
 2. **Null Bitmap**: Byte-aligned, placed after fields
 3. **Variable Data**: Padded to 8-byte boundary only when needed
 4. **Header**: May use relaxed alignment for smaller overhead

 #### Padding Bytes

 - All padding bytes must be set to zero
 - Ensures deterministic serialization output
 - Prevents information leakage from uninitialized memory

 ## Size Calculations

 ### Standard Row Size

 ```
 row_size = bitmap_size + num_fields * 8 + variable_data_size

 where:
   bitmap_size = ((num_fields + 63) / 64) * 8
   variable_data_size = sum of (padded_size for each variable field)
   padded_size = ((size + 7) / 8) * 8
 ```

 ### Compact Row Size

 ```
 row_size = fixed_region_size + bitmap_size + variable_data_size

 where:
   fixed_region_size = sum of (fixed_width(field) or 8 for each field)
   bitmap_size = all_non_nullable ? 0 : (num_nullable_fields + 7) / 8
   // May be rounded to 8-byte boundary if has variable fields
 ```

 ### Standard Array Size

 ```
 array_size = header_size + data_size

 where:
   header_size = 8 + ((num_elements + 63) / 64) * 8
   data_size = ((num_elements * element_size + 7) / 8) * 8
 ```

 ### Compact Array Size

 ```
 array_size = header_size + data_size

 where:
   header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)
   // header_size rounded to 8 if element_width % 8 == 0
   data_size = num_elements * element_width
 ```

 ### Map Size

 ```
 map_size = 8 + keys_array_size + values_array_size
 ```

 ## Summary Tables

 ### Layout Summary

 | Component        | Standard Format                 | Compact Format                        |
 | ---------------- | ------------------------------- | ------------------------------------- |
 | Row Header       | `((N + 63) / 64) * 8` bytes     | 0 or `(N + 7) / 8` bytes (at end)     |
 | Row Field Slots  | `N * 8` bytes                   | `sum(field_widths)` bytes             |
 | Array Header     | `8 + ((E + 63) / 64) * 8` bytes | `4 + (E + 7) / 8` bytes (if nullable) |
 | Array Elements   | `E * element_size` (8-aligned)  | `E * element_width`                   |
 | Map Header       | 8 bytes                         | 8 bytes                               |
 | Offset+Size Pair | 8 bytes (32-bit offset + size)  | 8 bytes (same)                        |

 Where N = number of fields, E = number of elements

 ### Type Width Summary

 | Category      | Storage Width | Standard Slot | Compact Slot |
 | ------------- | ------------- | ------------- | ------------ |
 | bool          | 1 byte        | 8 bytes       | 1 byte       |
 | int8          | 1 byte        | 8 bytes       | 1 byte       |
 | int16         | 2 bytes       | 8 bytes       | 2 bytes      |
 | int32         | 4 bytes       | 8 bytes       | 4 bytes      |
 | int64         | 8 bytes       | 8 bytes       | 8 bytes      |
 | float32       | 4 bytes       | 8 bytes       | 4 bytes      |
 | float64       | 8 bytes       | 8 bytes       | 8 bytes      |
 | string/binary | Variable      | 8 bytes       | 8 bytes      |
 | array         | Variable      | 8 bytes       | 8 bytes      |
 | map           | Variable      | 8 bytes       | 8 bytes      |
 | struct        | Variable      | 8 bytes       | inline or 8  |

 ## Implementation Notes

 ### Endianness

 - All multi-byte integers are stored in **little-endian** format
 - Floating-point values use native IEEE 754 representation

 ### Memory Safety

 - Writers must zero padding bytes to prevent information leakage
 - Readers must validate offsets and sizes before accessing data
 - Buffer bounds checking is required for untrusted input

 ### Performance Considerations

 **Standard Format**:

 - Fixed 8-byte slots enable O(1) field access with simple arithmetic
 - 8-byte alignment optimizes CPU cache line usage
 - Best for cross-language interoperability

 **Compact Format**:

 - Smaller row sizes reduce memory bandwidth
 - Field sorting minimizes padding waste
 - Inline structs eliminate pointer chasing
 - Relaxed alignment may have slight CPU overhead on some architectures

 ### When to Use Each Format

 | Scenario                         | Recommended Format |
 | -------------------------------- | ------------------ |
 | Cross-language data exchange     | Standard           |
 | Java-only, memory-constrained    | Compact            |
 | Many small primitive fields      | Compact            |
 | Many nested fixed-size structs   | Compact            |
 | Maximum read performance         | Standard           |
 | Interoperability with C++/Python | Standard           |
	---
	title: Row Format
	sidebar_position: 2
	id: row_format_spec
	license: \|
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to You under the Apache License, Version 2.0
	(the "License"); you may not use this file except in compliance with
	the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	---

	## Overview

	Apache Fory Row Format is a cache-friendly, random-access binary format designed for high-performance data processing. Unlike traditional serialization formats that require full deserialization, the row format enables:

	- Random Field Access: Read individual fields without deserializing the entire row
	- Zero-Copy Operations: Direct memory access without data transformation
	- Cache-Friendly Layout: Optimized memory layout for CPU cache efficiency
	- Cross-Language Support: Consistent binary format across Java, C++, and Python

	Fory provides two row format variants:

	\| Format \| Languages \| Use Case \|
	\| --------------- \| ----------------- \| ------------------------------ \|
	\| Standard Format \| Java, C++, Python \| Cross-language compatibility \|
	\| Compact Format \| Java only \| Space efficiency, smaller rows \|

	## Format Comparison

	\| Feature \| Standard Format \| Compact Format \|
	\| -------------------- \| ----------------------------- \| ----------------------------------- \|
	\| Field Slot Size \| Fixed 8 bytes \| Natural width (1, 2, 4, or 8 bytes) \|
	\| Null Bitmap Size \| 8-byte aligned \| Byte-aligned, can borrow padding \|
	\| Null Bitmap Position \| Before field slots \| After field slots (at end) \|
	\| Fixed-Size Structs \| Variable region (offset+size) \| Inline in fixed region \|
	\| Field Ordering \| Schema-defined order \| Sorted by alignment \|
	\| All Non-Nullable \| Bitmap still present \| Bitmap skipped entirely \|
	\| Alignment \| Strict 8-byte \| Relaxed (2, 4, or 8-byte) \|

	---

	## Standard Row Format

	The standard format prioritizes cross-language compatibility and simplicity with uniform 8-byte field slots.

	### Design Principles

	1. 8-Byte Alignment: All major structures are aligned to 8-byte boundaries for optimal memory access
	2. Fixed-Width Field Slots: Every field uses an 8-byte slot for uniform offset calculation
	3. Null Bitmap: Compact null tracking using bit vectors
	4. Relative Offsets: Variable-length data uses relative offsets for sub-buffer navigation

	### Row Binary Layout

	A row stores structured data with the following layout:

	```
	+----------------+------------------+------------------+-----+------------------+------------------+
	\| Null Bitmap \| Field 0 Slot \| Field 1 Slot \| ... \| Field N-1 Slot \| Variable Data \|
	+----------------+------------------+------------------+-----+------------------+------------------+
	\| B bytes \| 8 bytes \| 8 bytes \| \| 8 bytes \| Variable size \|
	```

	#### Null Bitmap

	The null bitmap tracks which fields contain null values:

	- Size: `((num_fields + 63) / 64) * 8` bytes (rounded up to nearest 8-byte word)
	- Encoding: Each bit corresponds to a field index
	- Bit value `1` = field is null
	- Bit value `0` = field is not null
	- Bit Order: Bit 0 of the first byte corresponds to field 0

	Example: For 10 fields, bitmap size = `((10 + 63) / 64) * 8 = 8` bytes

	#### Field Slots

	Each field occupies a fixed 8-byte slot regardless of its actual data type:

	- Slot Offset: `bitmap_size + field_index * 8`
	- Total Fixed Region: `bitmap_size + num_fields * 8` bytes

	Field Slot Contents by Type:

	\| Type Category \| Slot Contents \|
	\| -------------- \| ----------------------------------- \|
	\| Fixed-width \| Value stored directly (zero-padded) \|
	\| Variable-width \| Offset + Size encoding (see below) \|

	#### Variable-Width Data Encoding

	Variable-length fields (strings, arrays, maps, nested structs) store an offset-size pair in their slot:

	```
	+---------------------------+---------------------------+
	\| Relative Offset \| Size \|
	\| (32 bits) \| (32 bits) \|
	+---------------------------+---------------------------+
	\|<-------------- 64-bit field slot value -------------->\|
	```

	- Relative Offset (upper 32 bits): Offset from the row's base address
	- Size (lower 32 bits): Size of the variable-width data in bytes

	Encoding:

	```
	offset_and_size = (relative_offset << 32) \| size
	```

	Decoding:

	```
	relative_offset = (offset_and_size >> 32) & 0xFFFFFFFF
	size = offset_and_size & 0xFFFFFFFF
	```

	#### Variable Data Region

	Variable-length data is stored after the fixed region:

	- Data is written sequentially as fields are set
	- Each variable-length value is padded to 8-byte alignment
	- Padding bytes are zeroed for deterministic output

	### Array Binary Layout

	Arrays store homogeneous sequences of elements:

	```
	+------------------+------------------+------------------+
	\| Element Count \| Null Bitmap \| Element Data \|
	+------------------+------------------+------------------+
	\| 8 bytes \| B bytes \| Variable size \|
	```

	#### Array Header

	\| Field \| Size \| Description \|
	\| ------------- \| ------------------------------- \| --------------------------- \|
	\| Element Count \| 8 bytes \| Number of elements (uint64) \|
	\| Null Bitmap \| `((count + 63) / 64) * 8` bytes \| Per-element null flags \|

	Header Size: `8 + ((num_elements + 63) / 64) * 8` bytes

	#### Array Element Data

	Elements are stored contiguously after the header:

	- Fixed-width elements: Stored with their natural width (1, 2, 4, or 8 bytes)
	- Variable-width elements: Stored as 8-byte offset+size pairs

	Element Offset: `header_size + element_index * element_size`

	Data Region Size: Rounded up to nearest 8-byte boundary

	#### Array Element Sizes

	\| Element Type \| Element Size \|
	\| ---------------- \| --------------------- \|
	\| bool \| 1 byte \|
	\| int8 \| 1 byte \|
	\| int16 \| 2 bytes \|
	\| int32 \| 4 bytes \|
	\| int64 \| 8 bytes \|
	\| float32 \| 4 bytes \|
	\| float64 \| 8 bytes \|
	\| string/binary \| 8 bytes (offset+size) \|
	\| array/map/struct \| 8 bytes (offset+size) \|

	### Map Binary Layout

	Maps store key-value pairs as two separate arrays:

	```
	+------------------+------------------+------------------+
	\| Keys Array Size \| Keys Array \| Values Array \|
	+------------------+------------------+------------------+
	\| 8 bytes \| Variable size \| Variable size \|
	```

	#### Map Structure

	\| Field \| Size \| Description \|
	\| --------------- \| -------- \| --------------------------------- \|
	\| Keys Array Size \| 8 bytes \| Total size of keys array in bytes \|
	\| Keys Array \| Variable \| Full array structure for keys \|
	\| Values Array \| Variable \| Full array structure for values \|

	Keys Array Offset: `base_offset + 8`
	Values Array Offset: `base_offset + 8 + keys_array_size`

	Both keys and values arrays follow the standard array binary layout.

	### Nested Struct Layout

	Nested structs are stored as complete row structures within the variable data region:

	1. Parent field slot contains offset+size pointing to nested row
	2. Nested row has its own null bitmap and field slots
	3. Supports arbitrary nesting depth

	```
	Parent Row:
	+----------------+------------------+------------------+
	\| Null Bitmap \| ... Slots ... \| Nested Row Data \|
	+----------------+------------------+------------------+
	\| ^
	\| offset+size \|
	+------------------->+

	Nested Row:
	+----------------+------------------+------------------+
	\| Null Bitmap \| Field Slots \| Variable Data \|
	+----------------+------------------+------------------+
	```

	---

	## Compact Row Format (Java Only)

	The compact format provides space-efficient encoding with additional optimizations. It is currently implemented in Java only.

	> Note: The compact format is still under development and may not be stable yet.

	### Design Principles

	1. Natural Width Storage: Fixed-size fields use their natural byte width instead of 8 bytes
	2. Alignment-Based Field Sorting: Fields are sorted by alignment requirements to minimize padding
	3. Conditional Null Bitmap: Null bitmap is omitted when all fields are non-nullable
	4. Inline Fixed-Size Structs: Nested structs with all fixed-size fields are stored inline

	### Compact Row Binary Layout

	```
	+------------------+------------------+-----+------------------+----------------+------------------+
	\| Field 0 Value \| Field 1 Value \| ... \| Field N-1 Value \| Null Bitmap \| Variable Data \|
	+------------------+------------------+-----+------------------+----------------+------------------+
	\| W0 bytes \| W1 bytes \| \| WN-1 bytes \| B bytes (opt) \| Variable size \|
	```

	#### Key Differences from Standard Format

	1. Field Slot Sizes: Each field uses its natural width (Wi = type width or 8 for variable)
	2. Null Bitmap Position: Placed after field slots, can borrow alignment padding
	3. Field Order: Fields are sorted by alignment (8-byte → 4-byte → 2-byte → 1-byte → variable)
	4. Conditional Bitmap: Skipped entirely if all fields are non-nullable

	#### Null Bitmap (Compact)

	- Size: `(num_nullable_fields + 7) / 8` bytes (byte-aligned, not 8-byte aligned)
	- Skipped: When all fields are primitive/non-nullable
	- Position: After all fixed-size field slots, can use alignment padding space

	#### Field Sorting Algorithm

	Fields are sorted to minimize padding and optimize alignment:

	```
	Priority order (highest to lowest):
	1. Fields with 8-byte alignment (int64, float64, variable-width)
	2. Fields with 4-byte alignment (int32, float32)
	3. Fields with 2-byte alignment (int16)
	4. Fields with 1-byte alignment (int8, bool)
	```

	Within each alignment group, larger fields come first.

	#### Fixed-Size Struct Inlining

	Nested structs with all fixed-size fields are stored inline in the parent row:

	Standard Format (nested struct with 2 int32 fields):

	```
	Parent slot: [offset (4 bytes) \| size (4 bytes)] → Points to nested row (8+ bytes elsewhere)
	```

	Compact Format (same nested struct):

	```
	Parent slot: [int32 field 0 \| int32 field 1] → 8 bytes total, inline
	```

	This eliminates the offset+size indirection for fixed-size nested structures.

	#### Fixed-Width Calculation

	A field's fixed width is determined recursively:

	- Primitive types: Natural byte width (1, 2, 4, or 8)
	- Struct types: Sum of all child fixed widths (if all children are fixed-width)
	- Variable types (string, array, map): Returns -1 (uses 8-byte offset+size slot)

	```
	fixed_width(field) =
	if primitive: type_width
	if struct and all_children_fixed: header_bytes + sum(fixed_width(child) for each child)
	else: -1 (variable, uses 8-byte slot)
	```

	### Compact Array Binary Layout

	```
	+------------------+------------------+------------------+
	\| Element Count \| Null Bitmap \| Element Data \|
	+------------------+------------------+------------------+
	\| 4 bytes \| B bytes (opt) \| Variable size \|
	```

	#### Compact Array Header

	\| Field \| Size \| Description \|
	\| ------------- \| ----------------------------- \| -------------------------- \|
	\| Element Count \| 4 bytes \| Number of elements (int32) \|
	\| Null Bitmap \| `(count + 7) / 8` bytes (opt) \| Per-element null flags \|

	Header Size Calculation:

	```
	header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)

	// Round to 8-byte boundary only if element width is 8-byte aligned
	if (fixed_width % 8 == 0):
	header_size = round_to_8_bytes(header_size)
	```

	#### Key Differences from Standard Array

	1. Element Count: 4 bytes instead of 8 bytes
	2. Null Bitmap: Byte-aligned, skipped if elements are non-nullable
	3. Fixed-Size Structs: Inline storage for fixed-width struct elements

	---

	## Common Specifications

	The following specifications apply to both standard and compact formats.

	### Type Encoding

	#### Primitive Types

	\| Type \| Width \| Encoding \|
	\| ------- \| ------- \| ------------------------------- \|
	\| bool \| 1 byte \| `0x00` (false) or `0x01` (true) \|
	\| int8 \| 1 byte \| Two's complement \|
	\| int16 \| 2 bytes \| Two's complement, little-endian \|
	\| int32 \| 4 bytes \| Two's complement, little-endian \|
	\| int64 \| 8 bytes \| Two's complement, little-endian \|
	\| float32 \| 4 bytes \| IEEE 754 single precision \|
	\| float64 \| 8 bytes \| IEEE 754 double precision \|

	#### Temporal Types

	\| Type \| Width \| Encoding \|
	\| --------- \| ------- \| ------------------------------------- \|
	\| timestamp \| 8 bytes \| Microseconds since Unix epoch (int64) \|
	\| date32 \| 4 bytes \| Days since Unix epoch (int32) \|
	\| duration \| 8 bytes \| Duration in microseconds (int64) \|

	#### String and Binary

	- Encoding: UTF-8 for strings, raw bytes for binary
	- Storage: Offset+size pair in field slot, data in variable region
	- Padding: Data padded to 8-byte alignment (standard) or natural alignment (compact)

	### Null Handling

	#### Row Null Handling

	- Null fields have their corresponding bit set to 1 in the null bitmap
	- Field slot contents are undefined for null fields (standard) or zeroed (compact)
	- Reading a null field returns a null/empty value indicator

	#### Array Null Handling

	- Null elements have their corresponding bit set to 1 in the array's null bitmap
	- Element data is undefined for null elements
	- Compact format: Bitmap skipped if elements are non-nullable

	#### Variable-Width Null Semantics

	When reading variable-width data from a null field:

	- Returns size of -1 or equivalent null indicator
	- No data access is performed

	### Alignment and Padding

	#### Standard Format Alignment

	1. Null Bitmap: Size rounded to 8-byte boundary
	2. Field Slots: Always 8 bytes each
	3. Variable Data: Each value padded to 8-byte boundary
	4. Array Data: Total data region padded to 8-byte boundary

	#### Compact Format Alignment

	1. Field Slots: Natural width (1, 2, 4, or 8 bytes)
	2. Null Bitmap: Byte-aligned, placed after fields
	3. Variable Data: Padded to 8-byte boundary only when needed
	4. Header: May use relaxed alignment for smaller overhead

	#### Padding Bytes

	- All padding bytes must be set to zero
	- Ensures deterministic serialization output
	- Prevents information leakage from uninitialized memory

	## Size Calculations

	### Standard Row Size

	```
	row_size = bitmap_size + num_fields * 8 + variable_data_size

	where:
	bitmap_size = ((num_fields + 63) / 64) * 8
	variable_data_size = sum of (padded_size for each variable field)
	padded_size = ((size + 7) / 8) * 8
	```

	### Compact Row Size

	```
	row_size = fixed_region_size + bitmap_size + variable_data_size

	where:
	fixed_region_size = sum of (fixed_width(field) or 8 for each field)
	bitmap_size = all_non_nullable ? 0 : (num_nullable_fields + 7) / 8
	// May be rounded to 8-byte boundary if has variable fields
	```

	### Standard Array Size

	```
	array_size = header_size + data_size

	where:
	header_size = 8 + ((num_elements + 63) / 64) * 8
	data_size = ((num_elements * element_size + 7) / 8) * 8
	```

	### Compact Array Size

	```
	array_size = header_size + data_size

	where:
	header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)
	// header_size rounded to 8 if element_width % 8 == 0
	data_size = num_elements * element_width
	```

	### Map Size

	```
	map_size = 8 + keys_array_size + values_array_size
	```

	## Summary Tables

	### Layout Summary

	\| Component \| Standard Format \| Compact Format \|
	\| ---------------- \| ------------------------------- \| ------------------------------------- \|
	\| Row Header \| `((N + 63) / 64) * 8` bytes \| 0 or `(N + 7) / 8` bytes (at end) \|
	\| Row Field Slots \| `N * 8` bytes \| `sum(field_widths)` bytes \|
	\| Array Header \| `8 + ((E + 63) / 64) * 8` bytes \| `4 + (E + 7) / 8` bytes (if nullable) \|
	\| Array Elements \| `E * element_size` (8-aligned) \| `E * element_width` \|
	\| Map Header \| 8 bytes \| 8 bytes \|
	\| Offset+Size Pair \| 8 bytes (32-bit offset + size) \| 8 bytes (same) \|

	Where N = number of fields, E = number of elements

	### Type Width Summary

	\| Category \| Storage Width \| Standard Slot \| Compact Slot \|
	\| ------------- \| ------------- \| ------------- \| ------------ \|
	\| bool \| 1 byte \| 8 bytes \| 1 byte \|
	\| int8 \| 1 byte \| 8 bytes \| 1 byte \|
	\| int16 \| 2 bytes \| 8 bytes \| 2 bytes \|
	\| int32 \| 4 bytes \| 8 bytes \| 4 bytes \|
	\| int64 \| 8 bytes \| 8 bytes \| 8 bytes \|
	\| float32 \| 4 bytes \| 8 bytes \| 4 bytes \|
	\| float64 \| 8 bytes \| 8 bytes \| 8 bytes \|
	\| string/binary \| Variable \| 8 bytes \| 8 bytes \|
	\| array \| Variable \| 8 bytes \| 8 bytes \|
	\| map \| Variable \| 8 bytes \| 8 bytes \|
	\| struct \| Variable \| 8 bytes \| inline or 8 \|

	## Implementation Notes

	### Endianness

	- All multi-byte integers are stored in little-endian format
	- Floating-point values use native IEEE 754 representation

	### Memory Safety

	- Writers must zero padding bytes to prevent information leakage
	- Readers must validate offsets and sizes before accessing data
	- Buffer bounds checking is required for untrusted input

	### Performance Considerations

	Standard Format:

	- Fixed 8-byte slots enable O(1) field access with simple arithmetic
	- 8-byte alignment optimizes CPU cache line usage
	- Best for cross-language interoperability

	Compact Format:

	- Smaller row sizes reduce memory bandwidth
	- Field sorting minimizes padding waste
	- Inline structs eliminate pointer chasing
	- Relaxed alignment may have slight CPU overhead on some architectures

	### When to Use Each Format

	\| Scenario \| Recommended Format \|
	\| -------------------------------- \| ------------------ \|
	\| Cross-language data exchange \| Standard \|
	\| Java-only, memory-constrained \| Compact \|
	\| Many small primitive fields \| Compact \|
	\| Many nested fixed-size structs \| Compact \|
	\| Maximum read performance \| Standard \|
	\| Interoperability with C++/Python \| Standard \|