Vector Index Storage Format

This document describes the v1 on-disk formats written by the Rust core library. Version 1 is the first release format. Pre-release layouts are not part of the compatibility contract.

Compatibility Policy

• All multi-byte integers and f32 values are little-endian.
• The unified reader dispatches by the first 4-byte magic value.
• Magic names below show the u32 constants in human-readable big-endian form. Because the fields are little-endian, the raw file bytes for those constants appear in reverse ASCII order.
• Readers reject unknown magic values, unknown versions, unknown required flags, non-zero reserved bytes, invalid section sizes, negative counts, and malformed list payload metadata.
• Incompatible on-disk changes require a new format version. Version 1 readers do not attempt to read future versions.
• Reserved bytes are written as zero and must be read back as zero. Future extensions must use flags or a new format version rather than repurposing non-zero reserved bytes within v1.
• Index files have no outer container, footer, checksum, compression envelope, or schema registry. The complete file starts at byte offset 0 with one of the headers below.
• File integrity, including length and checksum validation, is guaranteed by the outer Paimon file/manifest layer rather than by an embedded index footer.
• Roaring row-id filters are a query-time API payload. They are not embedded in any index file format.

Common Encodings

Delta-Varint IDs

IVF-PQ, IVF-FLAT, and IVF-RQ v1 sort each non-empty list by signed row id before writing. The first id is stored as base_id: i64. The id stream then stores one unsigned LEB128 varint per id, including the first id's zero delta. Each delta is computed with wrapping unsigned subtraction from the previous signed id. Readers reject a decoded sequence that is not monotonically non-decreasing in signed order.

HNSW Graph Section

IVF-HNSW-FLAT and IVF-HNSW-SQ store one graph section per non-empty list. The section starts with a fixed header, followed by a contiguous sequence of unsigned LEB128 varints. Neighbor ids within each adjacency group are sorted by local vector id and stored as unsigned deltas from the previous neighbor id, with an initial previous id of 0:

Field	Count
graph_magic	1 little-endian u32, HWGR (0x48574752)
graph_version	1 little-endian u32, currently 1
graph_flags	1 little-endian u32; bit 0 delta-varint adjacency is required
graph_count	1 varint
entry_point	1 varint
max_observed_level	1 varint
level[node]	graph_count varints
degree[node][level] followed by neighbor id deltas	one group for each node level

Each node has levels 0..=level[node]. A level-0 node may have at most 2 * m neighbors, and higher levels may have at most m neighbors.

IVF-PQ v1

Magic: IVPQ (0x49565051). Version: 1. Header size: 64 bytes.

Offset	Size	Type	Field
0	4	u32	magic
4	4	u32	version
8	4	i32	dimension d
12	4	i32	IVF list count nlist
16	4	i32	PQ subquantizer count m
20	4	i32	centroid count per subquantizer ksub
24	4	i32	subvector dimension dsub
28	4	u32	metric (0=L2, 1=InnerProduct, 2=Cosine)
32	8	i64	total vector count
40	4	u32	flags
44	20	bytes	reserved

Flags:

Bit	Meaning
0	OPQ rotation matrix is present
1	PQ codes are trained/stored by residual
2	delta-varint ids are used; required in v1
3	PQ codes are transposed by subquantizer; required in v1

Sections after the header:

1. Optional OPQ rotation matrix: d * d f32 values when flag bit 0 is set.
2. IVF coarse centroids: nlist * d f32 values.
3. PQ centroids: m * ksub * dsub f32 values.
4. Offset table: nlist entries of (offset: i64, count: i32, id_bytes_len: i32).
5. List payloads.

For each non-empty list payload:

Field	Type	Notes
base_id	i64	first sorted row id
id_bytes_len	i32	byte length of encoded id stream
id_bytes	bytes	delta-varint ids
codes	bytes	transposed PQ codes

For 8-bit PQ, each vector has m code bytes and the stored code layout is codes[sub][vector]. For 4-bit PQ, each byte stores two subquantizers and the stored layout is codes[pair][vector].

IVF-FLAT v1

Magic: IVFL (0x4956464C). Version: 1. Header size: 64 bytes.

Offset	Size	Type	Field
0	4	u32	magic
4	4	u32	version
8	4	i32	dimension d
12	4	i32	IVF list count nlist
16	4	u32	metric (0=L2, 1=InnerProduct, 2=Cosine)
20	8	i64	total vector count
28	4	u32	flags
32	32	bytes	reserved

Flags:

Bit	Meaning
0	delta-varint ids are used; required in v1

Sections after the header:

1. IVF coarse centroids: nlist * d f32 values.
2. Offset table: nlist entries of (offset: i64, count: i32, id_bytes_len: i32).
3. List payloads.

For each non-empty list payload:

Field	Type	Notes
base_id	i64	first sorted row id
id_bytes_len	i32	byte length of encoded id stream
id_bytes	bytes	delta-varint ids
vectors	count * d f32	raw stored vectors

IVF-RQ v1

Magic: IVRQ (0x49565251). Version: 1. Header size: 64 bytes.

Offset	Size	Type	Field
0	4	u32	magic
4	4	u32	version
8	4	i32	dimension d
12	4	i32	IVF list count nlist
16	4	u32	metric (0=L2, 1=InnerProduct, 2=Cosine)
20	4	u32	flags
24	8	i64	total vector count
32	8	u64	deterministic rotation seed
40	4	u32	deterministic rotation rounds
44	4	i32	bytes per primary sign-code plane, ceil(d / 8)
48	4	u32	RQ num_bits; currently 1, reserved values 2, 4, and 8
52	4	u32	rotation_type; currently 1 for deterministic Kac rotation
56	4	u32	factor_layout; currently 1 for three RaBitQ f32 factors
60	4	u32	RQ format flags

Flags:

Bit	Meaning
0	delta-varint ids are used; required in v1

RQ format flags:

Bit	Meaning
0	optional ex_codes section is present
1	optional error_factor section is present

Sections after the header:

1. IVF coarse centroids: nlist * d f32 values.
2. Offset table: nlist entries of (offset: i64, count: i32, id_bytes_len: i32).
3. List payloads.

For each non-empty list payload:

Field	Type	Notes
base_id	i64	first sorted row id
id_bytes_len	i32	byte length of encoded id stream
id_bytes	bytes	delta-varint ids
codes	count * ceil(d / 8) bytes	primary RaBitQ sign-code plane over rotated residuals
ex_codes	optional bytes	reserved for additional code planes when num_bits > 1; present when RQ format flag bit 0 is set
factors	count * 3 f32	per-vector (residual_norm_sqr, vector_norm_sqr, dp_multiplier) correction factors
error_factor	optional count f32	reserved per-vector refinement factor; present when RQ format flag bit 1 is set

The deterministic rotation is derived from (d, rotation_seed, rotation_rounds) and applied to both indexed residuals and query residuals. The rotation parameters are stored in the header so independently written files can use the same distance estimator after being read by any v1 reader.

Current writers emit num_bits=1, rotation_type=1, factor_layout=1, and no optional RQ sections. Current readers validate these format fields and reject reserved multi-bit or optional-section encodings until the corresponding scanner is implemented.

query_bits is a search-time IVF-RQ parameter and is not serialized. 0 uses the default float-query byte-LUT estimator; 4 and 8 quantize the rotated query residual into sign and magnitude bit planes for bitwise/popcount scanning.

IVF-HNSW-FLAT v1

Magic: IHFL (0x4948464C). Version: 1. Header size: 64 bytes.

Offset	Size	Type	Field
0	4	u32	magic
4	4	u32	version
8	4	i32	dimension d
12	4	i32	IVF list count nlist
16	4	u32	metric (0=L2, 1=InnerProduct, 2=Cosine)
20	8	i64	total vector count
28	4	i32	HNSW m
32	4	i32	HNSW ef_construction
36	4	i32	HNSW max_level
40	4	u32	flags
44	20	bytes	reserved

Flags:

Bit	Meaning
0	sorted delta-varint ids are stored; required in v1
1	HNSW graph section uses the v1 delta-varint graph encoding; required in v1

Sections after the header:

1. IVF coarse centroids: nlist * d f32 values.
2. Offset table: nlist entries of (offset: i64, count: i32, graph_bytes_len: i32, payload_bytes_len: i64).
3. List payloads.

For each non-empty list payload:

Field	Type	Notes
base_id	i64	first sorted row id
id_bytes_len	i32	byte length of encoded id stream
id_bytes	bytes	delta-varint ids
vectors	count * d f32	raw stored vectors
graph	bytes	HNSW graph section

IVF-HNSW-SQ v1

Magic: IHSQ (0x49485351). Version: 1. Header size: 64 bytes.

Offset	Size	Type	Field
0	4	u32	magic
4	4	u32	version
8	4	i32	dimension d
12	4	i32	IVF list count nlist
16	4	u32	metric (0=L2, 1=InnerProduct, 2=Cosine)
20	8	i64	total vector count
28	4	i32	HNSW m
32	4	i32	HNSW ef_construction
36	4	i32	HNSW max_level
40	4	f32	global minimum SQ bound summary
44	4	f32	global maximum SQ bound summary
48	4	u32	flags
52	12	bytes	reserved

Flags:

Bit	Meaning
0	sorted delta-varint ids are stored; required in v1
1	HNSW graph section uses the v1 delta-varint graph encoding; required in v1

Sections after the header:

1. Global SQ min bounds: d f32 values.
2. Global SQ max bounds: d f32 values.
3. Per-list SQ bounds: for each list, d min f32 values followed by d max f32 values.
4. IVF coarse centroids: nlist * d f32 values.
5. Offset table: nlist entries of (offset: i64, count: i32, graph_bytes_len: i32, payload_bytes_len: i64).
6. List payloads.

For each non-empty list payload:

Field	Type	Notes
base_id	i64	first sorted row id
id_bytes_len	i32	byte length of encoded id stream
id_bytes	bytes	delta-varint ids
codes	bytes	scalar quantized residual codes, count * d bytes
graph	bytes	HNSW graph section over decoded vectors