docs/ivf-flat.html - paimon-vector-index - Git at Google

 <!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements. See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to you under the Apache License, Version 2.0.
 -->
 <!doctype html>
 <html lang="en">
 <head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><meta name="description" content="IVF-FLAT usage, design, search pipeline, tuning, capacity, and v1 storage layout."><title>IVF-FLAT · Paimon Vector Index</title><link rel="stylesheet" href="styles.css"><script src="docs.js" defer></script></head>
 <body>
   <a class="skip-link" href="#main">Skip to content</a>
   <header class="site-header"><div class="header-inner"><a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"><span class="brand-mark">VI</span><span>Paimon Vector Index</span></a><nav class="site-nav" data-site-nav aria-label="Documentation"><a href="index.html">Overview</a><a href="api.html">API</a><a href="development.html">Development</a><a href="ivf-flat.html" aria-current="page">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a href="ivf-sq.html">IVF-SQ</a><a href="diskann.html">DiskANN</a><a href="releases.html">Releases</a></nav><div class="header-actions"><button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button><button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button></div></div></header>
   <main id="main"><div class="page-shell">
     <section class="hero detail-hero"><p class="eyebrow">Exact scan inside selected lists</p><h1>IVF-FLAT</h1><p class="hero-lead">Use IVF coarse partitions to reduce the candidate set, then retain and scan full <code>f32</code> vectors inside each selected list. IVF-FLAT is the clearest baseline for separating partition recall loss from quantization loss.</p><div class="badge-row"><span class="badge strong">No in-list quantization</span><span class="badge">About 4d bytes / vector</span><span class="badge">Simple build</span><span class="badge">Magic: IVFL</span></div></section>
     <div class="doc-layout">
       <aside class="toc" aria-label="On this page"><strong>On this page</strong><a href="#position">Positioning</a><a href="#design">Design</a><a href="#benchmarks">Public benchmarks</a><a href="#usage">Usage</a><a href="#parameters">Parameters</a><a href="#storage">Storage layout</a><a href="#sizing">Capacity</a><a href="#tuning">Tuning</a><a href="#limits">Boundaries</a></aside>
       <article class="article">
         <section class="article-section" id="position">
           <h2>Positioning and trade-offs</h2>
           <div class="metric-strip"><div class="metric"><span class="label">In-list accuracy</span><span class="value">Exact raw-vector distance</span></div><div class="metric"><span class="label">Main storage</span><span class="value"><code>4 × d × N</code></span></div><div class="metric"><span class="label">Training</span><span class="value">IVF K-Means only</span></div><div class="metric"><span class="label">Query work</span><span class="value">Scan every vector in probed lists</span></div></div>
           <div class="split"><div class="pro-con"><h3>Good fit</h3><ul><li>Establishing recall and latency baselines.</li><li>Moderate vector counts or dimensions.</li><li>Accuracy matters more than index size.</li><li>A clear, low-risk implementation is preferred.</li></ul></div><div class="pro-con"><h3>Poor fit</h3><ul><li>High-dimensional collections with strict space budgets.</li><li>Large lists where scans dominate tail latency.</li><li>Very small remote-read budgets.</li></ul></div></div>
           <div class="callout"><strong>“Exact” has a boundary</strong>IVF-FLAT computes exact distances only inside the <code>nprobe</code> selected lists. Whenever <code>nprobe &lt; nlist</code>, the global search remains approximate.</div>
         </section>

         <section class="article-section" id="design">
           <h2>Design and data flow</h2>
           <h3>Build</h3>
           <div class="pipeline"><div class="pipeline-item"><span class="pipeline-index">1</span><div><h3>Preprocess training vectors</h3><p>Cosine mode applies L2 normalization. L2 and inner product keep the input representation.</p></div></div><div class="pipeline-item"><span class="pipeline-index">2</span><div><h3>Train coarse centroids</h3><p>K-Means produces <code>nlist × d</code> <code>f32</code> centroid values.</p></div></div><div class="pipeline-item"><span class="pipeline-index">3</span><div><h3>Assign vectors</h3><p>Each vector enters its nearest coarse list with its complete processed vector and row ID.</p></div></div><div class="pipeline-item"><span class="pipeline-index">4</span><div><h3>Sort and serialize</h3><p>The writer retains compact sort permutations and delta-varint IDs, then materializes and writes one sorted raw-vector list at a time instead of duplicating every list in memory.</p></div></div></div>
           <h3>Search</h3>
           <ol><li>Apply the same preprocessing used during construction.</li><li>Measure the query against every IVF centroid and select the closest <code>nprobe</code> lists.</li><li>Read those list payloads through the offset table in bounded concurrent multi-range calls.</li><li>Decode row IDs and compute the true metric against every raw vector. A Roaring filter skips disallowed IDs.</li><li>For at least 1,048,576 distance components, scan independent lists on Rayon workers. Batch search keeps list-major locality and scans every loaded list for all queries that selected it.</li><li>Merge list-local results in the original list order. Missing entries are padded with <code>-1 / f32::MAX</code>.</li></ol>
         </section>

         <section class="article-section" id="benchmarks">
           <h2>Public-corpus measurements</h2>
           <p>Apple M4 Pro, 12 Rayon workers, one million SIFT/GIST vectors or 1,183,514 GloVe vectors, 1,000 published queries, <code>nlist=1024</code>, <code>nprobe=64</code>, Top-10, and warm APFS pages. Times are release-build measurements from 25 July 2026.</p>
           <div class="table-wrap"><table><thead><tr><th>Dataset</th><th>Recall@10</th><th>Local P95</th><th>Local batch QPS</th><th>Read / query</th></tr></thead><tbody><tr><td>SIFT1M, 128d</td><td>0.9937</td><td>1.88 ms</td><td>8,510</td><td>33.19 MiB</td></tr><tr><td>GIST1M, 960d</td><td>0.9549</td><td>11.38 ms</td><td>875</td><td>283.40 MiB</td></tr><tr><td>GloVe-100, 100d</td><td>0.8832</td><td>1.40 ms</td><td>9,502</td><td>27.57 MiB</td></tr></tbody></table></div>
           <div class="callout"><strong>Latest complete same-file rerun</strong>The Reader receives each list directly into an <code>f32</code>-aligned allocation and scans the raw-vector suffix without allocating a second decoded payload. An internal prefix of at most three bytes compensates for the variable-length row-ID prefix; the persisted bytes do not change. Together with the strict partial-L2 cutoff, this changed SIFT/GIST/GloVe batch throughput from 6,570 / 559 / 6,345 to 8,510 / 875 / 9,502 QPS and P95 from 5.31 / 47.04 / 4.76 ms to 1.88 / 11.38 / 1.40 ms. Recall, file bytes, and read bytes stayed identical. The table reports the complete rerun rather than a best-of result.</div>
           <p>The remote model groups every query's selected ranges into calls capped at 64 MiB. SIFT/GloVe average one round; GIST averages five because its raw-vector payload is much larger. Fixed 2/20 ms latency therefore produced P95 7.31/21.80 ms on SIFT, 29.73/130.49 ms on GIST, and 6.60/21.22 ms on GloVe. Batch throughput was 6,763/3,076 QPS on SIFT, 846/349 on GIST, and 7,394/2,948 on GloVe. These modeled numbers do not charge bandwidth, so they should not be read as evidence that transferring tens or hundreds of MiB directly from an object store is cheap.</p>
         </section>

         <section class="article-section" id="usage">
           <h2>Usage</h2>
           <p>The trained state is one-shot: finish training, pass the state to a Writer, add production vectors in batches, and serialize one index file. Readers discover the type from the header.</p>
           <div class="code-block"><span class="code-label">Java · build</span><pre><code>Map&lt;String, String&gt; options = new HashMap&lt;&gt;();
 options.put("index.type", "ivf_flat");
 options.put("dimension", "128");
 options.put("nlist", "1024");
 options.put("metric", "l2");

 try (VectorIndexTraining training =
              VectorIndexTrainer.train(options, trainingVectors, trainingCount);
      VectorIndexWriter writer = new VectorIndexWriter(training)) {
     writer.addVectors(rowIds, vectors, vectorCount);
     writer.writeIndex(vectorIndexOutput);
 }</code></pre></div>
           <div class="code-block"><span class="code-label">Java · search</span><pre><code>try (VectorIndexReader reader = new VectorIndexReader(vectorIndexInput)) {
     reader.optimizeForSearch();
     VectorSearchParams params = new VectorSearchParams(10, 16);
     VectorSearchResult result = reader.search(query, params);
 }</code></pre></div>
           <div class="code-block"><span class="code-label">Rust · configuration</span><pre><code>let config = VectorIndexConfig::IvfFlat {
     dimension: 128,
     nlist: 1024,
     metric: MetricType::L2,
 };
 let params = VectorSearchParams::new(10, 16);</code></pre></div>
         </section>

         <section class="article-section" id="parameters">
           <h2>Parameters</h2>
           <div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Requirement</th><th>Purpose</th><th>Effect when increased</th></tr></thead><tbody><tr><td><code>dimension</code></td><td>Inferred by Java/Python one-shot training; otherwise required and &gt; 0</td><td>Input dimension</td><td>Linearly increases compute and vector payload</td></tr><tr><td><code>nlist</code></td><td>Auto from <code>expected-vector-count</code>, or explicit &gt; 0</td><td>IVF partition count</td><td>Shorter average lists and a larger centroid table; automatic <code>nprobe</code> follows the resolved value</td></tr><tr><td><code>metric</code></td><td>Required: <code>l2</code>, <code>inner_product</code>, or <code>cosine</code></td><td>Training, assignment, and search distance</td><td>Semantic, not inferred; it must match ground truth</td></tr><tr><td><code>top_k</code></td><td>Query-time, &gt; 0</td><td>Requested results</td><td>Increases heap and output work</td></tr><tr><td><code>nprobe</code></td><td>Automatic by default; explicit 1 to <code>nlist</code></td><td>Lists to probe</td><td>Auto accounts for K, average list size, and filter selectivity; explicit values remain available for measured overrides</td></tr></tbody></table></div>
         </section>

         <section class="article-section" id="storage">
           <h2>v1 storage layout</h2>
           <p>The file is little-endian and has no outer container. The magic constant is <code>IVFL / 0x4956464C</code>; raw file bytes appear in reverse ASCII order because the integer is serialized little-endian.</p>
           <div class="storage-map" aria-label="IVF-FLAT file layout"><div class="storage-block primary"><strong>64 B header</strong>Type, dimension, count, flags</div><div class="storage-block"><strong>Coarse centroids</strong><code>nlist × d × f32</code></div><div class="storage-block"><strong>Offset table</strong><code>nlist × 16 B</code></div><div class="storage-block"><strong>Lists 0..N</strong>IDs + raw vectors</div></div>
           <h3>Fixed 64-byte header</h3>
           <div class="table-wrap"><table><thead><tr><th>Offset</th><th>Size</th><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>4</td><td><code>magic: u32</code></td><td><code>IVFL</code></td></tr><tr><td>4</td><td>4</td><td><code>version: u32</code></td><td>Currently 1</td></tr><tr><td>8</td><td>4</td><td><code>dimension: i32</code></td><td><code>d</code></td></tr><tr><td>12</td><td>4</td><td><code>nlist: i32</code></td><td>IVF list count</td></tr><tr><td>16</td><td>4</td><td><code>metric: u32</code></td><td>0=L2, 1=IP, 2=Cosine</td></tr><tr><td>20</td><td>8</td><td><code>total_vectors: i64</code></td><td>Total vector count</td></tr><tr><td>28</td><td>4</td><td><code>flags: u32</code></td><td>Bit 0: delta-varint IDs; required in v1</td></tr><tr><td>32</td><td>32</td><td>Reserved</td><td>Must be all zero</td></tr></tbody></table></div>
           <h3>Offset table and list payloads</h3>
           <p>Each offset entry is <code>(offset: i64, count: i32, id_bytes_len: i32)</code>, or 16 bytes. The Reader validates that list counts match the header before serving a query. A non-empty list contains:</p>
           <div class="table-wrap"><table><thead><tr><th>Field</th><th>Type / size</th><th>Description</th></tr></thead><tbody><tr><td><code>base_id</code></td><td><code>i64</code></td><td>First sorted row ID</td></tr><tr><td><code>id_bytes_len</code></td><td><code>i32</code></td><td>Encoded ID stream length</td></tr><tr><td><code>id_bytes</code></td><td>Variable</td><td>One unsigned LEB128 delta per ID; the first delta is zero</td></tr><tr><td><code>vectors</code></td><td><code>count × d × f32</code></td><td>Raw vectors in sorted-ID order</td></tr></tbody></table></div>
         </section>

         <section class="article-section" id="sizing">
           <h2>Capacity estimate</h2>
           <div class="callout"><strong>Approximate file size</strong><code>64 + 4 × nlist × d + 16 × nlist + N × 4 × d + encoded_ids</code></div>
           <p>For <code>N=1,000,000</code> and <code>d=128</code>, vector payloads alone are about 488 MiB. Sorted delta-varint ID size depends on ID continuity and cannot be estimated as a fixed eight bytes. With <code>nlist=1024</code>, coarse centroids add about 0.5 MiB.</p>
         </section>

         <section class="article-section" id="tuning">
           <h2>Tuning order</h2>
           <ol><li><strong>Fix the metric.</strong> Build exact ground truth with the production metric and include zero-vector edge cases for cosine.</li><li><strong>Start automatic.</strong> Supply the final corpus count, inspect the resolved <code>nlist</code>, and use automatic query width.</li><li><strong>Calibrate only if needed.</strong> Sweep explicit <code>nprobe</code> around the automatic value and record Recall@K, P95/P99, selected lists, and bytes read.</li><li><strong>Measure batch search.</strong> Readers submit bounded multi-range batches, which often matters more than single-query latency on object stores.</li><li><strong>Only then compress.</strong> Try IVF-SQ for a one-byte-per-dimension scan, IVF-PQ for stronger compression, or IVF-RQ for the strongest measured compact-IVF recall.</li></ol>
         </section>

         <section class="article-section" id="limits">
           <h2>Implementation boundaries</h2>
           <ul><li>Index files do not contain checksums; the outer Paimon file and manifest layer provides integrity.</li><li>Roaring64 filters are query payloads. Negative row IDs cannot match the <code>RoaringTreemap</code> domain.</li><li>Readers reject unknown versions, non-zero reserved bytes, unknown flags, negative counts, mismatched total counts, and out-of-bounds sections.</li><li>The search-only list payload owns an aligned <code>f32</code> allocation and reads bytes into it directly; the public list-materialization API still returns owned row IDs and vectors.</li><li><code>optimizeForSearch()</code> does not change files or results; for IVF-FLAT it preloads centroids and the offset table in one contiguous read. The unified Reader reuses its type-dispatch header, so open plus metadata initialization takes two read rounds.</li></ul>
           <nav class="pager" aria-label="Index navigation"><a href="index.html"><small>Back</small>Index overview</a><a href="ivf-pq.html"><small>Next</small>IVF-PQ →</a></nav>
         </section>
       </article>
     </div>
   </div></main>
   <footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index</span><span>IVF-FLAT · v1</span></div></footer>
 </body>
 </html>
	<!--
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to you under the Apache License, Version 2.0.
	-->
	<!doctype html>
	<html lang="en">
	<head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><meta name="description" content="IVF-FLAT usage, design, search pipeline, tuning, capacity, and v1 storage layout."><title>IVF-FLAT · Paimon Vector Index</title><link rel="stylesheet" href="styles.css"><script src="docs.js" defer></script></head>
	<body>
	<a class="skip-link" href="#main">Skip to content</a>
	<header class="site-header"><div class="header-inner"><a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"><span class="brand-mark">VI</span><span>Paimon Vector Index</span></a><nav class="site-nav" data-site-nav aria-label="Documentation"><a href="index.html">Overview</a><a href="api.html">API</a><a href="development.html">Development</a><a href="ivf-flat.html" aria-current="page">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a href="ivf-sq.html">IVF-SQ</a><a href="diskann.html">DiskANN</a><a href="releases.html">Releases</a></nav><div class="header-actions"><button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button><button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button></div></div></header>
	<main id="main"><div class="page-shell">
	<section class="hero detail-hero"><p class="eyebrow">Exact scan inside selected lists</p><h1>IVF-FLAT</h1><p class="hero-lead">Use IVF coarse partitions to reduce the candidate set, then retain and scan full <code>f32</code> vectors inside each selected list. IVF-FLAT is the clearest baseline for separating partition recall loss from quantization loss.</p><div class="badge-row"><span class="badge strong">No in-list quantization</span><span class="badge">About 4d bytes / vector</span><span class="badge">Simple build</span><span class="badge">Magic: IVFL</span></div></section>
	<div class="doc-layout">
	<aside class="toc" aria-label="On this page"><strong>On this page</strong><a href="#position">Positioning</a><a href="#design">Design</a><a href="#benchmarks">Public benchmarks</a><a href="#usage">Usage</a><a href="#parameters">Parameters</a><a href="#storage">Storage layout</a><a href="#sizing">Capacity</a><a href="#tuning">Tuning</a><a href="#limits">Boundaries</a></aside>
	<article class="article">
	<section class="article-section" id="position">
	<h2>Positioning and trade-offs</h2>
	<div class="metric-strip"><div class="metric"><span class="label">In-list accuracy</span><span class="value">Exact raw-vector distance</span></div><div class="metric"><span class="label">Main storage</span><span class="value"><code>4 × d × N</code></span></div><div class="metric"><span class="label">Training</span><span class="value">IVF K-Means only</span></div><div class="metric"><span class="label">Query work</span><span class="value">Scan every vector in probed lists</span></div></div>
	<div class="split"><div class="pro-con"><h3>Good fit</h3><ul><li>Establishing recall and latency baselines.</li><li>Moderate vector counts or dimensions.</li><li>Accuracy matters more than index size.</li><li>A clear, low-risk implementation is preferred.</li></ul></div><div class="pro-con"><h3>Poor fit</h3><ul><li>High-dimensional collections with strict space budgets.</li><li>Large lists where scans dominate tail latency.</li><li>Very small remote-read budgets.</li></ul></div></div>
	<div class="callout"><strong>“Exact” has a boundary</strong>IVF-FLAT computes exact distances only inside the <code>nprobe</code> selected lists. Whenever <code>nprobe < nlist</code>, the global search remains approximate.</div>
	</section>

	<section class="article-section" id="design">
	<h2>Design and data flow</h2>
	<h3>Build</h3>
	<div class="pipeline"><div class="pipeline-item"><span class="pipeline-index">1</span><div><h3>Preprocess training vectors</h3><p>Cosine mode applies L2 normalization. L2 and inner product keep the input representation.</p></div></div><div class="pipeline-item"><span class="pipeline-index">2</span><div><h3>Train coarse centroids</h3><p>K-Means produces <code>nlist × d</code> <code>f32</code> centroid values.</p></div></div><div class="pipeline-item"><span class="pipeline-index">3</span><div><h3>Assign vectors</h3><p>Each vector enters its nearest coarse list with its complete processed vector and row ID.</p></div></div><div class="pipeline-item"><span class="pipeline-index">4</span><div><h3>Sort and serialize</h3><p>The writer retains compact sort permutations and delta-varint IDs, then materializes and writes one sorted raw-vector list at a time instead of duplicating every list in memory.</p></div></div></div>
	<h3>Search</h3>
	<ol><li>Apply the same preprocessing used during construction.</li><li>Measure the query against every IVF centroid and select the closest <code>nprobe</code> lists.</li><li>Read those list payloads through the offset table in bounded concurrent multi-range calls.</li><li>Decode row IDs and compute the true metric against every raw vector. A Roaring filter skips disallowed IDs.</li><li>For at least 1,048,576 distance components, scan independent lists on Rayon workers. Batch search keeps list-major locality and scans every loaded list for all queries that selected it.</li><li>Merge list-local results in the original list order. Missing entries are padded with <code>-1 / f32::MAX</code>.</li></ol>
	</section>

	<section class="article-section" id="benchmarks">
	<h2>Public-corpus measurements</h2>
	<p>Apple M4 Pro, 12 Rayon workers, one million SIFT/GIST vectors or 1,183,514 GloVe vectors, 1,000 published queries, <code>nlist=1024</code>, <code>nprobe=64</code>, Top-10, and warm APFS pages. Times are release-build measurements from 25 July 2026.</p>
	<div class="table-wrap"><table><thead><tr><th>Dataset</th><th>Recall@10</th><th>Local P95</th><th>Local batch QPS</th><th>Read / query</th></tr></thead><tbody><tr><td>SIFT1M, 128d</td><td>0.9937</td><td>1.88 ms</td><td>8,510</td><td>33.19 MiB</td></tr><tr><td>GIST1M, 960d</td><td>0.9549</td><td>11.38 ms</td><td>875</td><td>283.40 MiB</td></tr><tr><td>GloVe-100, 100d</td><td>0.8832</td><td>1.40 ms</td><td>9,502</td><td>27.57 MiB</td></tr></tbody></table></div>
	<div class="callout"><strong>Latest complete same-file rerun</strong>The Reader receives each list directly into an <code>f32</code>-aligned allocation and scans the raw-vector suffix without allocating a second decoded payload. An internal prefix of at most three bytes compensates for the variable-length row-ID prefix; the persisted bytes do not change. Together with the strict partial-L2 cutoff, this changed SIFT/GIST/GloVe batch throughput from 6,570 / 559 / 6,345 to 8,510 / 875 / 9,502 QPS and P95 from 5.31 / 47.04 / 4.76 ms to 1.88 / 11.38 / 1.40 ms. Recall, file bytes, and read bytes stayed identical. The table reports the complete rerun rather than a best-of result.</div>
	<p>The remote model groups every query's selected ranges into calls capped at 64 MiB. SIFT/GloVe average one round; GIST averages five because its raw-vector payload is much larger. Fixed 2/20 ms latency therefore produced P95 7.31/21.80 ms on SIFT, 29.73/130.49 ms on GIST, and 6.60/21.22 ms on GloVe. Batch throughput was 6,763/3,076 QPS on SIFT, 846/349 on GIST, and 7,394/2,948 on GloVe. These modeled numbers do not charge bandwidth, so they should not be read as evidence that transferring tens or hundreds of MiB directly from an object store is cheap.</p>
	</section>

	<section class="article-section" id="usage">
	<h2>Usage</h2>
	<p>The trained state is one-shot: finish training, pass the state to a Writer, add production vectors in batches, and serialize one index file. Readers discover the type from the header.</p>
	<div class="code-block"><span class="code-label">Java · build</span><pre><code>Map<String, String> options = new HashMap<>();
	options.put("index.type", "ivf_flat");
	options.put("dimension", "128");
	options.put("nlist", "1024");
	options.put("metric", "l2");

	try (VectorIndexTraining training =
	VectorIndexTrainer.train(options, trainingVectors, trainingCount);
	VectorIndexWriter writer = new VectorIndexWriter(training)) {
	writer.addVectors(rowIds, vectors, vectorCount);
	writer.writeIndex(vectorIndexOutput);
	}</code></pre></div>
	<div class="code-block"><span class="code-label">Java · search</span><pre><code>try (VectorIndexReader reader = new VectorIndexReader(vectorIndexInput)) {
	reader.optimizeForSearch();
	VectorSearchParams params = new VectorSearchParams(10, 16);
	VectorSearchResult result = reader.search(query, params);
	}</code></pre></div>
	<div class="code-block"><span class="code-label">Rust · configuration</span><pre><code>let config = VectorIndexConfig::IvfFlat {
	dimension: 128,
	nlist: 1024,
	metric: MetricType::L2,
	};
	let params = VectorSearchParams::new(10, 16);</code></pre></div>
	</section>

	<section class="article-section" id="parameters">
	<h2>Parameters</h2>
	<div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Requirement</th><th>Purpose</th><th>Effect when increased</th></tr></thead><tbody><tr><td><code>dimension</code></td><td>Inferred by Java/Python one-shot training; otherwise required and > 0</td><td>Input dimension</td><td>Linearly increases compute and vector payload</td></tr><tr><td><code>nlist</code></td><td>Auto from <code>expected-vector-count</code>, or explicit > 0</td><td>IVF partition count</td><td>Shorter average lists and a larger centroid table; automatic <code>nprobe</code> follows the resolved value</td></tr><tr><td><code>metric</code></td><td>Required: <code>l2</code>, <code>inner_product</code>, or <code>cosine</code></td><td>Training, assignment, and search distance</td><td>Semantic, not inferred; it must match ground truth</td></tr><tr><td><code>top_k</code></td><td>Query-time, > 0</td><td>Requested results</td><td>Increases heap and output work</td></tr><tr><td><code>nprobe</code></td><td>Automatic by default; explicit 1 to <code>nlist</code></td><td>Lists to probe</td><td>Auto accounts for K, average list size, and filter selectivity; explicit values remain available for measured overrides</td></tr></tbody></table></div>
	</section>

	<section class="article-section" id="storage">
	<h2>v1 storage layout</h2>
	<p>The file is little-endian and has no outer container. The magic constant is <code>IVFL / 0x4956464C</code>; raw file bytes appear in reverse ASCII order because the integer is serialized little-endian.</p>
	<div class="storage-map" aria-label="IVF-FLAT file layout"><div class="storage-block primary"><strong>64 B header</strong>Type, dimension, count, flags</div><div class="storage-block"><strong>Coarse centroids</strong><code>nlist × d × f32</code></div><div class="storage-block"><strong>Offset table</strong><code>nlist × 16 B</code></div><div class="storage-block"><strong>Lists 0..N</strong>IDs + raw vectors</div></div>
	<h3>Fixed 64-byte header</h3>
	<div class="table-wrap"><table><thead><tr><th>Offset</th><th>Size</th><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>4</td><td><code>magic: u32</code></td><td><code>IVFL</code></td></tr><tr><td>4</td><td>4</td><td><code>version: u32</code></td><td>Currently 1</td></tr><tr><td>8</td><td>4</td><td><code>dimension: i32</code></td><td><code>d</code></td></tr><tr><td>12</td><td>4</td><td><code>nlist: i32</code></td><td>IVF list count</td></tr><tr><td>16</td><td>4</td><td><code>metric: u32</code></td><td>0=L2, 1=IP, 2=Cosine</td></tr><tr><td>20</td><td>8</td><td><code>total_vectors: i64</code></td><td>Total vector count</td></tr><tr><td>28</td><td>4</td><td><code>flags: u32</code></td><td>Bit 0: delta-varint IDs; required in v1</td></tr><tr><td>32</td><td>32</td><td>Reserved</td><td>Must be all zero</td></tr></tbody></table></div>
	<h3>Offset table and list payloads</h3>
	<p>Each offset entry is <code>(offset: i64, count: i32, id_bytes_len: i32)</code>, or 16 bytes. The Reader validates that list counts match the header before serving a query. A non-empty list contains:</p>
	<div class="table-wrap"><table><thead><tr><th>Field</th><th>Type / size</th><th>Description</th></tr></thead><tbody><tr><td><code>base_id</code></td><td><code>i64</code></td><td>First sorted row ID</td></tr><tr><td><code>id_bytes_len</code></td><td><code>i32</code></td><td>Encoded ID stream length</td></tr><tr><td><code>id_bytes</code></td><td>Variable</td><td>One unsigned LEB128 delta per ID; the first delta is zero</td></tr><tr><td><code>vectors</code></td><td><code>count × d × f32</code></td><td>Raw vectors in sorted-ID order</td></tr></tbody></table></div>
	</section>

	<section class="article-section" id="sizing">
	<h2>Capacity estimate</h2>
	<div class="callout"><strong>Approximate file size</strong><code>64 + 4 × nlist × d + 16 × nlist + N × 4 × d + encoded_ids</code></div>
	<p>For <code>N=1,000,000</code> and <code>d=128</code>, vector payloads alone are about 488 MiB. Sorted delta-varint ID size depends on ID continuity and cannot be estimated as a fixed eight bytes. With <code>nlist=1024</code>, coarse centroids add about 0.5 MiB.</p>
	</section>

	<section class="article-section" id="tuning">
	<h2>Tuning order</h2>
	<ol><li><strong>Fix the metric.</strong> Build exact ground truth with the production metric and include zero-vector edge cases for cosine.</li><li><strong>Start automatic.</strong> Supply the final corpus count, inspect the resolved <code>nlist</code>, and use automatic query width.</li><li><strong>Calibrate only if needed.</strong> Sweep explicit <code>nprobe</code> around the automatic value and record Recall@K, P95/P99, selected lists, and bytes read.</li><li><strong>Measure batch search.</strong> Readers submit bounded multi-range batches, which often matters more than single-query latency on object stores.</li><li><strong>Only then compress.</strong> Try IVF-SQ for a one-byte-per-dimension scan, IVF-PQ for stronger compression, or IVF-RQ for the strongest measured compact-IVF recall.</li></ol>
	</section>

	<section class="article-section" id="limits">
	<h2>Implementation boundaries</h2>
	<ul><li>Index files do not contain checksums; the outer Paimon file and manifest layer provides integrity.</li><li>Roaring64 filters are query payloads. Negative row IDs cannot match the <code>RoaringTreemap</code> domain.</li><li>Readers reject unknown versions, non-zero reserved bytes, unknown flags, negative counts, mismatched total counts, and out-of-bounds sections.</li><li>The search-only list payload owns an aligned <code>f32</code> allocation and reads bytes into it directly; the public list-materialization API still returns owned row IDs and vectors.</li><li><code>optimizeForSearch()</code> does not change files or results; for IVF-FLAT it preloads centroids and the offset table in one contiguous read. The unified Reader reuses its type-dispatch header, so open plus metadata initialization takes two read rounds.</li></ul>
	<nav class="pager" aria-label="Index navigation"><a href="index.html"><small>Back</small>Index overview</a><a href="ivf-pq.html"><small>Next</small>IVF-PQ →</a></nav>
	</section>
	</article>
	</div>
	</div></main>
	<footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index</span><span>IVF-FLAT · v1</span></div></footer>
	</body>
	</html>