| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to you under the Apache License, Version 2.0. |
| --> |
| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="description" content="Compare the six Apache Paimon Vector Index families by accuracy, latency, storage, build cost, and on-disk layout."> |
| <title>Index Selection Guide · Paimon Vector Index</title> |
| <link rel="stylesheet" href="styles.css"> |
| <script src="docs.js" defer></script> |
| </head> |
| <body> |
| <a class="skip-link" href="#main">Skip to content</a> |
| <header class="site-header"> |
| <div class="header-inner"> |
| <a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"> |
| <span class="brand-mark">VI</span> |
| <span>Paimon Vector Index</span> |
| </a> |
| <nav class="site-nav" data-site-nav aria-label="Documentation"> |
| <a href="index.html" aria-current="page">Overview</a> |
| <a href="api.html">API</a> |
| <a href="development.html">Development</a> |
| <a href="ivf-flat.html">IVF-FLAT</a> |
| <a href="ivf-pq.html">IVF-PQ</a> |
| <a href="ivf-rq.html">IVF-RQ</a> |
| <a href="ivf-sq.html">IVF-SQ</a> |
| <a href="diskann.html">DiskANN</a> |
| <a href="releases.html">Releases</a> |
| </nav> |
| <div class="header-actions"> |
| <button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button> |
| <button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button> |
| </div> |
| </div> |
| </header> |
| |
| <main id="main"> |
| <div class="page-shell"> |
| <section class="hero"> |
| <div class="hero-grid"> |
| <div> |
| <p class="eyebrow">Index selection guide · v1</p> |
| <h1>Five indexes.<br>One selection map.</h1> |
| <p class="hero-lead">Four implementations use IVF to narrow the search space before scanning raw vectors or compact codes. DiskANN instead uses one global Vamana graph with resident PQ navigation and paged F16/F32 reranking. This guide compares recall, latency, storage, build cost, and object-store I/O.</p> |
| <div class="badge-row" aria-label="Shared capabilities"> |
| <span class="badge strong">All indexes: L2 / IP / Cosine</span> |
| <span class="badge">Batch search</span> |
| <span class="badge">Roaring row filters</span> |
| <span class="badge">Magic-based detection</span> |
| </div> |
| </div> |
| <aside class="hero-note"> |
| <strong>The short answer</strong> |
| Measure IVF-FLAT first. For compact indexes, choose IVF-SQ for batch throughput, IVF-RQ for higher recall, or IVF-PQ for the smallest files. Choose DiskANN for an immutable collection when high recall and small local-SSD reads justify a much slower build; validate its recall independently for the production metric. |
| </aside> |
| </div> |
| </section> |
| |
| <section class="section" id="shared-path"> |
| <div class="section-heading"> |
| <h2>IVF's shared two-stage search</h2> |
| <p>For the four IVF families, <code>nlist</code> controls the number of coarse partitions and <code>nprobe</code> controls how many partitions a query reads. Only the stored representation and scan kernel change.</p> |
| </div> |
| <div class="callout"><strong>DiskANN follows a different path</strong>It traverses one global Vamana graph using resident PQ codes, reads adjacency pages on demand, then reranks persisted F16 or F32 vectors from either the same interleaved pages or a separate compact section. It has no IVF lists; the tagged query API uses <code>l_search</code> and rejects an IVF <code>nprobe</code> override.</div> |
| <div class="flow" aria-label="Shared search pipeline"> |
| <div class="flow-step"><small>01</small><strong>Preprocess vectors<br>Normalize for cosine</strong></div> |
| <div class="flow-step"><small>02</small><strong>Measure distance to<br>IVF coarse centroids</strong></div> |
| <div class="flow-step"><small>03</small><strong>Select the nearest<br><code>nprobe</code> lists</strong></div> |
| <div class="flow-step"><small>04</small><strong>Scan raw vectors or<br>compact codes</strong></div> |
| <div class="flow-step"><small>05</small><strong>Merge candidates into<br>the global top K</strong></div> |
| </div> |
| </section> |
| |
| <section class="section" id="documentation"> |
| <div class="section-heading"> |
| <h2>Usage and development</h2> |
| <p>Index pages focus on algorithms and formats. Shared lifecycle, language bindings, filtering, benchmarks, and build commands live in dedicated guides so the same information is maintained only once.</p> |
| </div> |
| <div class="card-grid"> |
| <article class="card"><h3>API and language bindings</h3><p>The Trainer / Writer / Reader lifecycle with complete Rust, C, C++, Java/JNI, and Python examples.</p><a class="card-link" href="api.html">Open the API guide →</a></article> |
| <article class="card"><h3>Search and filtering</h3><p>Understand automatic width, explicit IVF <code>nprobe</code>, DiskANN <code>l_search</code> calibration, warm-up, and Roaring64 filters.</p><a class="card-link" href="api.html#params">Explore the search API →</a></article> |
| <article class="card"><h3>Development and benchmarks</h3><p>Repository modules, Rust checks, cross-language smoke tests, ANN benchmarks, and storage compatibility.</p><a class="card-link" href="development.html">Open the development guide →</a></article> |
| <article class="card"><h3>DiskANN deployment and tuning</h3><p>Understand the memory, local, remote, and object-store profiles, resident-memory model, build parameters, and production-readiness boundary.</p><a class="card-link" href="diskann.html">Open the DiskANN guide →</a></article> |
| <article class="card"><h3>Releases and verification</h3><p>Download signed source releases, create a release candidate, or independently verify one before voting.</p><a class="card-link" href="releases.html">Open the release guides →</a></article> |
| </div> |
| </section> |
| |
| <section class="section" id="comparison"> |
| <div class="section-heading"> |
| <h2>Core differences</h2> |
| <p>This table describes the implementation in this repository, not a general promise made by similarly named algorithms elsewhere. Storage estimates omit row IDs, model sections, offset tables, alignment, and headers unless noted.</p> |
| </div> |
| <div class="filter-row" aria-label="Filter indexes by priority"> |
| <button class="filter-button" type="button" data-filter="all" aria-pressed="true">All</button> |
| <button class="filter-button" type="button" data-filter="recall" aria-pressed="false">Recall first</button> |
| <button class="filter-button" type="button" data-filter="compact" aria-pressed="false">Space first</button> |
| <button class="filter-button" type="button" data-filter="latency" aria-pressed="false">Latency first</button> |
| <button class="filter-button" type="button" data-filter="simple" aria-pressed="false">Simple baseline</button> |
| </div> |
| <div class="table-wrap"> |
| <table class="comparison-table"> |
| <thead><tr><th>Index</th><th>Representation</th><th>Candidate search</th><th>Main payload per vector</th><th>Accuracy profile</th><th>Build cost</th><th>Primary controls</th><th>Best fit</th></tr></thead> |
| <tbody> |
| <tr data-index-row data-fit="compact latency"><td><a href="ivf-pq.html">IVF-PQ</a></td><td>8-bit PQ codes; optional OPQ</td><td>Distance-table lookup over compact codes</td><td>About <code>m</code> bytes</td><td>PQ reconstruction error; OPQ may improve uneven subspaces</td><td>Medium to high</td><td>Automatic <code>nlist</code>/<code>nprobe</code>/<code>pq.m</code>; target-based OPQ; explicit overrides</td><td>Minimum file and selected-list bytes among IVF when the measured 0.58–0.74 recall band is sufficient</td></tr> |
| <tr data-index-row data-fit="compact latency simple"><td><a href="ivf-sq.html">IVF-SQ</a></td><td>8-bit scalar residual codes with per-list bounds</td><td>SIMD code scan in probed lists</td><td>About <code>d</code> bytes</td><td>Per-coordinate scalar quantization loss</td><td>Low</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; explicit overrides</td><td>Highest measured compact batch throughput when 0.80–0.86 recall meets the target</td></tr> |
| <tr data-index-row data-fit="compact recall"><td><a href="ivf-rq.html">IVF-RQ</a></td><td>Multi-bit rotated residual levels + coarse/full factors</td><td>Bounded sign-plane scan, then full bit-plane refinement</td><td>Default about <code>padded_d/2+20</code> bytes</td><td>Measured 0.82–0.91 Recall@10 across GloVe-100, SIFT1M, and GIST1M at four bits</td><td>Low to medium</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; budget-based bits; explicit overrides</td><td>Best measured compact recall when 3–4.5× lower batch throughput than IVF-SQ is acceptable</td></tr> |
| <tr data-index-row data-fit="recall latency"><td><a href="diskann.html">DiskANN</a></td><td>Global Vamana + resident PQ + persisted rerank vectors</td><td>PQ-guided graph traversal and F32/F16 rerank</td><td><code>E·d + pq.m + 4(R+1)</code> bytes, approximately; <code>E=4</code> or <code>2</code></td><td>Approximate candidate discovery; F32-exact or F16-quantized distances for reranked candidates</td><td>Very high</td><td>Build preset + deployment/capacity objectives; calibrated automatic <code>l_search</code></td><td>Immutable L2 on local SSD when high recall and sub-MiB query reads repay the long build</td></tr> |
| <tr data-index-row data-fit="recall simple"><td><a href="ivf-flat.html">IVF-FLAT</a></td><td>Raw <code>f32</code> vectors</td><td>Exact distance scan in probed lists</td><td>About <code>4d</code> bytes</td><td>No quantization loss; recall mainly depends on <code>nprobe</code></td><td>Low</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; explicit overrides</td><td>Recall ceiling, frequent rebuilds, IP/cosine, or production sets whose scan bytes are affordable</td></tr> |
| </tbody> |
| </table> |
| </div> |
| </section> |
| |
| <section class="section" id="public-corpus-check"> |
| <div class="section-heading"> |
| <h2>Measured comparison: public SIFT1M, GIST1M, and GloVe-100 corpora</h2> |
| <p>This repository's unified benchmark builds all five indexes over standard public vectors, searches the same independent public queries, and scores every result against published exact neighbors. The results below are the homepage's sole performance evidence.</p> |
| </div> |
| |
| <h3>Benchmark setup</h3> |
| <div class="callout warning"><strong>This is not the zero-configuration benchmark</strong>Running <code>cargo bench -p paimon-vindex-core --bench ann_bench</code> without public file paths uses a 20k-vector, 64-dimensional generated smoke workload. Reproducing the results below requires the public files, recorded IVF and DiskANN search settings, a fixed worker count, and an output directory on the storage device being measured. File shape, training count, and multi-index process isolation are automatic.</div> |
| <p>The real-data run uses the public <a href="https://github.com/erikbern/ann-benchmarks">ANN-Benchmarks</a> SIFT1M, GIST1M, and GloVe-100 files, the first 1,000 independent test queries, and their published Top-100 exact neighbors. Recall@10 compares only the first ten published neighbors. SIFT and GIST contain one million base vectors with 128 and 960 dimensions. GloVe contains 1,183,514 vectors with 100 dimensions and angular ground truth; its base and query vectors are L2-normalized during conversion so the common L2 benchmark produces the same neighbor ordering as cosine. The benchmark supplies 65,536 base vectors to every trainer; DiskANN bounds PQ training memory with a deterministic reservoir of at most 50,000 vectors.</p> |
| <div class="table-wrap"><table><thead><tr><th>Source</th><th>Parameters</th><th>Recorded value</th><th>Reproduction rule</th></tr></thead><tbody> |
| <tr><td>Public data</td><td><code>ANN_BASE_FVECS</code>, <code>ANN_QUERY_FVECS</code>, <code>ANN_GROUND_TRUTH_IVECS</code></td><td>SIFT1M, GIST1M, or normalized GloVe-100 converted files</td><td>Set all three together; otherwise the benchmark generates synthetic vectors.</td></tr> |
| <tr><td>Dataset shape</td><td><code>ANN_N</code>, <code>ANN_NQ</code>, <code>ANN_D</code></td><td><code>1,000,000 / 1,183,514</code>, <code>1,000</code>, <code>128 / 960 / 100</code></td><td>Inferred from the public files. An explicit value becomes a shape assertion and fails before the full dataset is loaded if it differs.</td></tr> |
| <tr><td>Training input</td><td><code>ANN_TRAIN_N</code></td><td><code>65,536</code></td><td>Inferred as <code>min(N, max(65,536, 64 × nlist))</code>. DiskANN deterministically retains at most 50,000 of these vectors for bounded PQ training; the IVF trainers consume all 65,536.</td></tr> |
| <tr><td>IVF search</td><td><code>ANN_NLIST</code>, <code>ANN_NPROBE</code></td><td><code>1,024</code>, <code>64</code></td><td>Set explicitly; benchmark defaults are 64 and 8.</td></tr> |
| <tr><td>DiskANN search</td><td><code>ANN_DISKANN_L_SEARCH</code></td><td><code>100</code></td><td>May be omitted; this is the benchmark default and the automatic value for <code>k=10</code>.</td></tr> |
| <tr><td>Process and device isolation</td><td><code>ANN_INDEXES</code>, <code>ANN_OUTPUT_DIR</code>, <code>RAYON_NUM_THREADS</code></td><td>All five indexes, target APFS path, 12 threads</td><td>Public multi-index runs automatically spawn one child process per index. Set a subset only when needed; set the device path and worker count explicitly.</td></tr> |
| <tr><td>Matching defaults</td><td><code>ANN_K</code>, <code>ANN_PQ_CODE_RATIO</code>, <code>ANN_RQ_BITS</code></td><td><code>10</code>, <code>0.0625</code>, <code>4</code></td><td>May be omitted; the reproduction command pins them so a future default change cannot silently alter the comparison.</td></tr> |
| <tr><td>DiskANN build defaults</td><td><code>pq.bits</code>, <code>R</code>, <code>Lbuild</code>, <code>alpha</code>, memory budget, layout, raw-vector encoding, build distance</td><td><code>8</code>, <code>64</code>, <code>100</code>, <code>1.2</code>, 8 GiB, compact, F16, product-quantized</td><td>Use the benchmark defaults. Leave <code>ANN_PQ_M</code> unset so <code>pq.code-ratio</code> resolves the concrete value.</td></tr> |
| <tr><td>Reader/I/O model</td><td>Automatic read plan, Reader budget, simulated latency</td><td>Latency-derived local/remote/object-store plans, 4 GiB automatically partitioned budget, 0/2/20 ms per read round</td><td>Fixed by the current benchmark implementation; <code>ANN_STORAGE_CASES</code> selects a focused subset. DiskANN range reads use an I/O pool independent of query workers; all five indexes were refreshed after their current reader, storage, and parallel-scan work.</td></tr> |
| </tbody></table></div> |
| <div class="callout"><strong>Equal relative PQ budget</strong>The default <code>pq.code-ratio=0.0625</code> automatically resolves SIFT to <code>pq.m=32</code>, GIST to <code>pq.m=240</code>, and GloVe to <code>pq.m=25</code>. Every code occupies 6.25% as many bytes as its raw <code>f32</code> vector and leaves four dimensions per PQ sub-vector. The concrete value is persisted in index metadata; use explicit <code>pq.m</code> only as an expert override.</div> |
| <p>The cross-index run was recorded on 25 July 2026 using an Apple M4 Pro with 12 logical CPUs and 48 GiB RAM, a release build with Rust 1.95, real APFS files with warm operating-system pages, and the automatic 4 GiB DiskANN Reader budget. The IVF-RQ staged A/B and rebased IVF-PQ warm-local refresh were recorded on 30 July on the same host and toolchain. The reproduction command pins Rayon to 12 workers instead of relying on automatic host parallelism. The modeled serving profiles add 2 ms or 20 ms per positional-read round while executing all ranges in that round concurrently. DiskANN's benchmark adapter runs those ranges on a separate 12-worker I/O pool so a full query-worker pool cannot starve nested reads; this models the independent executor required of a production concurrent storage callback. For the 20 ms profile, open/optimization and sequential-query latency are computed as measured CPU/I/O time plus 20 ms per observed round; batch QPS retains literal delay injection so query overlap is measured. IVF multi-range calls are bounded to 64 MiB, so an all-query GIST batch uses 4 IVF-PQ, 15 IVF-SQ, or 59 IVF-FLAT payload rounds instead of submitting hundreds of MiB or several GiB as one unbounded call. Unified IVF Readers now reuse the 64-byte dispatch header, so opening and loading resident metadata takes two positional-read rounds rather than three. Each dataset's three DiskANN profile rows reuse the same built graph. Sequential and batch measurements use separately opened and optimized Readers, so the batch does not inherit query-dependent windows from the sequential sweep. Batch QPS measures one <code>search_batch</code> call over all 1,000 public queries; it is not concurrent-client QPS. See the <a href="development.html#ann">complete public-data command</a>.</p> |
| |
| <h4>Build, file, and peak process memory</h4> |
| <div class="table-wrap"><table><thead><tr><th>Index</th><th>SIFT build</th><th>SIFT file / RSS</th><th>GIST build</th><th>GIST file / RSS</th><th>GloVe build</th><th>GloVe file / RSS</th></tr></thead><tbody> |
| <tr><td>IVF-PQ</td><td>8.74 s</td><td>0.032 / 0.88 GiB</td><td>55.4 s</td><td>0.230 / 5.00 GiB</td><td>7.92 s</td><td>0.030 / 0.85 GiB</td></tr> |
| <tr><td>IVF-SQ</td><td>3.93 s</td><td>0.122 / 0.79 GiB</td><td>22.7 s</td><td>0.907 / 5.09 GiB</td><td>3.86 s</td><td>0.113 / 0.71 GiB</td></tr> |
| <tr><td>IVF-RQ</td><td>3.92 s</td><td>0.080 / 0.71 GiB</td><td>23.5 s</td><td>0.471 / 4.65 GiB</td><td>4.03 s</td><td>0.095 / 0.68 GiB</td></tr> |
| <tr><td>DiskANN</td><td>74.0 s</td><td>0.361 / 1.51 GiB</td><td>11 min 26 s</td><td>2.089 / 7.94 GiB</td><td>2 min 33 s</td><td>0.396 / 1.45 GiB</td></tr> |
| <tr><td>IVF-FLAT</td><td>4.05 s</td><td>0.479 / 1.83 GiB</td><td>24.9 s</td><td>3.582 / 12.74 GiB</td><td>4.10 s</td><td>0.443 / 1.60 GiB</td></tr> |
| </tbody></table></div> |
| <div class="callout"><strong>IVF-SQ build and scan refresh</strong>The add path now borrows L2/IP input, assigns rows once, and encodes lists in parallel with one residual scratch vector per active list task instead of materializing an additional <code>N × d</code> residual matrix. In the immediately preceding same-machine run, SIFT/GIST/GloVe peak RSS was 1.81 / 12.92 / 1.60 GiB; it is now 0.79 / 5.09 / 0.71 GiB. A Top-K threshold fast path skips hash work for candidates that cannot enter the heap: local P95 is now 0.79 / 3.56 / 0.71 ms and batch throughput is 11,082 / 1,502 / 12,962 QPS. An experimental list-major batch scan was slower on SIFT/GIST and was not retained. The blocked-code format, file size, read bytes, and measured Recall@10 remain unchanged.</div> |
| <div class="callout"><strong>30 July IVF-PQ batch-table reuse refresh</strong>The rebased Reader retains the v1 zero-copy/transposed-code, ordered-list, one-byte row-ID, and first-column accumulation fast paths. For large 8-bit residual-L2 batches, the default <code>Auto</code> mode now factors each distance table into reusable per-list and per-query components when the reuse heuristic and 64 MiB working-memory guard both pass; small or unsuitable batches keep the direct path, and callers may explicitly select <code>On</code> or <code>Off</code>. Six same-file runs per mode alternated execution order. SIFT/GIST/GloVe median batch throughput changed from 4,191 / 497 / 4,366 QPS with reuse disabled to 7,899 / 950 / 8,048 QPS with <code>Auto</code>, gains of 88.5% / 91.3% / 84.3%. The <code>Auto</code> medians used below are 0.72 / 2.47 / 0.64 ms P95 and 1,583 / 467 / 1,802 sequential QPS. File bytes, query bytes, and the v1 format are unchanged. GIST and GloVe Recall@10 are unchanged at four decimals; SIFT moved from 0.7143 to 0.7142 because the stable <code>f64</code> factored-table path is numerically close but not bit-identical to direct residual-table accumulation. A removed contiguous all-query table remains distinct from this bounded factorization. Faiss FastScan's 4-bit, 32-row design remains a different accuracy/format choice.</div> |
| <div class="callout"><strong>Latest IVF-FLAT storage and scan review</strong>The v1 writer retains only sort permutations and encoded IDs, materializes one sorted raw-vector list at a time, and reproduced all three prior public files byte-for-byte. The Reader now receives list bytes directly into an <code>f32</code>-aligned allocation; an internal prefix of at most three bytes keeps the raw-vector suffix aligned despite variable-length row IDs, so search no longer allocates and decodes a second vector payload. Together with the strict partial-L2 cutoff, the complete public rerun changed SIFT/GIST/GloVe local batch throughput from 6,570 / 559 / 6,345 to 8,510 / 875 / 9,502 QPS and P95 from 5.31 / 47.04 / 4.76 ms to 1.88 / 11.38 / 1.40 ms. Recall@10, file version, file bytes, and bytes read are unchanged; these are complete-run results, not best-of measurements.</div> |
| <div class="callout"><strong>Pre-release storage changes produced measurable savings</strong>The compact IVF-RQ factor layout removes one unused F32 value per row: SIFT/GIST/GloVe files fell from 90.3 / 509.8 / 106.8 MB to 86.3 / 505.8 / 102.1 MB without changing the estimator; the refreshed Recall@10 values are 0.9148 / 0.9039 / 0.8203. Changing the balanced DiskANN default from F32 to F16 rerank vectors reduced the same three files from 0.599 / 3.877 / 0.617 GiB to 0.361 / 2.089 / 0.396 GiB. The current warm-local Recall@10 values are 0.9915 / 0.9336 / 0.8355. Timing changes also include the intervening reader/cache and graph-build work.</div> |
| <div class="callout"><strong>30 July IVF-RQ staged scan A/B</strong>The four changes below were measured against one retained v1 index per corpus with the same 1,000 queries, <code>nlist=1,024</code>, <code>nprobe=64</code>, four stored RQ bits, 12 Rayon workers, and warm APFS pages. SIFT and GloVe use seven interleaved runs. Because GIST-960 showed thermal drift during long stage sweeps, its reported wall-clock changes use five baseline/final pairs with alternating execution order. Every staged binary returned the same Recall@10. The GIST timing A/B used the public Open VDB mirror and returned 0.9037 rather than the earlier ANN-Benchmarks file's 0.9039, so the cross-index table below retains the original recall row and this section uses timing deltas only.</div> |
| <div class="table-wrap"><table><thead><tr><th>IVF-RQ change</th><th>SIFT1M result</th><th>GIST1M result</th><th>GloVe-100 result</th></tr></thead><tbody> |
| <tr><td>Block-aggregated scan statistics</td><td>70.91% of batch candidates reached full bit-plane refinement</td><td>95.88% reached bit-plane refinement, but the complete approximate-distance bound reduced exact coarse reevaluation to 1.45%</td><td>96.03% reached full bit-plane refinement</td></tr> |
| <tr><td>Reuse IVF centroid distance</td><td>Removes 8.2 million repeated rotated centroid terms per 1,000-query run</td><td>Removes 61.4 million repeated terms; end-to-end movement stayed inside run variance because code scan dominates</td><td>Removes 8.2 million repeated padded terms per run</td></tr> |
| <tr><td>16-entry FastScan LUT + NEON/AVX2</td><td>Deliberately bypassed below padded dimension 256</td><td>Versus the optimized scalar scanner: P95 −8.3%, sequential QPS +6.6%, batch QPS +17.8% in paired medians</td><td>Deliberately bypassed below padded dimension 256</td></tr> |
| <tr><td>32-vector single-query seed threshold</td><td>Refinement work −4.3%; P95 −0.9% versus no seed</td><td>Exact final evaluations −33.1%; paired sequential QPS +1.3% while P95 was neutral</td><td>Refinement work −0.32%; P95 −2.3% versus no seed</td></tr> |
| </tbody></table></div> |
| <p>The first statistics prototype incremented counters inside byte-lookup loops and regressed GIST, so it was not retained. The final implementation derives lookup counts once per block or admitted candidate, keeps per-list statistics thread-local, and merges them after parallel work. Centroid reuse derives the RQ query terms from the distances already produced by IVF probing and stores only centroid norms in the Reader. FastScan quantizes two 16-entry nibble tables, evaluates 32 rows with NEON or AVX2, then uses a conservative complete-distance interval before doing exact coarse reevaluation; the final ranking still uses all persisted RQ bit planes and does not require original vectors. Small dimensions stay on the exact scalar byte-LUT path because their public A/B did not justify SIMD setup.</p> |
| <div class="table-wrap"><table><thead><tr><th>Same-file endpoint</th><th>SIFT P95 / sequential QPS / batch QPS</th><th>GIST P95 / sequential QPS / batch QPS</th><th>GloVe P95 / sequential QPS / batch QPS</th></tr></thead><tbody> |
| <tr><td>Pre-change baseline</td><td>1.101 ms / 1,020 / 2,706</td><td>5.599 ms / 203 / 325</td><td>1.071 ms / 1,041 / 2,855</td></tr> |
| <tr><td>Final scanner</td><td>1.081 ms / 1,042 / 2,743</td><td>5.190 ms / 218 / 399</td><td>1.046 ms / 1,057 / 2,839</td></tr> |
| </tbody></table></div> |
| <p>Against its paired baseline, the final GIST scanner improved median P95 by 8.7%, sequential QPS by 8.1%, and batch QPS by 20.0%. SIFT improved P95 by 1.8%, sequential QPS by 2.1%, and batch QPS by 1.4%. GloVe improved P95 by 2.3% and sequential QPS by 1.6%; its 0.6% batch-QPS decrease is treated as noise, not an improvement. File bytes, bytes read, storage format, and compressed-domain ranking are unchanged.</p> |
| <div class="callout"><strong>DiskANN read-path review</strong>DiskANN converts F16 rerank vectors and accumulates L2 directly in one AArch64 NEON loop; its local profile coalesces 16 KiB windows. The final compact-layout batch rerank groups candidate windows with a hash table, then sorts only the unique windows into deterministic I/O order. Against the immediately preceding ordered-map control, median local batch time changed from 117 / 223 / 162 ms to 111 / 215 / 159 ms on SIFT/GIST/GloVe. Local P95 remains 1.50 / 1.83 / 1.90 ms; representative batch throughput is 9,009 / 4,651 / 6,289 QPS with a separate range-I/O executor. Remote and object-store plans remain 32 / 64 KiB. A broader Vec sort/dedup replacement for graph window planners was not retained because it regressed local SIFT/GIST batch time by 7–13%.</div> |
| <div class="callout"><strong>Final open-source cross-check and format decision</strong><a href="https://github.com/facebookresearch/faiss/wiki/Fast-accumulation-of-PQ-and-AQ-codes-%28FastScan%29">Faiss FastScan</a> still trades down to 4-bit lookup tables and a 32-row layout, so it is not a transparent replacement for the published 8-bit IVF-PQ v1. Faiss Panorama's additional level-oriented energy data was not needed to keep the existing IVF-FLAT v1 progressive cutoff. Lance's partition prefetch and transposed PQ storage match the current batched Readers; its prepared transposed L2 target is explicitly aimed at small target sets such as PQ codebooks, not large flat lists. Faiss RaBitQ's blocked multi-bit scan remains structurally aligned with IVF-RQ, while a direct-factor RQ payload experiment regressed SIFT batch throughput by 10–15% and was removed. <a href="https://github.com/microsoft/DiskANN">DiskANN3</a>'s asynchronous provider, beam, and working-set model remains aligned with <code>SeekRead</code>, the latency-derived read planner, and the bounded caches. No measured result justified a v2 migration for IVF-PQ or IVF-FLAT, and no byte-layout change was retained for the pre-release IVF-SQ, IVF-RQ, or DiskANN formats.</div> |
| <p>DiskANN spends almost all build time constructing one global graph: about 18× IVF-FLAT on SIFT, 28× on GIST, and 37× on GloVe. The balanced F16 default makes its files about 42% smaller than IVF-FLAT on GIST and 11% smaller on GloVe, but they remain much larger than the compact IVF encodings because persisted rerank vectors, resident codes, and graph edges are all material. Peak RSS remains below the raw IVF writers because the DiskANN writer does not retain a second full raw-vector organization.</p> |
| |
| <h4>Warm local-storage result</h4> |
| <div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / read</th><th>GIST Recall / P95 / batch QPS / read</th><th>GloVe Recall / P95 / batch QPS / read</th></tr></thead><tbody> |
| <tr><td>IVF-PQ</td><td>0.7142 / 0.72 ms / 7,899 / 2.18 MiB</td><td>0.7410 / 2.47 ms / 950 / 17.84 MiB</td><td>0.5819 / 0.64 ms / 8,048 / 1.84 MiB</td></tr> |
| <tr><td>IVF-SQ</td><td>0.8627 / 0.79 ms / 11,082 / 8.38 MiB</td><td>0.8577 / 3.56 ms / 1,502 / 70.95 MiB</td><td>0.8036 / 0.71 ms / 12,962 / 6.99 MiB</td></tr> |
| <tr><td>IVF-RQ</td><td>0.9148 / 1.20 ms / 3,074 / 5.54 MiB</td><td>0.9039 / 4.41 ms / 444 / 37.02 MiB</td><td>0.8203 / 1.23 ms / 2,917 / 5.89 MiB</td></tr> |
| <tr><td>DiskANN</td><td>0.9915 / 1.50 ms / 9,009 / 0.66 MiB</td><td>0.9336 / 1.83 ms / 4,651 / 0.83 MiB</td><td>0.8355 / 1.90 ms / 6,289 / 0.96 MiB</td></tr> |
| <tr><td>IVF-FLAT</td><td>0.9937 / 1.88 ms / 8,510 / 33.19 MiB</td><td>0.9549 / 11.38 ms / 875 / 283.40 MiB</td><td>0.8832 / 1.40 ms / 9,502 / 27.57 MiB</td></tr> |
| </tbody></table></div> |
| <p>IVF-SQ is the compact-throughput choice: it leads the compact indexes on all three local batch runs. IVF-RQ uses smaller files and raises recall from 0.8627 / 0.8577 / 0.8036 to 0.9148 / 0.9039 / 0.8203, but batch throughput falls by about 3.4–4.4×. IVF-PQ is smaller and faster than RQ, but its 0.5819–0.7410 recall makes it a capacity-first choice rather than a default accuracy compromise. DiskANN is compelling on SIFT and especially GIST: it reads below 1 MiB per query on both, and on GIST greatly outpaces IVF-FLAT. It is not automatically best on GloVe, where IVF-FLAT has higher recall, lower P95, and higher batch throughput at the recorded settings. Treat <code>l_search</code> and <code>nprobe</code> as calibration points whenever the displayed recall misses the production gate.</p> |
| |
| <h4>Remote cache with 2 ms per I/O round</h4> |
| <div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / rounds</th><th>GIST Recall / P95 / batch QPS / rounds</th><th>GloVe Recall / P95 / batch QPS / rounds</th></tr></thead><tbody> |
| <tr><td>IVF-PQ</td><td>0.7142 / 6.16 ms / 7,282 / 1.0</td><td>0.7410 / 7.12 ms / 1,162 / 1.0</td><td>0.5819 / 5.94 ms / 8,243 / 1.0</td></tr> |
| <tr><td>IVF-SQ</td><td>0.8627 / 6.38 ms / 9,345 / 1.0</td><td>0.8577 / 11.36 ms / 1,359 / 1.9</td><td>0.8036 / 6.16 ms / 10,502 / 1.0</td></tr> |
| <tr><td>IVF-RQ</td><td>0.9148 / 7.05 ms / 2,916 / 1.0</td><td>0.9039 / 7.80 ms / 434 / 1.0</td><td>0.8203 / 6.74 ms / 3,080 / 1.0</td></tr> |
| <tr><td>DiskANN</td><td>0.9808 / 15.97 ms / 3,089 / 1.7</td><td>0.8482 / 14.22 ms / 2,175 / 1.5</td><td>0.8029 / 18.18 ms / 2,203 / 2.1</td></tr> |
| <tr><td>IVF-FLAT</td><td>0.9937 / 7.31 ms / 6,763 / 1.0</td><td>0.9549 / 29.73 ms / 846 / 5.0</td><td>0.8832 / 6.60 ms / 7,394 / 1.0</td></tr> |
| </tbody></table></div> |
| <p>IVF-PQ and IVF-RQ load each query's selected compact lists in one concurrent multi-range round. IVF-SQ does the same on SIFT/GloVe; GIST's 960-dimensional payload crosses the 64 MiB per-call guard on 89% of queries and averages 1.9 rounds. IVF-FLAT also uses bounded concurrent multi-range reads: SIFT/GloVe fit in one round, while the 283 MiB GIST payload averages 5.0. IVF-RQ preserves the strongest compact-IVF recall with 7.05 / 7.80 / 6.74 ms P95. Parallel IVF-FLAT has much higher recall and competitive fixed-latency results on SIFT/GloVe, but transfers 28–33 MiB per query; this model does not charge bandwidth. DiskANN's adaptive coalescing and caches reduce the average sequential request count to 1.5–1.7 rounds on SIFT/GIST and 2.1 rounds on GloVe.</p> |
| |
| <h4>Object store with 20 ms per I/O round</h4> |
| <div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / rounds</th><th>GIST Recall / P95 / batch QPS / rounds</th><th>GloVe Recall / P95 / batch QPS / rounds</th></tr></thead><tbody> |
| <tr><td>IVF-PQ</td><td>0.7142 / 20.70 ms / 6,831 / 1.0</td><td>0.7410 / 22.14 ms / 1,054 / 1.0</td><td>0.5819 / 20.62 ms / 8,299 / 1.0</td></tr> |
| <tr><td>IVF-SQ</td><td>0.8627 / 20.77 ms / 6,781 / 1.0</td><td>0.8577 / 43.50 ms / 854 / 1.9</td><td>0.8036 / 20.68 ms / 7,175 / 1.0</td></tr> |
| <tr><td>IVF-RQ</td><td>0.9148 / 21.05 ms / 2,688 / 1.0</td><td>0.9039 / 24.34 ms / 392 / 1.0</td><td>0.8203 / 21.04 ms / 2,767 / 1.0</td></tr> |
| <tr><td>DiskANN</td><td>0.9808 / 60.57 ms / 537 / 1.0</td><td>0.8483 / 41.19 ms / 1,011 / 1.0</td><td>0.8033 / 80.84 ms / 421 / 1.2</td></tr> |
| <tr><td>IVF-FLAT</td><td>0.9937 / 21.80 ms / 3,076 / 1.0</td><td>0.9549 / 130.49 ms / 349 / 5.0</td><td>0.8832 / 21.22 ms / 2,948 / 1.0</td></tr> |
| </tbody></table></div> |
| <p>At 20 ms per round, IVF-RQ is the strongest measured compact one-round option: it reaches 0.90-class recall on SIFT/GIST and 0.8203 on GloVe. IVF-SQ is faster when its lower recall is enough, and IVF-PQ is smaller when stronger quantization loss is acceptable. IVF-FLAT now looks competitive on one-round SIFT/GloVe in this fixed-latency model, but that result assumes 28–33 MiB transfers have no bandwidth cost; GIST's 283 MiB and five rounds expose the boundary. DiskANN averages about one modeled round after warmup, but dependent graph rounds remain visible in P95. A complete local SSD cache remains its preferred deployment.</p> |
| <div class="callout warning"><strong>The automatic read plan affects approximate search</strong>The latency-derived local tier uses graph beam 4 while remote and object-store tiers use beam 16, so the same <code>l_search</code> can return different approximate candidates; this is visible for both GIST and GloVe at <code>l_search=100</code>. The tiers use 16 KiB, 32 KiB, and 64 KiB coalescing windows respectively. Storage latency itself does not change ground truth. Compare indexes with the same latency and capability hints when isolating media effects.</div> |
| <div class="callout warning"><strong>Remote-model boundary</strong>The vectors and exact neighbors are public corpus data, but the 2 ms and 20 ms profiles are controlled I/O models rather than measurements from a production cache or object store. They add fixed latency without modeling bandwidth, cache misses, TLS, retries, throttling, request limits, or tail-latency variance.</div> |
| </section> |
| |
| <section class="section" id="decision"> |
| <div class="section-heading"> |
| <h2>Choose by constraint</h2> |
| <p>There is no best index independent of data distribution. Narrow the field to one or two candidates, then evaluate Recall@K, P95/P99 latency, file size, build time, and object-store bytes on real queries.</p> |
| </div> |
| <div class="callout"><strong>Practical default order</strong>First reject indexes that cannot meet the measured recall target. Build IVF-FLAT to establish the corpus-specific IVF ceiling. If a compact representation is required, choose IVF-SQ for throughput, IVF-RQ for recall, or IVF-PQ for minimum bytes. Evaluate DiskANN separately for immutable data served from local SSD; do not select it only because the collection is large or assume L2 results transfer to another metric.</div> |
| <h3>Measured recommendation matrix</h3> |
| <div class="table-wrap"><table><thead><tr><th>Production constraint</th><th>Start with</th><th>Evidence from this run</th><th>Move away when</th></tr></thead><tbody> |
| <tr><td>Establish a recall ceiling or debug ranking quality</td><td><a href="ivf-flat.html">IVF-FLAT</a></td><td>Highest measured recall on all three corpora: 0.9937 / 0.9549 / 0.8832, with roughly four-second SIFT/GloVe builds.</td><td>The 28–283 MiB selected-list reads or raw-vector file size exceed the serving budget.</td></tr> |
| <tr><td>Highest compact batch throughput</td><td><a href="ivf-sq.html">IVF-SQ</a></td><td>11,082 / 1,502 / 12,962 local batch QPS at 0.8627 / 0.8577 / 0.8036 recall; files are about one quarter of IVF-FLAT.</td><td>The recall gate is above SQ, or one byte per dimension is still too large.</td></tr> |
| <tr><td>Strongest recall in a compact IVF file</td><td><a href="ivf-rq.html">IVF-RQ</a></td><td>0.9148 / 0.9039 / 0.8203 recall in files smaller than IVF-SQ, with one sequential read round per query in all three modeled profiles.</td><td>Batch throughput is the primary SLO; the four-bit scanner is 3–4.5× slower than SQ in the local run.</td></tr> |
| <tr><td>Minimum index file and compact-IVF scan bytes</td><td><a href="ivf-pq.html">IVF-PQ</a></td><td>The smallest files—0.032 / 0.230 / 0.030 GiB—and the smallest IVF selected-list reads at 1.84–17.84 MiB, with strong batch throughput.</td><td>0.5819–0.7410 recall is below the gate; increase the PQ budget or choose SQ/RQ instead.</td></tr> |
| <tr><td>High-recall immutable data on local SSD</td><td><a href="diskann.html">DiskANN</a>, checked against IVF-FLAT for the same metric</td><td>The recorded L2-equivalent run reached 0.9915 / 0.9336 / 0.8355 recall with 0.66 / 0.83 / 0.96 MiB reads; SIFT/GIST P95 is 1.50 / 1.83 ms.</td><td>Metric-specific recall misses the gate, rebuilds are frequent, the file is not locally cached, preview maturity is unacceptable, or the corpus behaves like GloVe at <code>l_search=100</code>.</td></tr> |
| <tr><td>Frequent rebuilds or rapidly changing snapshots</td><td><a href="ivf-flat.html">IVF-FLAT</a>, <a href="ivf-sq.html">IVF-SQ</a>, or <a href="ivf-rq.html">IVF-RQ</a></td><td>These build in about 4 seconds on SIFT/GloVe and 23–25 seconds on GIST; IVF-PQ is about 2× slower and DiskANN is 18–37× slower than IVF-FLAT.</td><td>The serving phase dominates lifetime cost enough to justify PQ training or graph construction.</td></tr> |
| <tr><td>Direct 2/20 ms remote or object-store reads</td><td>Compact IVF selected by recall: PQ → SQ → RQ</td><td>PQ and RQ use one sequential multi-range round here; SQ does so on SIFT/GloVe and averages 1.9 rounds on GIST. Choose successively more recall at greater bytes or CPU cost.</td><td>Bandwidth, request limits, or real tail latency invalidate the fixed-latency model; prefer a complete local SSD cache and rerun the benchmark.</td></tr> |
| <tr><td>Inner product or cosine</td><td>IVF-FLAT as the recall control; DiskANN as an additional candidate for immutable local-SSD serving</td><td>All five implementations support L2, IP, and cosine. DiskANN normalizes cosine internally and uses metric-aware graph construction and exact reranking, but the displayed public-corpus matrix was recorded through the L2-equivalent benchmark path.</td><td>The selected configuration misses its metric-specific recall gate—retune <code>nprobe</code>, representation width, OPQ, or <code>l_search</code> before deployment.</td></tr> |
| </tbody></table></div> |
| <div class="callout warning"><strong>A displayed winner can still be the wrong choice</strong>These recommendations apply to the recorded <code>nlist=1024</code>, <code>nprobe=64</code>, PQ ratio, RQ bits, and <code>l_search=100</code>. For example, the current GloVe run does not reach 0.90 recall with any index, and current GIST reaches 0.95 only with IVF-FLAT. If a required recall threshold is not present in the table, tune and rebuild rather than choosing the closest result.</div> |
| <div class="card-grid"> |
| <article class="card"><h3>I need a trustworthy baseline</h3><p>Start with IVF-FLAT. It exposes the IVF partition ceiling without quantization loss and rebuilds quickly.</p><a class="card-link" href="ivf-flat.html">Explore IVF-FLAT →</a></article> |
| <article class="card"><h3>I need compact high recall</h3><p>Choose IVF-RQ when its 0.82–0.91 measured recall matters more than batch throughput; compare every result with the IVF-FLAT ceiling.</p><a class="card-link" href="ivf-rq.html">Explore IVF-RQ →</a></article> |
| <article class="card"><h3>I need the smallest index</h3><p>Choose IVF-PQ when its corpus-specific recall passes the gate. It is the capacity-first option, not the automatic middle ground.</p><a class="card-link" href="ivf-pq.html">Explore IVF-PQ →</a></article> |
| <article class="card"><h3>I need compact batch speed</h3><p>Choose IVF-SQ when one byte per dimension fits and its measured 0.80–0.86 recall is enough; it is the fastest compact scanner here.</p><a class="card-link" href="ivf-sq.html">Explore IVF-SQ →</a></article> |
| <article class="card"><h3>Raw vectors exceed RAM but fit local SSD</h3><p>Evaluate DiskANN for immutable L2, IP, or cosine data when high recall and sub-MiB query reads justify a much slower build; retain IVF-FLAT as the metric-specific accuracy control.</p><a class="card-link" href="diskann.html">Explore DiskANN →</a></article> |
| <article class="card"><h3>Data lives in S3, OSS, or HDFS</h3><p>Prefer durable publication plus a complete local SSD cache. For direct remote reads, start with a compact IVF index when one-round scans meet recall; use DiskANN only after measuring its corpus-dependent coalesced graph rounds.</p><a class="card-link" href="diskann.html#deployment">Compare deployment modes →</a></article> |
| </div> |
| </section> |
| |
| <section class="section" id="parameters"> |
| <div class="section-heading"><h2>How parameters interact</h2><p>Build parameters define static structures; query parameters define per-request work. For IVF, changing <code>nlist</code> usually changes the useful <code>nprobe</code> range. DiskANN instead couples graph build quality with online <code>l_search</code>.</p></div> |
| <div class="callout"><strong>Automate numeric work, keep semantics explicit</strong><code>index.type</code> and <code>metric</code> remain required because changing either changes persistence and result meaning. Rust callers can use <code>recommend_index</code> as an advisory starting point and must explicitly accept its result. For measured offline sweeps, <code>select_calibrated_candidate</code> chooses the smallest candidate satisfying supplied recall, byte, and build-time objectives and returns no result when the sample does not meet them.</div> |
| <div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Stage</th><th>Indexes</th><th>Typical effect when increased</th><th>Constraint / default</th></tr></thead><tbody> |
| <tr><td><code>index.type</code></td><td>Build</td><td>All</td><td>Changes the persisted algorithm and its capability boundary</td><td>Required; recommendation is advisory, never silently applied</td></tr> |
| <tr><td><code>metric</code></td><td>Build</td><td>All</td><td>Changes training, ranking, and ground-truth semantics</td><td>Required and never inferred</td></tr> |
| <tr><td><code>dimension</code></td><td>Build</td><td>All</td><td>Changes representation width and distance work</td><td>Inferred by Java/Python one-shot training; required by streaming APIs</td></tr> |
| <tr><td><code>nlist</code></td><td>Build</td><td>IVF families</td><td>Shorter lists and more coarse centroids; a fixed <code>nprobe</code> covers less of the collection</td><td>Auto: nearest power of two around √N, with at least 64 rows of training density per list; requires <code>expected-vector-count</code></td></tr> |
| <tr><td><code>nprobe</code></td><td>Query</td><td>IVF families</td><td>Reads more lists; recall usually rises with latency and I/O</td><td>Auto is K-, N-, nlist-, and filter-selectivity-aware; explicit values are expert overrides</td></tr> |
| <tr><td><code>pq.code-ratio</code></td><td>Build</td><td>IVF-PQ, DiskANN</td><td>Raises or lowers the automatically inferred code bytes and subquantizer count</td><td>Default 0.0625; finite and positive</td></tr> |
| <tr><td><code>pq.m</code></td><td>Build</td><td>IVF-PQ, DiskANN</td><td>Expert override for the inferred subquantizer count; larger values often reduce quantization error but add lookup work</td><td>Optional; <code>d % m == 0</code></td></tr> |
| <tr><td><code>rq.bits</code></td><td>Build</td><td>IVF-RQ</td><td>More persisted bit planes improve reconstruction and usually recall while increasing file bytes, I/O, and scan work</td><td>Auto from <code>max-bytes-per-vector</code>; otherwise <code>4</code></td></tr> |
| <tr><td><code>use-opq</code></td><td>Build</td><td>IVF-PQ</td><td>Adds training and matrix cost; may improve PQ quality</td><td>Auto enables at <code>target-recall ≥ 0.9</code>; explicit true/false wins</td></tr> |
| <tr><td><code>target-recall</code></td><td>Build objective</td><td>IVF-PQ, DiskANN</td><td>Selects OPQ and a coherent DiskANN build preset</td><td>Starting policy only; validate measured recall on held-out queries</td></tr> |
| <tr><td><code>max-bytes-per-vector</code></td><td>Build objective and preflight bound</td><td>IVF-PQ, IVF-RQ, DiskANN</td><td>Reduces code width and may select 4-bit/F16 DiskANN storage; rejects configurations whose conservative persisted-size estimate exceeds the bound</td><td>Includes estimated row bytes and, when <code>expected-vector-count</code> is set, amortized fixed data; not an exact final-file-size promise</td></tr> |
| <tr><td><code>max-build-seconds</code></td><td>Offline calibration objective</td><td>Measured candidate sets</td><td>Rejects candidates whose measured build time exceeds the target</td><td>Accepted by <code>VectorIndexBuildPlan</code>; direct Trainer creation rejects it because build time cannot be safely guessed from hardware</td></tr> |
| <tr><td><code>diskann.build-preset</code></td><td>Build</td><td>DiskANN</td><td>Moves together across degree, construction width, encoding, and build distance</td><td><code>fast_build</code>, <code>balanced</code>, or <code>high_recall</code>; inferred from target recall</td></tr> |
| <tr><td><code>deployment-profile</code></td><td>Build</td><td>DiskANN</td><td>Selects interleaved layout for eligible memory/local serving and compact layout for remote/object serving</td><td>Explicit layout/encoding/build-distance overrides always win</td></tr> |
| <tr><td><code>estimated_random_read_latency_nanos</code></td><td>Reader input capability</td><td>DiskANN</td><td>Selects the internal read window, graph beam, and automatic cache partition without probe I/O</td><td>0 measures the mandatory header read; positive values are useful for known remote/cache latency</td></tr> |
| <tr><td><code>l_search</code></td><td>Query</td><td>DiskANN</td><td>Larger DiskANN candidate list, usually higher recall and latency</td><td>Auto uses calibrated 100/200/400 when available, otherwise <code>max(100, 2k)</code></td></tr> |
| <tr><td><code>memory_budget_bytes</code></td><td>Reader</td><td>DiskANN</td><td>Controls required resident state plus automatically partitioned adjacency/raw-vector caches</td><td>4 GiB; cache sub-budgets are internal</td></tr> |
| </tbody></table></div> |
| </section> |
| |
| <section class="section" id="io"> |
| <div class="section-heading"><h2>Data-lake storage and I/O</h2><p>IVF files begin with a 64-byte v1 header and use model/list sections. DiskANN uses a 256-byte header, page-aligned resident/adjacency data, and either densely packed compact vector records or interleaved page-contained records. The Reader dispatches on the first four-byte magic and uses positional reads for both layouts.</p></div> |
| <div class="metric-strip"><div class="metric"><span class="label">Byte order</span><span class="value">Little-endian</span></div><div class="metric"><span class="label">Type discovery</span><span class="value">First 4-byte magic</span></div><div class="metric"><span class="label">Row IDs</span><span class="value">IVF delta varints / DiskANN adaptive packed</span></div><div class="metric"><span class="label">Integrity</span><span class="value">Outer Paimon file layer</span></div></div> |
| <div class="callout warning"><strong>Format boundary</strong>v1 files have no footer, checksum, compression envelope, or schema registry. Roaring filters are query payloads and are not embedded. Readers reject unknown versions, required flags, non-zero reserved bytes, and malformed sections.</div> |
| </section> |
| |
| <section class="section" id="evaluation"> |
| <div class="section-heading"><h2>A practical evaluation order</h2><p>Fix the dataset and query set, then introduce approximation one layer at a time. This makes it possible to attribute loss to IVF selection, vector quantization, or graph traversal.</p></div> |
| <div class="pipeline"> |
| <div class="pipeline-item"><span class="pipeline-index">1</span><div><h3>Build ground truth</h3><p>Generate exact top K using the production metric, realistic filters, and edge cases such as zero vectors.</p></div></div> |
| <div class="pipeline-item"><span class="pipeline-index">2</span><div><h3>Measure IVF-FLAT</h3><p>Establish the recall ceiling caused by probing only <code>nprobe</code> lists and record bytes read.</p></div></div> |
| <div class="pipeline-item"><span class="pipeline-index">3</span><div><h3>Compare compression</h3><p>Use the same <code>nlist/nprobe</code> for IVF-SQ, IVF-PQ, and IVF-RQ to isolate added approximation and storage savings.</p></div></div> |
| <div class="pipeline-item"><span class="pipeline-index">4</span><div><h3>Evaluate disk-backed search</h3><p>When raw data exceeds RAM, compare DiskANN on local SSD and realistic remote storage, including cold/warm caches and read rounds.</p></div></div> |
| <div class="pipeline-item"><span class="pipeline-index">5</span><div><h3>Apply production budgets</h3><p>Set thresholds for Recall@K, P99, file size, build time, RSS, and remote bytes rather than optimizing average latency alone.</p></div></div> |
| </div> |
| </section> |
| </div> |
| </main> |
| <footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index · Implementation guide</span><span>Based on the current Rust core and v1 storage format</span></div></footer> |
| </body> |
| </html> |
