| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to you under the Apache License, Version 2.0. |
| --> |
| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="description" content="Unified Paimon Vector Index APIs for Rust, C, C++, Java, and Python, including warm-up and metadata filter pushdown."> |
| <title>API and Language Bindings · Paimon Vector Index</title> |
| <link rel="stylesheet" href="styles.css"> |
| <script src="docs.js" defer></script> |
| </head> |
| <body> |
| <a class="skip-link" href="#main">Skip to content</a> |
| <header class="site-header"><div class="header-inner"> |
| <a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"><span class="brand-mark">VI</span><span>Paimon Vector Index</span></a> |
| <nav class="site-nav" data-site-nav aria-label="Documentation"><a href="index.html">Overview</a><a href="api.html" aria-current="page">API</a><a href="development.html">Development</a><a href="ivf-flat.html">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a href="ivf-sq.html">IVF-SQ</a><a href="diskann.html">DiskANN</a><a href="releases.html">Releases</a></nav> |
| <div class="header-actions"><button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button><button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button></div> |
| </div></header> |
| |
| <main id="main"><div class="page-shell"> |
| <section class="hero detail-hero"> |
| <p class="eyebrow">One lifecycle · five language surfaces</p> |
| <h1>API and language bindings</h1> |
| <p class="hero-lead">The Rust core, C FFI, C++ RAII layer, Java/JNI, and Python ctypes package share one train → write → serialize → detect → search model. Build options select the index type; Readers discover it from the file magic.</p> |
| <div class="badge-row"><span class="badge strong">Unified lifecycle</span><span class="badge">Positional I/O</span><span class="badge">Single and batch search</span><span class="badge">Roaring64 prefilter</span></div> |
| </section> |
| |
| <div class="doc-layout"> |
| <aside class="toc" aria-label="On this page"><strong>On this page</strong><a href="#workspace">Modules</a><a href="#lifecycle">Lifecycle</a><a href="#params">Search parameters</a><a href="#reader-options">Reader options</a><a href="#warmup">Warm-up</a><a href="#rust">Rust</a><a href="#c">C FFI</a><a href="#cpp">C++</a><a href="#java">Java/JNI</a><a href="#python">Python</a><a href="#filter">Filter pushdown</a></aside> |
| <article class="article"> |
| <section class="article-section" id="workspace"> |
| <h2>Public integration layers</h2> |
| <div class="table-wrap"><table><thead><tr><th>Module</th><th>Role</th><th>Primary entry point</th></tr></thead><tbody> |
| <tr><td><code>core</code></td><td>Pure Rust indexes, file Readers/Writers, and benchmarks</td><td><code>paimon_vindex_core::index</code></td></tr> |
| <tr><td><code>ffi</code></td><td>C ABI over the Rust core; builds <code>libpaimon_vindex_ffi</code></td><td>Generated <code>include/paimon_vindex.h</code></td></tr> |
| <tr><td><code>include</code></td><td>C++ RAII wrapper</td><td><code>paimon_vindex.hpp</code></td></tr> |
| <tr><td><code>jni</code> + <code>java</code></td><td>JNI implementation and Java API</td><td><code>org.apache.paimon.index.vector</code></td></tr> |
| <tr><td><code>python</code></td><td>Pure Python package loading the C FFI through ctypes</td><td><code>paimon_vindex</code></td></tr> |
| </tbody></table></div> |
| <p>The top-level Cargo workspace contains <code>core</code>, <code>ffi</code>, and <code>jni</code>. The Python package loads the shared FFI library at runtime.</p> |
| </section> |
| |
| <section class="article-section" id="lifecycle"> |
| <h2>Shared lifecycle</h2> |
| <div class="flow" aria-label="Unified API lifecycle"><div class="flow-step"><small>01</small><strong>Create a Trainer<br>Parse and validate options</strong></div><div class="flow-step"><small>02</small><strong>Submit one or more<br>training batches</strong></div><div class="flow-step"><small>03</small><strong>Finish training and<br>create a one-shot Writer</strong></div><div class="flow-step"><small>04</small><strong>Add row IDs / vectors<br>and write the file</strong></div><div class="flow-step"><small>05</small><strong>Detect file magic<br>and execute searches</strong></div></div> |
| <ul><li>Vectors are contiguous <code>f32</code> values; length must equal <code>vector_count × dimension</code>.</li><li>Training data may arrive in batches. Every IVF trainer keeps a deterministic reservoir of at most <code>max(65,536, 64 × resolved nlist)</code> vectors. DiskANN starts from a 50,000-row cap and lowers it when necessary so the retained sample, optional cosine-normalized copy, codebook, and parallel PQ-training scratch fit <code>diskann.memory-budget-bytes</code>. Sampling is independent of batch boundaries.</li><li>The Python and Java one-shot <code>train</code> helpers infer <code>dimension</code> from the matrix and use its row count for automatic <code>nlist</code>. When the matrix is only a sample, pass the final corpus size as <code>expected-vector-count</code>. Streaming Trainer APIs require a concrete dimension before their first batch.</li><li>A Writer may receive production vectors in multiple batches. Row-ID count must equal vector count.</li><li>Readers expose metadata, single-query search, batch search, and Roaring64-filtered variants.</li><li>Files carry their type and resolved model sections. Callers do not pass index options again when opening a Reader.</li></ul> |
| </section> |
| |
| <section class="article-section" id="params"> |
| <h2>Shared search parameters</h2> |
| <div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Applies to</th><th>Description</th></tr></thead><tbody> |
| <tr><td><code>top_k</code></td><td>All indexes</td><td>Number of nearest neighbors returned for each query.</td></tr> |
| <tr><td><code>Auto</code> width</td><td>All indexes</td><td>IVF starts at <code>max(8, ceil(nlist/16))</code>, adds enough average-list capacity for at least <code>4 × top_k</code> candidates, scales for filter selectivity, and progressively doubles when a filtered result is short. DiskANN uses a calibrated width when present, otherwise <code>max(100, 2 × top_k)</code>.</td></tr> |
| <tr><td><code>nprobe</code></td><td>IVF families</td><td>Explicit expert override for the number of lists probed. A tagged IVF width is rejected by DiskANN rather than silently ignored.</td></tr> |
| <tr><td><code>l_search</code></td><td>DiskANN</td><td>Explicit expert override for graph search-list size; it is clamped to at least <code>top_k</code>. A tagged DiskANN width is rejected by IVF indexes.</td></tr> |
| </tbody></table></div> |
| <p>Use the binding's <code>automatic</code> search-parameter constructor. For DiskANN, <code>calibrate_search_width</code> / <code>calibrateSearchWidth</code> evaluates widths 100, 200, and 400 on representative queries and remembers the smallest adjacent pair with at least 98% Top-K result overlap. This is a stability proxy, not a ground-truth recall guarantee; explicit widths always win.</p> |
| <div class="callout"><strong>IVF-RQ width is a build option</strong><code>rq.bits</code> accepts 1–8 and defaults to 4. It is persisted in the file and reported as <code>rq_bits</code> / <code>rqBits</code> metadata; search has no separate bit-width switch.</div> |
| </section> |
| |
| <section class="article-section" id="reader-options"> |
| <h2>Reader options for DiskANN</h2> |
| <p>Reader options are accepted by every binding and affect DiskANN only. Other index families continue to use their existing positional-I/O behavior.</p> |
| <div class="table-wrap"><table><thead><tr><th>Concept</th><th>Rust / Python</th><th>C / C++ / Java</th><th>Default</th></tr></thead><tbody> |
| <tr><td>Random-read latency hint</td><td>Input capability <code>estimated_random_read_latency_nanos</code></td><td>C/C++ field / Java <code>estimatedRandomReadLatencyNanos()</code></td><td>0: reuse the mandatory header read's elapsed time; positive values bypass measurement</td></tr> |
| <tr><td>Total Reader memory</td><td><code>memory_budget_bytes</code></td><td><code>memory_budget_bytes</code> / constructor argument</td><td>4 GiB; automatically partitioned among required resident state, a profile-sized adjacency prefix, a bounded cold-adjacency LRU, and a bounded raw-vector LRU</td></tr> |
| </tbody></table></div> |
| <p>The Reader has no public storage-tier switch. DiskANN selects an internal tier once during <code>open</code> from the latency hint or mandatory header-read timing; its latency, window, and beam policy then remain stable. <code>read_plan</code> / <code>readPlan</code> exposes that policy together with the current effective preload and shared-cache capacities. Those capacities are zero before resident initialization and may shrink when lazy row-ID lookup state consumes the same total budget. Cache partitioning remains an internal decision. The input callback receives all positional ranges for one round together and should execute them concurrently.</p> |
| <p>A storage adapter may additionally advertise <code>preferred_window_bytes</code> and a maximum range count named <code>max_ranges_per_pread</code> in Rust or <code>max_ranges_per_read</code> in C/C++/Python. Zero means unspecified. DiskANN rounds the requested window to complete 4 KiB logical pages, bounds it to 1 MiB, limits each <code>pread</code> batch to the advertised range count, and keeps physical alignment concerns inside the storage adapter. Build-time <code>deployment-profile</code> remains separate because it can select the persisted compact or interleaved layout. See <a href="diskann.html#reader-parameters">DiskANN reader tuning</a>.</p> |
| </section> |
| |
| <section class="article-section" id="warmup"> |
| <h2>Search warm-up</h2> |
| <p>After opening a Reader and before repeated searches, initialize resident state and, for DiskANN, optionally replay a small representative query set. Warm-up builds process-local caches without changing the file or search results.</p> |
| <div class="table-wrap"><table><thead><tr><th>Language</th><th>Resident initialization</th><th>Representative-query warm-up</th><th>DiskANN width calibration</th></tr></thead><tbody> |
| <tr><td>Rust</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr> |
| <tr><td>C</td><td><code>paimon_vindex_reader_optimize_for_search</code></td><td><code>paimon_vindex_reader_warmup_queries</code></td><td><code>paimon_vindex_reader_calibrate_search_width</code></td></tr> |
| <tr><td>C++</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr> |
| <tr><td>Java</td><td><code>optimizeForSearch</code></td><td><code>warmupQueries</code></td><td><code>calibrateSearchWidth</code></td></tr> |
| <tr><td>Python</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr> |
| </tbody></table></div> |
| <p><code>optimize_for_search</code> is optional for correctness: the first search performs the same initialization lazily. It builds IVF-PQ residual-L2 tables or DiskANN resident PQ/row-ID state and its automatically budgeted hot adjacency prefix. DiskANN <code>warmup_queries</code> then executes top-1 graph traversal and persisted-vector rerank for each supplied query, priming immutable adjacency and raw-vector LRUs. Other index families treat representative warm-up as resident initialization only.</p> |
| </section> |
| |
| <section class="article-section" id="rust"> |
| <h2>Rust</h2> |
| <div class="code-block"><span class="code-label">Rust · train, write, and search</span><pre><code>use std::fs::File; |
| |
| use paimon_vindex_core::distance::MetricType; |
| use paimon_vindex_core::index::{ |
| VectorIndexConfig, VectorIndexReader, VectorIndexTrainer, |
| VectorIndexWriter, VectorSearchParams, |
| }; |
| use paimon_vindex_core::io::PosWriter; |
| |
| let config = VectorIndexConfig::IvfSq { |
| dimension: 128, |
| nlist: 1024, |
| metric: MetricType::L2, |
| }; |
| |
| let training = VectorIndexTrainer::train( |
| config, &training_vectors, training_count)?; |
| let mut writer = VectorIndexWriter::new(training); |
| writer.add_vectors(&row_ids, &vectors, vector_count)?; |
| |
| let mut file = File::create("vectors.pvindex")?; |
| let mut out = PosWriter::new(&mut file); |
| writer.write(&mut out)?; |
| |
| let file = File::open("vectors.pvindex")?; |
| let mut reader = VectorIndexReader::open(file)?; |
| reader.optimize_for_search()?; |
| let params = VectorSearchParams::automatic(10); |
| let (ids, distances) = reader.search(&query, params)?;</code></pre></div> |
| <div class="code-block"><span class="code-label">Rust · other configurations</span><pre><code>VectorIndexConfig::IvfFlat { |
| dimension: 128, nlist: 1024, metric: MetricType::L2, |
| }; |
| VectorIndexConfig::ivf_pq( |
| 128, 1024, MetricType::L2, false, |
| )?; |
| VectorIndexConfig::IvfRq { |
| dimension: 128, nlist: 1024, bits: 4, metric: MetricType::L2, |
| }; |
| VectorIndexConfig::IvfSq { |
| dimension: 128, nlist: 1024, metric: MetricType::L2, |
| };</code></pre></div> |
| <p>The IVF-PQ constructor uses the default relative PQ-code budget and resolves a concrete <code>m</code>. In every option-map API, <code>pq.m</code> is optional: <code>pq.code-ratio=0.0625</code> is the default, and an explicit <code>pq.m</code> takes precedence. Metadata and the on-disk header expose the resolved value.</p> |
| </section> |
| |
| <section class="article-section" id="c"> |
| <h2>C FFI</h2> |
| <p>The C ABI builds <code>libpaimon_vindex_ffi</code>; cbindgen produces the public header. Status-returning operations return <code>0</code> on success and <code>-1</code> on failure. Handle-producing functions such as <code>paimon_vindex_trainer_open()</code>, <code>paimon_vindex_trainer_finish()</code>, <code>paimon_vindex_writer_open()</code>, and <code>paimon_vindex_reader_open()</code> return a handle on success or <code>NULL</code> on failure. The corresponding free functions return <code>void</code> and accept <code>NULL</code>. After a failure, <code>paimon_vindex_last_error()</code> returns the thread-local diagnostic string, or <code>NULL</code> if no error has been recorded.</p> |
| <div class="code-block"><span class="code-label">C · complete lifecycle</span><pre><code>#include "paimon_vindex.h" |
| |
| const char *keys[] = {"index.type", "dimension", "nlist", "metric"}; |
| const char *values[] = {"ivf_flat", "128", "1024", "l2"}; |
| |
| PaimonVindexTrainerHandle *trainer = |
| paimon_vindex_trainer_open(keys, values, 4); |
| paimon_vindex_trainer_add_training_vectors( |
| trainer, training_vectors, training_count); |
| PaimonVindexTrainingHandle *training = |
| paimon_vindex_trainer_finish(trainer); |
| paimon_vindex_trainer_free(trainer); |
| |
| PaimonVindexWriterHandle *writer = paimon_vindex_writer_open(training); |
| paimon_vindex_training_free(training); |
| paimon_vindex_writer_add_vectors(writer, row_ids, vectors, vector_count); |
| paimon_vindex_writer_write_index(writer, output_file); |
| paimon_vindex_writer_free(writer); |
| |
| PaimonVindexReaderHandle *reader = paimon_vindex_reader_open(input_file); |
| PaimonVindexMetadata metadata; |
| paimon_vindex_reader_metadata(reader, &metadata); |
| paimon_vindex_reader_optimize_for_search(reader); |
| |
| int64_t ids[10]; |
| float distances[10]; |
| PaimonVindexSearchParams params = { |
| .top_k = 10, |
| .search_width = PAIMON_VINDEX_SEARCH_WIDTH_AUTO, |
| .width = 0, |
| }; |
| paimon_vindex_reader_warmup_queries(reader, representative_queries, 8, 0); |
| paimon_vindex_reader_search(reader, query, params, ids, distances, 10); |
| paimon_vindex_reader_free(reader);</code></pre></div> |
| <p><code>PaimonVindexOutputFile</code> and <code>PaimonVindexInputFile</code> are callback structures. The input callback receives every positional range in one I/O batch, allowing object-store implementations to issue reads concurrently. DiskANN batch search may also invoke the callback concurrently from separate query workers, so the callback and its context must be thread-safe. The input structure also carries the three optional read-capability hints described above. C callers own result buffers and pass their capacity explicitly. Metadata reports <code>pq_m</code>/<code>pq_bits</code> for PQ-backed indexes and <code>rq_bits</code> for IVF-RQ. IVF-SQ reports <code>pq_m=0</code> and <code>pq_bits=8</code> to expose its scalar-code width.</p> |
| </section> |
| |
| <section class="article-section" id="cpp"> |
| <h2>C++ RAII</h2> |
| <p><code>include/paimon_vindex.hpp</code> wraps the C ABI with explicit ownership and automatic cleanup.</p> |
| <div class="code-block"><span class="code-label">C++</span><pre><code>#include "paimon_vindex.hpp" |
| |
| std::vector<std::pair<std::string, std::string>> options = { |
| {"index.type", "ivf_flat"}, {"dimension", "128"}, |
| {"nlist", "1024"}, {"metric", "l2"}, |
| }; |
| |
| paimon::vindex::Training training = |
| paimon::vindex::Trainer::train( |
| options, training_vectors.data(), training_count); |
| paimon::vindex::Writer writer(std::move(training)); |
| writer.add_vectors(row_ids.data(), vectors.data(), vector_count); |
| writer.write_index(output_file); |
| |
| paimon::vindex::Reader reader(input_file); |
| auto metadata = reader.metadata(); |
| reader.optimize_for_search(); |
| reader.warmup_queries(representative_queries.data(), 8); |
| auto result = reader.search( |
| query.data(), paimon::vindex::SearchParams::automatic(10));</code></pre></div> |
| </section> |
| |
| <section class="article-section" id="java"> |
| <h2>Java / JNI</h2> |
| <p>The package is <code>org.apache.paimon.index.vector</code>. String options map directly to Paimon table and index properties; Rust parses and validates them when a Trainer is created.</p> |
| <p>The Maven JAR contains the supported Linux x86-64, Linux aarch64, macOS arm64, and Windows x86-64 JNI libraries. The first native API call selects the current platform, extracts that library to a private temporary file, and loads it automatically. To use a separately built library instead, start the JVM with <code>-Dpaimon.vindex.native.path=/absolute/path/to/the/library</code>.</p> |
| <div class="code-block"><span class="code-label">Java · build and search</span><pre><code>Map<String, String> options = new HashMap<>(); |
| options.put("index.type", "ivf_sq"); |
| options.put("metric", "l2"); |
| |
| try (VectorIndexTraining training = |
| VectorIndexTrainer.train(options, trainingVectors, trainingCount); |
| VectorIndexWriter writer = new VectorIndexWriter(training)) { |
| writer.addVectors(rowIds, vectors, vectorCount); |
| writer.writeIndex(vectorIndexOutput); |
| } |
| |
| try (VectorIndexReader reader = new VectorIndexReader(vectorIndexInput)) { |
| VectorIndexMetadata metadata = reader.metadata(); |
| reader.optimizeForSearch(); |
| VectorSearchResult result = reader.search( |
| query, VectorSearchParams.automatic(10)); |
| }</code></pre></div> |
| <h3>Batching a large training set</h3> |
| <div class="code-block"><span class="code-label">Java · Trainer-owned native staging</span><pre><code>try (VectorIndexTrainer trainer = VectorIndexTrainer.create(options)) { |
| for (float[] batch : trainingBatches) { |
| trainer.addTrainingVectors(batch, batch.length / dimension); |
| } |
| try (VectorIndexTraining training = trainer.finishTraining(); |
| VectorIndexWriter writer = new VectorIndexWriter(training)) { |
| writer.addVectors(rowIds, vectors, vectorCount); |
| writer.writeIndex(vectorIndexOutput); |
| } |
| }</code></pre></div> |
| <p>Staging avoids one very large Java <code>float[]</code> and its array-length limit. Native reservoir sampling bounds retained training rows, but streaming creation still needs concrete <code>dimension</code> and either concrete <code>nlist</code> or <code>expected-vector-count</code>.</p> |
| </section> |
| |
| <section class="article-section" id="python"> |
| <h2>Python</h2> |
| <p>The C++, Java, and Python Readers serialize operations on one native handle across threads. Calling another method on that same handle from its input/output callback is rejected as reentrant—even when DiskANN invokes the callback from a native query worker—instead of deadlocking or re-entering a mutable native handle; use a separate handle if a callback needs vector-index work. The Python package loads <code>libpaimon_vindex_ffi</code> with ctypes. A single search returns one-dimensional NumPy arrays. <code>search_batch</code> accepts a two-dimensional query array and returns arrays shaped <code>(query_count, top_k)</code>.</p> |
| <div class="code-block"><span class="code-label">Python</span><pre><code>from paimon_vindex import ( |
| SearchParams, VectorIndexReader, |
| VectorIndexTrainer, VectorIndexWriter, |
| ) |
| |
| class VectorIndexInput: |
| def __init__(self, data: bytes): |
| self.data = data |
| self.estimated_random_read_latency_nanos = 20_000_000 |
| self.preferred_window_bytes = 64 * 1024 |
| self.max_ranges_per_read = 32 |
| |
| def pread_many(self, ranges): |
| return [self.data[pos : pos + length] for pos, length in ranges] |
| |
| options = {"index.type": "ivf_sq", "metric": "l2"} |
| training = VectorIndexTrainer.train(options, training_vectors) |
| writer = VectorIndexWriter(training) |
| writer.add_vectors(row_ids, vectors) |
| writer.write(output) |
| |
| reader = VectorIndexReader(VectorIndexInput(index_bytes)) |
| reader.optimize_for_search() |
| reader.warmup_queries(representative_queries) |
| ids, distances = reader.search( |
| query, SearchParams.automatic(top_k=10))</code></pre></div> |
| </section> |
| |
| <section class="article-section" id="filter"> |
| <h2>Metadata filter pushdown</h2> |
| <p>The query layer can evaluate metadata predicates through Paimon table or scalar indexes, serialize the allowed row IDs as a 64-bit Roaring bitmap, and pass that set into ANN search as a prefilter. Every binding uses the same wire format:</p> |
| <div class="table-wrap"><table><thead><tr><th>Language</th><th>Single / batch entry points</th></tr></thead><tbody> |
| <tr><td>Rust</td><td><code>search_with_roaring_filter</code> / <code>search_batch_with_roaring_filter</code></td></tr> |
| <tr><td>C</td><td><code>paimon_vindex_reader_search_with_roaring_filter</code> / <code>...search_batch_with_roaring_filter</code></td></tr> |
| <tr><td>Java</td><td><code>VectorIndexReader.search(..., byte[])</code> / <code>searchBatch(..., byte[])</code></td></tr> |
| <tr><td>Python</td><td><code>search(..., filter_bytes=...)</code> / <code>search_batch(..., filter_bytes=...)</code></td></tr> |
| </tbody></table></div> |
| <div class="callout warning"><strong>Row-ID constraint</strong><code>RoaringTreemap</code> uses the <code>u64</code> domain, so row IDs must be non-negative to match the filter. Filters are query payloads and are never written into index files.</div> |
| <p>DiskANN maps sparse filters through its persisted row-ID lookup after first use; dense filters scan resident row IDs once per batch. It normally scans PQ codes only for matching nodes, then exactly reranks a bounded candidate set. A quality-gated broad filter may use ordinary graph navigation and post-filter its candidates, with a complete matching-node PQ fallback before raw-vector reads. Benchmark filter-heavy traffic separately because the safe PQ path remains linear in the number of matching nodes.</p> |
| <nav class="pager" aria-label="Related documentation"><a href="index.html"><small>Choose an index</small>← Index overview</a><a href="development.html"><small>Next</small>Development and benchmarks →</a></nav> |
| </section> |
| </article> |
| </div> |
| </div></main> |
| <footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index</span><span>API and language bindings</span></div></footer> |
| </body> |
| </html> |
