Install numpy for Python API tests
Pure Rust IVF-PQ implementation for Apache Paimon. Designed for data lake (S3/HDFS/OSS) with seek-based I/O, supporting both 8-bit and 4-bit PQ with SIMD acceleration.
The vector index accepts a serialized 64-bit Roaring bitmap of allowed row IDs during reader search. This lets the Paimon query layer evaluate metadata predicates with table/scalar indexes first, then pass the matching row-id set into IVF-PQ as an ANN prefilter.
Bindings expose the same wire format:
search_with_reader_roaring_filter and search_batch_reader_roaring_filterIVFPQReader.search(..., byte[]) and IVFPQReader.searchBatch(..., byte[])IVFPQReader.search(..., filter_bytes=...) and IVFPQReader.search_batch(..., filter_bytes=...)Row IDs must be non-negative to map directly into RoaringTreemap's u64 domain.
The Java binding provides small lifecycle-safe facades over the JNI symbols: IVFPQWriter builds and writes an index, IVFPQReader opens an index and runs single-query or batch search, and result containers expose defensive copies of IDs and distances.
The Python binding mirrors that flow with IVFPQWriter and IVFPQReader. search returns one-dimensional NumPy arrays for a single query, while search_batch accepts a two-dimensional query array and returns two-dimensional NumPy arrays shaped as (query_count, top_k).
Apache Paimon Vector Index is an exciting project currently under active development. Whether you're looking to use it in your projects or contribute to its growth, there are several ways you can get involved:
Submit issues for bug report or asking questions in discussion.
Licensed under Apache License, Version 2.0.