Apache Paimon Full Text: Rust full text index for data lake search.

Clone this repo:
  1. 00a2e9c Start 0.3.0 development after cutting 0.2.0 by JingsongLi · 2 weeks ago main
  2. 3684c2a Fix release workflows for current ASF action approvals by JingsongLi · 2 weeks ago
  3. 09aa5fc Add ICU tokenizers for mixed-language full-text search (#24) by Jingsong Lee · 2 weeks ago
  4. ee83ca7 Test Python 3.8 compatibility in CI (#19) by jianguotian · 4 months ago
  5. fa1c437 ci: preserve legal file line endings on Windows (#22) by XiaoHongbo · 4 months ago

Apache Paimon Full Text Index

Standalone Tantivy-based full-text index library for Apache Paimon-style data lake storage. The project follows the same shape as paimon-vector-index:

  • core: Rust implementation and v1 storage format.
  • ffi: C ABI over the Rust core.
  • jni: Java JNI bridge over the Rust core.
  • java: public Java API.
  • python: Python ctypes API over the C ABI.

The index file is self-describing. Readers only need positional pread I/O and do not depend on Paimon manifest metadata.

Current Status

Implemented:

  • Rust writer, reader, v1 envelope, and search.
  • Single-field, multi-field, repeated-field string-array, and dotted-path text fields.
  • Match, fuzzy match (fuzziness, auto, max_expansions, prefix_length), phrase, boolean, multi-match, and boost-demotion queries.
  • C FFI writer/reader/search with query JSON strings, including serialized 64-bit Roaring row-id filters.
  • Java API and JNI bridge.
  • Python ctypes package.
  • Cross-boundary round-trip tests for Rust core, FFI, Java/JNI, and Python.

Supported tokenizers in this first implementation:

  • default
  • simple
  • whitespace
  • raw
  • ngram
  • jieba
  • icu
  • icu/split

Default tokenizer behavior uses English full-text defaults: lower-case, stemming, stop-word removal, ASCII folding, max token length 40, and positions. Set with-position=false only when phrase search is not needed.

For mixed Chinese and English text, set fulltext.tokenizer=icu to use Unicode word boundaries and built-in ICU dictionaries; no model download is required. Use fulltext.tokenizer=icu/split to additionally split tokens on non-alphanumeric characters, for example hello_world into hello and world, and alpha.beta into alpha and beta. The split mode only emits the parts, without retaining the original token. Neither mode splits camelCase identifiers. Both modes use the same configurable filters as the existing tokenizers. For technical text, consider fulltext.stem=false and fulltext.remove-stop-words=false to preserve identifiers and common words.

Readers expose archived Tantivy files through a seek-on-demand directory, so opening an index reads the envelope and Tantivy metadata without loading all segment files into memory.

Build

cargo test -p paimon-ftindex-core
cargo test -p paimon-ftindex-ffi
cargo build -p paimon-ftindex-ffi
cargo build -p paimon-ftindex-jni
mvn -q -f java/pom.xml test
PYTHONPATH=python python3 -m pytest -q python/tests

Rust Example

use paimon_ftindex_core::io::{PosWriter, SliceReader};
use paimon_ftindex_core::{FullTextIndexConfig, FullTextIndexReader, FullTextIndexWriter};

let mut writer = FullTextIndexWriter::new(FullTextIndexConfig::new())?;
writer.add_document(1, "Apache Paimon full text search")?;

let mut bytes = Vec::new();
writer.write(&mut PosWriter::new(&mut bytes))?;

let reader = FullTextIndexReader::open(SliceReader::new(bytes))?;
reader.prewarm()?;
let result = reader.search(r#"{"match":{"query":"paimon","column":"text"}}"#, 10)?;

Multi-field indexes can be configured with named fields:

let config = FullTextIndexConfig::new().with_text_fields(["title", "body"]);
let mut writer = FullTextIndexWriter::new(config)?;
writer.add_document_fields(
    1,
    [
        ("title".to_string(), "Apache Paimon".to_string()),
        ("body".to_string(), "lake storage".to_string()),
    ],
)?;

When a match query omits column, the reader searches all indexed text fields. This lets a Paimon adapter populate extra fields internally without requiring callers to build a multi_match query.

To restrict search to an upstream candidate set, pass a serialized RoaringTreemap of allowed row ids:

let filtered = reader.search_with_roaring_filter(
    r#"{"match":{"query":"paimon","column":"text"}}"#,
    10,
    roaring_filter_bytes,
)?;

Python Example

from io import BytesIO
from paimon_ftindex import FullTextIndexReader, FullTextIndexWriter

out = BytesIO()
with FullTextIndexWriter({"text-fields": "title,body"}) as writer:
    writer.add_document_fields(1, {"title": "Apache Paimon", "body": "lake storage"})
    writer.write(out)

class Input:
    def __init__(self, data):
        self.data = data
    def pread(self, pos, length):
        return self.data[pos:pos + length]

with FullTextIndexReader(Input(out.getvalue())) as reader:
    reader.prewarm()
    ids, scores = reader.search('{"match":{"query":"paimon"}}', limit=10)
    filtered_ids, filtered_scores = reader.search(
        '{"match":{"query":"paimno","column":"title","fuzziness":1}}',
        limit=10,
        filter_bytes=roaring_filter_bytes,
    )
    metrics = reader.read_metrics()

Search APIs accept the query DSL as a JSON string.

prewarm() eagerly initializes the underlying search reader and archive cache before a query burst. read_metrics() reports positional read calls/bytes and archive cache hit/miss counters for tuning reader reuse and object-store access patterns.