paimon-eslib: ES Index (Multi-Column Global Index)

paimon-eslib provides the es-index global index: a Lucene-backed, multi-column, multi-modal index. A single es-index file indexes a primary column (typically a vector) together with optional companion columns — full-text, keyword, scalar, and date — so vector search and full-text search can be served from the same index, and companion columns stay available for scoring and filtering inside one index definition.

Unlike the single-purpose vector and full-text global indexes — each of which indexes one column with one dedicated engine — es-index groups a primary column plus optional companion columns into one index. The primary column determines the main modality (usually a vector); companion columns are indexed according to their data type and per-field options.

Plugin required. es-index is provided by this paimon-eslib module. Make sure paimon-eslib is on the classpath of your Spark or Flink job (and of any reader) before building or querying an es-index. The module and its ESLib/Lucene dependencies require Java 11 or newer. It is excluded from the default Maven reactor; build it explicitly with -Ppaimon-eslib on JDK 11 or newer.

paimon-eslib is distributed as a thin JAR and does not embed ESLib or Lucene classes. Maven and Gradle resolve these dependencies transitively. When installing JARs manually, place eslib-core-lucene9 and eslib-simdvec-lucene9 1.0.7, together with their Lucene 9.12 dependencies, in the same runtime classloader as paimon-eslib.

ESLib source and artifacts

• ESLib source code: CrownChu/es-paimon-lib
• Published ESLib JARs: CrownChu/es-paimon-lib-releases

The ESLib source and binary repositories are maintained independently of the Apache Software Foundation. The exact ESLib artifact versions used by this module are declared in pom.xml.

See the general Global Index documentation for the required Data Evolution table properties, coverage/freshness behavior, and shared build options.

How columns map to index types

For each indexed column, es-index decides the sub-index type from the column data type, unless you override it with fields.<field>.type:

Text columns always provide both capabilities. A fulltext primary field gets a <column>.keyword exact-match sub-field; a keyword primary field gets a <column>.fulltext analyzed sub-field.

Prerequisites

Create a Data Evolution table with the required properties:

CREATE TABLE my_table (
    id INT,
    embedding ARRAY<FLOAT>,
    content STRING,
    category STRING,
    price INT
) TBLPROPERTIES (
    'bucket' = '-1',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true'
);

Build ES Index

Build the index with sys.create_global_index and index_type => 'es-index'. To index several columns in one es-index, pass a comma-separated index_column list: the first column is the primary column, the rest are companion columns. Per-column behavior is configured through options under the global-index.es-index. prefix.

Spark SQL

-- Single vector column with an HNSW index
CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=hnsw,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine'
);

-- Multi-column: vector (primary) + full-text + keyword + scalar in one index.
-- 'content' becomes full-text because an analyzer is set; 'category' stays keyword; 'price' scalar.
CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding,content,category,price',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=diskbbq,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine,global-index.es-index.fields.content.analyzer=standard,global-index.es-index.fields.category.type=keyword'
);

Flink SQL

-- Single vector column with an HNSW index
CALL sys.create_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=hnsw,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine'
);

-- Multi-column: vector (primary) + full-text + keyword + scalar in one index.
CALL sys.create_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding,content,category,price',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=diskbbq,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine,global-index.es-index.fields.content.analyzer=standard,global-index.es-index.fields.category.type=keyword'
);

You can also set the same options in TBLPROPERTIES at table-creation time instead of passing them per build. Options are resolved with field-level keys taking precedence over index-type-level keys: global-index.es-index.fields.<field>.<key> overrides global-index.es-index.<key>.

Options

Index-type-level keys use the global-index.es-index.<key> prefix and apply to every column of that kind; field-level keys use global-index.es-index.fields.<field>.<key> and override them for one column.

Query

es-index implements the same search API as the other global indexes, so querying does not depend on which engine built the index — only the build step differs.

Spark SQL

-- Vector search: top-5 nearest neighbors on the primary vector column
SELECT * FROM vector_search('my_table', 'embedding', array(1.0f, 2.0f, 3.0f), 5);

-- Full-text search on a companion full-text column, using the JSON query DSL
SELECT * FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format"}}',
    10
);

Vector routes and full-text routes over an es-index can also be combined with the hybrid_search(...) table-valued function.

Flink SQL

Flink exposes vector search as a CALL procedure. The procedure returns JSON-serialized rows as strings.

-- Vector search: top-5 nearest neighbors on the primary vector column
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5
);

-- Optionally restrict the returned columns
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5,
    projection => 'id,content'
);

Note. Flink SQL currently exposes vector search only. For full-text search and hybrid search over an es-index, use Spark SQL (the full_text_search / hybrid_search table-valued functions) or the Java API (Table.newFullTextSearchBuilder() / Table.newHybridSearchBuilder()).

Drop ES Index

Spark SQL

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index'
);

Flink SQL

CALL sys.drop_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index'
);