name: Formal RFC about: Submit a formal Request For Comments for consideration by the team. title: ‘Mango JSON indexes in FoundationDB’ labels: rfc, discussion assignees: ‘’
This document describes the data model, querying and indexing management for Mango JSON indexes with FoundationDB.
This document details the data model for storing Mango indexes. Indexes will be updated in the transaction that a document is written to FoundationDB. When an index is created on an existing database, a background task will build the index up to the Sequence that the index was created at.
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
Sequence
: a 13-byte value formed by combining the current Incarnation
of the database and the Versionstamp
of the transaction. Sequences are monotonically increasing even when a database is relocated across FoundationDB clusters. See (RFC002)[LINK TBD] for a full explanation.
Mango is a declarative JSON querying syntax that allows a user to retrieve documents based on a selector. Indexes can be defined to improve query performance. In CouchDB Mango is a query layer built on top of Map/Reduce indexes. Each Mango query follows a two-step process, first a subset of the selector is converted into a map query to be used with a predefined index or falling back to _all_docs
if no indexes are available. Each document retrieved from the index is then matched against the full query selector.
With CouchDB on FoundationDB, all new created Mango indexes have the interactive: true
option set. Thereby Mango indexes will be indexed in the same transaction that a document is add/updated to the database.
A Mango index is defined as:
{ "name": "view-name", "index": { "fields": ["fieldA", "fieldB"] }, "partial_filter_selector": {} }
The above index definition would be converted into a map index that looks like this:
{ "_id": "_design/ddoc", "language": "query", "views": { "view-name": { "map": { "fields": [{ "fieldA": "asc" }, { "fieldB": "asc" }], "selector": {} } } }, "options": [{ "autoupdate": false }, { "interactive": true }] }
{"autoupdate": false}
means that the index will not be auto updated in the background{"interactive": true}
configures the index to be updated in the document update transactionMango indexes are a layer on top of map indexes. So the index definition is the same as the map index definition.
This design has certain defined limits for it to work correctly:
name
, fields
and partial_filter_selector
) cannot exceed 64 KB FDB value limitWhen an index is created on an existing database, the index will be updated in a background job up to the versionstamp that the index was added to the database at. The process for building a new index would be:
building
so that is it not used to service any queries until it is updated. Add a job to couch_jobs
to build the index.couch_jobs
will start reading sections of the changes feed and building the index, this background process will keep processing the changes read until it reaches the creation versionstamp. Once it reaches that point, the index is up to date and build_status
will be marked as active
and the index can be used to service queries.Index limitations aside, this design preserves all of the existing API options for working with CouchDB documents.
The mango
application will be modified to work with FoundationDB
When querying any of the _index
endpoints an extra field, build_status
, will be added to the index definition. The build_status
will either be building
or active
.
None,
None have been identified.
Original mailing list discussion
thanks to following in participating in the design discussion