name: Pagination support about: Add bookmark based pagination interface title: ‘Pagination API’ labels: rfc, discussion assignees: ''
Implement pagination support for query like endpoints
The main driver for this proposal is the introduction of FoundationDB as a storage engine. FoundationDB imposes limits on transaction duration and size. Therefore we need to find way to restrict amount of data we return to customers. We could simply set the maximum limit
which would cap amount of rows client can request. However this “solution” has a big disadvantage. Which is it would require clients to write pagination recipe in their code. Current pagination scheme requires complex logic on the client side. There are quite a few corner cases to handle.
The main addition is to add new bookmark based pagination scheme to all quiery like endpoints. As a first step we would keep _all_dbs
, _dbs_info
and _changes
out of scope for the following reasons:
The endpoints in scope are:
In a nutshell the idea is:
page_size
query field to control number of rows on each page and to flag that client expects paginated responsefirst
, previous
, next
fields which contain bookmark part of URIbookmark
query field to retrieve bookmarked pageThe key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
bookmark
- is opaque token which would contain information needed to retrieve bookmarked page. The format of the token value MUST NOT be relied upon in the client.bookmark
to following endpointspage_size
if it is set we would use paginated endpoint otherwise use old code path[request_limits] _all_docs = 5000 _all_docs/queries = 5000 _all_dbs = 5000 _dbs_info = 5000 _view = 2500 _view/queries = 2500 _find = 2500
"first": "12345678945621321689", "previous": "983uiwfjkdsdf", "next": "12343tyekf3"
first
/next
/last
keys in the response are represented as path which includes the bookmark query key. This means the bookmark token size contributes to total URI length and is subject to a max URL length (around 2000 characters). This means storing keys
in a bookmark is not an option. For that reason POST
method is not supported when pagination is enabledrequest_limit
. However with streaming we've already sent a return code.bookmark
field is providedpage_size
query key is specified and when it is below the max limitbookmark
field is set and other query fields are presentpage_size
query key is specified and it is greater than the max limitprevious
/next
/first
keys are optional and we omit them for the cases they don't make sensepage_size
to limit
if provided limit
is less than value configured in request_limit
of default.ini
for the given endpointpage_size
to the value configured in request_limit
of default.ini
for the given endpointlimit
is reached the final response will not have a “next” bookmarkskip
query parameter is limited to the same page_size
or value configured in request_limit
whatever is lesspage_size
the response will not have a “next” bookmarkpage_size
is used with _all_docs/queries
or {db}/_design/{ddoc}/_view/{view}/queries
the specified limit applies to number of queries provided in the request._all_docs/queries
and {db}/_design/{ddoc}/_view/{view}/queries
the total number of rows returned shouldn't exceed provided page_size
or configured max limit (whatever is less){restart_tx, true}
option for FDB calls._all_docs/queries
and {db}/_design/{ddoc}/_view/{view}/queries
can include bookmarks:{"queries": [ {"bookmark": "bookmarkForQuery1PageL"}, {"bookmark": "bookmarkForQuery2PageM"}, {"bookmark": "bookmarkForQuery3PageN"} ] }
_all_docs/queries
and {db}/_design/{ddoc}/_view/{view}/queries
can be submitted via separate request to _all_docs
and {db}/_design/{ddoc}/_view/{view}
correspondly.The page size limits are configured in default.ini
(or another ini
file) in request_limit
section as follows:
[request_limits] _all_docs = 5000 _all_docs/queries = 5000 _all_dbs = 5000 _dbs_info = 5000 _view = 2500 _view/queries = 2500 _find = 2500
_changes
endpoint_all_dbs
and _dbs_info
which would change response type to be object (using versioned API feature)N/A
N/A
No changes to the security model
Thank you to all partitipants in the discussion threads. You all helped to shape and refine this proposal in one form or another.