Protocol Documentation
Table of Contents
banyandb/common/v1/common.proto
Group
Group is an internal object for Group management
| Field | Type | Label | Description |
|---|
| metadata | Metadata | | metadata define the group's identity |
| catalog | Catalog | | catalog denotes which type of data the group contains |
| resource_opts | ResourceOpts | | resourceOpts indicates the structure of the underlying kv storage |
| updated_at | google.protobuf.Timestamp | | updated_at indicates when resources of the group are updated |
Metadata
Metadata is for multi-tenant, multi-model use
| Field | Type | Label | Description |
|---|
| group | string | | group contains a set of options, like retention policy, max |
| name | string | | name of the entity |
| id | uint32 | | |
| create_revision | int64 | | readonly. create_revision is the revision of last creation on this key. |
| mod_revision | int64 | | readonly. mod_revision is the revision of last modification on this key. |
ResourceOpts
| Field | Type | Label | Description |
|---|
| shard_num | uint32 | | shard_num is the number of shards |
| block_num | uint32 | | block_num specific how many blocks in a segment |
| ttl | string | | ttl indicates time to live, how long the data will be cached |
Catalog
| Name | Number | Description |
|---|
| CATALOG_UNSPECIFIED | 0 | |
| CATALOG_STREAM | 1 | |
| CATALOG_MEASURE | 2 | |
banyandb/database/v1/database.proto
Node
Shard
banyandb/model/v1/common.proto
FieldValue
ID
| Field | Type | Label | Description |
|---|
| value | string | | |
Int
| Field | Type | Label | Description |
|---|
| value | int64 | | |
IntArray
| Field | Type | Label | Description |
|---|
| value | int64 | repeated | |
Str
| Field | Type | Label | Description |
|---|
| value | string | | |
StrArray
| Field | Type | Label | Description |
|---|
| value | string | repeated | |
TagFamilyForWrite
| Field | Type | Label | Description |
|---|
| tags | TagValue | repeated | |
TagValue
AggregationFunction
| Name | Number | Description |
|---|
| AGGREGATION_FUNCTION_UNSPECIFIED | 0 | |
| AGGREGATION_FUNCTION_MEAN | 1 | |
| AGGREGATION_FUNCTION_MAX | 2 | |
| AGGREGATION_FUNCTION_MIN | 3 | |
| AGGREGATION_FUNCTION_COUNT | 4 | |
| AGGREGATION_FUNCTION_SUM | 5 | |
banyandb/model/v1/query.proto
Condition
Condition consists of the query condition with a single binary operator to be imposed For 1:1 BinaryOp, values in condition must be an array with length = 1, while for 1:N BinaryOp, values can be an array with length >= 1.
Criteria
tag_families are indexed.
QueryOrder
QueryOrder means a Sort operation to be done for a given index rule. The index_rule_name refers to the name of a index rule bound to the subject.
| Field | Type | Label | Description |
|---|
| index_rule_name | string | | |
| sort | Sort | | |
Tag
Pair is the building block of a record which is equivalent to a key-value pair. In the context of Trace, it could be metadata of a trace such as service_name, service_instance, etc. Besides, other tags are organized in key-value pair in the underlying storage layer. One should notice that the values can be a multi-value.
TagFamily
| Field | Type | Label | Description |
|---|
| name | string | | |
| tags | Tag | repeated | |
TagProjection
TagProjection is used to select the names of keys to be returned.
TagProjection.TagFamily
TimeRange
TimeRange is a range query for uint64, the range here follows left-inclusive and right-exclusive rule, i.e. [begin, end) if both edges exist
Condition.BinaryOp
BinaryOp specifies the operation imposed to the given query condition For EQ, NE, LT, GT, LE and GE, only one operand should be given, i.e. one-to-one relationship. HAVING and NOT_HAVING allow multi-value to be the operand such as array/vector, i.e. one-to-many relationship. For example, "keyA" contains "valueA" and "valueB"
| Name | Number | Description |
|---|
| BINARY_OP_UNSPECIFIED | 0 | |
| BINARY_OP_EQ | 1 | |
| BINARY_OP_NE | 2 | |
| BINARY_OP_LT | 3 | |
| BINARY_OP_GT | 4 | |
| BINARY_OP_LE | 5 | |
| BINARY_OP_GE | 6 | |
| BINARY_OP_HAVING | 7 | |
| BINARY_OP_NOT_HAVING | 8 | |
| BINARY_OP_IN | 9 | |
| BINARY_OP_NOT_IN | 10 | |
Sort
| Name | Number | Description |
|---|
| SORT_UNSPECIFIED | 0 | |
| SORT_DESC | 1 | |
| SORT_ASC | 2 | |
banyandb/database/v1/schema.proto
Entity
| Field | Type | Label | Description |
|---|
| tag_names | string | repeated | |
FieldSpec
FieldSpec is the specification of field
| Field | Type | Label | Description |
|---|
| name | string | | name is the identity of a field |
| field_type | FieldType | | field_type denotes the type of field value |
| encoding_method | EncodingMethod | | encoding_method indicates how to encode data during writing |
| compression_method | CompressionMethod | | compression_method indicates how to compress data during writing |
IndexRule
IndexRule defines how to generate indices based on tags and the index type IndexRule should bind to a subject through an IndexRuleBinding to generate proper indices.
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata define the rule's identity |
| tags | string | repeated | tags are the combination that refers to an indexed object If the elements in tags are more than 1, the object will generate a multi-tag index Caveat: All tags in a multi-tag MUST have an identical IndexType |
| type | IndexRule.Type | | type is the IndexType of this IndexObject. |
| location | IndexRule.Location | | location indicates where to store index. |
| updated_at | google.protobuf.Timestamp | | updated_at indicates when the IndexRule is updated |
IndexRuleBinding
IndexRuleBinding is a bridge to connect severalIndexRules to a subject This binding is valid between begin_at_nanoseconds and expire_at_nanoseconds, that provides flexible strategies to control how to generate time series indices.
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata is the identity of this binding |
| rules | string | repeated | rules refers to the IndexRule |
| subject | Subject | | subject indicates the subject of binding action |
| begin_at | google.protobuf.Timestamp | | begin_at_nanoseconds is the timestamp, after which the binding will be active |
| expire_at | google.protobuf.Timestamp | | expire_at_nanoseconds it the timestamp, after which the binding will be inactive expire_at_nanoseconds must be larger than begin_at_nanoseconds |
| updated_at | google.protobuf.Timestamp | | updated_at indicates when the IndexRuleBinding is updated |
Measure
Measure intends to store data point
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata is the identity of a measure |
| tag_families | TagFamilySpec | repeated | tag_families are for filter measures |
| fields | FieldSpec | repeated | fields denote measure values |
| entity | Entity | | entity indicates which tags will be to generate a series and shard a measure |
| interval | string | | interval indicates how frequently to send a data point valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h", "d". |
| updated_at | google.protobuf.Timestamp | | updated_at indicates when the measure is updated |
Stream
Stream intends to store streaming data, for example, traces or logs
Subject
Subject defines which stream or measure would generate indices
| Field | Type | Label | Description |
|---|
| catalog | banyandb.common.v1.Catalog | | catalog is where the subject belongs to |
| name | string | | name refers to a stream or measure in a particular catalog |
TagFamilySpec
| Field | Type | Label | Description |
|---|
| name | string | | |
| tags | TagSpec | repeated | tags defines accepted tags |
TagSpec
TopNAggregation
TopNAggregation generates offline TopN statistics for a measure's TopN approximation
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata is the identity of an aggregation |
| source_measure | banyandb.common.v1.Metadata | | source_measure denotes the data source of this aggregation |
| field_name | string | | field_name is the name of field used for ranking |
| field_value_sort | banyandb.model.v1.Sort | | field_value_sort indicates how to sort fields ASC: bottomN DESC: topN UNSPECIFIED: topN + bottomN |
| group_by_tag_names | string | repeated | group_by_tag_names groups data points into statistical counters |
| criteria | banyandb.model.v1.Criteria | repeated | criteria select partial data points from measure |
| counters_number | int32 | | counters_number sets the number of counters to be tracked. The default value is 1000 |
| updated_at | google.protobuf.Timestamp | | updated_at indicates when the measure is updated |
CompressionMethod
| Name | Number | Description |
|---|
| COMPRESSION_METHOD_UNSPECIFIED | 0 | |
| COMPRESSION_METHOD_ZSTD | 1 | |
EncodingMethod
| Name | Number | Description |
|---|
| ENCODING_METHOD_UNSPECIFIED | 0 | |
| ENCODING_METHOD_GORILLA | 1 | |
FieldType
| Name | Number | Description |
|---|
| FIELD_TYPE_UNSPECIFIED | 0 | |
| FIELD_TYPE_STRING | 1 | |
| FIELD_TYPE_INT | 2 | |
| FIELD_TYPE_DATA_BINARY | 3 | |
IndexRule.Location
| Name | Number | Description |
|---|
| LOCATION_UNSPECIFIED | 0 | |
| LOCATION_SERIES | 1 | |
| LOCATION_GLOBAL | 2 | |
IndexRule.Type
Type determine the index structure under the hood
| Name | Number | Description |
|---|
| TYPE_UNSPECIFIED | 0 | |
| TYPE_TREE | 1 | |
| TYPE_INVERTED | 2 | |
TagType
| Name | Number | Description |
|---|
| TAG_TYPE_UNSPECIFIED | 0 | |
| TAG_TYPE_STRING | 1 | |
| TAG_TYPE_INT | 2 | |
| TAG_TYPE_STRING_ARRAY | 3 | |
| TAG_TYPE_INT_ARRAY | 4 | |
| TAG_TYPE_DATA_BINARY | 5 | |
| TAG_TYPE_ID | 6 | |
banyandb/database/v1/event.proto
EntityEvent
EntityEvent.TagLocator
| Field | Type | Label | Description |
|---|
| family_offset | uint32 | | |
| tag_offset | uint32 | | |
ShardEvent
Action
| Name | Number | Description |
|---|
| ACTION_UNSPECIFIED | 0 | |
| ACTION_PUT | 1 | |
| ACTION_DELETE | 2 | |
banyandb/database/v1/rpc.proto
GroupRegistryServiceCreateRequest
GroupRegistryServiceCreateResponse
GroupRegistryServiceDeleteRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
GroupRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
GroupRegistryServiceGetRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
GroupRegistryServiceGetResponse
GroupRegistryServiceListRequest
GroupRegistryServiceListResponse
GroupRegistryServiceUpdateRequest
GroupRegistryServiceUpdateResponse
IndexRuleBindingRegistryServiceCreateRequest
IndexRuleBindingRegistryServiceCreateResponse
IndexRuleBindingRegistryServiceDeleteRequest
IndexRuleBindingRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
IndexRuleBindingRegistryServiceGetRequest
IndexRuleBindingRegistryServiceGetResponse
IndexRuleBindingRegistryServiceListRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
IndexRuleBindingRegistryServiceListResponse
IndexRuleBindingRegistryServiceUpdateRequest
IndexRuleBindingRegistryServiceUpdateResponse
IndexRuleRegistryServiceCreateRequest
IndexRuleRegistryServiceCreateResponse
IndexRuleRegistryServiceDeleteRequest
IndexRuleRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
IndexRuleRegistryServiceGetRequest
IndexRuleRegistryServiceGetResponse
IndexRuleRegistryServiceListRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
IndexRuleRegistryServiceListResponse
| Field | Type | Label | Description |
|---|
| index_rule | IndexRule | repeated | |
IndexRuleRegistryServiceUpdateRequest
IndexRuleRegistryServiceUpdateResponse
MeasureRegistryServiceCreateRequest
| Field | Type | Label | Description |
|---|
| measure | Measure | | |
MeasureRegistryServiceCreateResponse
MeasureRegistryServiceDeleteRequest
MeasureRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
MeasureRegistryServiceGetRequest
MeasureRegistryServiceGetResponse
| Field | Type | Label | Description |
|---|
| measure | Measure | | |
MeasureRegistryServiceListRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
MeasureRegistryServiceListResponse
| Field | Type | Label | Description |
|---|
| measure | Measure | repeated | |
MeasureRegistryServiceUpdateRequest
| Field | Type | Label | Description |
|---|
| measure | Measure | | |
MeasureRegistryServiceUpdateResponse
StreamRegistryServiceCreateRequest
| Field | Type | Label | Description |
|---|
| stream | Stream | | |
StreamRegistryServiceCreateResponse
StreamRegistryServiceDeleteRequest
StreamRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
StreamRegistryServiceGetRequest
StreamRegistryServiceGetResponse
| Field | Type | Label | Description |
|---|
| stream | Stream | | |
StreamRegistryServiceListRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
StreamRegistryServiceListResponse
| Field | Type | Label | Description |
|---|
| stream | Stream | repeated | |
StreamRegistryServiceUpdateRequest
| Field | Type | Label | Description |
|---|
| stream | Stream | | |
StreamRegistryServiceUpdateResponse
TopNAggregationRegistryServiceCreateRequest
TopNAggregationRegistryServiceCreateResponse
TopNAggregationRegistryServiceDeleteRequest
TopNAggregationRegistryServiceDeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
TopNAggregationRegistryServiceGetRequest
TopNAggregationRegistryServiceGetResponse
TopNAggregationRegistryServiceListRequest
| Field | Type | Label | Description |
|---|
| group | string | | |
TopNAggregationRegistryServiceListResponse
TopNAggregationRegistryServiceUpdateRequest
TopNAggregationRegistryServiceUpdateResponse
GroupRegistryService
IndexRuleBindingRegistryService
IndexRuleRegistryService
MeasureRegistryService
StreamRegistryService
TopNAggregationRegistryService
banyandb/measure/v1/query.proto
DataPoint
DataPoint is stored in Measures
DataPoint.Field
QueryRequest
QueryRequest is the request contract for query.
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata is required |
| time_range | banyandb.model.v1.TimeRange | | time_range is a range query with begin/end time of entities in the timeunit of milliseconds. |
| criteria | banyandb.model.v1.Criteria | repeated | tag_families are indexed. |
| tag_projection | banyandb.model.v1.TagProjection | | tag_projection can be used to select tags of the data points in the response |
| field_projection | QueryRequest.FieldProjection | | field_projection can be used to select fields of the data points in the response |
| group_by | QueryRequest.GroupBy | | group_by groups data points based on their field value for a specific tag and use field_name as the projection name |
| agg | QueryRequest.Aggregation | | agg aggregates data points based on a field |
| top | QueryRequest.Top | | top limits the result based on a particular field. If order_by is specificed, top sorts the dataset based on order_by's output |
| offset | uint32 | | offset is used to support pagination, together with the following limit. If top is sepcificed, offset processes the dataset based on top's output |
| limit | uint32 | | limit is used to impose a boundary on the number of records being returned. If top is sepcificed, limit processes the dataset based on top's output |
| order_by | banyandb.model.v1.QueryOrder | | order_by is given to specify the sort for a tag. |
QueryRequest.Aggregation
QueryRequest.FieldProjection
| Field | Type | Label | Description |
|---|
| names | string | repeated | |
QueryRequest.GroupBy
| Field | Type | Label | Description |
|---|
| tag_projection | banyandb.model.v1.TagProjection | | tag_projection must be a subset of the tag_projection of QueryRequest |
| field_name | string | | field_name must be one of fields indicated by field_projection |
QueryRequest.Top
| Field | Type | Label | Description |
|---|
| number | int32 | | number set the how many items should be returned |
| field_name | string | | field_name must be one of files indicated by the field_projection |
| field_value_sort | banyandb.model.v1.Sort | | field_value_sort indicates how to sort fields ASC: bottomN DESC: topN UNSPECIFIED: topN |
QueryResponse
QueryResponse is the response for a query to the Query module.
| Field | Type | Label | Description |
|---|
| data_points | DataPoint | repeated | data_points are the actual data returned |
banyandb/measure/v1/write.proto
DataPointValue
DataPointValue is the data point for writing. It only contains values.
InternalWriteRequest
WriteRequest
WriteRequest is the request contract for write
WriteResponse
WriteResponse is the response contract for write
banyandb/measure/v1/topn.proto
TopNList
TopNList contains a series of topN items
TopNList.Item
TopNRequest
TopNRequest is the request contract for query.
TopNResponse
TopNResponse is the response for a query to the Query module.
| Field | Type | Label | Description |
|---|
| lists | TopNList | repeated | lists contain a series topN lists ranked by timestamp if agg_func in query request is specified, lists' size should be one. |
banyandb/measure/v1/rpc.proto
MeasureService
banyandb/property/v1/property.proto
Metadata
Metadata is for multi-tenant use
Property
Property stores the user defined data
banyandb/property/v1/rpc.proto
CreateRequest
| Field | Type | Label | Description |
|---|
| property | Property | | |
CreateResponse
DeleteRequest
| Field | Type | Label | Description |
|---|
| metadata | Metadata | | |
DeleteResponse
| Field | Type | Label | Description |
|---|
| deleted | bool | | |
GetRequest
| Field | Type | Label | Description |
|---|
| metadata | Metadata | | |
GetResponse
| Field | Type | Label | Description |
|---|
| property | Property | | |
ListRequest
ListResponse
| Field | Type | Label | Description |
|---|
| property | Property | repeated | |
UpdateRequest
| Field | Type | Label | Description |
|---|
| property | Property | | |
UpdateResponse
PropertyService
banyandb/stream/v1/query.proto
Element
Element represents (stream context) a Span defined in Google Dapper paper or equivalently a Segment in Skywalking. (Log context) a log
| Field | Type | Label | Description |
|---|
| element_id | string | | element_id could be span_id of a Span or segment_id of a Segment in the context of stream |
| timestamp | google.protobuf.Timestamp | | timestamp represents a millisecond 1) either the start time of a Span/Segment, 2) or the timestamp of a log |
| tag_families | banyandb.model.v1.TagFamily | repeated | fields contains all indexed Field. Some typical names, - stream_id - duration - service_name - service_instance_id - end_time_milliseconds |
QueryRequest
QueryRequest is the request contract for query.
| Field | Type | Label | Description |
|---|
| metadata | banyandb.common.v1.Metadata | | metadata is required |
| time_range | banyandb.model.v1.TimeRange | | time_range is a range query with begin/end time of entities in the timeunit of milliseconds. In the context of stream, it represents the range of the startTime for spans/segments, while in the context of Log, it means the range of the timestamp(s) for logs. it is always recommended to specify time range for performance reason |
| offset | uint32 | | offset is used to support pagination, together with the following limit |
| limit | uint32 | | limit is used to impose a boundary on the number of records being returned |
| order_by | banyandb.model.v1.QueryOrder | | order_by is given to specify the sort for a field. So far, only fields in the type of Integer are supported |
| criteria | banyandb.model.v1.Criteria | repeated | tag_families are indexed. |
| projection | banyandb.model.v1.TagProjection | | projection can be used to select the key names of the element in the response |
QueryResponse
QueryResponse is the response for a query to the Query module.
| Field | Type | Label | Description |
|---|
| elements | Element | repeated | elements are the actual data returned |
banyandb/stream/v1/write.proto
ElementValue
| Field | Type | Label | Description |
|---|
| element_id | string | | element_id could be span_id of a Span or segment_id of a Segment in the context of stream |
| timestamp | google.protobuf.Timestamp | | timestamp is in the timeunit of milliseconds. It represents 1) either the start time of a Span/Segment, 2) or the timestamp of a log |
| tag_families | banyandb.model.v1.TagFamilyForWrite | repeated | the order of tag_families' items match the stream schema |
InternalWriteRequest
WriteRequest
WriteResponse
banyandb/stream/v1/rpc.proto
StreamService
Scalar Value Types
| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
|---|
| double | | double | double | float | float64 | double | float | Float |
| float | | float | float | float | float32 | float | float | Float |
| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass |
| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |