docs/src/dev/io/graphbinary.asciidoc

////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
////

[[graphbinary]]
= GraphBinary

GraphBinary is a binary serialization format suitable for object trees, designed to reduce serialization
overhead on both the client and the server, as well as limiting the size of the payload that is transmitted over the
wire.

It describes arbitrary object graphs with a fully-qualified format:

[source]
----
{type_code}{type_info}{value_flag}{value}
----

Where:

* `{type_code}` is a single unsigned byte representing the type number.
* `{type_info}` is an optional sequence of bytes providing additional information of the type represented. This is
especially useful for representing complex and custom types.
* `{value_flag}` is a single byte providing information about the value. Each type may have its own specific flags so
see each type for more details. Generally, flags have the following meaning:
** `0x00` The value is `non-null`, which means it is followed by a `{value}`.
** `0x01` The value is `null`. When this flag is set, no bytes for `{value}` will be provided.
* `{value}` is a sequence of bytes which content is determined by the type.

All encodings are big-endian.

Quick examples, using hexadecimal notation to represent each byte:

- `01 00 00 00 00 01`: a 32-bit integer number, that represents the decimal number 1. It’s composed by the
type_code `0x01`, and non-null value_flag `0x00` and four bytes to describe the value.
- `01 00 00 00 00 ff`: a 32-bit integer, representing the number 255.
- `01 01`: a null value for a 32-bit integer. It’s composed by the type_code `0x01`, and a null flag value `0x01`.
- `02 00 00 00 00 00 00 00 00 01`: a 64-bit integer number 1. It’s composed by the type_code `0x02`, non-null value_flag and
eight bytes to describe the value.

== Version 4.0

=== Changes from Version 1.0

==== Removed Types

The following types have been removed:

- `Date` (0x04), `Timestamp` (0x05) - replaced by `DateTime`
- `Class` (0x06)
- `Barrier` (0x13), `Binding` (0x14), `Bytecode` (0x15), `Cardinality` (0x16), `Column` (0x17), `Operator` (0x19), 
`Order` (0x1a), `Pick` (0x1b), `Pop` (0x1c), `Lambda` (0x1d), `P` (0x1e), `Scope` (0x1f), `Traverser` (0x21), `TextP` 
(0x28), `TraversalStrategy` (0x29), `DT` (0x2f), `GType` (0x30)
- `BulkSet` (0x2a) - functionality merged into `List`
- `Metrics` (0x2c), `TraversalMetrics` (0x2d)
- `Custom` (0x00) - replaced by `CompositePDT` and `PrimitivePDT`
- `InetAddress` (0x82)
- `Instant` (0x83), `LocalDate` (0x84), `LocalDateTime` (0x85), `LocalTime` (0x86), `MonthDay` (0x87), 
`OffsetDateTime` (0x88), `OffsetTime` (0x89), `Period` (0x8a), `Year` (0x8b), `YearMonth` (0x8c), 
`ZonedDateTime` (0x8d), `ZoneOffset` (0x8e) - Java-specific time types replaced by `DateTime`

==== Added Types

- `DateTime` (0x04) - unified date-time representation with timezone offset
- `CompositePDT` (0xf0) - provider-defined composite type with typed fields
- `PrimitivePDT` (0xf1) - provider-defined primitive type with stringified value
- `Marker` (0xfd) - separator between result data and status in response messages

==== Modified Types

===== List

The {value_flag} now indicates the format variant:

0x00: Standard format
0x01: Null
0x02: Bulked format where each item is followed by a Long bulk count

This change incorporates the functionality previously provided by BulkSet.

===== Map

The {value_flag} now indicates ordering:

0x00: Standard format (no ordering defined)
0x01: Null
0x02: Ordered

===== Labels and Properties in Edge, Graph, Vertex, and VertexProperty

The {label} field is now a List of Strings instead of a String value. Also, the {properties} field used to always be
null but can now contain a `List` of properties (this change actually started in a revision to v1).

===== Byte

Changed from unsigned 8-bit integer to signed two's complement representation.

===== ByteBuffer

Renamed to `Binary` (0x25). Format remains unchanged.

===== Graph

Renamed from TinkerGraph. Type code (0x10) remains unchanged. Label format has changed.

==== Request Message Changes

The request message format has been updated:

- Version 1.0: {version}{request_id}{op}{processor}{args}
- Version 4.0: {version}{fields}{gremlin}

`OpProcessors` have been removed so {op}{processor} were removed and {args} is essentially renamed to {fields}. The
version byte changed from `0x81` to `0x84`.

==== Response Message Changes

The response message format has been restructured:

- Version 1.0: {version}{request_id}{status_code}{status_message}{status_attributes}{result_meta}{result_data}
- Version 4.0: {version}{bulked}{result_data}{marker}{status_code}{status_message}{exception}

This new format expects all the results in one ResponseMessage (like it is in HTTP).

- Added {bulked} flag to indicate whether result data contains bulk counts since there are no Traverser serializers
- Added {marker} to separate result data from status information
- Version byte changed from 0x81 to 0x84

=== Forward Compatibility

The serialization format supports new types being added without the need to introduce a new version.

Changes to existing types require new revision.

=== Data Type Codes

- `0x01`: Int
- `0x02`: Long
- `0x03`: String
- `0x04`: DateTime
- `0x07`: Double
- `0x08`: Float
- `0x09`: List
- `0x0a`: Map
- `0x0b`: Set
- `0x0c`: UUID
- `0x0d`: Edge
- `0x0e`: Path
- `0x0f`: Property
- `0x10`: Graph
- `0x11`: Vertex
- `0x12`: VertexProperty
- `0x18`: Direction
- `0x20`: T
- `0x22`: BigDecimal
- `0x23`: BigInteger
- `0x24`: Byte
- `0x25`: Binary
- `0x26`: Short
- `0x27`: Boolean
- `0x2b`: Tree
- `0x2e`: Merge
- `0xf0`: CompositePDT
- `0xf1`: PrimitivePDT
- `0xfd`: Marker
- `0xfe`: Unspecified null object
- `0x80`: Char
- `0x81`: Duration

=== Null Handling

The serialization format defines two ways to represent null values:

- Unspecified null object
- Fully-qualified null

When a parent type can contain any subtype e.g., a object collection, a `null` value must be represented using the
"Unspecified Null Object" type code and the null value flag.

In contrast, when the parent type contains a type parameter that must be specified, a `null` value is represented using
a fully-qualified object using the appropriate type code and type information.

=== Data Type Formats

==== Int

Format: 4-byte two's complement integer.

Example values:

- `00 00 00 01`: 32-bit integer number 1.
- `00 00 01 01`: 32-bit integer number 257.
- `ff ff ff ff`: 32-bit integer number -1.
- `ff ff ff fe`: 32-bit integer number -2.

==== Long

Format: 8-byte two's complement integer.

Example values:

- `00 00 00 00 00 00 00 01`: 64-bit integer number 1.
- `ff ff ff ff ff ff ff fe`: 64-bit integer number -2.

==== String

Format: `{length}{text_value}`

Where:

- `{length}` is an `Int {value}` describing the byte length of the text. Length is a positive number or zero to represent
the empty string.
- `{text_value}` is a sequence of bytes representing the string value in UTF8 encoding.

Example values:

- `00 00 00 03 61 62 63`: the string 'abc'.
- `00 00 00 04 61 62 63 64`: the string 'abcd'.
- `00 00 00 00`: the empty string ''.

==== DateTime

A date-time with an offset from UTC/Greenwich in the ISO-8601 calendar system, such as 2007-12-03T10:15:30+01:00.

Format: `{year}{month}{day}{time}{offset}`

Where:

- `{year}` is an `Int {value}` from -999,999,999 to 999,999,999.
- `{month}` is a `Byte {value}` to represent the month, from 1 (January) to 12 (December)
- `{day}` is a `Byte {value}` from 1 to 31.
- `{time}` is a `Long {value}` to represent nanoseconds since midnight, from 0 to 86399999999999
- `{offset}` is an `Int {value}` to represent total zone offset in seconds, from -64800 (-18:00) to 64800 (+18:00).

==== Double

Format: 8 bytes representing IEEE 754 double-precision binary floating-point format.

Example values:

- `3f f0 00 00 00 00 00 00`: Double 1
- `3f 70 00 00 00 00 00 00`: Double 0.00390625
- `3f b9 99 99 99 99 99 9a`: Double 0.1

==== Float

Format: 4 bytes representing IEEE 754 single-precision binary floating-point format.

Example values:

- `3f 80 00 00`: Float 1
- `3e c0 00 00`: Float 0.375

==== List

An ordered collection of items. The format depends on the {value_flag}.

Format (value_flag=0x00): `{length}{item_0}...{item_n}`

Where:

- `{length}` is an `Int {value}` describing the length of the collection.
- `{item_0}...{item_n}` are the items of the list. `{item_i}` is a fully-qualified typed value composed of
`{type_code}{type_info}{value_flag}{value}`.

value_flag=0x01 represents null.

Format (value_flag=0x02): `{length}{item_0}{bulk_0}...{item_n}{bulk_n}`

Where:

- `{length}` is an `Int {value}` describing the length of the collection.
- `{item_0}...{item_n}` are the items of the list. `{item_i}` is a fully-qualified typed value composed of
`{type_code}{type_info}{value_flag}{value}`.
- `{bulk_0}...{bulk_n}` are `Long {value}s` that represent how many times that item should be repeated in the expanded list.

==== Set

A collection that contains no duplicate elements.

Format: Same as `List` for `0x00` and `0x01` {value_flag} formats. No support for bulk (`0x02`).

==== Map

A dictionary of keys to values. Both keys and values can be any type specified in this document.

The {value_flag} are defined as:
- `0x00` unordered
- `0x01` null
- `0x02` ordered

Format: `{length}{item_0}...{item_n}`

Where:

- `{length}` is an `Int {value}` describing the length of the map.
- `{item_0}...{item_n}` are the items of the map. `{item_i}` is sequence of 2 fully-qualified typed values one
representing the key and the following representing the value, each composed of
`{type_code}{type_info}{value_flag}{value}`.

==== UUID

A 128-bit universally unique identifier.

Format: 16 bytes representing the uuid.

Example

- `00 11 22 33 44 55 66 77 88 99 aa bb cc dd ee ff`: Uuid 00112233-4455-6677-8899-aabbccddeeff.

==== Edge

Format: `{id}{label}{inVId}{inVLabel}{outVId}{outVLabel}{parent}{properties}`

Where:

- `{id}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{label}` is a `List {value}`.
- `{inVId}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{inVLabel}` is a `List {value}`.
- `{outVId}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{outVLabel}` is a `List {value}`.
- `{parent}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}` which contains
the parent `Vertex`. Note that as TinkerPop currently send "references" only, this value will always be `null`.
- `{properties}` is a fully-qualified `List` of `Property` objects.

Example values:

[source,text]
----
01 00  00 00 00 0d                                            id is 13
00 00 00 01  03 00  00 00 00 08  64 65 76 65 6c 6f 70 73      label is a size 1 list with string 'develops'
01 00  00 00 00 0a                                            inVId is 10
00 00 00 01  03 00  00 00 00 08  73 6f 66 74 77 61 72 65      inVLabel is a size 1 list with string 'software'
01 00  00 00 00 01                                            outVId is 1
00 00 00 01  03 00  00 00 00 06  70 65 72 73 6f 6e            outVLabel is a size 1 list with string 'person'
fe 01                                                         parent is always null
09 00  00 00 00 01                                            properties is a size 1 list
0f 00 00 00 00 05  73 69 6e 63 65  01 00 00 00 07 d9  fe 01   property with key 'since' and value 2009 and null parent
----

==== Path

Format: `{labels}{objects}`

Where:

- `{labels}` is a fully-qualified `List` in which each item is a fully-qualified `Set` of `String`.
- `{objects}` is a fully-qualified `List` of fully-qualified typed values.

==== Property

Format: `{key}{value}{parent}`

Where:

- `{key}` is a `String {value}`.
- `{value}`  is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{parent}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}` which is either
an `Edge` or `VertexProperty`. Note that as TinkerPop currently sends "references" only this value will always be
`null`.

==== Graph

A collection of vertices and edges. Note that while similar the vertex/edge formats here hold some differences as
compared to the `Vertex` and `Edge` formats used for standard serialization/deserialization of a single graph element.

Format: `{vlength}{vertex_0}...{vertex_n}{elength}{edge_0}...{edge_n}`

Where:

- `{vlength}` is an `Int {value}` describing the number of vertices.
- `{vertex_0}...{vertex_n}` are vertices as described below.
- `{elength}` is an `Int {value}` describing the number of edges.
- `{edge_0}...{edge_n}` are edges as described below.

Vertex Format: `{id}{label}{plength}{property_0}...{property_n}`

- `{id}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{label}` is a `List {value}`.
- `{plength}` is an `Int {value}` describing the number of properties on the vertex.
- `{property_0}...{property_n}` are the vertex properties consisting of `{id}{label}{value}{parent}{properties}` where
the `{parent}` is always `null` and `{properties}` is a `List {value}` of `Property` objects.

Edge Format: `{id}{label}{inVId}{inVLabel}{outVId}{outVLabel}{parent}{properties}`

Where:

- `{id}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{label}` is a `List {value}`.
- `{inVId}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{inVLabel}` is always `null`.
- `{outVId}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{outVLabel}` is always `null`.
- `{parent}` is always `null`.
- `{properties}` is a `List {value}` of `Property` objects.

==== Vertex

Format: `{id}{label}{properties}`

Where:

- `{id}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{label}` is a `List {value}`.
- `{properties}` is a fully-qualified `List` of `VertexProperty` values.

Example values:

[source,text]
----
01 00  00 00 00 01                                        id is int 1
00 00 00 01  03 00  00 00 00 06  70 65 72 73 6f 6e        label is size 1 list with string 'person'
09 00  00 00 00 01  12 00  02 00  00 00 00 00 00 00 00 09 properties is a size 1 list with VertexProperty id 9
00 00 00 01  03 00  00 00 00 08  6c 6f 63 61 74 69 6f 6e  VertexProperty label is size 1 list with string 'location'
03 00  00 00 00 08  73 61 6e 74 61 20 66 65               VertexProperty value is string 'santa fe'
fe 01                                                     VertexProperty parent is always null
09 00  00 00 00 01                                        VertexProperty has a size 1 list
0f 00  00 00 00 09 73 74 61 72 74 54 69 6d 65             metaproperty with string key 'startTime'
01 00  00 00 07 d5  fe 01                                 VertexProperty metaproperty value is 2005 with null parent
----

==== VertexProperty

Format: `{id}{label}{value}{parent}{properties}`

Where:

- `{id}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{label}` is a `List {value}`.
- `{value}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
- `{parent}` is a fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}` which contains
the parent `Vertex`. Note that as TinkerPop currently send "references" only, this value will always be `null`.
- `{properties}` is a fully-qualified `List` of `Property` objects.

Example values:

[source,text]
----
02 00  00 00 00 00 00 00 00 00                id is Long 0
00 00 00 01  03 00  00 00 00 04  6e 61 6d 65  label is size 1 list with string 'name'
03 00  00 00 00 05  6d 61 72 6b 6f            value is string 'marko'
fe 01                                         parent is always null
09 00  00 00 00 00                            metaproperties is empty list
----

==== Direction

Format: a fully-qualified single `String` representing the enum value.

Example values:

- `03 00 00 00 00 03 4F 55 54`: OUT
- `03 00 00 00 00 02 49 4E`: IN

==== T

Format: a fully-qualified single `String` representing the enum value.

Example values:

- `03 00 00 00 00 05 6C 61 62 65 6C`: label
- `03 00 00 00 00 02 69 64`: id

==== Merge

Format: a fully-qualified single `String` representing the enum value.

Example values:

- `03 00 00 00 00 08 6F 6E 43 72 65 61 74 65`: onCreate
- `03 00 00 00 00 07 6F 6E 4D 61 74 63 68`: onMatch
- `03 00 00 00 00 04 6F 75 74 56`: outV
- `03 00 00 00 00 03 69 6E 56`: inV

==== BigDecimal

Represents an arbitrary-precision signed decimal number, consisting of an arbitrary precision integer unscaled value
and a 32-bit integer scale.

Format: `{scale}{unscaled_value}`

Where:

- `{scale}` is an `Int {value}`.
- `{unscaled_value}` is a `BigInteger {value}`.

==== BigInteger

A variable-length two's complement encoding of a signed integer.

Format: `{length}{value}`

Where:

- `{length}` is an `Int {value}` describing the size of `{value}` in bytes.
- `{value}` is the two's complement of the `BigInteger`.

Example values of the two's complement `{value}`:

- `00`: Integer 0.
- `01`: Integer 1.
- `7f`: Integer 127.
- `00 80`: Integer 128.
- `ff`: Integer -1.
- `80`: Integer -128.
- `ff 7f`: Integer -129.

==== Byte

Format: 1-byte two's complement integer.

Example values:

- `01`: 8-bit integer number 1.
- `ff`: 8-bit integer number -1.

==== Binary

Format: `{length}{value}`

Where:

- `{length}` is an `Int {value}` representing the amount of bytes contained in the value.
- `{value}` sequence of bytes.

==== Short

Format: 2-byte two's complement integer.

Example values:

- `00 01`: 16-bit integer number 1.
- `01 02`: 16-bit integer number 258.

==== Boolean

Format: A single byte containing the value `0x01` when it's `true` and `0` otherwise.

==== Tree

Format: `{length}{item_0}...{item_n}`

Where:

- `{length}` is an `Int {value}` describing the amount of items.
- `{item_0}...{item_n}` are the items of the `Tree`. `{item_i}` is composed of a `{key}` which is a fully-qualified typed value
followed by a `{Tree}`.

==== Marker

A 1-byte marker used to separate the end of the data and the beginning of the status of a `ResponseMessage`. This is
mainly used by language variants during deserialization.

Format: 1-byte integer with a value of `00`.

==== CompositePDT

A composite custom type, represented as a type and a map of values.

Format: `{type}{fields}`

Where:

- `{type}` is a fully-qualified `String` containing the implementation specific text identifier of the custom type.
- `{fields}` is a fully-qualified `Map` representing the fields of the composite type.

Example values:

[source,text]
----
03 00 00 00 00 05 50 6F 69 6E 74: the string "Point"
0A 00 00 00 00 02: a length 2 map
03 00 00 00 00 01 78 01 00 00 00 00 01: {x:1}
03 00 00 00 00 01 79 01 00 00 00 00 02: {y:2}
----

==== PrimitivePDT

A primitive custom type, represented as a type and the stringified value.

Format: `{type}{value}`

Where:

- `{type}` is a fully-qualified `String` containing the implementation specific text identifier of the custom type.
- `{value}` is a fully-qualified `String` representing the string version of the value.

Example values:

[source,text]
----
03 00 00 00 00 05 55 69 6E 74 38: the string "Uint8"
03 00 00 00 00 02 31 30: the string "10"
----

==== Unspecified Null Object

A `null` value for an unspecified Object value.

It's represented using the null `{value_flag}` set and no sequence of bytes (which is `FE 01`).

==== Char

Format: one to four bytes representing a single UTF8 char, according to the Unicode standard.

For characters `0x00`-`0x7F`, UTF-8 encodes the character as a single byte.

For characters `0x80`-`0x07FF`, UTF-8 uses 2 bytes: the first byte is binary `110` followed by the 5 high bits of the
character, while the second byte is binary 10 followed by the 6 low bits of the character.

The 3 and 4-byte encodings are similar to the 2-byte encoding, except that the first byte of the 3-byte encoding starts
with `1110` and the first byte of the 4-byte encoding starts with `11110`.

Example values (hex bytes):

- `61`: Character 'a'.
- `c2 a2`: Character '¢'.
- `e2 82 ac`: Character '€'

==== Duration

A time-based amount of time.

Format: `{seconds}{nanos}`

Where:

- `{seconds}` is a `Long {value}`.
- `{nanos}` is an `Int {value}`.

=== Request and Response Messages

Request and response messages are special container types used to represent messages from client to the server and the
other way around. These messages are independent from the transport layer.

==== Request Message

Represents a message from the client to the server.

Format: `{version}{fields}{gremlin}`

Where:

- `{version}` is a `Byte {value}` representing the specification version, with the most significant bit set to one. For this
version of the format, the value expected is `0x84` (`10000100`).
- `{fields}` is a `Map {value}`.
- `{gremlin}` is a `String {value}`.

The total length is not part of the message as the transport layer will provide it. For example: in HTTP, there is the
`Content-Length` header which defines the payload size.

==== Response Message

Format: `{version}{bulked}{result_data}{marker}{status_code}{status_message}{exception}`

Where:

- `{version}` is a `Byte {value}` representing the protocol version, with the most significant bit set to one. For this version
of the protocol, the value expected is `0x84` (`10000100`).
- `{bulked}` is a `Byte {value}` representing whether `{result_data}` is bulked. `00` is false and `01` is true.
- `{result_data}` is a sequence of fully-qualified typed value composed of `{type_code}{type_info}{value_flag}{value}`.
If `{bulked}` is `01` then each value is followed by a `Long` value denoting the bulk of the preceding value.
- `{marker}` is a fully-qualified `Marker`.
- `{status_code}` is an `Int {value}`.
- `{status_message}` is a nullable `{value_flag}` and `String {value}`.
- `{exception}` is a nullable `{value_flag}` and `String {value}`.

The total length is not part of the message as the transport layer will provide it.