manual/developer/native_protocol/README.md - cassandra-java-driver - Git at Google

 <!--
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
 -->

 ## Native protocol layer

 The native protocol layer encodes protocol messages into binary, before they are sent over the
 network.

 This part of the code lives in its own project:
 [native-protocol](https://github.com/datastax/native-protocol). We extracted it to make it reusable
 ([Simulacron](https://github.com/datastax/simulacron) also uses it).

 The protocol specifications are available in
 [native-protocol/src/main/resources](https://github.com/datastax/native-protocol/tree/1.x/src/main/resources).
 These files originally come from Cassandra, we copy them over for easy access. Note that, if the
 latest version is a beta (this is the case for v5 at the time of writing -- September 2019), the
 specification might not be up to date. Always compare with the latest revision in
 [cassandra/doc](https://github.com/apache/cassandra/tree/trunk/doc).


 For a broad overview of how protocol types are used in the driver, let's step through an example:

 * the user calls `session.execute()` with a `SimpleStatement`. The protocol message for a
   non-prepared request is `QUERY`;
 * `CqlRequestHandler` uses `Conversions.toMessage` to convert the statement into a
   `c.d.o.protocol.internal.request.Query`;
 * `InflightHandler.write` assigns a stream id to that message, and wraps it into a
   `c.d.o.protocol.internal.Frame`;
 * `FrameEncoder` uses `c.d.o.protocol.internal.FrameCodec` to convert the frame to binary.

 (All types prefixed with `c.d.o.protocol.internal` belong to the native-protocol project.)

 A similar process happens on the response path: decode the incoming binary payload into a protocol
 message, then convert the message into higher-level driver objects: `ResultSet`, `ExecutionInfo`,
 etc.

 ### Native protocol types

 #### Messages

 Every protocol message is identified by an opcode, and has a corresponding `Message` subclass.

 A `Frame` wraps a message to add metadata, such as the protocol version and stream id.

 ```ditaa
 +-------+ contains +------------+
 | Frame +--------->+ Message    +
 +-------+          +------------+
                    | int opcode |
                    +--+---------+
                       |
                       |    +---------+
                       +----+ Query   |
                       |    +---------+
                       |
                       |    +---------+
                       +----+ Execute |
                       |    +---------+
                       |
                       |    +---------+
                       +----+ Rows    |
                            +---------+

                            etc.
 ```

 All value classes are immutable, but for efficiency they don't make defensive copies of their
 fields. If these fields are mutable (for example collections), they shouldn't be modified after
 creating a message instance.

 The code makes very few assumptions about how the messages will be used. Data is often represented
 in the most simple way. For example, `ProtocolConstants` uses simple integer constants to represent
 protocol codes (enums wouldn't work at that level, because we need to add new codes in the DSE
 driver); the driver generally rewraps them in more type-safe structures before exposing them to
 higher-level layers.

 #### Encoding/decoding

 For every message, there is a corresponding `Message.Codec` for encoding and decoding. A
 `FrameCodec` relies on a set of message codecs, for one or more protocol versions. Given an incoming
 frame, it looks up the right message codec to use, based on the protocol version and opcode.
 Optionally, it compresses frame bodies with a `Compressor`.


 ```ditaa
 +-----------------+                +-------------------+
 | FrameCodec[B]   +----------------+ PrimitiveCodec[B] |
 +-----------------+                +-------------------+
 | B encode(Frame) |
 | Frame decode(B) +-------+        +---------------+
 +------+----------+       +--------+ Compressor[B] |
        |                           +---------------+
        |
        |                           +-------------------+
        +---------------------------+ Message.Codec     |
            1 codec per opcode      +-------------------+
            and protocol version    | B encode(Message) |
                                    | Message decode(B) |
                                    +-------------------+
 ```

 Most of the time, you'll want to use the full set of message codecs for a given protocol version.
 `CodecGroup` provides a convenient way to register multiple codecs at once. The project provides
 default implementations for all supported protocol version, both for clients like the driver (e.g.
 encode `QUERY`, decode `RESULT`), or servers like Simulacron (decode `QUERY` encode `RESULT`).


 ```ditaa
 +-------------+
 |  CodecGroup |
 +------+------+
        |
        |    +------------------------+
        +----+ ProtocolV3ClientCodecs |
        |    +------------------------+
        |
        |    +------------------------+
        +----+ ProtocolV3ServerCodecs |
        |    +------------------------+
        |
        |    +------------------------+
        +----+ ProtocolV4ClientCodecs |
        |    +------------------------+
        |
        |    +------------------------+
        +----+ ProtocolV4ClientCodecs |
        |    +------------------------+
        |
        |    +------------------------+
        +----+ ProtocolV5ClientCodecs |
        |    +------------------------+
        |
        |    +------------------------+
        +----+ ProtocolV5ClientCodecs |
             +------------------------+
 ```

 The native protocol layer is agnostic to the actual binary representation. In the driver, this
 happens to be a Netty `ByteBuf`, but the encoding logic doesn't need to be aware of that. This is
 expressed by the type parameter `B` in `FrameCodec<B>`. `PrimitiveCodec<B>` abstracts the basic
 primitives to work with a `B`: how to create an instance, read and write data to it, etc.

 ```java
 public interface PrimitiveCodec<B> {
   B allocate(int size);
   int readInt(B source);
   void writeInt(int i, B dest);
   ...
 }
 ```

 Everything else builds upon those primitives. By just switching the `PrimitiveCodec` implementation,
 the whole protocol layer could be reused with a different type, such as `byte[]`.

 In summary, to initialize a `FrameCodec`, you need:

 * a `PrimitiveCodec`;
 * a `Compressor` (optional);
 * one or more `CodecGroup`s.

 ### Integration in the driver

 The driver initializes its `FrameCodec` in `DefaultDriverContext.buildFrameCodec()`.

 * the primitive codec is `ByteBufPrimitiveCodec`, which implements the basic primitives for Netty's
   `ByteBuf`;
 * the compressor comes from `DefaultDriverContext.buildCompressor()`, which determines the
   implementation from the configuration;
 * it is built with `FrameCodec.defaultClient`, which is a shortcut to use the default client groups:
   `ProtocolV3ClientCodecs`, `ProtocolV4ClientCodecs` and `ProtocolV5ClientCodecs`.

 ### Extension points

 The default frame codec can be replaced by [extending the
 context](../common/context/#overriding-a-context-component) to override `buildFrameCodec`. This
 can be used to add or remove a protocol version, or replace a particular codec.

 If protocol versions change, `ProtocolVersionRegistry` will likely be affected as well.

 Also, depending on the nature of the protocol changes, the driver's [request
 processors](../request_execution/#request-processors) might require some adjustments: either replace
 them, or introduce separate ones (possibly with new `executeXxx()` methods on a custom session
 interface).
	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->

	## Native protocol layer

	The native protocol layer encodes protocol messages into binary, before they are sent over the
	network.

	This part of the code lives in its own project:
	[native-protocol](https://github.com/datastax/native-protocol). We extracted it to make it reusable
	([Simulacron](https://github.com/datastax/simulacron) also uses it).

	The protocol specifications are available in
	[native-protocol/src/main/resources](https://github.com/datastax/native-protocol/tree/1.x/src/main/resources).
	These files originally come from Cassandra, we copy them over for easy access. Note that, if the
	latest version is a beta (this is the case for v5 at the time of writing -- September 2019), the
	specification might not be up to date. Always compare with the latest revision in
	[cassandra/doc](https://github.com/apache/cassandra/tree/trunk/doc).


	For a broad overview of how protocol types are used in the driver, let's step through an example:

	* the user calls `session.execute()` with a `SimpleStatement`. The protocol message for a
	non-prepared request is `QUERY`;
	* `CqlRequestHandler` uses `Conversions.toMessage` to convert the statement into a
	`c.d.o.protocol.internal.request.Query`;
	* `InflightHandler.write` assigns a stream id to that message, and wraps it into a
	`c.d.o.protocol.internal.Frame`;
	* `FrameEncoder` uses `c.d.o.protocol.internal.FrameCodec` to convert the frame to binary.

	(All types prefixed with `c.d.o.protocol.internal` belong to the native-protocol project.)

	A similar process happens on the response path: decode the incoming binary payload into a protocol
	message, then convert the message into higher-level driver objects: `ResultSet`, `ExecutionInfo`,
	etc.

	### Native protocol types

	#### Messages

	Every protocol message is identified by an opcode, and has a corresponding `Message` subclass.

	A `Frame` wraps a message to add metadata, such as the protocol version and stream id.

	```ditaa
	+-------+ contains +------------+
	\| Frame +--------->+ Message +
	+-------+ +------------+
	\| int opcode \|
	+--+---------+
	\|
	\| +---------+
	+----+ Query \|
	\| +---------+
	\|
	\| +---------+
	+----+ Execute \|
	\| +---------+
	\|
	\| +---------+
	+----+ Rows \|
	+---------+

	etc.
	```

	All value classes are immutable, but for efficiency they don't make defensive copies of their
	fields. If these fields are mutable (for example collections), they shouldn't be modified after
	creating a message instance.

	The code makes very few assumptions about how the messages will be used. Data is often represented
	in the most simple way. For example, `ProtocolConstants` uses simple integer constants to represent
	protocol codes (enums wouldn't work at that level, because we need to add new codes in the DSE
	driver); the driver generally rewraps them in more type-safe structures before exposing them to
	higher-level layers.

	#### Encoding/decoding

	For every message, there is a corresponding `Message.Codec` for encoding and decoding. A
	`FrameCodec` relies on a set of message codecs, for one or more protocol versions. Given an incoming
	frame, it looks up the right message codec to use, based on the protocol version and opcode.
	Optionally, it compresses frame bodies with a `Compressor`.


	```ditaa
	+-----------------+ +-------------------+
	\| FrameCodec[B] +----------------+ PrimitiveCodec[B] \|
	+-----------------+ +-------------------+
	\| B encode(Frame) \|
	\| Frame decode(B) +-------+ +---------------+
	+------+----------+ +--------+ Compressor[B] \|
	\| +---------------+
	\|
	\| +-------------------+
	+---------------------------+ Message.Codec \|
	1 codec per opcode +-------------------+
	and protocol version \| B encode(Message) \|
	\| Message decode(B) \|
	+-------------------+
	```

	Most of the time, you'll want to use the full set of message codecs for a given protocol version.
	`CodecGroup` provides a convenient way to register multiple codecs at once. The project provides
	default implementations for all supported protocol version, both for clients like the driver (e.g.
	encode `QUERY`, decode `RESULT`), or servers like Simulacron (decode `QUERY` encode `RESULT`).


	```ditaa
	+-------------+
	\| CodecGroup \|
	+------+------+
	\|
	\| +------------------------+
	+----+ ProtocolV3ClientCodecs \|
	\| +------------------------+
	\|
	\| +------------------------+
	+----+ ProtocolV3ServerCodecs \|
	\| +------------------------+
	\|
	\| +------------------------+
	+----+ ProtocolV4ClientCodecs \|
	\| +------------------------+
	\|
	\| +------------------------+
	+----+ ProtocolV4ClientCodecs \|
	\| +------------------------+
	\|
	\| +------------------------+
	+----+ ProtocolV5ClientCodecs \|
	\| +------------------------+
	\|
	\| +------------------------+
	+----+ ProtocolV5ClientCodecs \|
	+------------------------+
	```

	The native protocol layer is agnostic to the actual binary representation. In the driver, this
	happens to be a Netty `ByteBuf`, but the encoding logic doesn't need to be aware of that. This is
	expressed by the type parameter `B` in `FrameCodec<B>`. `PrimitiveCodec<B>` abstracts the basic
	primitives to work with a `B`: how to create an instance, read and write data to it, etc.

	```java
	public interface PrimitiveCodec<B> {
	B allocate(int size);
	int readInt(B source);
	void writeInt(int i, B dest);
	...
	}
	```

	Everything else builds upon those primitives. By just switching the `PrimitiveCodec` implementation,
	the whole protocol layer could be reused with a different type, such as `byte[]`.

	In summary, to initialize a `FrameCodec`, you need:

	* a `PrimitiveCodec`;
	* a `Compressor` (optional);
	* one or more `CodecGroup`s.

	### Integration in the driver

	The driver initializes its `FrameCodec` in `DefaultDriverContext.buildFrameCodec()`.

	* the primitive codec is `ByteBufPrimitiveCodec`, which implements the basic primitives for Netty's
	`ByteBuf`;
	* the compressor comes from `DefaultDriverContext.buildCompressor()`, which determines the
	implementation from the configuration;
	* it is built with `FrameCodec.defaultClient`, which is a shortcut to use the default client groups:
	`ProtocolV3ClientCodecs`, `ProtocolV4ClientCodecs` and `ProtocolV5ClientCodecs`.

	### Extension points

	The default frame codec can be replaced by [extending the
	context](../common/context/#overriding-a-context-component) to override `buildFrameCodec`. This
	can be used to add or remove a protocol version, or replace a particular codec.

	If protocol versions change, `ProtocolVersionRegistry` will likely be affected as well.

	Also, depending on the nature of the protocol changes, the driver's [request
	processors](../request_execution/#request-processors) might require some adjustments: either replace
	them, or introduce separate ones (possibly with new `executeXxx()` methods on a custom session
	interface).