| --- |
| title: Framing |
| --- |
| |
| Every message on a binary transport is a fixed **256-byte header** followed by an optional body. The header is a `#[repr(C)]` struct decoded by pointer cast (zero-copy), so field offsets are fixed and enforced at compile time in the server. Three header shapes cross the client boundary: |
| |
| - `RequestHeader`: client to server. |
| - `ReplyHeader`: server to client, answers one request. |
| - `EvictionHeader`: server to client, session-terminal rejection with no body. |
| |
| The header shape is identified by the `command` byte at offset 60. Values a client sends or receives: |
| |
| | `command` | Value | Direction | Meaning | |
| |-----------|-------|-----------|---------| |
| | `Request` | 5 | client to server | A command request | |
| | `Reply` | 8 | server to client | Answer to one request | |
| | `Eviction` | 13 | server to client | Session is dead, no per-request correlation | |
| |
| Other `command` values (Prepare, PrepareOk, view-change traffic, and so on) are replica-to-replica consensus messages and never appear on a client connection. They are documented on the [Server-to-server](/docs/binary-protocol/cluster) page. |
| |
| The `size` field at offset 48 is the total frame length in bytes, header included. A body, when present, immediately follows the 256 header bytes and has length `size - 256`. Readers can decode `size` before typing the header. That offset is a protocol constant. |
| |
| ## RequestHeader |
| |
| Client to server. 256 bytes. |
| |
| | Offset | Size | Field | Type | Description | |
| |--------|------|-------|------|-------------| |
| | 0 | 16 | `checksum` | u128 | Frame seal. Not used on client-facing frames; send zero. | |
| | 16 | 16 | `checksum_body` | u128 | Body seal. Not used on client-facing frames; send zero. | |
| | 32 | 16 | `cluster` | u128 | Cluster id. Send zero. | |
| | 48 | 4 | `size` | u32 | Total frame length: 256 + body length. | |
| | 52 | 4 | `view` | u32 | Consensus view. Send zero. | |
| | 56 | 4 | `release` | u32 | Must be zero. | |
| | 60 | 1 | `command` | u8 | `5` (`Request`). | |
| | 61 | 1 | `replica` | u8 | Send zero. | |
| | 62 | 66 | reserved | bytes | Zero. | |
| | 128 | 16 | `client` | u128 | Client-chosen session identity, non-zero. Minted fresh for each registration. | |
| | 144 | 16 | `request_checksum` | u128 | Optional integrity stamp over the request body (the Rust SDK uses XxHash3-64 widened to u128). Lets the server's client table catch a `request` number reused for different arguments. Zero disables the comparison. Stamped only for metadata-plane operations; zero for partition-plane and non-replicated ones. | |
| | 160 | 8 | `timestamp` | u64 | Informational; the server echoes it into `ReplyHeader.timestamp`. May be zero. | |
| | 168 | 8 | `request` | u64 | Request number, per client. See [request numbering](#request-numbering). | |
| | 176 | 1 | `operation` | u8 | The [`Operation`](#operation-discriminants) discriminant. | |
| | 177 | 7 | padding | bytes | Zero. | |
| | 184 | 8 | `session` | u64 | Session fence epoch from the login reply. Zero on the login-register request itself and on non-replicated operations sent before login (only `PING` is accepted unauthenticated). | |
| | 192 | 4 | `user_id` | u32 | Ignored on the wire; the server stamps the authenticated user itself. Send zero. | |
| | 196 | 60 | reserved | bytes | Zero, except: for `operation = 2` (`NonReplicated`) bytes 196..200 carry the u32 command code, little-endian. | |
| |
| ## Operation discriminants |
| |
| The `operation` byte tells the server which state-machine operation the request carries. Replicated operations are identified by `operation` alone. Non-replicated operations (reads, ping) all use `operation = 2` and carry their concrete u32 command code in header bytes 196..200. |
| |
| Values a client may send: |
| |
| | Value | Operation | Plane | |
| |-------|-----------|-------| |
| | 1 | `Register` | session (login handshake, codes 40 and 45) | |
| | 2 | `NonReplicated` | reads and ping; the u32 code rides bytes 196..200 | |
| | 3 | `Logout` | session (code 39) | |
| | 128 | `CreateStream` | metadata | |
| | 129 | `UpdateStream` | metadata | |
| | 130 | `DeleteStream` | metadata | |
| | 131 | `PurgeStream` | metadata | |
| | 132 | `CreateTopic` | metadata | |
| | 133 | `UpdateTopic` | metadata | |
| | 134 | `DeleteTopic` | metadata | |
| | 135 | `PurgeTopic` | metadata | |
| | 136 | `CreatePartitions` | metadata | |
| | 137 | `DeletePartitions` | metadata | |
| | 138 | `DeleteSegments` | metadata (resolved to an internal partition truncation) | |
| | 139 | `CreateConsumerGroup` | metadata | |
| | 140 | `DeleteConsumerGroup` | metadata | |
| | 141 | `CreateUser` | metadata | |
| | 142 | `UpdateUser` | metadata | |
| | 143 | `DeleteUser` | metadata | |
| | 144 | `ChangePassword` | metadata | |
| | 145 | `UpdatePermissions` | metadata | |
| | 146 | `CreatePersonalAccessToken` | metadata | |
| | 147 | `DeletePersonalAccessToken` | metadata | |
| | 148 | `JoinConsumerGroup` | metadata | |
| | 149 | `LeaveConsumerGroup` | metadata | |
| | 160 | `SendMessages` | partition | |
| | 161 | `StoreConsumerOffset` | partition | |
| | 162 | `DeleteConsumerOffset` | partition | |
| |
| Value 0 is reserved and rejected. The 64..127 range is reserved for server-internal operations (currently 64 through 68 are assigned) and every value in it is refused from clients. |
| |
| The planes matter for delivery semantics: |
| |
| - **Metadata operations** replicate through the metadata consensus group. The server deduplicates them by `(client, request)` and caches replies, so a retried request gets the cached answer instead of a double apply (exactly-once). |
| - **Partition operations** replicate through their partition's consensus group. They are at-least-once: no reply cache, a replay may apply again. |
| - **Non-replicated operations** are reads. They bypass consensus and deduplication entirely. |
| |
| ## Request numbering |
| |
| `request` is a per-client counter the server's client table tracks for metadata-plane operations: |
| |
| - Metadata operations must send a strictly increasing `request` (the SDK advances the counter per metadata request). |
| - Partition operations and non-replicated operations send the current counter value without advancing it. The server doesn't track theirs. |
| - The counter is a watermark, not a contiguous sequence: any value above the last accepted one is admissible. |
| |
| `session` is the fence epoch: the value handed back by the login reply. Every request after login must echo it. When the same `client` id registers again, the new registration mints a higher epoch and requests carrying the old one are fenced (rejected as zombies). |
| |
| ## ReplyHeader |
| |
| Server to client. 256 bytes, followed by `size - 256` bytes of body. |
| |
| | Offset | Size | Field | Type | Description | |
| |--------|------|-------|------|-------------| |
| | 0 | 16 | `checksum` | u128 | Zero on client-facing frames. | |
| | 16 | 16 | `checksum_body` | u128 | Zero on client-facing frames. | |
| | 32 | 16 | `cluster` | u128 | Cluster id. | |
| | 48 | 4 | `size` | u32 | Total frame length: 256 + body length. | |
| | 52 | 4 | `view` | u32 | Consensus view the reply was produced in. | |
| | 56 | 4 | `release` | u32 | Zero. | |
| | 60 | 1 | `command` | u8 | `8` (`Reply`). | |
| | 61 | 1 | `replica` | u8 | Answering replica index. | |
| | 62 | 66 | reserved | bytes | Zero. | |
| | 128 | 16 | `request_checksum` | u128 | Echoed from the request. | |
| | 144 | 16 | `context` | u128 | Server context. | |
| | 160 | 16 | `client` | u128 | Echoed client id. | |
| | 176 | 8 | `op` | u64 | Log position of the committed operation. | |
| | 184 | 8 | `commit` | u64 | Commit point at reply time. | |
| | 192 | 8 | `timestamp` | u64 | Echo of the request `timestamp`. | |
| | 200 | 8 | `request` | u64 | Echoed request number; correlate replies by this. | |
| | 208 | 1 | `operation` | u8 | Echoed operation discriminant. | |
| | 209 | 7 | padding | bytes | Zero. | |
| | 216 | 4 | `status` | u32 | `0` = accepted. Nonzero = an `IggyError` code for a failure decided **before** commit (authorization denial, admission reject). A nonzero status always comes with an empty body. | |
| | 220 | 36 | reserved | bytes | Zero. | |
| |
| Decode order for a client: |
| |
| 1. Read 256 bytes, check `command`. `13` (`Eviction`) means the session is dead: map the eviction reason to an error and stop. `8` (`Reply`) continues. |
| 2. Read the remaining `size - 256` body bytes. |
| 3. If `status` is nonzero, the request failed pre-commit. The status value is the `IggyError` code and the body is empty. |
| 4. Otherwise decode the body. For result-framed operations, strip the [result section](#result-section) first. |
| |
| ## Result section |
| |
| Replies for all metadata operations and for the partition-plane consumer-offset writes (`StoreConsumerOffset`, `DeleteConsumerOffset`) are **result-framed**: the body starts with a committed-result section ahead of the typed payload. |
| |
| ```text |
| [count: u32] |
| count x { index: u32, result: u32 } |
| ``` |
| |
| - Success: `count = 0`, and the typed response payload (if any) follows the 4 count bytes. |
| - Committed business rejection: one entry `{ index: 0, result: error_code }` and no payload. |
| |
| The header `status` channel and the result section are mutually exclusive by construction: a reply either failed pre-commit (`status` nonzero, empty body) or committed (`status` zero, result section present). A login-register reply carries the result section only when non-empty. On success its body starts directly with the [login response payload](/docs/binary-protocol/connection-lifecycle#login-register). |
| |
| Replies to non-replicated commands aren't result-framed: after `status = 0` the body is the response payload directly. |
| |
| ## EvictionHeader |
| |
| Server to client. 256 bytes, never a body. An eviction is session-terminal: it says "this session is dead", carries no per-request correlation, and a client should deinitialize and re-login. |
| |
| | Offset | Size | Field | Type | Description | |
| |--------|------|-------|------|-------------| |
| | 0 | 16 | `checksum` | u128 | Zero on client-facing frames. | |
| | 16 | 16 | `checksum_body` | u128 | Zero on client-facing frames. | |
| | 32 | 16 | `cluster` | u128 | Cluster id. | |
| | 48 | 4 | `size` | u32 | Always 256: an eviction has no body. | |
| | 52 | 4 | `view` | u32 | | |
| | 56 | 4 | `release` | u32 | Zero. | |
| | 60 | 1 | `command` | u8 | `13` (`Eviction`). | |
| | 61 | 1 | `replica` | u8 | | |
| | 62 | 66 | reserved | bytes | Zero. | |
| | 128 | 16 | `client` | u128 | The evicted client id. | |
| | 144 | 4 | `server_protocol_version` | u32 | Accepted protocol window (max), packed semver. Set only for reason 14, zero otherwise. | |
| | 148 | 4 | `server_protocol_version_min` | u32 | Accepted protocol window (min). Set only for reason 14, zero otherwise. | |
| | 152 | 103 | reserved | bytes | Zero. | |
| | 255 | 1 | `reason` | u8 | Eviction reason, see below. | |
| |
| Eviction reasons: |
| |
| | Value | Reason | Meaning | |
| |-------|--------|---------| |
| | 1 | `NoSession` | No session exists for this client id | |
| | 2 | `ClientReleaseTooLow` | Client release below cluster minimum | |
| | 3 | `ClientReleaseTooHigh` | Client release above cluster maximum | |
| | 4 | `InvalidRequestOperation` | Unknown `operation` discriminant | |
| | 5 | `InvalidRequestBody` | Body failed validation | |
| | 6 | `InvalidRequestBodySize` | Body size mismatch | |
| | 7 | `SessionTooLow` | Session epoch below the cluster's retained minimum | |
| | 8 | `SessionReleaseMismatch` | Session bound to a different release | |
| | 9 | `InvalidCredentials` | Login refused: bad username or password | |
| | 10 | `InvalidToken` | Login refused: bad personal access token | |
| | 11 | `UserInactive` | Login refused: user inactive | |
| | 12 | `SessionError` | Session-level failure | |
| | 13 | `StaleClient` | Missed heartbeats; server evicted the session | |
| | 14 | `IncompatibleProtocol` | Client protocol version outside the accepted window (see bytes 144..152) | |
| | 15 | `MalformedLogin` | Login body without a decodable version prefix, or a legacy login code | |
| |
| Value 0 is reserved and never sent. |