| <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="Frame a stream of bytes based on a length prefix"><meta name="keywords" content="rust, rustlang, rust-lang, length_delimited"><title>tokio_util::codec::length_delimited - Rust</title><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../SourceSerif4-Regular.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../FiraSans-Regular.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../FiraSans-Medium.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../SourceCodePro-Regular.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../SourceSerif4-Bold.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../../SourceCodePro-Semibold.ttf.woff2"><link rel="stylesheet" href="../../../normalize.css"><link rel="stylesheet" href="../../../rustdoc.css" id="mainThemeStyle"><link rel="stylesheet" href="../../../ayu.css" disabled><link rel="stylesheet" href="../../../dark.css" disabled><link rel="stylesheet" href="../../../light.css" id="themeStyle"><script id="default-settings" ></script><script src="../../../storage.js"></script><script defer src="../../../main.js"></script><noscript><link rel="stylesheet" href="../../../noscript.css"></noscript><link rel="alternate icon" type="image/png" href="../../../favicon-16x16.png"><link rel="alternate icon" type="image/png" href="../../../favicon-32x32.png"><link rel="icon" type="image/svg+xml" href="../../../favicon.svg"></head><body class="rustdoc mod"><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><nav class="mobile-topbar"><button class="sidebar-menu-toggle">☰</button><a class="sidebar-logo" href="../../../tokio_util/index.html"><div class="logo-container"><img class="rust-logo" src="../../../rust-logo.svg" alt="logo"></div></a><h2></h2></nav><nav class="sidebar"><a class="sidebar-logo" href="../../../tokio_util/index.html"><div class="logo-container"><img class="rust-logo" src="../../../rust-logo.svg" alt="logo"></div></a><h2 class="location"><a href="#">Module length_delimited</a></h2><div class="sidebar-elems"><section><ul class="block"><li><a href="#structs">Structs</a></li></ul></section></div></nav><main><div class="width-limiter"><nav class="sub"><form class="search-form"><div class="search-container"><span></span><input class="search-input" name="search" autocomplete="off" spellcheck="false" placeholder="Click or press ‘S’ to search, ‘?’ for more options…" type="search"><div id="help-button" title="help" tabindex="-1"><a href="../../../help.html">?</a></div><div id="settings-menu" tabindex="-1"><a href="../../../settings.html" title="settings"><img width="22" height="22" alt="Change settings" src="../../../wheel.svg"></a></div></div></form></nav><section id="main-content" class="content"><div class="main-heading"><h1 class="fqn">Module <a href="../../index.html">tokio_util</a>::<wbr><a href="../index.html">codec</a>::<wbr><a class="mod" href="#">length_delimited</a><button id="copy-path" onclick="copy_path(this)" title="Copy item path to clipboard"><img src="../../../clipboard.svg" width="19" height="18" alt="Copy item path"></button></h1><span class="out-of-band"><a class="srclink" href="../../../src/tokio_util/codec/length_delimited.rs.html#1-1043">source</a> · <a id="toggle-all-docs" href="javascript:void(0)" title="collapse all docs">[<span class="inner">−</span>]</a></span></div><details class="rustdoc-toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>Frame a stream of bytes based on a length prefix</p> |
| <p>Many protocols delimit their frames by prefacing frame data with a |
| frame head that specifies the length of the frame. The |
| <code>length_delimited</code> module provides utilities for handling the length |
| based framing. This allows the consumer to work with entire frames |
| without having to worry about buffering or other framing logic.</p> |
| <h2 id="getting-started"><a href="#getting-started">Getting started</a></h2> |
| <p>If implementing a protocol from scratch, using length delimited framing |
| is an easy way to get started. <a href="struct.LengthDelimitedCodec.html#method.new"><code>LengthDelimitedCodec::new()</code></a> will |
| return a length delimited codec using default configuration values. |
| This can then be used to construct a framer to adapt a full-duplex |
| byte stream into a stream of frames.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio::io::{AsyncRead, AsyncWrite}; |
| <span class="kw">use </span>tokio_util::codec::{Framed, LengthDelimitedCodec}; |
| |
| <span class="kw">fn </span>bind_transport<T: AsyncRead + AsyncWrite>(io: T) |
| -> Framed<T, LengthDelimitedCodec> |
| { |
| Framed::new(io, LengthDelimitedCodec::new()) |
| }</code></pre></div> |
| <p>The returned transport implements <code>Sink + Stream</code> for <code>BytesMut</code>. It |
| encodes the frame with a big-endian <code>u32</code> header denoting the frame |
| payload length:</p> |
| <div class="example-wrap"><pre class="language-text"><code>+----------+--------------------------------+ |
| | len: u32 | frame payload | |
| +----------+--------------------------------+</code></pre></div> |
| <p>Specifically, given the following:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio::io::{AsyncRead, AsyncWrite}; |
| <span class="kw">use </span>tokio_util::codec::{Framed, LengthDelimitedCodec}; |
| |
| <span class="kw">use </span>futures::SinkExt; |
| <span class="kw">use </span>bytes::Bytes; |
| |
| <span class="kw">async fn </span>write_frame<T>(io: T) -> <span class="prelude-ty">Result</span><(), Box<<span class="kw">dyn </span>std::error::Error>> |
| <span class="kw">where |
| </span>T: AsyncRead + AsyncWrite + Unpin, |
| { |
| <span class="kw">let </span><span class="kw-2">mut </span>transport = Framed::new(io, LengthDelimitedCodec::new()); |
| <span class="kw">let </span>frame = Bytes::from(<span class="string">"hello world"</span>); |
| |
| transport.send(frame).<span class="kw">await</span><span class="question-mark">?</span>; |
| <span class="prelude-val">Ok</span>(()) |
| }</code></pre></div> |
| <p>The encoded frame will look like this:</p> |
| <div class="example-wrap"><pre class="language-text"><code>+---- len: u32 ----+---- data ----+ |
| | \x00\x00\x00\x0b | hello world | |
| +------------------+--------------+</code></pre></div><h2 id="decoding"><a href="#decoding">Decoding</a></h2> |
| <p><a href="../struct.FramedRead.html"><code>FramedRead</code></a> adapts an <a href="../../../tokio/io/async_read/trait.AsyncRead.html"><code>AsyncRead</code></a> into a <code>Stream</code> of <a href="../../../bytes/bytes_mut/struct.BytesMut.html"><code>BytesMut</code></a>, |
| such that each yielded <a href="../../../bytes/bytes_mut/struct.BytesMut.html"><code>BytesMut</code></a> value contains the contents of an |
| entire frame. There are many configuration parameters enabling |
| <a href="../struct.FramedRead.html"><code>FramedRead</code></a> to handle a wide range of protocols. Here are some |
| examples that will cover the various options at a high level.</p> |
| <h3 id="example-1"><a href="#example-1">Example 1</a></h3> |
| <p>The following will parse a <code>u16</code> length field at offset 0, including the |
| frame head in the yielded <code>BytesMut</code>.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">0</span>) <span class="comment">// default value |
| </span>.length_field_type::<u16>() |
| .length_adjustment(<span class="number">0</span>) <span class="comment">// default value |
| </span>.num_skip(<span class="number">0</span>) <span class="comment">// Do not strip frame header |
| </span>.new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT DECODED |
| +-- len ---+--- Payload ---+ +-- len ---+--- Payload ---+ |
| | \x00\x0B | Hello world | --> | \x00\x0B | Hello world | |
| +----------+---------------+ +----------+---------------+</code></pre></div> |
| <p>The value of the length field is 11 (<code>\x0B</code>) which represents the length |
| of the payload, <code>hello world</code>. By default, <a href="../struct.FramedRead.html"><code>FramedRead</code></a> assumes that |
| the length field represents the number of bytes that <strong>follows</strong> the |
| length field. Thus, the entire frame has a length of 13: 2 bytes for the |
| frame head + 11 bytes for the payload.</p> |
| <h3 id="example-2"><a href="#example-2">Example 2</a></h3> |
| <p>The following will parse a <code>u16</code> length field at offset 0, omitting the |
| frame head in the yielded <code>BytesMut</code>.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">0</span>) <span class="comment">// default value |
| </span>.length_field_type::<u16>() |
| .length_adjustment(<span class="number">0</span>) <span class="comment">// default value |
| // `num_skip` is not needed, the default is to skip |
| </span>.new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT DECODED |
| +-- len ---+--- Payload ---+ +--- Payload ---+ |
| | \x00\x0B | Hello world | --> | Hello world | |
| +----------+---------------+ +---------------+</code></pre></div> |
| <p>This is similar to the first example, the only difference is that the |
| frame head is <strong>not</strong> included in the yielded <code>BytesMut</code> value.</p> |
| <h3 id="example-3"><a href="#example-3">Example 3</a></h3> |
| <p>The following will parse a <code>u16</code> length field at offset 0, including the |
| frame head in the yielded <code>BytesMut</code>. In this case, the length field |
| <strong>includes</strong> the frame head length.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">0</span>) <span class="comment">// default value |
| </span>.length_field_type::<u16>() |
| .length_adjustment(-<span class="number">2</span>) <span class="comment">// size of head |
| </span>.num_skip(<span class="number">0</span>) |
| .new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT DECODED |
| +-- len ---+--- Payload ---+ +-- len ---+--- Payload ---+ |
| | \x00\x0D | Hello world | --> | \x00\x0D | Hello world | |
| +----------+---------------+ +----------+---------------+</code></pre></div> |
| <p>In most cases, the length field represents the length of the payload |
| only, as shown in the previous examples. However, in some protocols the |
| length field represents the length of the whole frame, including the |
| head. In such cases, we specify a negative <code>length_adjustment</code> to adjust |
| the value provided in the frame head to represent the payload length.</p> |
| <h3 id="example-4"><a href="#example-4">Example 4</a></h3> |
| <p>The following will parse a 3 byte length field at offset 0 in a 5 byte |
| frame head, including the frame head in the yielded <code>BytesMut</code>.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">0</span>) <span class="comment">// default value |
| </span>.length_field_length(<span class="number">3</span>) |
| .length_adjustment(<span class="number">2</span>) <span class="comment">// remaining head |
| </span>.num_skip(<span class="number">0</span>) |
| .new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT |
| +---- len -----+- head -+--- Payload ---+ |
| | \x00\x00\x0B | \xCAFE | Hello world | |
| +--------------+--------+---------------+ |
| |
| DECODED |
| +---- len -----+- head -+--- Payload ---+ |
| | \x00\x00\x0B | \xCAFE | Hello world | |
| +--------------+--------+---------------+</code></pre></div> |
| <p>A more advanced example that shows a case where there is extra frame |
| head data between the length field and the payload. In such cases, it is |
| usually desirable to include the frame head as part of the yielded |
| <code>BytesMut</code>. This lets consumers of the length delimited framer to |
| process the frame head as needed.</p> |
| <p>The positive <code>length_adjustment</code> value lets <code>FramedRead</code> factor in the |
| additional head into the frame length calculation.</p> |
| <h3 id="example-5"><a href="#example-5">Example 5</a></h3> |
| <p>The following will parse a <code>u16</code> length field at offset 1 of a 4 byte |
| frame head. The first byte and the length field will be omitted from the |
| yielded <code>BytesMut</code>, but the trailing 2 bytes of the frame head will be |
| included.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">1</span>) <span class="comment">// length of hdr1 |
| </span>.length_field_type::<u16>() |
| .length_adjustment(<span class="number">1</span>) <span class="comment">// length of hdr2 |
| </span>.num_skip(<span class="number">3</span>) <span class="comment">// length of hdr1 + LEN |
| </span>.new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT |
| +- hdr1 -+-- len ---+- hdr2 -+--- Payload ---+ |
| | \xCA | \x00\x0B | \xFE | Hello world | |
| +--------+----------+--------+---------------+ |
| |
| DECODED |
| +- hdr2 -+--- Payload ---+ |
| | \xFE | Hello world | |
| +--------+---------------+</code></pre></div> |
| <p>The length field is situated in the middle of the frame head. In this |
| case, the first byte in the frame head could be a version or some other |
| identifier that is not needed for processing. On the other hand, the |
| second half of the head is needed.</p> |
| <p><code>length_field_offset</code> indicates how many bytes to skip before starting |
| to read the length field. <code>length_adjustment</code> is the number of bytes to |
| skip starting at the end of the length field. In this case, it is the |
| second half of the head.</p> |
| <h3 id="example-6"><a href="#example-6">Example 6</a></h3> |
| <p>The following will parse a <code>u16</code> length field at offset 1 of a 4 byte |
| frame head. The first byte and the length field will be omitted from the |
| yielded <code>BytesMut</code>, but the trailing 2 bytes of the frame head will be |
| included. In this case, the length field <strong>includes</strong> the frame head |
| length.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">1</span>) <span class="comment">// length of hdr1 |
| </span>.length_field_type::<u16>() |
| .length_adjustment(-<span class="number">3</span>) <span class="comment">// length of hdr1 + LEN, negative |
| </span>.num_skip(<span class="number">3</span>) |
| .new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT |
| +- hdr1 -+-- len ---+- hdr2 -+--- Payload ---+ |
| | \xCA | \x00\x0F | \xFE | Hello world | |
| +--------+----------+--------+---------------+ |
| |
| DECODED |
| +- hdr2 -+--- Payload ---+ |
| | \xFE | Hello world | |
| +--------+---------------+</code></pre></div> |
| <p>Similar to the example above, the difference is that the length field |
| represents the length of the entire frame instead of just the payload. |
| The length of <code>hdr1</code> and <code>len</code> must be counted in <code>length_adjustment</code>. |
| Note that the length of <code>hdr2</code> does <strong>not</strong> need to be explicitly set |
| anywhere because it already is factored into the total frame length that |
| is read from the byte stream.</p> |
| <h3 id="example-7"><a href="#example-7">Example 7</a></h3> |
| <p>The following will parse a 3 byte length field at offset 0 in a 4 byte |
| frame head, excluding the 4th byte from the yielded <code>BytesMut</code>.</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_offset(<span class="number">0</span>) <span class="comment">// default value |
| </span>.length_field_length(<span class="number">3</span>) |
| .length_adjustment(<span class="number">0</span>) <span class="comment">// default value |
| </span>.num_skip(<span class="number">4</span>) <span class="comment">// skip the first 4 bytes |
| </span>.new_read(io);</code></pre></div> |
| <p>The following frame will be decoded as such:</p> |
| <div class="example-wrap"><pre class="language-text"><code> INPUT DECODED |
| +------- len ------+--- Payload ---+ +--- Payload ---+ |
| | \x00\x00\x0B\xFF | Hello world | => | Hello world | |
| +------------------+---------------+ +---------------+</code></pre></div> |
| <p>A simple example where there are unused bytes between the length field |
| and the payload.</p> |
| <h2 id="encoding"><a href="#encoding">Encoding</a></h2> |
| <p><a href="../struct.FramedWrite.html"><code>FramedWrite</code></a> adapts an <a href="../../../tokio/io/async_write/trait.AsyncWrite.html"><code>AsyncWrite</code></a> into a <code>Sink</code> of <a href="../../../bytes/bytes_mut/struct.BytesMut.html"><code>BytesMut</code></a>, |
| such that each submitted <a href="../../../bytes/bytes_mut/struct.BytesMut.html"><code>BytesMut</code></a> is prefaced by a length field. |
| There are fewer configuration options than <a href="../struct.FramedRead.html"><code>FramedRead</code></a>. Given |
| protocols that have more complex frame heads, an encoder should probably |
| be written by hand using <a href="../trait.Encoder.html"><code>Encoder</code></a>.</p> |
| <p>Here is a simple example, given a <code>FramedWrite</code> with the following |
| configuration:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code>LengthDelimitedCodec::builder() |
| .length_field_type::<u16>() |
| .new_write(io);</code></pre></div> |
| <p>A payload of <code>hello world</code> will be encoded as:</p> |
| <div class="example-wrap"><pre class="language-text"><code>+- len: u16 -+---- data ----+ |
| | \x00\x0b | hello world | |
| +------------+--------------+</code></pre></div></div></details><h2 id="structs" class="small-section-header"><a href="#structs">Structs</a></h2><div class="item-table"><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.Builder.html" title="tokio_util::codec::length_delimited::Builder struct">Builder</a></div><div class="item-right docblock-short">Configure length delimited <code>LengthDelimitedCodec</code>s.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.LengthDelimitedCodec.html" title="tokio_util::codec::length_delimited::LengthDelimitedCodec struct">LengthDelimitedCodec</a></div><div class="item-right docblock-short">A codec for frames delimited by a frame head specifying their lengths.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.LengthDelimitedCodecError.html" title="tokio_util::codec::length_delimited::LengthDelimitedCodecError struct">LengthDelimitedCodecError</a></div><div class="item-right docblock-short">An error when the number of bytes read is more than max frame length.</div></div></div></section></div></main><div id="rustdoc-vars" data-root-path="../../../" data-current-crate="tokio_util" data-themes="ayu,dark,light" data-resource-suffix="" data-rustdoc-version="1.66.0-nightly (5c8bff74b 2022-10-21)" ></div></body></html> |
