| <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="Adaptors from AsyncRead/AsyncWrite to Stream/Sink"><meta name="keywords" content="rust, rustlang, rust-lang, codec"><title>tokio_util::codec - Rust</title><link rel="preload" as="font" type="font/woff2" crossorigin href="../../SourceSerif4-Regular.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../FiraSans-Regular.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../FiraSans-Medium.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../SourceCodePro-Regular.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../SourceSerif4-Bold.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../../SourceCodePro-Semibold.ttf.woff2"><link rel="stylesheet" href="../../normalize.css"><link rel="stylesheet" href="../../rustdoc.css" id="mainThemeStyle"><link rel="stylesheet" href="../../ayu.css" disabled><link rel="stylesheet" href="../../dark.css" disabled><link rel="stylesheet" href="../../light.css" id="themeStyle"><script id="default-settings" ></script><script src="../../storage.js"></script><script defer src="../../main.js"></script><noscript><link rel="stylesheet" href="../../noscript.css"></noscript><link rel="alternate icon" type="image/png" href="../../favicon-16x16.png"><link rel="alternate icon" type="image/png" href="../../favicon-32x32.png"><link rel="icon" type="image/svg+xml" href="../../favicon.svg"></head><body class="rustdoc mod"><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><nav class="mobile-topbar"><button class="sidebar-menu-toggle">☰</button><a class="sidebar-logo" href="../../tokio_util/index.html"><div class="logo-container"><img class="rust-logo" src="../../rust-logo.svg" alt="logo"></div></a><h2></h2></nav><nav class="sidebar"><a class="sidebar-logo" href="../../tokio_util/index.html"><div class="logo-container"><img class="rust-logo" src="../../rust-logo.svg" alt="logo"></div></a><h2 class="location"><a href="#">Module codec</a></h2><div class="sidebar-elems"><section><ul class="block"><li><a href="#reexports">Re-exports</a></li><li><a href="#modules">Modules</a></li><li><a href="#structs">Structs</a></li><li><a href="#enums">Enums</a></li><li><a href="#traits">Traits</a></li></ul></section></div></nav><main><div class="width-limiter"><nav class="sub"><form class="search-form"><div class="search-container"><span></span><input class="search-input" name="search" autocomplete="off" spellcheck="false" placeholder="Click or press ‘S’ to search, ‘?’ for more options…" type="search"><div id="help-button" title="help" tabindex="-1"><a href="../../help.html">?</a></div><div id="settings-menu" tabindex="-1"><a href="../../settings.html" title="settings"><img width="22" height="22" alt="Change settings" src="../../wheel.svg"></a></div></div></form></nav><section id="main-content" class="content"><div class="main-heading"><h1 class="fqn">Module <a href="../index.html">tokio_util</a>::<wbr><a class="mod" href="#">codec</a><button id="copy-path" onclick="copy_path(this)" title="Copy item path to clipboard"><img src="../../clipboard.svg" width="19" height="18" alt="Copy item path"></button></h1><span class="out-of-band"><a class="srclink" href="../../src/tokio_util/codec/mod.rs.html#1-290">source</a> · <a id="toggle-all-docs" href="javascript:void(0)" title="collapse all docs">[<span class="inner">−</span>]</a></span></div><details class="rustdoc-toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>Adaptors from AsyncRead/AsyncWrite to Stream/Sink</p> |
| <p>Raw I/O objects work with byte sequences, but higher-level code usually |
| wants to batch these into meaningful chunks, called “frames”.</p> |
| <p>This module contains adapters to go from streams of bytes, <a href="../../tokio/io/async_read/trait.AsyncRead.html"><code>AsyncRead</code></a> and |
| <a href="../../tokio/io/async_write/trait.AsyncWrite.html"><code>AsyncWrite</code></a>, to framed streams implementing <a href="../../futures_sink/trait.Sink.html"><code>Sink</code></a> and <a href="../../futures_core/stream/trait.Stream.html"><code>Stream</code></a>. |
| Framed streams are also known as transports.</p> |
| <h2 id="the-decoder-trait"><a href="#the-decoder-trait">The Decoder trait</a></h2> |
| <p>A <a href="trait.Decoder.html"><code>Decoder</code></a> is used together with <a href="struct.FramedRead.html"><code>FramedRead</code></a> or <a href="struct.Framed.html"><code>Framed</code></a> to turn an |
| <a href="../../tokio/io/async_read/trait.AsyncRead.html"><code>AsyncRead</code></a> into a <a href="../../futures_core/stream/trait.Stream.html"><code>Stream</code></a>. The job of the decoder trait is to specify |
| how sequences of bytes are turned into a sequence of frames, and to |
| determine where the boundaries between frames are. The job of the |
| <code>FramedRead</code> is to repeatedly switch between reading more data from the IO |
| resource, and asking the decoder whether we have received enough data to |
| decode another frame of data.</p> |
| <p>The main method on the <code>Decoder</code> trait is the <a href="trait.Decoder.html#tymethod.decode"><code>decode</code></a> method. This method |
| takes as argument the data that has been read so far, and when it is called, |
| it will be in one of the following situations:</p> |
| <ol> |
| <li>The buffer contains less than a full frame.</li> |
| <li>The buffer contains exactly a full frame.</li> |
| <li>The buffer contains more than a full frame.</li> |
| </ol> |
| <p>In the first situation, the decoder should return <code>Ok(None)</code>.</p> |
| <p>In the second situation, the decoder should clear the provided buffer and |
| return <code>Ok(Some(the_decoded_frame))</code>.</p> |
| <p>In the third situation, the decoder should use a method such as <a href="../../bytes/bytes_mut/struct.BytesMut.html#method.split_to"><code>split_to</code></a> |
| or <a href="../../bytes/buf/buf_impl/trait.Buf.html#tymethod.advance"><code>advance</code></a> to modify the buffer such that the frame is removed from the |
| buffer, but any data in the buffer after that frame should still remain in |
| the buffer. The decoder should also return <code>Ok(Some(the_decoded_frame))</code> in |
| this case.</p> |
| <p>Finally the decoder may return an error if the data is invalid in some way. |
| The decoder should <em>not</em> return an error just because it has yet to receive |
| a full frame.</p> |
| <p>It is guaranteed that, from one call to <code>decode</code> to another, the provided |
| buffer will contain the exact same data as before, except that if more data |
| has arrived through the IO resource, that data will have been appended to |
| the buffer. This means that reading frames from a <code>FramedRead</code> is |
| essentially equivalent to the following loop:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio::io::AsyncReadExt; |
| |
| <span class="kw">let </span><span class="kw-2">mut </span>buf = bytes::BytesMut::new(); |
| <span class="kw">loop </span>{ |
| <span class="comment">// The read_buf call will append to buf rather than overwrite existing data. |
| </span><span class="kw">let </span>len = io_resource.read_buf(<span class="kw-2">&mut </span>buf).<span class="kw">await</span><span class="question-mark">?</span>; |
| |
| <span class="kw">if </span>len == <span class="number">0 </span>{ |
| <span class="kw">while let </span><span class="prelude-val">Some</span>(frame) = decoder.decode_eof(<span class="kw-2">&mut </span>buf)<span class="question-mark">? </span>{ |
| <span class="kw">yield </span>frame; |
| } |
| <span class="kw">break</span>; |
| } |
| |
| <span class="kw">while let </span><span class="prelude-val">Some</span>(frame) = decoder.decode(<span class="kw-2">&mut </span>buf)<span class="question-mark">? </span>{ |
| <span class="kw">yield </span>frame; |
| } |
| }</code></pre></div> |
| <p>The example above uses <code>yield</code> whenever the <code>Stream</code> produces an item.</p> |
| <h3 id="example-decoder"><a href="#example-decoder">Example decoder</a></h3> |
| <p>As an example, consider a protocol that can be used to send strings where |
| each frame is a four byte integer that contains the length of the frame, |
| followed by that many bytes of string data. The decoder fails with an error |
| if the string data is not valid utf-8 or too long.</p> |
| <p>Such a decoder can be written like this:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio_util::codec::Decoder; |
| <span class="kw">use </span>bytes::{BytesMut, Buf}; |
| |
| <span class="kw">struct </span>MyStringDecoder {} |
| |
| <span class="kw">const </span>MAX: usize = <span class="number">8 </span>* <span class="number">1024 </span>* <span class="number">1024</span>; |
| |
| <span class="kw">impl </span>Decoder <span class="kw">for </span>MyStringDecoder { |
| <span class="kw">type </span>Item = String; |
| <span class="kw">type </span>Error = std::io::Error; |
| |
| <span class="kw">fn </span>decode( |
| <span class="kw-2">&mut </span><span class="self">self</span>, |
| src: <span class="kw-2">&mut </span>BytesMut |
| ) -> <span class="prelude-ty">Result</span><<span class="prelude-ty">Option</span><<span class="self">Self</span>::Item>, <span class="self">Self</span>::Error> { |
| <span class="kw">if </span>src.len() < <span class="number">4 </span>{ |
| <span class="comment">// Not enough data to read length marker. |
| </span><span class="kw">return </span><span class="prelude-val">Ok</span>(<span class="prelude-val">None</span>); |
| } |
| |
| <span class="comment">// Read length marker. |
| </span><span class="kw">let </span><span class="kw-2">mut </span>length_bytes = [<span class="number">0u8</span>; <span class="number">4</span>]; |
| length_bytes.copy_from_slice(<span class="kw-2">&</span>src[..<span class="number">4</span>]); |
| <span class="kw">let </span>length = u32::from_le_bytes(length_bytes) <span class="kw">as </span>usize; |
| |
| <span class="comment">// Check that the length is not too large to avoid a denial of |
| // service attack where the server runs out of memory. |
| </span><span class="kw">if </span>length > MAX { |
| <span class="kw">return </span><span class="prelude-val">Err</span>(std::io::Error::new( |
| std::io::ErrorKind::InvalidData, |
| <span class="macro">format!</span>(<span class="string">"Frame of length {} is too large."</span>, length) |
| )); |
| } |
| |
| <span class="kw">if </span>src.len() < <span class="number">4 </span>+ length { |
| <span class="comment">// The full string has not yet arrived. |
| // |
| // We reserve more space in the buffer. This is not strictly |
| // necessary, but is a good idea performance-wise. |
| </span>src.reserve(<span class="number">4 </span>+ length - src.len()); |
| |
| <span class="comment">// We inform the Framed that we need more bytes to form the next |
| // frame. |
| </span><span class="kw">return </span><span class="prelude-val">Ok</span>(<span class="prelude-val">None</span>); |
| } |
| |
| <span class="comment">// Use advance to modify src such that it no longer contains |
| // this frame. |
| </span><span class="kw">let </span>data = src[<span class="number">4</span>..<span class="number">4 </span>+ length].to_vec(); |
| src.advance(<span class="number">4 </span>+ length); |
| |
| <span class="comment">// Convert the data to a string, or fail if it is not valid utf-8. |
| </span><span class="kw">match </span>String::from_utf8(data) { |
| <span class="prelude-val">Ok</span>(string) => <span class="prelude-val">Ok</span>(<span class="prelude-val">Some</span>(string)), |
| <span class="prelude-val">Err</span>(utf8_error) => { |
| <span class="prelude-val">Err</span>(std::io::Error::new( |
| std::io::ErrorKind::InvalidData, |
| utf8_error.utf8_error(), |
| )) |
| }, |
| } |
| } |
| }</code></pre></div> |
| <h2 id="the-encoder-trait"><a href="#the-encoder-trait">The Encoder trait</a></h2> |
| <p>An <a href="trait.Encoder.html" title="Encoder"><code>Encoder</code></a> is used together with <a href="struct.FramedWrite.html"><code>FramedWrite</code></a> or <a href="struct.Framed.html"><code>Framed</code></a> to turn |
| an <a href="../../tokio/io/async_write/trait.AsyncWrite.html"><code>AsyncWrite</code></a> into a <a href="../../futures_sink/trait.Sink.html"><code>Sink</code></a>. The job of the encoder trait is to |
| specify how frames are turned into a sequences of bytes. The job of the |
| <code>FramedWrite</code> is to take the resulting sequence of bytes and write it to the |
| IO resource.</p> |
| <p>The main method on the <code>Encoder</code> trait is the <a href="trait.Encoder.html#tymethod.encode"><code>encode</code></a> method. This method |
| takes an item that is being written, and a buffer to write the item to. The |
| buffer may already contain data, and in this case, the encoder should append |
| the new frame the to buffer rather than overwrite the existing data.</p> |
| <p>It is guaranteed that, from one call to <code>encode</code> to another, the provided |
| buffer will contain the exact same data as before, except that some of the |
| data may have been removed from the front of the buffer. Writing to a |
| <code>FramedWrite</code> is essentially equivalent to the following loop:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio::io::AsyncWriteExt; |
| <span class="kw">use </span>bytes::Buf; <span class="comment">// for advance |
| |
| </span><span class="kw">const </span>MAX: usize = <span class="number">8192</span>; |
| |
| <span class="kw">let </span><span class="kw-2">mut </span>buf = bytes::BytesMut::new(); |
| <span class="kw">loop </span>{ |
| <span class="macro">tokio::select! </span>{ |
| num_written = io_resource.write(<span class="kw-2">&</span>buf), <span class="kw">if </span>!buf.is_empty() => { |
| buf.advance(num_written<span class="question-mark">?</span>); |
| }, |
| frame = next_frame(), <span class="kw">if </span>buf.len() < MAX => { |
| encoder.encode(frame, <span class="kw-2">&mut </span>buf)<span class="question-mark">?</span>; |
| }, |
| <span class="kw">_ </span>= no_more_frames() => { |
| io_resource.write_all(<span class="kw-2">&</span>buf).<span class="kw">await</span><span class="question-mark">?</span>; |
| io_resource.shutdown().<span class="kw">await</span><span class="question-mark">?</span>; |
| <span class="kw">return </span><span class="prelude-val">Ok</span>(()); |
| }, |
| } |
| }</code></pre></div> |
| <p>Here the <code>next_frame</code> method corresponds to any frames you write to the |
| <code>FramedWrite</code>. The <code>no_more_frames</code> method corresponds to closing the |
| <code>FramedWrite</code> with <a href="https://docs.rs/futures/0.3/futures/sink/trait.SinkExt.html#method.close"><code>SinkExt::close</code></a>.</p> |
| <h3 id="example-encoder"><a href="#example-encoder">Example encoder</a></h3> |
| <p>As an example, consider a protocol that can be used to send strings where |
| each frame is a four byte integer that contains the length of the frame, |
| followed by that many bytes of string data. The encoder will fail if the |
| string is too long.</p> |
| <p>Such an encoder can be written like this:</p> |
| |
| <div class="example-wrap"><pre class="rust rust-example-rendered"><code><span class="kw">use </span>tokio_util::codec::Encoder; |
| <span class="kw">use </span>bytes::BytesMut; |
| |
| <span class="kw">struct </span>MyStringEncoder {} |
| |
| <span class="kw">const </span>MAX: usize = <span class="number">8 </span>* <span class="number">1024 </span>* <span class="number">1024</span>; |
| |
| <span class="kw">impl </span>Encoder<String> <span class="kw">for </span>MyStringEncoder { |
| <span class="kw">type </span>Error = std::io::Error; |
| |
| <span class="kw">fn </span>encode(<span class="kw-2">&mut </span><span class="self">self</span>, item: String, dst: <span class="kw-2">&mut </span>BytesMut) -> <span class="prelude-ty">Result</span><(), <span class="self">Self</span>::Error> { |
| <span class="comment">// Don't send a string if it is longer than the other end will |
| // accept. |
| </span><span class="kw">if </span>item.len() > MAX { |
| <span class="kw">return </span><span class="prelude-val">Err</span>(std::io::Error::new( |
| std::io::ErrorKind::InvalidData, |
| <span class="macro">format!</span>(<span class="string">"Frame of length {} is too large."</span>, item.len()) |
| )); |
| } |
| |
| <span class="comment">// Convert the length into a byte array. |
| // The cast to u32 cannot overflow due to the length check above. |
| </span><span class="kw">let </span>len_slice = u32::to_le_bytes(item.len() <span class="kw">as </span>u32); |
| |
| <span class="comment">// Reserve space in the buffer. |
| </span>dst.reserve(<span class="number">4 </span>+ item.len()); |
| |
| <span class="comment">// Write the length and string to the buffer. |
| </span>dst.extend_from_slice(<span class="kw-2">&</span>len_slice); |
| dst.extend_from_slice(item.as_bytes()); |
| <span class="prelude-val">Ok</span>(()) |
| } |
| }</code></pre></div> |
| </div></details><h2 id="reexports" class="small-section-header"><a href="#reexports">Re-exports</a></h2><div class="item-table"><div class="item-row"><div class="item-left import-item" id="reexport.LengthDelimitedCodec"><code>pub use self::length_delimited::<a class="struct" href="length_delimited/struct.LengthDelimitedCodec.html" title="struct tokio_util::codec::length_delimited::LengthDelimitedCodec">LengthDelimitedCodec</a>;</code></div></div><div class="item-row"><div class="item-left import-item" id="reexport.LengthDelimitedCodecError"><code>pub use self::length_delimited::<a class="struct" href="length_delimited/struct.LengthDelimitedCodecError.html" title="struct tokio_util::codec::length_delimited::LengthDelimitedCodecError">LengthDelimitedCodecError</a>;</code></div></div></div><h2 id="modules" class="small-section-header"><a href="#modules">Modules</a></h2><div class="item-table"><div class="item-row"><div class="item-left module-item"><a class="mod" href="length_delimited/index.html" title="tokio_util::codec::length_delimited mod">length_delimited</a></div><div class="item-right docblock-short">Frame a stream of bytes based on a length prefix</div></div></div><h2 id="structs" class="small-section-header"><a href="#structs">Structs</a></h2><div class="item-table"><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.AnyDelimiterCodec.html" title="tokio_util::codec::AnyDelimiterCodec struct">AnyDelimiterCodec</a></div><div class="item-right docblock-short">A simple <a href="trait.Decoder.html"><code>Decoder</code></a> and <a href="trait.Encoder.html"><code>Encoder</code></a> implementation that splits up data into chunks based on any character in the given delimiter string.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.BytesCodec.html" title="tokio_util::codec::BytesCodec struct">BytesCodec</a></div><div class="item-right docblock-short">A simple <a href="trait.Decoder.html"><code>Decoder</code></a> and <a href="trait.Encoder.html"><code>Encoder</code></a> implementation that just ships bytes around.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.Framed.html" title="tokio_util::codec::Framed struct">Framed</a></div><div class="item-right docblock-short">A unified <a href="../../futures_core/stream/trait.Stream.html"><code>Stream</code></a> and <a href="../../futures_sink/trait.Sink.html"><code>Sink</code></a> interface to an underlying I/O object, using |
| the <code>Encoder</code> and <code>Decoder</code> traits to encode and decode frames.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.FramedParts.html" title="tokio_util::codec::FramedParts struct">FramedParts</a></div><div class="item-right docblock-short"><code>FramedParts</code> contains an export of the data of a Framed transport. |
| It can be used to construct a new <a href="struct.Framed.html"><code>Framed</code></a> with a different codec. |
| It contains all current buffers and the inner transport.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.FramedRead.html" title="tokio_util::codec::FramedRead struct">FramedRead</a></div><div class="item-right docblock-short">A <a href="../../futures_core/stream/trait.Stream.html"><code>Stream</code></a> of messages decoded from an <a href="../../tokio/io/async_read/trait.AsyncRead.html"><code>AsyncRead</code></a>.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.FramedWrite.html" title="tokio_util::codec::FramedWrite struct">FramedWrite</a></div><div class="item-right docblock-short">A <a href="../../futures_sink/trait.Sink.html"><code>Sink</code></a> of frames encoded to an <code>AsyncWrite</code>.</div></div><div class="item-row"><div class="item-left module-item"><a class="struct" href="struct.LinesCodec.html" title="tokio_util::codec::LinesCodec struct">LinesCodec</a></div><div class="item-right docblock-short">A simple <a href="trait.Decoder.html"><code>Decoder</code></a> and <a href="trait.Encoder.html"><code>Encoder</code></a> implementation that splits up data into lines.</div></div></div><h2 id="enums" class="small-section-header"><a href="#enums">Enums</a></h2><div class="item-table"><div class="item-row"><div class="item-left module-item"><a class="enum" href="enum.AnyDelimiterCodecError.html" title="tokio_util::codec::AnyDelimiterCodecError enum">AnyDelimiterCodecError</a></div><div class="item-right docblock-short">An error occurred while encoding or decoding a chunk.</div></div><div class="item-row"><div class="item-left module-item"><a class="enum" href="enum.LinesCodecError.html" title="tokio_util::codec::LinesCodecError enum">LinesCodecError</a></div><div class="item-right docblock-short">An error occurred while encoding or decoding a line.</div></div></div><h2 id="traits" class="small-section-header"><a href="#traits">Traits</a></h2><div class="item-table"><div class="item-row"><div class="item-left module-item"><a class="trait" href="trait.Decoder.html" title="tokio_util::codec::Decoder trait">Decoder</a></div><div class="item-right docblock-short">Decoding of frames via buffers.</div></div><div class="item-row"><div class="item-left module-item"><a class="trait" href="trait.Encoder.html" title="tokio_util::codec::Encoder trait">Encoder</a></div><div class="item-right docblock-short">Trait of helper objects to write out messages as bytes, for use with |
| <a href="struct.FramedWrite.html"><code>FramedWrite</code></a>.</div></div></div></section></div></main><div id="rustdoc-vars" data-root-path="../../" data-current-crate="tokio_util" data-themes="ayu,dark,light" data-resource-suffix="" data-rustdoc-version="1.66.0-nightly (5c8bff74b 2022-10-21)" ></div></body></html> |
