Mbufs
=====

The mbuf (short for memory buffer) is a common concept in networking
stacks. The mbuf is used to hold packet data as it traverses the stack.
The mbuf also generally stores header information or other networking
stack information that is carried around with the packet. The mbuf and
its associated library of functions were developed to make common
networking stack operations (like stripping and adding protocol headers)
efficient and as copy-free as possible.

In its simplest form, an mbuf is a memory block with some space reserved
for internal information and a pointer which is used to "chain" memory
blocks together in order to create a "packet". This is a very important
aspect of the mbuf: the ability to chain mbufs together to create larger
"packets" (chains of mbufs).

Why use mbufs?
----------------

The main reason is to conserve memory. Consider a networking protocol
that generally sends small packets but occasionally sends large ones.
The Bluetooth Low Energy (BLE) protocol is one such example. A flat
buffer would need to be sized so that the maximum packet size could be
contained by the buffer. With the mbuf, a number of mbufs can be chained
together so that the occasional large packet can be handled while
leaving more packet buffers available to the networking stack for
smaller packets.

Packet Header mbuf
--------------------

Not all mbufs are created equal. The first mbuf in a chain of mbufs is a
special mbuf called a "packet header mbuf". The reason that this mbuf is
special is that it contains the length of all the data contained by the
chain of mbufs (the packet length, in other words). The packet header
mbuf may also contain a user defined structure (called a "user header")
so that networking protocol specific information can be conveyed to
various layers of the networking stack. Any mbufs that are part of the
packet (i.e. in the mbuf chain but not the first one) are "normal" (i.e.
non-packet header) mbufs. A normal mbuf does not have any packet header
or user packet header structures in them; they only contain the basic
mbuf header (:c:type:`struct os_mbuf`). Figure 1 illustrates these two types
of mbufs. Note that the numbers/text in parentheses denote the size of
the structures/elements (in bytes) and that MBLEN is the memory block
length of the memory pool used by the mbuf pool.

.. figure:: pics/mbuf_fig1.png
   :alt: Packet header mbuf

   Packet header mbuf

Normal mbuf
--------------

Now let's take a deeper dive into the mbuf structure. Figure 2
illustrates a normal mbuf and breaks out the various fields in the
c:type:`os_mbuf` structure.

-  The :c:member:`om_data` field is a pointer to where the data starts inside the
   data buffer. Typically, mbufs that are allocated from the mbuf pool
   (discussed later) have their :c:member:`om_data` pointer set to the start of the
   data buffer but there are cases where this may not be desirable
   (added a protocol header to a packet, for example).
-  The :c:member:`om_flags` field is a set of flags used internally by the mbuf
   library. Currently, no flags have been defined.
-  The :c:member:`om_pkthdr_len` field is the total length of all packet headers
   in the mbuf. For normal mbufs this is set to 0 as there is no packet
   or user packet headers. For packet header mbufs, this would be set to
   the length of the packet header structure (16) plus the size of the
   user packet header (if any). Note that it is this field which
   differentiates packet header mbufs from normal mbufs (i.e. if
   :c:member:`om_pkthdr_len` is zero, this is a normal mbuf; otherwise it is a
   packet header mbuf).
-  The :c:member:`om_len` field contains the amount of user data in the data
   buffer. When initially allocated, this field is 0 as there is no user
   data in the mbuf.
-  The :c:member:`omp_pool` field is a pointer to the pool from which this mbuf
   has been allocated. This is used internally by the mbuf library.
-  The :c:member:`omp_next` field is a linked list element which is used to chain
   mbufs.

Figure 2 also shows a normal mbuf with actual values in the :c:type:`os_mbuf`
structure. This mbuf starts at address 0x1000 and is 256 bytes in total
length. In this example, the user has copied 33 bytes into the data
buffer starting at address 0x1010 (this is where :c:member:`om_data` points). Note
that the packet header length in this mbuf is 0 as it is not a packet
header mbuf.

.. figure:: pics/mbuf_fig2.png
   :alt: OS mbuf structure

   OS mbuf structure

Figure 3 illustrates the packet header mbuf along with some chained
mbufs (i.e a "packet"). In this example, the user header structure is
defined to be 8 bytes. Note that in figure 3 we show a number of
different mbufs with varying :c:member:`om_data` pointers and lengths since we
want to show various examples of valid mbufs. For all the mbufs (both
packet header and normal ones) the total length of the memory block is
128 bytes.

.. figure:: pics/mbuf_fig3.png
   :alt: Packet

   Packet

Mbuf pools
---------------

Mbufs are collected into "mbuf pools" much like memory blocks. The mbuf
pool itself contains a pointer to a memory pool. The memory blocks in
this memory pool are the actual mbufs; both normal and packet header
mbufs. Thus, the memory block (and corresponding memory pool) must be
sized correctly. In other words, the memory blocks which make up the
memory pool used by the mbuf pool must be at least: sizeof(struct
os\_mbuf) + sizeof(struct os\_mbuf\_pkthdr) + sizeof(struct
user\_defined\_header) + desired minimum data buffer length. For
example, if the developer wants mbufs to contain at least 64 bytes of
user data and they have a user header of 12 bytes, the size of the
memory block would be (at least): 64 + 12 + 16 + 8, or 100 bytes. Yes,
this is a fair amount of overhead. However, the flexibility provided by
the mbuf library usually outweighs overhead concerns.

Create mbuf pool
-----------------

Creating an mbuf pool is fairly simple: create a memory pool and then
create the mbuf pool using that memory pool. Once the developer has
determined the size of the user data needed per mbuf (this is based on
the application/networking stack and is outside the scope of this
discussion) and the size of the user header (if any), the memory blocks
can be sized. In the example shown below, the application requires 64
bytes of user data per mbuf and also allocates a user header (called
struct user\_hdr). Note that we do not show the user header data
structure as there really is no need; all we need to do is to account
for it when creating the memory pool. In the example, we use the macro
*MBUF\_PKTHDR\_OVERHEAD* to denote the amount of packet header overhead
per mbuf and *MBUF\_MEMBLOCK\_OVERHEAD* to denote the total amount of
overhead required per memory block. The macro *MBUF\_BUF\_SIZE* is used
to denote the amount of payload that the application requires (aligned
on a 32-bit boundary in this case). All this leads to the total memory
block size required, denoted by the macro *MBUF\_MEMBLOCK\_OVERHEAD*.

.. code-block:: c


    #define MBUF_PKTHDR_OVERHEAD    sizeof(struct os_mbuf_pkthdr) + sizeof(struct user_hdr)
    #define MBUF_MEMBLOCK_OVERHEAD  sizeof(struct os_mbuf) + MBUF_PKTHDR_OVERHEAD

    #define MBUF_NUM_MBUFS      (32)
    #define MBUF_PAYLOAD_SIZE   (64)
    #define MBUF_BUF_SIZE       OS_ALIGN(MBUF_PAYLOAD_SIZE, 4)
    #define MBUF_MEMBLOCK_SIZE  (MBUF_BUF_SIZE + MBUF_MEMBLOCK_OVERHEAD)
    #define MBUF_MEMPOOL_SIZE   OS_MEMPOOL_SIZE(MBUF_NUM_MBUFS, MBUF_MEMBLOCK_SIZE)

    struct os_mbuf_pool g_mbuf_pool; 
    struct os_mempool g_mbuf_mempool;
    os_membuf_t g_mbuf_buffer[MBUF_MEMPOOL_SIZE];

    void
    create_mbuf_pool(void)
    {
        int rc;
        
        rc = os_mempool_init(&g_mbuf_mempool, MBUF_NUM_MBUFS, 
                              MBUF_MEMBLOCK_SIZE, &g_mbuf_buffer[0], "mbuf_pool");
        assert(rc == 0);

        rc = os_mbuf_pool_init(&g_mbuf_pool, &g_mbuf_mempool, MBUF_MEMBLOCK_SIZE, 
                               MBUF_NUM_MBUFS);
        assert(rc == 0);
    }

Msys
-----

Msys stands for "system mbufs" and is a set of API built on top of the
mbuf code. The basic idea behind msys is the following. The developer
can create different size mbuf pools and register them with msys. The
application then allocates mbufs using the msys API (as opposed to the
mbuf API). The msys code will choose the mbuf pool with the smallest
mbufs that can accommodate the requested size.

Let us walk through an example where the user registers three mbuf pools
with msys: one with 32 byte mbufs, one with 256 and one with 2048. If
the user requests an mbuf with 10 bytes, the 32-byte mbuf pool is used.
If the request is for 33 bytes the 256 byte mbuf pool is used. If an
mbuf data size is requested that is larger than any of the pools (say,
4000 bytes) the largest pool is used. While this behaviour may not be
optimal in all cases that is the currently implemented behaviour. All
this means is that the user is not guaranteed that a single mbuf can
hold the requested data.

The msys code will not allocate an mbuf from a larger pool if the chosen
mbuf pool is empty. Similarly, the msys code will not chain together a
number of smaller mbufs to accommodate the requested size. While this
behaviour may change in future implementations the current code will
simply return NULL. Using the above example, say the user requests 250
bytes. The msys code chooses the appropriate pool (i.e. the 256 byte
mbuf pool) and attempts to allocate an mbuf from that pool. If that pool
is empty, NULL is returned even though the 32 and 2048 byte pools are
not empty.

Note that no added descriptions on how to use the msys API are presented
here (other than in the API descriptions themselves) as the msys API is
used in exactly the same manner as the mbuf API. The only difference is
that mbuf pools are added to msys by calling ``os_msys_register().``


Using mbufs
--------------

The following examples illustrate typical mbuf usage. There are two
basic mbuf allocation API: c:func:`os_mbuf_get()` and
:c:func:`os_mbuf_get_pkthdr()`. The first API obtains a normal mbuf whereas
the latter obtains a packet header mbuf. Typically, application
developers use :c:func:`os_mbuf_get_pkthdr()` and rarely, if ever, need to
call :c:func:`os_mbuf_get()` as the rest of the mbuf API (e.g.
:c:func:`os_mbuf_append()`, :c:func:`os_mbuf_copyinto()`, etc.) typically 
deal with allocating and chaining mbufs. It is recommended to use the 
provided API to copy data into/out of mbuf chains and/or manipulate mbufs.

In ``example1``, the developer creates a packet and then sends the
packet to a networking interface. The code sample also provides an
example of copying data out of an mbuf as well as use of the "pullup"
api (another very common mbuf api).

.. code-block:: c


    void
    mbuf_usage_example1(uint8_t *mydata, int mydata_length)
    {
        int rc;
        struct os_mbuf *om;

        /* get a packet header mbuf */
        om = os_mbuf_get_pkthdr(&g_mbuf_pool, sizeof(struct user_hdr));
        if (om) {
            /* 
             * Copy user data into mbuf. NOTE: if mydata_length is greater than the
             * mbuf payload size (64 bytes using above example), mbufs are allocated
             * and chained together to accommodate the total packet length.
             */
            rc = os_mbuf_copyinto(om, 0, mydata, len);
            if (rc) {
                /* Error! Could not allocate enough mbufs for total packet length */
                return -1;
            }
            
            /* Send packet to networking interface */
            send_pkt(om);
        }
    }

In ``example2`` we show use of the pullup api as this illustrates some
of the typical pitfalls developers encounter when using mbufs. The first
pitfall is one of alignment/padding. Depending on the processor and/or
compiler, the sizeof() a structure may vary. Thus, the size of
*my\_protocol\_header* may be different inside the packet data of the
mbuf than the size of the structure on the stack or as a global
variable, for instance. While some networking protcols may align
protocol information on convenient processor boundaries many others try
to conserve bytes "on the air" (i.e inside the packet data). Typical
methods used to deal with this are "packing" the structure (i.e. force
compiler to not pad) or creating protocol headers that do not require
padding. ``example2`` assumes that one of these methods was used when
defining the *my\_protocol\_header* structure.

Another common pitfall occurs around endianness. A network protocol may
be little endian or big endian; it all depends on the protocol
specification. Processors also have an endianness; this means that the
developer has to be careful that the processor endianness and the
protocol endianness are handled correctly. In ``example2``, some common
networking functions are used: ``ntohs()`` and ``ntohl()``. These are
shorthand for "network order to host order, short" and "network order to
host order, long". Basically, these functions convert data of a certain
size (i.e. 16 bits, 32 bits, etc) to the endianness of the host. Network
byte order is big-endian (most significant byte first), so these
functions convert big-endian byte order to host order (thus, the
implementation of these functions is host dependent). Note that the BLE
networking stack "on the air" format is least signigicant byte first
(i.e. little endian), so a "bletoh" function would have to take little
endian format and convert to host format.

A long story short: the developer must take care when copying structure
data to/from mbufs and flat buffers!

A final note: these examples assume the same mbuf struture and
definitions used in the first example.

.. code-block:: c

    void
    mbuf_usage_example2(struct mbuf *rxpkt)
    {
        int rc;
        uint8_t packet_data[16];
        struct mbuf *om;
        struct my_protocol_header *phdr;

        /* Make sure that "my_protocol_header" bytes are contiguous in mbuf */
        om = os_mbuf_pullup(&g_mbuf_pool, sizeof(struct my_protocol_header));
        if (!om) {
            /* Not able to pull up data into contiguous area */
            return -1;
        }
        
        /* 
         * Get the protocol information from the packet. In this example we presume that we
         * are interested in protocol types that are equal to MY_PROTOCOL_TYPE, are not zero
         * length, and have had some time in flight.
         */
        phdr = OS_MBUF_DATA(om, struct my_protocol_header *);
        type = ntohs(phdr->prot_type);
        length = ntohs(phdr->prot_length);
        time_in_flight = ntohl(phdr->prot_tif);
        
        if ((type == MY_PROTOCOL_TYPE) && (length > 0) && (time_in_flight > 0)) {
            rc = os_mbuf_copydata(rxpkt, sizeof(struct my_protocol_header), 16, packet_data);
            if (!rc) {
                /* Success! Perform operations on packet data */
                <... user code here ...>
            }
        }
        
        /* Free passed in packet (mbuf chain) since we don't need it anymore */
        os_mbuf_free_chain(om);
    }


Mqueue
-------

The mqueue construct allows a task to wake up when it receives data.
Typically, this data is in the form of packets received over a network.
A common networking stack operation is to put a packet on a queue and
post an event to the task monitoring that queue. When the task handles
the event, it processes each packet on the packet queue.

Using Mqueue
--------------

The following code sample demonstrates how to use an mqueue. In this
example:

-  packets are put on a receive queue
-  a task processes each packet on the queue (increments a receive
   counter)

Not shown in the code example is a call ``my_task_rx_data_func``.
Presumably, some other code will call this API.

.. code-block:: c

    uint32_t pkts_rxd;
    struct os_mqueue rxpkt_q;
    struct os_eventq my_task_evq;

    /**
     * Removes each packet from the receive queue and processes it.
     */
    void
    process_rx_data_queue(void)
    {
        struct os_mbuf *om;

        while ((om = os_mqueue_get(&rxpkt_q)) != NULL) {
            ++pkts_rxd;
            os_mbuf_free_chain(om);
        }
    }

    /**
     * Called when a packet is received.
     */
    int
    my_task_rx_data_func(struct os_mbuf *om)
    {
        int rc;

        /* Enqueue the received packet and wake up the listening task. */
        rc = os_mqueue_put(&rxpkt_q, &my_task_evq, om);
        if (rc != 0) {
            return -1;
        }

        return 0;
    }

    void
    my_task_handler(void *arg)
    {
        struct os_event *ev;
        struct os_callout_func *cf;
        int rc;

        /* Initialize eventq */
        os_eventq_init(&my_task_evq);

        /* Initialize mqueue */
        os_mqueue_init(&rxpkt_q, NULL);

        /* Process each event posted to our eventq.  When there are no events to
         * process, sleep until one arrives.
         */
        while (1) {
            os_eventq_run(&my_task_evq);
        }
    }


API
-----------------

.. doxygengroup:: OSMbuf
    :content-only:
    :members: