operator_guide.md

# An operator's guide to IOQ

IOQ handles the prioritisation of IO operations in the database. It has
two main responsibilities:

 1. Providing configurable prioritisation of interactive requests and
    background requests such as compaction and internal replication.
 2. Providing equal prioritisation for interactive requests by backend/database.

From an operational perspective point 1 is of most interest as it provides a set
of levers that can be pulled to change the behaviour of the cluster in favour
of particular workloads or operational concerns.

## Basic overview

From an operational point-of-view, IOQ carries out two fundamental operations:

 1. Enqueueing requests into one of a number of available channels.
 2. Selecting and submitting a request from the available channels according
    to configured priorities.

IOQ categorises IO requests by class and by priority. The class of a request
dictates the channel into which it will be enqueued and the priority influences
the probability that a given request will be dequeued and executed.

The following table lists the IOQ classes and the corresponding priorities. Note
that the mapping of IOQ classes to class priorities is not 1:1.

```
|---------------+---------------+--------------------------------------------|
| IOQ class     | IOQ priority  | Description                                |
|---------------+---------------+--------------------------------------------|
| interactive   | reads, writes | IO requests related to requests made by    |
|               |               | users via the http layer.                  |
|               |               |                                            |
| db_update     | writes        | Interactive IO requests which are database |
|               |               | write operations.                          |
|               |               |                                            |
| view_update   | views         | IO requests related to view index builds.  |
|               |               |                                            |
| db_compact    | compaction    | IO requests related to database            |
|               |               | compactions.                               |
|               |               |                                            |
| view_compact  | compaction    | IO requests related to view compactions.   |
|               |               |                                            |
| internal_repl | replication   | IO requests related to internal            |
|               |               | replication.                               |
|               |               |                                            |
| low           | low           | IO requests related to requests made by    |
|               |               | users via the http layer where the         |
|               |               | "x-cloudant-priority: low" header is set.  |
|               |               |                                            |
| other         | undefined     | IO requests that do not fit any of the     |
|               |               | above classes. This includes search IO     |
|               |               | requests.                                  |
|---------------+---------------+--------------------------------------------|
```

## Internals

To understand the relationship between the IOQ classes and the IOQ priorities
it is helpful to understand the channels into which IO requests are enqueued.

IOQ uses the following four channels:

 - `Compaction`
 - `Internal replication`
 - `Low`
 - `Customer`

The `Customer` channel is effectively a meta-channel where each item in the
queue represents a backend/dbname combination that consists of a further three
channels:

 - `Interactive`
 - `DB update`
 - `View update`

Requests are enqueued according to the following scheme:

 - Requests with class `internal_repl`, `low`, `db_compact` or `view_compact`
   are enqueued into `Internal replication`, `Low` or `Compaction` channels
   respectively.
 - Requests with class `interactive`, `db_update` or `view_update` are enqueued
   into the `Interactive`, `DB update` or `View update` channel of the relevant
   `Customer` channel for the backend/database combination.
 - Requests with class `other` are enqueued into the `Interactive` queue of a
   `Customer` channel reserved for `other` IOQ requests.

Requests are submitted as follows:

 - The next item is selected from either the `Compaction`,
   `Internal replication`, `Low` or `Customer` channel according to the
   configured priorities (`compaction`, `replication`, `low` and `customer`).
 - If the item is obtained from the `Compaction`, `Internal replication` or
   `Low` channels then the request is submitted for execution.
 - If the item is obtained from the `Customer` channel then the request is
   selected from either the `Interactive`, `DB update` or `View update` channel
   according to the configured priorities (`reads`, `writes`, and `views`).

## Configuration

Unless there is prior knowledge of the IOQ configuration required to support the
intended workload of a cluster on a given hardware specification it is
recommended that IOQ is initially left with the default configuration values. As
more becomes known about the behaviour of a cluster under load the IOQ settings
can be tuned to provide optimal performance for the production workload.

Note that tuning IOQ is not the answer to all performance problems and there are
a finite number of gains to be had (possibly zero). You should also be
considering the total load on the cluster, the capabilities of the underlying
hardware and the usage patterns and design of the applications which sit on top
of the data layer.

### Priorities

IOQ ships with a default configuration which gives interactive reads/writes and
view builds a high priority (`1.0`) and the background requests a much lower
priority (`0.001` for compaction and `0.0001` for replication and low).

You can set the priorities to other values using the config app in a remsh as
follows:

    config:set("ioq", "views", "0.5", "FBXXXXX reduce views IOQ priority").

To return to the default value just delete the configuration value:

    config:delete("ioq", "views", "FBXXXXX revert to default priority").

The following sections describe typical situations where tuning IOQ priorities
might be appropriate.

#### Internal replication backlog

If cluster nodes are frequently exhibiting an internal replication backlog
then it might be worth increasing the `replication` priority.

A backlog can be confirmed by checking the following graphite target:

    net.cloudant.cluster001.db*.erlang.internal_replication_jobs

If this value is consistently elevated by more than a few hundred changes then
try increasing the `replication` IOQ priority:

    config:set("ioq", "replication", "0.5", "FBXXXXX speed up internal replication").

If this has been effective you should notice a change in the rate at which the
metric decreases. It is worth experimenting with values as high as `1.0` however
you will need to keep an eye on HTTP request latencies to make sure there is no
adverse impact on other aspects of cluster performance.

#### Compactions not completing quickly enough

If disk usage is rising on cluster nodes and there is a corresponding backlog
in compaction work then it might be worth increasing the `compaction` priority.

Check the volume of pending changes for ongoing compaction jobs in graphite:

    net.cloudant.cluster001.db1.dbcore.active_tasks.changes_pending.*compaction

Increase the priority for `compaction`:

    config:set("ioq", "compaction", "0.5", "FBXXXXX speed up compaction").

Now monitor the changes_pending metrics to see if the rate at which changes are
being processed has increased.

The notes in previous section apply here - experiment with values as high as
"1.0" if necessary and keep a close eye on cluster performance whilst you
do so.

#### Interactive requests and views competing for IO resource

Metrics might show that read/write performance worsens when views are building
or conversely that view build performance slows when read/write load increases.
If the performance requirements of the cluster are such that a particular
type of request is more critical to the application it supports then it might be
worth reducing the other IOQ priorities, for example:

    config:set("ioq", "views", "0.1", "FBXXXXX attempt to improve read/write performance").

### Concurrency

The concurrency defines the total number of concurrent IO operations allowed by
IOQ. The default value is `20` however it can be worth increasing if the
answer to the following questions is yes:

 1. Either `net.cloudant.cluster001.db1.erlang.io_queue.active_requests` or
    `net.cloudant.cluster001.db1.couchdb.io_queue.latency` is consistently
    elevated.

 2. Disk utilisation is significantly less than 100%.

If performance is being impacted by request waiting in the queues then it is
worth bumping IOQ concurrency (sensible values to try are `30`, `50` and `100`)
and observing the resulting effect.

Note that increasing this value beyond a certain point can result in the disks
being overloaded and overall performance degradation. The exact point depends
on the cluster workload and hardware so it is very important to monitor the
cluster when making changes here.

### Bypasses

In extreme cases it is possible that IOQ itself is the bottleneck for certain
request classes. If this is case then you can bypass IOQ for that request
class altogether, e.g. for interactive requests:

    config:set("ioq.bypass", "interactive", "true", "FBXXXXX attempt to improve interactive performance").

Note that bypasses are set for IOQ *classes* not IOQ priorities. This means if
you wanted to bypass all compaction requests you would need to set a bypass for
`db_compact` and `view_compact`.

The following warnings should be heeded when considering setting an IOQ bypass:

 - Other request classes will continue to be routed through IOQ so will not
   be able to compete with the bypassed requests. You should therefore monitor
   the cluster carefully to determine that overall performance is acceptable.
   Keep a close eye on compaction in particular (unless it is being bypassed)
   as if the rate of compaction slows too much the disk may start filling up.

 - The bypass effectively shifts the bottleneck to another part of the system
   which is typically evident in `couch_file` and `couch_db_updater` message
   queue backups.

 - Disk performance may also become saturated which could lead to various
   resulting performance degradations.

A good rule of thumb is to avoid IOQ bypasses altogether unless the customer
is in immediate pain.