IOQ handles the prioritisation of IO operations in the database. It has two main responsibilities:
From an operational perspective point 1 is of most interest as it provides a set of levers that can be pulled to change the behaviour of the cluster in favour of particular workloads or operational concerns.
From an operational point-of-view, IOQ carries out two fundamental operations:
IOQ categorises IO requests by class and by priority. The class of a request dictates the channel into which it will be enqueued and the priority influences the probability that a given request will be dequeued and executed.
The following table lists the IOQ classes and the corresponding priorities. Note that the mapping of IOQ classes to class priorities is not 1:1.
|---------------+---------------+--------------------------------------------| | IOQ class | IOQ priority | Description | |---------------+---------------+--------------------------------------------| | interactive | reads, writes | IO requests related to requests made by | | | | users via the http layer. | | | | | | db_update | writes | Interactive IO requests which are database | | | | write operations. | | | | | | view_update | views | IO requests related to view index builds. | | | | | | db_compact | compaction | IO requests related to database | | | | compactions. | | | | | | view_compact | compaction | IO requests related to view compactions. | | | | | | internal_repl | replication | IO requests related to internal | | | | replication. | | | | | | low | low | IO requests related to requests made by | | | | users via the http layer where the | | | | "x-cloudant-priority: low" header is set. | | | | | | other | undefined | IO requests that do not fit any of the | | | | above classes. This includes search IO | | | | requests. | |---------------+---------------+--------------------------------------------|
To understand the relationship between the IOQ classes and the IOQ priorities it is helpful to understand the channels into which IO requests are enqueued.
IOQ uses the following four channels:
Compaction
Internal replication
Low
Customer
The Customer
channel is effectively a meta-channel where each item in the queue represents a backend/dbname combination that consists of a further three channels:
Interactive
DB update
View update
Requests are enqueued according to the following scheme:
internal_repl
, low
, db_compact
or view_compact
are enqueued into Internal replication
, Low
or Compaction
channels respectively.interactive
, db_update
or view_update
are enqueued into the Interactive
, DB update
or View update
channel of the relevant Customer
channel for the backend/database combination.other
are enqueued into the Interactive
queue of a Customer
channel reserved for other
IOQ requests.Requests are submitted as follows:
Compaction
, Internal replication
, Low
or Customer
channel according to the configured priorities (compaction
, replication
, low
and customer
).Compaction
, Internal replication
or Low
channels then the request is submitted for execution.Customer
channel then the request is selected from either the Interactive
, DB update
or View update
channel according to the configured priorities (reads
, writes
, and views
).Unless there is prior knowledge of the IOQ configuration required to support the intended workload of a cluster on a given hardware specification it is recommended that IOQ is initially left with the default configuration values. As more becomes known about the behaviour of a cluster under load the IOQ settings can be tuned to provide optimal performance for the production workload.
Note that tuning IOQ is not the answer to all performance problems and there are a finite number of gains to be had (possibly zero). You should also be considering the total load on the cluster, the capabilities of the underlying hardware and the usage patterns and design of the applications which sit on top of the data layer.
IOQ ships with a default configuration which gives interactive reads/writes and view builds a high priority (1.0
) and the background requests a much lower priority (0.001
for compaction and 0.0001
for replication and low).
You can set the priorities to other values using the config app in a remsh as follows:
config:set("ioq", "views", "0.5", "FBXXXXX reduce views IOQ priority").
To return to the default value just delete the configuration value:
config:delete("ioq", "views", "FBXXXXX revert to default priority").
The following sections describe typical situations where tuning IOQ priorities might be appropriate.
If cluster nodes are frequently exhibiting an internal replication backlog then it might be worth increasing the replication
priority.
A backlog can be confirmed by checking the following graphite target:
net.cloudant.cluster001.db*.erlang.internal_replication_jobs
If this value is consistently elevated by more than a few hundred changes then try increasing the replication
IOQ priority:
config:set("ioq", "replication", "0.5", "FBXXXXX speed up internal replication").
If this has been effective you should notice a change in the rate at which the metric decreases. It is worth experimenting with values as high as 1.0
however you will need to keep an eye on HTTP request latencies to make sure there is no adverse impact on other aspects of cluster performance.
If disk usage is rising on cluster nodes and there is a corresponding backlog in compaction work then it might be worth increasing the compaction
priority.
Check the volume of pending changes for ongoing compaction jobs in graphite:
net.cloudant.cluster001.db1.dbcore.active_tasks.changes_pending.*compaction
Increase the priority for compaction
:
config:set("ioq", "compaction", "0.5", "FBXXXXX speed up compaction").
Now monitor the changes_pending metrics to see if the rate at which changes are being processed has increased.
The notes in previous section apply here - experiment with values as high as “1.0” if necessary and keep a close eye on cluster performance whilst you do so.
Metrics might show that read/write performance worsens when views are building or conversely that view build performance slows when read/write load increases. If the performance requirements of the cluster are such that a particular type of request is more critical to the application it supports then it might be worth reducing the other IOQ priorities, for example:
config:set("ioq", "views", "0.1", "FBXXXXX attempt to improve read/write performance").
The concurrency defines the total number of concurrent IO operations allowed by IOQ. The default value is 20
however it can be worth increasing if the answer to the following questions is yes:
Either net.cloudant.cluster001.db1.erlang.io_queue.active_requests
or net.cloudant.cluster001.db1.couchdb.io_queue.latency
is consistently elevated.
Disk utilisation is significantly less than 100%.
If performance is being impacted by request waiting in the queues then it is worth bumping IOQ concurrency (sensible values to try are 30
, 50
and 100
) and observing the resulting effect.
Note that increasing this value beyond a certain point can result in the disks being overloaded and overall performance degradation. The exact point depends on the cluster workload and hardware so it is very important to monitor the cluster when making changes here.
In extreme cases it is possible that IOQ itself is the bottleneck for certain request classes. If this is case then you can bypass IOQ for that request class altogether, e.g. for interactive requests:
config:set("ioq.bypass", "interactive", "true", "FBXXXXX attempt to improve interactive performance").
Note that bypasses are set for IOQ classes not IOQ priorities. This means if you wanted to bypass all compaction requests you would need to set a bypass for db_compact
and view_compact
.
The following warnings should be heeded when considering setting an IOQ bypass:
Other request classes will continue to be routed through IOQ so will not be able to compete with the bypassed requests. You should therefore monitor the cluster carefully to determine that overall performance is acceptable. Keep a close eye on compaction in particular (unless it is being bypassed) as if the rate of compaction slows too much the disk may start filling up.
The bypass effectively shifts the bottleneck to another part of the system which is typically evident in couch_file
and couch_db_updater
message queue backups.
Disk performance may also become saturated which could lead to various resulting performance degradations.
A good rule of thumb is to avoid IOQ bypasses altogether unless the customer is in immediate pain.