IOQ handles the prioritisation of IO operations in the database. It has two main responsibilities:
From an operational perspective point 1 is of most interest as it provides a set of levers that can be pulled to change the behaviour of the cluster in favour of particular workloads or operational concerns.
From an operational point-of-view, IOQ carries out two fundamental operations:
IOQ categorises IO requests by class and by priority. The class of a request dictates the channel into which it will be enqueued and the priority influences the probability that a given request will be dequeued and executed.
The following table lists the IOQ classes and the corresponding priorities. Note that the mapping of IOQ classes to class priorities is not 1:1.
|---------------+---------------+--------------------------------------------| | IOQ class | IOQ priority | Description | |---------------+---------------+--------------------------------------------| | interactive | reads, writes | IO requests related to requests made by | | | | users via the http layer. | | | | | | db_update | writes | Interactive IO requests which are database | | | | write operations. | | | | | | view_update | views | IO requests related to view index builds. | | | | | | db_compact | compaction | IO requests related to database | | | | compactions. | | | | | | view_compact | compaction | IO requests related to view compactions. | | | | | | internal_repl | replication | IO requests related to internal | | | | replication. | | | | | | low | low | IO requests related to requests made by | | | | users via the http layer where the | | | | "x-cloudant-priority: low" header is set. | | | | | | other | undefined | IO requests that do not fit any of the | | | | above classes. This includes search IO | | | | requests. | |---------------+---------------+--------------------------------------------|
To understand the relationship between the IOQ classes and the IOQ priorities it is helpful to understand the channels into which IO requests are enqueued.
IOQ uses the following four channels:
Customer channel is effectively a meta-channel where each item in the queue represents a backend/dbname combination that consists of a further three channels:
Requests are enqueued according to the following scheme:
view_compactare enqueued into
view_updateare enqueued into the
View updatechannel of the relevant
Customerchannel for the backend/database combination.
otherare enqueued into the
Interactivequeue of a
Customerchannel reserved for
Requests are submitted as follows:
Customerchannel according to the configured priorities (
Lowchannels then the request is submitted for execution.
Customerchannel then the request is selected from either the
View updatechannel according to the configured priorities (
Unless there is prior knowledge of the IOQ configuration required to support the intended workload of a cluster on a given hardware specification it is recommended that IOQ is initially left with the default configuration values. As more becomes known about the behaviour of a cluster under load the IOQ settings can be tuned to provide optimal performance for the production workload.
Note that tuning IOQ is not the answer to all performance problems and there are a finite number of gains to be had (possibly zero). You should also be considering the total load on the cluster, the capabilities of the underlying hardware and the usage patterns and design of the applications which sit on top of the data layer.
IOQ ships with a default configuration which gives interactive reads/writes and view builds a high priority (
1.0) and the background requests a much lower priority (
0.001 for compaction and
0.0001 for replication and low).
You can set the priorities to other values using the config app in a remsh as follows:
config:set("ioq", "views", "0.5", "FBXXXXX reduce views IOQ priority").
To return to the default value just delete the configuration value:
config:delete("ioq", "views", "FBXXXXX revert to default priority").
The following sections describe typical situations where tuning IOQ priorities might be appropriate.
If cluster nodes are frequently exhibiting an internal replication backlog then it might be worth increasing the
A backlog can be confirmed by checking the following graphite target:
If this value is consistently elevated by more than a few hundred changes then try increasing the
replication IOQ priority:
config:set("ioq", "replication", "0.5", "FBXXXXX speed up internal replication").
If this has been effective you should notice a change in the rate at which the metric decreases. It is worth experimenting with values as high as
1.0 however you will need to keep an eye on HTTP request latencies to make sure there is no adverse impact on other aspects of cluster performance.
If disk usage is rising on cluster nodes and there is a corresponding backlog in compaction work then it might be worth increasing the
Check the volume of pending changes for ongoing compaction jobs in graphite:
Increase the priority for
config:set("ioq", "compaction", "0.5", "FBXXXXX speed up compaction").
Now monitor the changes_pending metrics to see if the rate at which changes are being processed has increased.
The notes in previous section apply here - experiment with values as high as “1.0” if necessary and keep a close eye on cluster performance whilst you do so.
Metrics might show that read/write performance worsens when views are building or conversely that view build performance slows when read/write load increases. If the performance requirements of the cluster are such that a particular type of request is more critical to the application it supports then it might be worth reducing the other IOQ priorities, for example:
config:set("ioq", "views", "0.1", "FBXXXXX attempt to improve read/write performance").
The concurrency defines the total number of concurrent IO operations allowed by IOQ. The default value is
20 however it can be worth increasing if the answer to the following questions is yes:
net.cloudant.cluster001.db1.couchdb.io_queue.latency is consistently elevated.
Disk utilisation is significantly less than 100%.
If performance is being impacted by request waiting in the queues then it is worth bumping IOQ concurrency (sensible values to try are
100) and observing the resulting effect.
Note that increasing this value beyond a certain point can result in the disks being overloaded and overall performance degradation. The exact point depends on the cluster workload and hardware so it is very important to monitor the cluster when making changes here.
In extreme cases it is possible that IOQ itself is the bottleneck for certain request classes. If this is case then you can bypass IOQ for that request class altogether, e.g. for interactive requests:
config:set("ioq.bypass", "interactive", "true", "FBXXXXX attempt to improve interactive performance").
Note that bypasses are set for IOQ classes not IOQ priorities. This means if you wanted to bypass all compaction requests you would need to set a bypass for
The following warnings should be heeded when considering setting an IOQ bypass:
Other request classes will continue to be routed through IOQ so will not be able to compete with the bypassed requests. You should therefore monitor the cluster carefully to determine that overall performance is acceptable. Keep a close eye on compaction in particular (unless it is being bypassed) as if the rate of compaction slows too much the disk may start filling up.
The bypass effectively shifts the bottleneck to another part of the system which is typically evident in
couch_db_updater message queue backups.
Disk performance may also become saturated which could lead to various resulting performance degradations.
A good rule of thumb is to avoid IOQ bypasses altogether unless the customer is in immediate pain.