Apache couchdb

Clone this repo:

Branches

  1. 78eba84 Add unit tests for scheduling window logic by Adam Kocoloski · 6 weeks ago master
  2. dddaa3c Support scheduling compactions during time windows by Adam Kocoloski · 7 weeks ago
  3. 8553ee6 Merge pull request #2 from cloudant/fix-compilation-warnings by iilyak · 8 months ago 1.0.1
  4. 55bf6a3 Avoid using `erlang:now/0` by ILYA Khlopotov · 8 months ago
  5. d85183d Merge pull request #1 from cloudant/update_handle_config_terminate by iilyak · 8 months ago

Smoosh

Smoosh is CouchDB's auto-compaction daemon. It is notified when databases and views are updated and may then elect to enqueue them for compaction.

API

All API functions are in smoosh.erl and only the exported functions in this module should be called from outside of the smoosh application.

Additionally, smoosh responds to config changes dynamically and these changes are the principal means of interacting with smoosh.

Top-Level Settings

The main settings one interacts with are:

Sometimes it's necessary to use the following:

Channel Settings

A channel has several important settings that control runtime behavior.

Structure

Smoosh consists of a central gen_server (smoosh_server) which manages a number of subordinate smoosh_channel gen_servers. This is not properly managed by OTP yet.

Compaction Scheduling Algorithm

Smoosh decides whether to compact a database or view by evaluating the item against the selection criteria of each channel in the order they are configured. By default there are two channels for databases (“ratio_dbs” and “slack_dbs”), and two channels for views (“ratio_views” and “slack_views”)

Smoosh will enqueue the new item to the first channel that accepts it. If none accept it, the item is not enqueued for compaction.

Notes on the data_size value

Every database and view shard has a data_size value. In CouchDB this accurately reflects the post-compaction file size. In DbCore, it is the size of the file that we bill for. It excludes the b+tree and database footer overhead. We also bill customers for the uncompressed size of their documents, though we store them compressed on disk. These two systems were developed independently (ours predates CouchDB's) and DbCore only calculates the billing size value.

Because of the way our data_size is currently calculated, it can sometimes be necessary to enqueue databases and views with very low ratios. Due to this, it is also currently impossible to tell how optimally compacted a cluster is.

Example config commands

Change the set of database channels;

config:set("smoosh", "db_channels", "small_dbs,medium_dbs,large_dbs").

Change the set of database channels on all live nodes in the cluster;

rpc:multicall(config, set, ["smoosh", "db_channels", "small_dbs,medium_dbs,large_dbs"]).

Change the concurrency of the ratio_dbs database channel to 2

config:set("smoosh.ratio_dbs", "concurrency", "2").

Change it on all live nodes in the cluster;

rpc:multicall(config, set, ["smoosh.ratio_dbs", "concurrency", "2"]).

Example API commands

smoosh:status()

This prints the state of each channel; how many jobs they are currently running and how many jobs are enqueued (as well as the lowest and highest priority of those enqueued items). The idea is to provide, at a glance, sufficient insight into smoosh that an operator can assess whether smoosh is adequately targeting the reclaimable space in the cluster. In general, a healthy status output will have items in the ratio_dbs and ratio_views channels. Owing to the default settings, the slack_dbs and slack_views will almost certainly have items in them. Historically, we've not found that the slack channels, on their own, are particularly adept at keeping things well compacted.

smoosh:enqueue_all_dbs(), smoosh:enqueue_all_views()

These functions do just what they say but should not generally need to be called, smoosh is supposed to be autonomous. Call them if you get alerted to a disk space issue, they might well help. If they do, that indicates a bug in smoosh as it should already have enqueued eligible shards once they met the configured settings.