| = Use Nodetool |
| |
| Cassandra's `nodetool` allows you to narrow problems from the cluster |
| down to a particular node and gives a lot of insight into the state of |
| the Cassandra process itself. There are dozens of useful commands (see |
| `nodetool help` for all the commands), but briefly some of the most |
| useful for troubleshooting: |
| |
| [[nodetool-status]] |
| == Cluster Status |
| |
| You can use `nodetool status` to assess status of the cluster: |
| |
| [source, bash] |
| ---- |
| $ nodetool status <optional keyspace> |
| |
| Datacenter: dc1 |
| ======================= |
| Status=Up/Down |
| |/ State=Normal/Leaving/Joining/Moving |
| -- Address Load Tokens Owns (effective) Host ID Rack |
| UN 127.0.1.1 4.69 GiB 1 100.0% 35ea8c9f-b7a2-40a7-b9c5-0ee8b91fdd0e r1 |
| UN 127.0.1.2 4.71 GiB 1 100.0% 752e278f-b7c5-4f58-974b-9328455af73f r2 |
| UN 127.0.1.3 4.69 GiB 1 100.0% 9dc1a293-2cc0-40fa-a6fd-9e6054da04a7 r3 |
| ---- |
| |
| In this case we can see that we have three nodes in one datacenter with |
| about 4.6GB of data each and they are all "up". The up/down status of a |
| node is independently determined by every node in the cluster, so you |
| may have to run `nodetool status` on multiple nodes in a cluster to see |
| the full view. |
| |
| You can use `nodetool status` plus a little grep to see which nodes are |
| down: |
| |
| [source, bash] |
| ---- |
| $ nodetool status | grep -v '^UN' |
| Datacenter: dc1 |
| =============== |
| Status=Up/Down |
| |/ State=Normal/Leaving/Joining/Moving |
| -- Address Load Tokens Owns (effective) Host ID Rack |
| Datacenter: dc2 |
| =============== |
| Status=Up/Down |
| |/ State=Normal/Leaving/Joining/Moving |
| -- Address Load Tokens Owns (effective) Host ID Rack |
| DN 127.0.0.5 105.73 KiB 1 33.3% df303ac7-61de-46e9-ac79-6e630115fd75 r1 |
| ---- |
| |
| In this case there are two datacenters and there is one node down in |
| datacenter `dc2` and rack `r1`. This may indicate an issue on |
| `127.0.0.5` warranting investigation. |
| |
| [[nodetool-proxyhistograms]] |
| == Coordinator Query Latency |
| |
| You can view latency distributions of coordinator read and write latency |
| to help narrow down latency issues using `nodetool proxyhistograms`: |
| |
| [source, bash] |
| ---- |
| $ nodetool proxyhistograms |
| Percentile Read Latency Write Latency Range Latency CAS Read Latency CAS Write Latency View Write Latency |
| (micros) (micros) (micros) (micros) (micros) (micros) |
| 50% 454.83 219.34 0.00 0.00 0.00 0.00 |
| 75% 545.79 263.21 0.00 0.00 0.00 0.00 |
| 95% 654.95 315.85 0.00 0.00 0.00 0.00 |
| 98% 785.94 379.02 0.00 0.00 0.00 0.00 |
| 99% 3379.39 2346.80 0.00 0.00 0.00 0.00 |
| Min 42.51 105.78 0.00 0.00 0.00 0.00 |
| Max 25109.16 43388.63 0.00 0.00 0.00 0.00 |
| ---- |
| |
| Here you can see the full latency distribution of reads, writes, range |
| requests (e.g. `select * from keyspace.table`), CAS read (compare phase |
| of CAS) and CAS write (set phase of compare and set). These can be |
| useful for narrowing down high level latency problems, for example in |
| this case if a client had a 20 millisecond timeout on their reads they |
| might experience the occasional timeout from this node but less than 1% |
| (since the 99% read latency is 3.3 milliseconds < 20 milliseconds). |
| |
| [[nodetool-tablehistograms]] |
| == Local Query Latency |
| |
| If you know which table is having latency/error issues, you can use |
| `nodetool tablehistograms` to get a better idea of what is happening |
| locally on a node: |
| |
| [source, bash] |
| ---- |
| $ nodetool tablehistograms keyspace table |
| Percentile SSTables Write Latency Read Latency Partition Size Cell Count |
| (micros) (micros) (bytes) |
| 50% 0.00 73.46 182.79 17084 103 |
| 75% 1.00 88.15 315.85 17084 103 |
| 95% 2.00 126.93 545.79 17084 103 |
| 98% 2.00 152.32 654.95 17084 103 |
| 99% 2.00 182.79 785.94 17084 103 |
| Min 0.00 42.51 24.60 14238 87 |
| Max 2.00 12108.97 17436.92 17084 103 |
| ---- |
| |
| This shows you percentile breakdowns particularly critical metrics. |
| |
| The first column contains how many sstables were read per logical read. |
| A very high number here indicates that you may have chosen the wrong |
| compaction strategy, e.g. `SizeTieredCompactionStrategy` typically has |
| many more reads per read than `LeveledCompactionStrategy` does for |
| update heavy workloads. |
| |
| The second column shows you a latency breakdown of _local_ write |
| latency. In this case we see that while the p50 is quite good at 73 |
| microseconds, the maximum latency is quite slow at 12 milliseconds. High |
| write max latencies often indicate a slow commitlog volume (slow to |
| fsync) or large writes that quickly saturate commitlog segments. |
| |
| The third column shows you a latency breakdown of _local_ read latency. |
| We can see that local Cassandra reads are (as expected) slower than |
| local writes, and the read speed correlates highly with the number of |
| sstables read per read. |
| |
| The fourth and fifth columns show distributions of partition size and |
| column count per partition. These are useful for determining if the |
| table has on average skinny or wide partitions and can help you isolate |
| bad data patterns. For example if you have a single cell that is 2 |
| megabytes, that is probably going to cause some heap pressure when it's |
| read. |
| |
| [[nodetool-tpstats]] |
| == Threadpool State |
| |
| You can use `nodetool tpstats` to view the current outstanding requests |
| on a particular node. This is useful for trying to find out which |
| resource (read threads, write threads, compaction, request response |
| threads) the Cassandra process lacks. For example: |
| |
| [source, bash] |
| ---- |
| $ nodetool tpstats |
| Pool Name Active Pending Completed Blocked All time blocked |
| ReadStage 2 0 12 0 0 |
| MiscStage 0 0 0 0 0 |
| CompactionExecutor 0 0 1940 0 0 |
| MutationStage 0 0 0 0 0 |
| GossipStage 0 0 10293 0 0 |
| Repair-Task 0 0 0 0 0 |
| RequestResponseStage 0 0 16 0 0 |
| ReadRepairStage 0 0 0 0 0 |
| CounterMutationStage 0 0 0 0 0 |
| MemtablePostFlush 0 0 83 0 0 |
| ValidationExecutor 0 0 0 0 0 |
| MemtableFlushWriter 0 0 30 0 0 |
| ViewMutationStage 0 0 0 0 0 |
| CacheCleanupExecutor 0 0 0 0 0 |
| MemtableReclaimMemory 0 0 30 0 0 |
| PendingRangeCalculator 0 0 11 0 0 |
| SecondaryIndexManagement 0 0 0 0 0 |
| HintsDispatcher 0 0 0 0 0 |
| Native-Transport-Requests 0 0 192 0 0 |
| MigrationStage 0 0 14 0 0 |
| PerDiskMemtableFlushWriter_0 0 0 30 0 0 |
| Sampler 0 0 0 0 0 |
| ViewBuildExecutor 0 0 0 0 0 |
| InternalResponseStage 0 0 0 0 0 |
| AntiEntropyStage 0 0 0 0 0 |
| |
| Message type Dropped Latency waiting in queue (micros) |
| 50% 95% 99% Max |
| READ 0 N/A N/A N/A N/A |
| RANGE_SLICE 0 0.00 0.00 0.00 0.00 |
| _TRACE 0 N/A N/A N/A N/A |
| HINT 0 N/A N/A N/A N/A |
| MUTATION 0 N/A N/A N/A N/A |
| COUNTER_MUTATION 0 N/A N/A N/A N/A |
| BATCH_STORE 0 N/A N/A N/A N/A |
| BATCH_REMOVE 0 N/A N/A N/A N/A |
| REQUEST_RESPONSE 0 0.00 0.00 0.00 0.00 |
| PAGED_RANGE 0 N/A N/A N/A N/A |
| READ_REPAIR 0 N/A N/A N/A N/A |
| ---- |
| |
| This command shows you all kinds of interesting statistics. The first |
| section shows a detailed breakdown of threadpools for each Cassandra |
| stage, including how many threads are current executing (Active) and how |
| many are waiting to run (Pending). Typically if you see pending |
| executions in a particular threadpool that indicates a problem localized |
| to that type of operation. For example if the `RequestResponseState` |
| queue is backing up, that means that the coordinators are waiting on a |
| lot of downstream replica requests and may indicate a lack of token |
| awareness, or very high consistency levels being used on read requests |
| (for example reading at `ALL` ties up RF `RequestResponseState` threads |
| whereas `LOCAL_ONE` only uses a single thread in the `ReadStage` |
| threadpool). On the other hand if you see a lot of pending compactions |
| that may indicate that your compaction threads cannot keep up with the |
| volume of writes and you may need to tune either the compaction strategy |
| or the `concurrent_compactors` or `compaction_throughput` options. |
| |
| The second section shows drops (errors) and latency distributions for |
| all the major request types. Drops are cumulative since process start, |
| but if you have any that indicate a serious problem as the default |
| timeouts to qualify as a drop are quite high (~5-10 seconds). Dropped |
| messages often warrants further investigation. |
| |
| [[nodetool-compactionstats]] |
| == Compaction State |
| |
| As Cassandra is a LSM datastore, Cassandra sometimes has to compact |
| sstables together, which can have adverse effects on performance. In |
| particular, compaction uses a reasonable quantity of CPU resources, |
| invalidates large quantities of the OS |
| https://en.wikipedia.org/wiki/Page_cache[page cache], and can put a lot |
| of load on your disk drives. There are great `os tools <os-iostat>` to |
| determine if this is the case, but often it's a good idea to check if |
| compactions are even running using `nodetool compactionstats`: |
| |
| [source, bash] |
| ---- |
| $ nodetool compactionstats |
| pending tasks: 2 |
| - keyspace.table: 2 |
| |
| id compaction type keyspace table completed total unit progress |
| 2062b290-7f3a-11e8-9358-cd941b956e60 Compaction keyspace table 21848273 97867583 bytes 22.32% |
| Active compaction remaining time : 0h00m04s |
| ---- |
| |
| In this case there is a single compaction running on the |
| `keyspace.table` table, has completed 21.8 megabytes of 97 and Cassandra |
| estimates (based on the configured compaction throughput) that this will |
| take 4 seconds. You can also pass `-H` to get the units in a human |
| readable format. |
| |
| Generally each running compaction can consume a single core, but the |
| more you do in parallel the faster data compacts. Compaction is crucial |
| to ensuring good read performance so having the right balance of |
| concurrent compactions such that compactions complete quickly but don't |
| take too many resources away from query threads is very important for |
| performance. If you notice compaction unable to keep up, try tuning |
| Cassandra's `concurrent_compactors` or `compaction_throughput` options. |
