.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at
..
..     http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.

.. highlight:: none

.. _monitoring-metrics:

Monitoring
----------

Metrics in Cassandra are managed using the `Dropwizard Metrics <http://metrics.dropwizard.io>`__ library. These metrics
can be queried via JMX or pushed to external monitoring systems using a number of `built in
<http://metrics.dropwizard.io/3.1.0/getting-started/#other-reporting>`__ and `third party
<http://metrics.dropwizard.io/3.1.0/manual/third-party/>`__ reporter plugins.

Metrics are collected for a single node. It's up to the operator to use an external monitoring system to aggregate them.

Metric Types
^^^^^^^^^^^^
All metrics reported by cassandra fit into one of the following types.

``Gauge``
    An instantaneous measurement of a value.

``Counter``
    A gauge for an ``AtomicLong`` instance. Typically this is consumed by monitoring the change since the last call to
    see if there is a large increase compared to the norm.

``Histogram``
    Measures the statistical distribution of values in a stream of data.

    In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th
    percentiles.

``Timer``
    Measures both the rate that a particular piece of code is called and the histogram of its duration.

``Latency``
    Special type that tracks latency (in microseconds) with a ``Timer`` plus a ``Counter`` that tracks the total latency
    accrued since starting. The former is useful if you track the change in total latency since the last check. Each
    metric name of this type will have 'Latency' and 'TotalLatency' appended to it.

``Meter``
    A meter metric which measures mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving
    average throughputs.

.. _table-metrics:

Table Metrics
^^^^^^^^^^^^^

Each table in Cassandra has metrics responsible for tracking its state and performance.

The metric names are all appended with the specific ``Keyspace`` and ``Table`` name.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Table.<MetricName>.<Keyspace>.<Table>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Table,keyspace=<Keyspace>,scope=<Table>,name=<MetricName>``

.. NOTE::
    There is a special table called '``all``' without a keyspace. This represents the aggregation of metrics across
    **all** tables and keyspaces on the node.


=============================================== ============== ===========
Name                                            Type           Description
=============================================== ============== ===========
AdditionalWrites                                Counter        Total number of additional writes sent to the replicas (other than the intial contacted ones).
AllMemtablesLiveDataSize                        Gauge<Long>    Total amount of live data stored in the memtables (2i and pending flush memtables included) that resides off-heap, excluding any data structure overhead.
AllMemtablesOffHeapDataSize                     Gauge<Long>    Total amount of data stored in the memtables (2i and pending flush memtables included) that resides **off**-heap.
AllMemtablesOffHeapSize (Deprecated)            Gauge<Long>    Total amount of data stored in the memtables (2i and pending flush memtables included) that resides **off**-heap.
AllMemtablesOnHeapDataSize                      Gauge<Long>    Total amount of data stored in the memtables (2i and pending flush memtables included) that resides **on**-heap.
AllMemtablesOnHeapSize (Deprecated)             Gauge<Long>    Total amount of data stored in the memtables (2i and pending flush memtables included) that resides **on**-heap.
AnticompactionTime                              Timer          Time spent anticompacting before a consistent repair.
BloomFilterDiskSpaceUsed                        Gauge<Long>    Disk space used by bloom filter (in bytes).
BloomFilterFalsePositives                       Gauge<Long>    Number of false positives on table's bloom filter.
BloomFilterFalseRatio                           Gauge<Double>  False positive ratio of table's bloom filter.
BloomFilterOffHeapMemoryUsed                    Gauge<Long>    Off-heap memory used by bloom filter.
BytesAnticompacted                              Counter        How many bytes we anticompacted.
BytesFlushed                                    Counter        Total number of bytes flushed since server [re]start.
BytesMutatedAnticompaction                      Counter        How many bytes we avoided anticompacting because the sstable was fully contained in the repaired range.
BytesPendingRepair                              Gauge<Long>    Size of table data isolated for an ongoing incremental repair
BytesRepaired                                   Gauge<Long>    Size of table data repaired on disk
BytesUnrepaired                                 Gauge<Long>    Size of table data unrepaired on disk
BytesValidated                                  Histogram      Histogram over the amount of bytes read during validation.
CasCommit                                       Latency        Latency of paxos commit round.
CasPrepare                                      Latency        Latency of paxos prepare round.
CasPropose                                      Latency        Latency of paxos propose round.
ColUpdateTimeDeltaHistogram                     Histogram      Histogram of column update time delta on this table.
CompactionBytesWritten                          Counter        Total number of bytes written by compaction since server [re]start.
CompressionMetadataOffHeapMemoryUsed            Gauge<Long>    Off-heap memory used by compression meta data.
CompressionRatio                                Gauge<Double>  Current compression ratio for all SSTables.
CoordinatorReadLatency                          Timer          Coordinator read latency for this table.
CoordinatorScanLatency                          Timer          Coordinator range scan latency for this table.
CoordinatorWriteLatency                         Timer          Coordinator write latency for this table.
DroppedMutations                                Counter        Number of dropped mutations on this table.
EstimatedColumnCountHistogram                   Gauge<long[]>  Histogram of estimated number of columns.
EstimatedPartitionCount                         Gauge<Long>    Approximate number of keys in table.
EstimatedPartitionSizeHistogram                 Gauge<long[]>  Histogram of estimated partition size (in bytes).
IndexSummaryOffHeapMemoryUsed                   Gauge<Long>    Off-heap memory used by index summary.
KeyCacheHitRate                                 Gauge<Double>  Key cache hit rate for this table.
LiveDiskSpaceUsed                               Counter        Disk space used by SSTables belonging to this table (in bytes).
LiveSSTableCount                                Gauge<Integer> Number of SSTables on disk for this table.
LiveScannedHistogram                            Histogram      Histogram of live cells scanned in queries on this table.
MaxPartitionSize                                Gauge<Long>    Size of the largest compacted partition (in bytes).
MeanPartitionSize                               Gauge<Long>    Size of the average compacted partition (in bytes).
MemtableColumnsCount                            Gauge<Long>    Total number of columns present in the memtable.
MemtableLiveDataSize                            Gauge<Long>    Total amount of live data stored in the memtable, excluding any data structure overhead.
MemtableOffHeapDataSize                         Gauge<Long>    Total amount of data stored in the memtable that resides **off**-heap, including column related overhead and partitions overwritten.
MemtableOffHeapSize (Deprecated)                Gauge<Long>    Total amount of data stored in the memtable that resides **off**-heap, including column related overhead and partitions overwritten.
MemtableOnHeapDataSize                          Gauge<Long>    Total amount of data stored in the memtable that resides **on**-heap, including column related overhead and partitions overwritten.
MemtableOnHeapSize (Deprecated)                 Gauge<Long>    Total amount of data stored in the memtable that resides **on**-heap, including column related overhead and partitions overwritten.
MemtableSwitchCount                             Counter        Number of times flush has resulted in the memtable being switched out.
MinPartitionSize                                Gauge<Long>    Size of the smallest compacted partition (in bytes).
MutatedAnticompactionGauge                      Gauge<Double>  Ratio of bytes mutated vs total bytes repaired.
PartitionsValidated                             Histogram      Histogram over the number of partitions read during validation.
PendingCompactions                              Gauge<Integer> Estimate of number of pending compactions for this table.
PendingFlushes                                  Counter        Estimated number of flush tasks pending for this table.
PercentRepaired                                 Gauge<Double>  Percent of table data that is repaired on disk.
RangeLatency                                    Latency        Local range scan latency for this table.
ReadLatency                                     Latency        Local read latency for this table.
ReadRepairRequests                              Meter          Throughput for mutations generated by read-repair.
RepairJobsCompleted                             Counter        Total number of completed repairs as coordinator on this table.
RepairJobsStarted                               Counter        Total number of started repairs as coordinator on this table.
RepairSyncTime                                  Timer          Time spent doing streaming during repair.
RepairedDataTrackingOverreadRows                Histogram      Histogram of the amount of the additonal rows of the repaired data read.
RepairedDataTrackingOverreadTime                Timer          Time spent reading the additional rows of the repaired data.
ReplicaFilteringProtectionRequests              Meter          Throughput for row completion requests during replica filtering protection.
ReplicaFilteringProtectionRowsCachedPerQuery    Histogram      Histogram of the number of rows cached per query when replica filtering protection is engaged.
RowCacheHit                                     Counter        Number of table row cache hits.
RowCacheHitOutOfRange                           Counter        Number of table row cache hits that do not satisfy the query filter, thus went to disk.
RowCacheMiss                                    Counter        Number of table row cache misses.
SSTablesPerReadHistogram                        Histogram      Histogram of the number of sstable data files accessed per single partition read. SSTables skipped due to Bloom Filters, min-max key or partition index lookup are not taken into acoount.
ShortReadProtectionRequests                     Meter          Throughput for requests to get extra rows during short read protection.
SpeculativeFailedRetries                        Counter        Number of speculative retries that failed to prevent a timeout
SpeculativeInsufficientReplicas                 Counter        Number of speculative retries that couldn't be attempted due to lack of replicas
SpeculativeRetries                              Counter        Number of times speculative retries were sent for this table.
SpeculativeSampleLatencyNanos                   Gauge<Long>    Number of nanoseconds to wait before speculation is attempted. Value may be statically configured or updated periodically based on coordinator latency.
TombstoneScannedHistogram                       Histogram      Histogram of tombstones scanned in queries on this table.
TotalDiskSpaceUsed                              Counter        Total disk space used by SSTables belonging to this table, including obsolete ones waiting to be GC'd.
TrueSnapshotsSize                               Gauge<Long>    Disk space used by snapshots of this table including all SSTable components.
ValidationTime                                  Timer          Time spent doing validation compaction during repair.
ViewLockAcquireTime                             Timer          Time taken acquiring a partition lock for materialized view updates on this table.
ViewReadTime                                    Timer          Time taken during the local read of a materialized view update.
WaitingOnFreeMemtableSpace                      Histogram      Histogram of time spent waiting for free memtable space, either on- or off-heap.
WriteLatency                                    Latency        Local write latency for this table.
=============================================== ============== ===========

Keyspace Metrics
^^^^^^^^^^^^^^^^
Each keyspace in Cassandra has metrics responsible for tracking its state and performance.

Most of these metrics are the same as the ``Table Metrics`` above, only they are aggregated at the Keyspace level. Only the keyspace specific metrics are specified in the table below.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.keyspace.<MetricName>.<Keyspace>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Keyspace,keyspace=<Keyspace>,name=<MetricName>``


======================================= ============== ===========
Name                                    Type           Description
======================================= ============== ===========
IdealCLWriteLatency                     Latency        Coordinator latency of writes at the configured ideal consistency level. No values are recorded if ideal consistency level is not configured
RepairPrepareTime                       Timer          Total time spent preparing for repair.
RepairTime                              Timer          Total time spent as repair coordinator.
WriteFailedIdealCL                      Counter        Number of writes that failed to achieve the configured ideal consistency level or 0 if none is configured
======================================= ============== ===========

ThreadPool Metrics
^^^^^^^^^^^^^^^^^^

Cassandra splits work of a particular type into its own thread pool.  This provides back-pressure and asynchrony for
requests on a node.  It's important to monitor the state of these thread pools since they can tell you how saturated a
node is.

The metric names are all appended with the specific ``ThreadPool`` name.  The thread pools are also categorized under a
specific type.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.ThreadPools.<MetricName>.<Path>.<ThreadPoolName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=ThreadPools,path=<Path>,scope=<ThreadPoolName>,name=<MetricName>``

===================== ============== ===========
Name                  Type           Description
===================== ============== ===========
ActiveTasks           Gauge<Integer> Number of tasks being actively worked on by this pool.
CompletedTasks        Counter        Number of tasks completed.
CurrentlyBlockedTask  Counter        Number of tasks that are currently blocked due to queue saturation but on retry will become unblocked.
MaxPoolSize           Gauge<Integer> The maximum number of threads in this pool.
MaxTasksQueued        Gauge<Integer> The maximum number of tasks queued before a task get blocked.
PendingTasks          Gauge<Integer> Number of queued tasks queued up on this pool.
TotalBlockedTasks     Counter        Number of tasks that were blocked due to queue saturation.
===================== ============== ===========

The following thread pools can be monitored.

============================ ============== ===========
Name                         Type           Description
============================ ============== ===========
AntiEntropyStage             internal       Builds merkle tree for repairs
CacheCleanupExecutor         internal       Cache maintenance performed on this thread pool
CompactionExecutor           internal       Compactions are run on these threads
CounterMutationStage         request        Responsible for counter writes
GossipStage                  internal       Handles gossip requests
HintsDispatcher              internal       Performs hinted handoff
InternalResponseStage        internal       Responsible for intra-cluster callbacks
MemtableFlushWriter          internal       Writes memtables to disk
MemtablePostFlush            internal       Cleans up commit log after memtable is written to disk
MemtableReclaimMemory        internal       Memtable recycling
MigrationStage               internal       Runs schema migrations
MiscStage                    internal       Misceleneous tasks run here
MutationStage                request        Responsible for all other writes
Native-Transport-Requests    transport      Handles client CQL requests
PendingRangeCalculator       internal       Calculates token range
PerDiskMemtableFlushWriter_0 internal       Responsible for writing a spec (there is one of these per disk 0-N)
ReadRepairStage              request        ReadRepair happens on this thread pool
ReadStage                    request        Local reads run on this thread pool
RequestResponseStage         request        Coordinator requests to the cluster run on this thread pool
Sampler                      internal       Responsible for re-sampling the index summaries of SStables
SecondaryIndexManagement     internal       Performs updates to secondary indexes
ValidationExecutor           internal       Performs validation compaction or scrubbing
ViewBuildExecutor            internal       Performs materialized views initial build
ViewMutationStage            request        Responsible for materialized view writes
============================ ============== ===========

.. |nbsp| unicode:: 0xA0 .. nonbreaking space

Client Request Metrics
^^^^^^^^^^^^^^^^^^^^^^

Client requests have their own set of metrics that encapsulate the work happening at coordinator level.

Different types of client requests are broken down by ``RequestType``.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.ClientRequest.<MetricName>.<RequestType>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=ClientRequest,scope=<RequestType>,name=<MetricName>``


:RequestType: CASRead
:Description: Metrics related to transactional read requests.
:Metrics:
    ===================== ============== =============================================================
    Name                  Type           Description
    ===================== ============== =============================================================
    ConditionNotMet       Counter        Number of transaction preconditions did not match current values.
    ContentionHistogram   Histogram      How many contended reads were encountered
    Failures              Counter        Number of transaction failures encountered.
    Timeouts              Counter        Number of timeouts encountered.
    Unavailables          Counter        Number of unavailable exceptions encountered.
    UnfinishedCommit      Counter        Number of transactions that were committed on read.
    |nbsp|                Latency        Transaction read latency.
    ===================== ============== =============================================================

:RequestType: CASWrite
:Description: Metrics related to transactional write requests.
:Metrics:
    ===================== ============== =============================================================
    Name                  Type           Description
    ===================== ============== =============================================================
    ConditionNotMet       Counter        Number of transaction preconditions did not match current values.
    ContentionHistogram   Histogram      How many contended writes were encountered
    Failures              Counter        Number of transaction failures encountered.
    MutationSizeHistogram Histogram      Total size in bytes of the requests mutations.
    Timeouts              Counter        Number of timeouts encountered.
    UnfinishedCommit      Counter        Number of transactions that were committed on write.
    |nbsp|                Latency        Transaction write latency.
    ===================== ============== =============================================================


:RequestType: Read
:Description: Metrics related to standard read requests.
:Metrics:
    ===================== ============== =============================================================
    Name                  Type           Description
    ===================== ============== =============================================================
    Failures              Counter        Number of read failures encountered.
    Timeouts              Counter        Number of timeouts encountered.
    Unavailables          Counter        Number of unavailable exceptions encountered.
    |nbsp|                Latency        Read latency.
    ===================== ============== =============================================================

:RequestType: RangeSlice
:Description: Metrics related to token range read requests.
:Metrics:
    ===================== ============== =============================================================
    Name                  Type           Description
    ===================== ============== =============================================================
    Failures              Counter        Number of range query failures encountered.
    Timeouts              Counter        Number of timeouts encountered.
    Unavailables          Counter        Number of unavailable exceptions encountered.
    |nbsp|                Latency        Range query latency.
    ===================== ============== =============================================================

:RequestType: Write
:Description: Metrics related to regular write requests.
:Metrics:
    ===================== ============== =============================================================
    Name                  Type           Description
    ===================== ============== =============================================================
    Failures              Counter        Number of write failures encountered.
    MutationSizeHistogram Histogram      Total size in bytes of the requests mutations.
    Timeouts              Counter        Number of timeouts encountered.
    Unavailables          Counter        Number of unavailable exceptions encountered.
    |nbsp|                Latency        Write latency.
    ===================== ============== =============================================================


:RequestType: ViewWrite
:Description: Metrics related to materialized view write wrtes.
:Metrics:
    ===================== ============== =============================================================
    Failures              Counter        Number of transaction failures encountered.
    Timeouts              Counter        Number of timeouts encountered.
    Unavailables          Counter        Number of unavailable exceptions encountered.
    ViewPendingMutations  Gauge<Long>    ViewReplicasAttempted - ViewReplicasSuccess.
    ViewReplicasAttempted Counter        Total number of attempted view replica writes.
    ViewReplicasSuccess   Counter        Total number of succeded view replica writes.
    ViewWriteLatency      Timer          Time between when mutation is applied to base table and when CL.ONE is achieved on view.
    ===================== ============== =============================================================

Cache Metrics
^^^^^^^^^^^^^

Cassandra caches have metrics to track the effectivness of the caches. Though the ``Table Metrics`` might be more useful.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Cache.<MetricName>.<CacheName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Cache,scope=<CacheName>,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
Capacity                   Gauge<Long>    Cache capacity in bytes.
Entries                    Gauge<Integer> Total number of cache entries.
FifteenMinuteCacheHitRate  Gauge<Double>  15m cache hit rate.
FiveMinuteCacheHitRate     Gauge<Double>  5m cache hit rate.
HitRate                    Gauge<Double>  All time cache hit rate.
Hits                       Meter          Total number of cache hits.
MissLatency                Timer          Latency of misses.
Misses                     Meter          Total number of cache misses.
OneMinuteCacheHitRate      Gauge<Double>  1m cache hit rate.
Requests                   Gauge<Long>    Total number of cache requests.
Size                       Gauge<Long>    Total size of occupied cache, in bytes.
========================== ============== ===========

The following caches are covered:

============================ ===========
Name                         Description
============================ ===========
ChunkCache                   In process uncompressed page cache.
CounterCache                 Keeps hot counters in memory for performance.
KeyCache                     Cache for partition to sstable offsets.
RowCache                     Cache for rows kept in memory.
============================ ===========

.. NOTE::
    Misses and MissLatency are only defined for the ChunkCache

CQL Metrics
^^^^^^^^^^^

Metrics specific to CQL prepared statement caching.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.CQL.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=CQL,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
PreparedStatementsCount    Gauge<Integer> Number of cached prepared statements.
PreparedStatementsEvicted  Counter        Number of prepared statements evicted from the prepared statement cache
PreparedStatementsExecuted Counter        Number of prepared statements executed.
PreparedStatementsRatio    Gauge<Double>  Percentage of statements that are prepared vs unprepared.
RegularStatementsExecuted  Counter        Number of **non** prepared statements executed.
========================== ============== ===========

.. _dropped-metrics:

DroppedMessage Metrics
^^^^^^^^^^^^^^^^^^^^^^

Metrics specific to tracking dropped messages for different types of requests.
Dropped writes are stored and retried by ``Hinted Handoff``

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.DroppedMessage.<MetricName>.<Type>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=DroppedMessage,scope=<Type>,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
CrossNodeDroppedLatency    Timer          The dropped latency across nodes.
Dropped                    Meter          Number of dropped messages.
InternalDroppedLatency     Timer          The dropped latency within node.
========================== ============== ===========

The different types of messages tracked are:

============================ ===========
Name                         Description
============================ ===========
BATCH_REMOVE                 Batchlog cleanup (after succesfully applied)
BATCH_STORE                  Batchlog write
COUNTER_MUTATION             Counter writes
HINT                         Hint replay
MUTATION                     Regular writes
PAGED_SLICE                  Paged read
RANGE_SLICE                  Token range read
READ                         Regular reads
READ_REPAIR                  Read repair
REQUEST_RESPONSE             RPC Callbacks
_TRACE                       Tracing writes
============================ ===========

Streaming Metrics
^^^^^^^^^^^^^^^^^

Metrics reported during ``Streaming`` operations, such as repair, bootstrap, rebuild.

These metrics are specific to a peer endpoint, with the source node being the node you are pulling the metrics from.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Streaming.<MetricName>.<PeerIP>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Streaming,scope=<PeerIP>,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
IncomingBytes              Counter        Number of bytes streamed to this node from the peer.
IncomingProcessTime        Timer          The time spent on processing the incoming stream message from the peer.
OutgoingBytes              Counter        Number of bytes streamed to the peer endpoint from this node.
========================== ============== ===========


Compaction Metrics
^^^^^^^^^^^^^^^^^^

Metrics specific to ``Compaction`` work.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Compaction.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Compaction,name=<MetricName>``

========================== ======================================== ===============================================
Name                       Type                                     Description
========================== ======================================== ===============================================
BytesCompacted             Counter                                  Total number of bytes compacted since server [re]start.
CompletedTasks             Gauge<Long>                              Number of completed compactions since server [re]start.
PendingTasks               Gauge<Integer>                           Estimated number of compactions remaining to perform.
PendingTasksByTableName    Gauge<Map<String, Map<String, Integer>>> Estimated number of compactions remaining to perform, grouped by keyspace and then table name. This info is also kept in ``Table Metrics``.
TotalCompactionsCompleted  Meter                                    Throughput of completed compactions since server [re]start.
========================== ======================================== ===============================================

CommitLog Metrics
^^^^^^^^^^^^^^^^^

Metrics specific to the ``CommitLog``

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.CommitLog.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=CommitLog,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
CompletedTasks             Gauge<Long>    Total number of commit log messages written since [re]start.
OverSizedMutations         Meter          Throughput for mutations that exceed limit.
PendingTasks               Gauge<Long>    Number of commit log messages written but yet to be fsync'd.
TotalCommitLogSize         Gauge<Long>    Current size, in bytes, used by all the commit log segments.
WaitingOnCommit            Timer          Time spent waiting on CL fsync; for Periodic this is only occurs when the sync is lagging its sync interval.
WaitingOnSegmentAllocation Timer          Time spent waiting for a CommitLogSegment to be allocated - under normal conditions this should be zero.
========================== ============== ===========

Storage Metrics
^^^^^^^^^^^^^^^

Metrics specific to the storage engine.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Storage.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Storage,name=<MetricName>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
Exceptions                 Counter        Number of internal exceptions caught. Under normal exceptions this should be zero.
Load                       Counter        Size, in bytes, of the on disk data size this node manages.
TotalHints                 Counter        Number of hint messages written to this node since [re]start. Includes one entry for each host to be hinted per hint.
TotalHintsInProgress       Counter        Number of hints attemping to be sent currently.
========================== ============== ===========

.. _hintsservice-metrics:

HintsService Metrics
^^^^^^^^^^^^^^^^^^^^

Metrics specific to the Hints delivery service.  There are also some metrics related to hints tracked in ``Storage Metrics``

These metrics include the peer endpoint **in the metric name**

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.HintsService.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=HintsService,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
Hint_delays                 Histogram      Histogram of hint delivery delays (in milliseconds)
Hint_delays-<PeerIP>        Histogram      Histogram of hint delivery delays (in milliseconds) per peer
HintsFailed                 Meter          A meter of the hints that failed deliver
HintsSucceeded              Meter          A meter of the hints successfully delivered
HintsTimedOut               Meter          A meter of the hints that timed out
=========================== ============== ===========

SSTable Index Metrics
^^^^^^^^^^^^^^^^^^^^^

Metrics specific to the SSTable index metadata.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Index.<MetricName>.RowIndexEntry``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Index scope=RowIndexEntry,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
IndexInfoCount              Histogram      Histogram of the number of on-heap index entries managed across all SSTables.
IndexInfoGets               Histogram      Histogram of the number index seeks performed per SSTable.
IndexedEntrySize            Histogram      Histogram of the on-heap size, in bytes, of the index across all SSTables.
=========================== ============== ===========

BufferPool Metrics
^^^^^^^^^^^^^^^^^^

Metrics specific to the internal recycled buffer pool Cassandra manages.  This pool is meant to keep allocations and GC
lower by recycling on and off heap buffers.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.BufferPool.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=BufferPool,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
Misses                      Meter          The rate of misses in the pool. The higher this is the more allocations incurred.
Size                        Gauge<Long>    Size, in bytes, of the managed buffer pool
=========================== ============== ===========

Client Metrics
^^^^^^^^^^^^^^

Metrics specifc to client managment.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Client.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Client,name=<MetricName>``

============================== ================================ ===========
Name                           Type                             Description
============================== ================================ ===========
ClientsByProtocolVersion       Gauge<List<Map<String, String>>> List of up to last 100 connections including protocol version. Can be reset with clearConnectionHistory operation in org.apache.cassandra.db:StorageService mbean.
ConnectedNativeClients         Gauge<Integer>                   Number of clients connected to this nodes native protocol server
ConnectedNativeClientsByUser   Gauge<Map<String, Int>           Number of connnective native clients by username
Connections                    Gauge<List<Map<String, String>>  List of all connections and their state information
RequestsSize                   Gauge<Long>                      How many concurrent bytes used in currently processing requests
RequestsSizeByIpDistribution   Histogram                        How many concurrent bytes used in currently processing requests by different ips
============================== ================================ ===========

Batch Metrics
^^^^^^^^^^^^^

Metrics specific to batch statements.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Batch.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Batch,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
PartitionsPerCounterBatch   Histogram      Distribution of the number of partitions processed per counter batch
PartitionsPerLoggedBatch    Histogram      Distribution of the number of partitions processed per logged batch
PartitionsPerUnloggedBatch  Histogram      Distribution of the number of partitions processed per unlogged batch
=========================== ============== ===========

Read Repair Metrics
^^^^^^^^^^^^^^^^^^^

Metrics specific to read repair operations.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.ReadRepair.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=ReadRepair,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
ReconcileRead               Meter          The rate of read-only read repairs, which do not mutate the replicas
RepairedBlocking            Meter          The rate of blocking read repairs
SpeculatedRead              Meter          The rate of speculated reads during read repairs
SpeculatedWrite             Meter          The rate of speculated writes during read repairs
=========================== ============== ===========

Messaging Metrics
^^^^^^^^^^^^^^^^^

Metrics for internode messaging.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Messaging.<MetricName>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Messaging,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
<DC>-Latency                Timer          Latency of all internode messageing between this node and the datacenters.
<VERB>-WaitLatency          Timer          Latency between the message creation time and the time being executed by VERB
CrossNodeLatency            Timer          Latency of all internode messaging between this node and the peers
=========================== ============== ===========

Internode Inbound Connection Metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Metrics specific to inbound connections of internode messaging.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.InboundConnection.<MetricName>.<IP>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=InboundConnection,scope=<IP>,name=<MetricName>``

=========================== ============== ===========
Name                        Type           Description
=========================== ============== ===========
CorruptFramesRecovered      Guage<Long>    Estimated number of corrupted frames recovered
CorruptFramesUnrecovered    Guage<Long>    Estimated number of corrupted frames unrecovered
ErrorBytes                  Guage<Long>    Estimated number of error bytes
ErrorCount                  Guage<Long>    Estimated number of error count
ExpiredBytes                Guage<Long>    Estimated number of expired bytes
ExpiredCount                Guage<Long>    Estimated number of expired count
ScheduledBytes              Guage<Long>    Estimated number of bytes that are pending execution
ScheduledCount              Guage<Long>    Estimated number of message that are pending execution
ProcessedBytes              Guage<Long>    Estimated number of bytes processed
ProcessedCount              Guage<Long>    Estimated number of messages processed
ReceivedBytes               Guage<Long>    Estimated number of bytes received
ReceivedCount               Guage<Long>    Estimated number of messages received
ThrottledCount              Guage<Long>    Estimated number of messages throttled
ThrottledNanos              Guage<Long>    Estimated duration of throttling in nanoseconds
=========================== ============== ===========

Internode Outbound Connection Metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Metrics specific to outbound connections of internode messaging.

Reported name format:

**Metric Name**
    ``org.apache.cassandra.metrics.Connection.<MetricName>.<IP>``

**JMX MBean**
    ``org.apache.cassandra.metrics:type=Connection,scope=<IP>,name=<MetricName>``

======================================================= ============== ===========
Name                                                    Type           Description
======================================================= ============== ===========
[Large|Small|Urgent]MessagePendingTasks                 Guage<Long>    Estimated number of pending (Large|Small|Urgent) messages queued
[Large|Small|Urgent]MessagePendingBytes                 Guage<Long>    Estimated number of bytes of pending (Large|Small|Urgent) messages queued
[Large|Small|Urgent]MessageCompletedTasks               Guage<Long>    Estimated number of (Large|Small|Urgent) messages sent
[Large|Small|Urgent]MessageCompletedBytes               Guage<Long>    Estimated number of bytes of (Large|Small|Urgent) messages sent
[Large|Small|Urgent]MessageDroppedTasks                 Guage<Long>    Estimated number of dropped (Large|Small|Urgent) messages
[Large|Small|Urgent]MessageDroppedTasksDueToOverload    Guage<Long>    Estimated number of dropped (Large|Small|Urgent) messages due to overload
[Large|Small|Urgent]MessageDroppedBytesDueToOverload    Guage<Long>    Estimated number of bytes of dropped (Large|Small|Urgent) messages due to overload
[Large|Small|Urgent]MessageDroppedTasksDueToTimeout     Guage<Long>    Estimated number of dropped (Large|Small|Urgent) messages due to timeout
[Large|Small|Urgent]MessageDroppedBytesDueToTimeout     Guage<Long>    Estimated number of bytes of dropped (Large|Small|Urgent) messages due to timeout
[Large|Small|Urgent]MessageDroppedTasksDueToError       Guage<Long>    Estimated number of dropped (Large|Small|Urgent) messages due to error
[Large|Small|Urgent]MessageDroppedBytesDueToError       Guage<Long>    Estimated number of bytes of dropped (Large|Small|Urgent) messages due to error
======================================================= ============== ===========

JVM Metrics
^^^^^^^^^^^

JVM metrics such as memory and garbage collection statistics can either be accessed by connecting to the JVM using JMX or can be exported using `Metric Reporters`_.

BufferPool
++++++++++

**Metric Name**
    ``jvm.buffers.<direct|mapped>.<MetricName>``

**JMX MBean**
    ``java.nio:type=BufferPool,name=<direct|mapped>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
Capacity                   Gauge<Long>    Estimated total capacity of the buffers in this pool
Count                      Gauge<Long>    Estimated number of buffers in the pool
Used                       Gauge<Long>    Estimated memory that the Java virtual machine is using for this buffer pool
========================== ============== ===========

FileDescriptorRatio
+++++++++++++++++++

**Metric Name**
    ``jvm.fd.<MetricName>``

**JMX MBean**
    ``java.lang:type=OperatingSystem,name=<OpenFileDescriptorCount|MaxFileDescriptorCount>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
Usage                      Ratio          Ratio of used to total file descriptors
========================== ============== ===========

GarbageCollector
++++++++++++++++

**Metric Name**
    ``jvm.gc.<gc_type>.<MetricName>``

**JMX MBean**
    ``java.lang:type=GarbageCollector,name=<gc_type>``

========================== ============== ===========
Name                       Type           Description
========================== ============== ===========
Count                      Gauge<Long>    Total number of collections that have occurred
Time                       Gauge<Long>    Approximate accumulated collection elapsed time in milliseconds
========================== ============== ===========

Memory
++++++

**Metric Name**
    ``jvm.memory.<heap/non-heap/total>.<MetricName>``

**JMX MBean**
    ``java.lang:type=Memory``

========================== ============== ===========
Committed                  Gauge<Long>    Amount of memory in bytes that is committed for the JVM to use
Init                       Gauge<Long>    Amount of memory in bytes that the JVM initially requests from the OS
Max                        Gauge<Long>    Maximum amount of memory in bytes that can be used for memory management
Usage                      Ratio          Ratio of used to maximum memory
Used                       Gauge<Long>    Amount of used memory in bytes
========================== ============== ===========

MemoryPool
++++++++++

**Metric Name**
    ``jvm.memory.pools.<memory_pool>.<MetricName>``

**JMX MBean**
    ``java.lang:type=MemoryPool,name=<memory_pool>``

========================== ============== ===========
Committed                  Gauge<Long>    Amount of memory in bytes that is committed for the JVM to use
Init                       Gauge<Long>    Amount of memory in bytes that the JVM initially requests from the OS
Max                        Gauge<Long>    Maximum amount of memory in bytes that can be used for memory management
Usage                      Ratio          Ratio of used to maximum memory
Used                       Gauge<Long>    Amount of used memory in bytes
========================== ============== ===========

JMX
^^^

Any JMX based client can access metrics from cassandra.

If you wish to access JMX metrics over http it's possible to download `Mx4jTool <http://mx4j.sourceforge.net/>`__ and
place ``mx4j-tools.jar`` into the classpath.  On startup you will see in the log::

    HttpAdaptor version 3.0.2 started on port 8081

To choose a different port (8081 is the default) or a different listen address (0.0.0.0 is not the default) edit
``conf/cassandra-env.sh`` and uncomment::

    #MX4J_ADDRESS="-Dmx4jaddress=0.0.0.0"

    #MX4J_PORT="-Dmx4jport=8081"


Metric Reporters
^^^^^^^^^^^^^^^^

As mentioned at the top of this section on monitoring the Cassandra metrics can be exported to a number of monitoring
system a number of `built in <http://metrics.dropwizard.io/3.1.0/getting-started/#other-reporting>`__ and `third party
<http://metrics.dropwizard.io/3.1.0/manual/third-party/>`__ reporter plugins.

The configuration of these plugins is managed by the `metrics reporter config project
<https://github.com/addthis/metrics-reporter-config>`__. There is a sample configuration file located at
``conf/metrics-reporter-config-sample.yaml``.

Once configured, you simply start cassandra with the flag
``-Dcassandra.metricsReporterConfigFile=metrics-reporter-config.yaml``. The specified .yaml file plus any 3rd party
reporter jars must all be in Cassandra's classpath.