RabbitMQ monitoring

SkyWalking leverages rabbitmq_prometheus plugin for collecting metrics data from RabbitMQ. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

  1. The rabbitmq_prometheus plugin collect metrics data from RabbitMQ. Note: The RabbitMQ version is required to be 3.8.0+. The rabbitmq_prometheus plugin is built-in since RabbitMQ v3.8.0.
  2. OpenTelemetry Collector fetches metrics from rabbitmq_prometheus plugin via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Setup

  1. Setup rabbitmq_prometheus.
  2. Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
  3. Config SkyWalking OpenTelemetry receiver.

RabbitMQ Monitoring

RabbitMQ monitoring provides multidimensional metrics monitoring of RabbitMQ cluster as Layer: RABBITMQ Service in the OAP. In each cluster, the nodes are represented as Instance.

RabbitMQ Cluster Supported Metrics

Monitoring PanelMetric NameDescriptionData Source
Memory Available Before Publishers Blocked (MB)meter_rabbitmq_memory_available_before_publisher_blockedIf the value is zero or less, the memory alarm will be triggered and all publishing connections across all cluster nodes will be blocked.rabbitmq_prometheus plugin
Disk Space Available Before Publishers Blocked (GB)meter_rabbitmq_disk_space_available_before_publisher_blockedThis metric is reported for the partition where the RabbitMQ data directory is stored.rabbitmq_prometheus plugin
File Descriptors Availablemeter_rabbitmq_file_descriptors_availableWhen this value reaches zero, new connections will not be accepted and disk write operations may fail.rabbitmq_prometheus plugin
TCP Sockets Availablemeter_rabbitmq_tcp_socket_availableWhen this value reaches zero, new connections will not be accepted.rabbitmq_prometheus plugin
Messages Ready To Be Delivered To Consumersmeter_rabbitmq_message_ready_delivered_consumersTotal number of ready messages ready to be delivered to consumers.rabbitmq_prometheus plugin
Messages Pending Consumer Acknowledgementmeter_rabbitmq_message_unacknowledged_delivered_consumersThe total number of messages that are either in-flight to consumers, currently being processed by consumers or simply waiting for the consumer acknowledgements to be processed by the queue. Until the queue processes the message acknowledgement, the message will remain unacknowledged.rabbitmq_prometheus plugin
Messages Publishedmeter_rabbitmq_messages_publishedThe incoming message rate before any routing rules are applied.rabbitmq_prometheus plugin
Messages Confirmed To Publishersmeter_rabbitmq_messages_confirmedThe rate of messages confirmed by the broker to publishers. Publishers must opt-in to receive message confirmations.rabbitmq_prometheus plugin
Messages Unconfirmed To Publishersmeter_rabbitmq_messages_unconfirmedThe rate of messages received from publishers that have publisher confirms enabled and the broker has not confirmed yet.rabbitmq_prometheus plugin
Messages Routed To Queuesmeter_rabbitmq_messages_routedThe rate of messages received from publishers and successfully routed to the master queue replicas.rabbitmq_prometheus plugin
Unroutable Messages Returned To Publishersmeter_rabbitmq_messages_unroutable_returnedThe rate of messages that cannot be routed and are returned back to publishers.rabbitmq_prometheus plugin
Unroutable Messages Droppedmeter_rabbitmq_messages_unroutable_droppedThe rate of messages that cannot be routed and are dropped.rabbitmq_prometheus plugin
Queues Totalmeter_rabbitmq_queuesTotal number of queue masters per node.rabbitmq_prometheus plugin
Queues Declaredmeter_rabbitmq_queues_declared_totalThe rate of queue declarations performed by clients.rabbitmq_prometheus plugin
Queues Createdmeter_rabbitmq_queues_created_totalThe rate of new queues created (as opposed to redeclarations).rabbitmq_prometheus plugin
Queues Deletedmeter_rabbitmq_queues_deleted_totalThe rate of queues deleted.rabbitmq_prometheus plugin
Channels Totalmeter_rabbitmq_channelsTotal number of channels on all currently opened connections.rabbitmq_prometheus plugin
Channels Openedmeter_rabbitmq_channels_opened_totalThe rate of new channels opened by applications across all connections. Channels are expected to be long-lived.rabbitmq_prometheus plugin
Channels Closedmeter_rabbitmq_channels_closed_totalThe rate of channels closed by applications across all connections. Channels are expected to be long-lived.rabbitmq_prometheus plugin
Connections Totalmeter_rabbitmq_connectionsTotal number of client connections.rabbitmq_prometheus plugin
Connections Openedmeter_rabbitmq_connections_opened_totalThe rate of new connections opened by clients. Connections are expected to be long-lived.rabbitmq_prometheus plugin
Connections Closedmeter_rabbitmq_connections_closed_totalThe rate of connections closed. Connections are expected to be long-lived.rabbitmq_prometheus plugin

RabbitMQ Node Supported Metrics

Monitoring PanelUnitMetric NameDescriptionData Source
Ready Messagesmeter_rabbitmq_node_queue_messages_readyTotal number of ready messages ready to be delivered to consumers.rabbitmq_prometheus plugin
Unacknowledged Messagesmeter_rabbitmq_node_unacknowledged_messagesMessages delivered to consumers but not yet acknowledgedrabbitmq_prometheus plugin
Incoming Messagesmeter_rabbitmq_node_incoming_messagesThe incoming message rate before any routing rules are applied.rabbitmq_prometheus plugin
Outgoing Messagesmeter_rabbitmq_node_outgoing_messages_totalThe outgoing message rate before any routing rules are applied.rabbitmq_prometheus plugin
Publishersmeter_rabbitmq_node_publisher_totalPublishersrabbitmq_prometheus plugin
Consumersmeter_rabbitmq_node_consumer_totalConsumers currently connectrabbitmq_prometheus plugin
Collectionsmeter_rabbitmq_node_connections_totalConnections currently openrabbitmq_prometheus plugin
Channelsmeter_rabbitmq_node_channel_totalChannels currently openrabbitmq_prometheus plugin
Queuesmeter_rabbitmq_node_queue_totalQueues availablerabbitmq_prometheus plugin
Allocated Used%meter_rabbitmq_node_allocated_used_percentErlang VM memory utilisation from erts_alloc perspectiverabbitmq_prometheus plugin
Allocated Unused%meter_rabbitmq_node_allocated_unused_percentErlang VM memory utilisation from erts_alloc perspectiverabbitmq_prometheus plugin
Allocated UsedMBmeter_rabbitmq_node_allocated_used_bytesErlang VM memory utilisation from erts_alloc perspectiverabbitmq_prometheus plugin
Allocated UnusedMBmeter_rabbitmq_node_allocated_unused_bytesErlang VM memory utilisation from erts_alloc perspectiverabbitmq_prometheus plugin
Allocated TotalMBmeter_rabbitmq_node_allocated_total_bytesErlang VM memory utilisation from erts_alloc perspectiverabbitmq_prometheus plugin
Resident Set SizeMBmeter_rabbitmq_node_process_resident_memory_bytesErlang VM Resident Set Size (RSS) As reported by the OSrabbitmq_prometheus plugin
AllocatorsMBmeter_rabbitmq_node_allocated_unused_bytes
meter_rabbitmq_node_allocated_total_bytes
meter_rabbitmq_node_process_resident_memory_bytes
rabbitmq_prometheus plugin
Allocated By TypeMBmeter_rabbitmq_node_allocated_by_typeAllocated by allocator typerabbitmq_prometheus plugin
Multiblock UsedMBmeter_rabbitmq_node_allocated_multiblock_usedMulti block usedrabbitmq_prometheus plugin
Multiblock UnusedMBmeter_rabbitmq_node_allocated_multiblock_unusedMulti block usedrabbitmq_prometheus plugin
Multiblock Pool UsedMBmeter_rabbitmq_node_allocated_multiblock_pool_usedMulti block pool usedrabbitmq_prometheus plugin
Multiblock Pool UnusedMBmeter_rabbitmq_node_allocated_multiblock_pool_unusedMulti block pool unusedrabbitmq_prometheus plugin
Singleblock UsedMBmeter_rabbitmq_node_allocated_singleblock_usedSingle block usedrabbitmq_prometheus plugin
Singleblock UnusedMBmeter_rabbitmq_node_allocated_singleblock_unusedSingle block unusedrabbitmq_prometheus plugin

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/rabbitmq/rabbitmq-cluster.yaml, /config/otel-rules/rabbitmq/rabbitmq-node.yaml. The RabbitMQ dashboard panel configurations are found in /config/ui-initialized-templates/rabbitmq.