The Accumulo Monitor provides a web UI with information on the health and status of Accumulo.
The monitor can be viewed at:
http://<MONITOR_HOST>:9995/ - if Accumulo is running on a clusterThe Overview page (shown below) contains some summary information about the Accumulo instance and graphs displaying various metrics over time. These include ingest and scan performance and other useful measurements.
Note: If the Zookeeper information is not available on the Overview, your version of zookeeper may require setting the four-letter-word command whitelist to allow the stat command. See [Configuration - Zookeeper Considerations] for more information.
The Master Server, Tablet Servers, and Tables pages display metrics grouped in different ways (e.g. by tablet server or by table). Metrics typically include number of entries (key/value pairs), ingest and query rates. The number of running scans, major and minor compactions are in the form number_running (number_queued). Another important metric is hold time, which is the amount of time a tablet has been waiting but unable to flush its memory in a minor compaction.
The Server Activity page graphically displays tablet server status, with each server represented as a circle or square. Different metrics may be assigned to the nodes' color and speed of oscillation. The Overall Avg metric is only used on the Server Activity page, and represents the average of all the other metrics (after normalization). Similarly, the Overall Max metric picks the metric with the maximum normalized value.
The Garbage Collector page displays a list of garbage collection cycles, the number of files found of each type (including deletion candidates in use and files actually deleted), and the length of the deletion cycle. The Traces page displays data for recent traces performed (see the following section for information on [tracing][tracing]). The Recent Logs page displays warning and error logs forwarded to the monitor from all Accumulo processes. Also, the XML and JSON links provide metrics in XML and JSON formats, respectively.
The Accumulo monitor does a best-effort to not display any sensitive information to users; however, the monitor is intended to be a tool used with care. It is not a production-grade webservice. It is a good idea to whitelist access to the monitor via an authentication proxy or firewall. It is strongly recommended that the Monitor is not exposed to any publicly-accessible networks.
Things highlighted in yellow may be in need of attention. If anything is highlighted in red on the monitor page, it is something that definitely needs attention.
SSL may be enabled for the monitor page by setting the following properties in the accumulo.properties file:
If the Accumulo conf directory has been configured (in particular the accumulo-env.sh file must be set up), the accumulo-util gen-monitor-cert command can be used to create the keystore and truststore files with random passwords. The command will print out the properties that need to be added to the accumulo.properties file. The stores can also be generated manually with the Java keytool command, whose usage can be seen in the accumulo-util script.
If desired, the SSL ciphers allowed for connections can be controlled via the following properties in accumulo.properties:
If SSL is enabled, the monitor URL can only be accessed via https. This also allows you to access the Accumulo shell through the monitor page. The left navigation bar will have a new link to Shell. An Accumulo user name and password must be entered for access to the shell.
Accumulo can expose metrics through a legacy metrics library and using the Hadoop Metrics2 library.
Accumulo has a legacy metrics library that can be exposes metrics using JMX endpoints or file-based logging. These metrics can be enabled by setting {% plink general.legacy.metrics %} to true in accumulo.properties and placing the accumulo-metrics.xml configuration file on the classpath (which is typically done by placing the file in the conf/ directory). A template for accumulo-metrics.xml can be found in conf/templates of the Accumulo tarball.
Hadoop Metrics2 is a library which allows for routing of metrics generated by registered MetricsSources to configured MetricsSinks. Examples of sinks that are implemented by Hadoop include file-based logging, Graphite and Ganglia. All metric sources are exposed via JMX when using Metrics2.
Metrics2 is configured by examining the classpath for a file that matches hadoop-metrics2*.properties. The Accumulo tarball contains an example hadoop-metrics2-accumulo.properties file in conf/templates which can be copied to conf/ to place on classpath. This file is used to enable file, Graphite or Ganglia sinks (some minimal configuration required for Graphite and Ganglia). Because the Hadoop configuration is also on the Accumulo classpath, be sure that you do not have multiple Metrics2 configuration files. It is recommended to consolidate metrics in a single properties file in a central location to remove ambiguity. The contents of hadoop-metrics2-accumulo.properties can be added to a central hadoop-metrics2.properties in $HADOOP_CONF_DIR.
As a note for configuring the file sink, the provided path should be absolute. A relative path or file name will be created relative to the directory in which the Accumulo process was started. External tools, such as logrotate, can be used to prevent these files from growing without bound.
Each server process should have log messages from the Metrics2 library about the sinks that were created. Be sure to check the Accumulo processes log files when debugging missing metrics output.
Below are additional resources for configuring Metrics2:
[grafana-post]: {{ site.baseurl }}/blog/2018/03/22/view-metrics-in-grafana.html [metrics2-javadoc]: https://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html [tracing]: {% durl troubleshooting/tracing %} [Configuration - Zookeeper Considerations]: {% durl configuration/overview#monitor-and-zookeeper-four-letter-word-command-whitelist %}