blob: 1ef651f390ea63366259ce704dd105521c5419ff [file]
---
layout: post
status: PUBLISHED
published: true
title: Introduction "hbtop", a real-time monitoring tool for HBase modeled after Unix's
'top' command
id: c8bca77b-813e-4bec-bfeb-fbf625ef804e
date: '2019-09-09 17:50:24 -0400'
categories: hbase
tags:
- monitoring
- hbase
- top
permalink: hbase/entry/introduction-hbtop-a-real-time
---
<p><font size="2">by Toshihiro Suzuki, HBase Committer</font></p>
<p> <font size="2"><code style="box-sizing: border-box; font-family: "SFMono- Regular", Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">hbtop</code> is a real-time monitoring tool for HBase modeled after Unix's _top_ command. It can display summary information as well as metrics per Region/Namespace/Table/RegionServer. With this tool, you can see metrics sorted by a selected field and filter the metrics to see only metrics you are interested in. Also, with the drill-down feature, you dig in on hot-spotting Regions.</font> </p>
<p> <font size="2">hbtop (<a href="https://issues.apache.org/jira/browse/HBASE-11062" title="HBASE-11062">https://issues.apache.org/jira/browse/HBASE-11062</a>) is available in the coming hbase 2.1.7, 2.2.2, and 2.3.0 releases and is being actively backported to branch-1.</font> </p>
<p> <font size="2">Run hbtop with the following command:</font> </p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: #f6f8fa; border-radius: 3px; color: #24292e;"><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0px; margin: 0px; background: initial; border-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;"><font size="2">$ hbase hbtop
</font></code></pre>
<p> <font size="2">In this case, the values of <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">hbase.client.zookeeper.quorum</code> and <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">zookeeper.znode.parent</code> in <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">hbase-site.xml</code> in the classpath or their default values are used to connect. Or, specify your own zookeeper quorum and znode parent as follows:</font> </p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: #f6f8fa; border-radius: 3px; color: #24292e;"><font size="2"><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0px; margin: 0px; background: initial; border-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;">$ hbase hbtop -Dhbase.client.zookeeper.quorum=<zookeeper quorum> -Dzookeeper.znode.parent=<znode parent></code></font>
</pre>
<p> <font size="2"> </p>
<p>
Run hbtop and you'll see something like the following:
</p>
<p> </font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/top_screen.gif" alt="Top screen" title="Top screen" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2">The top screen consists of a summary and metrics section. In the summary section, you can see <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">HBase Version</code>, <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">Cluster ID</code>, <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">The number of region servers</code>, <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">Region count</code>, <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">Average Cluster Load</code> and <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">Aggregated Request/s</code>. In the metrics portion, you can see metrics per Region/Namespace/Table/RegionServer depending on the selected mode. The top screen is refreshed on a period &ndash; 3 seconds by default.<br />
</font> </p>
<p> <font size="2">You can scroll the metric records in the metrics section.</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/scrolling_metric_records.gif" alt="Scrolling metric records" title="Scrolling metric records" style="box-sizing: initial; max-width: 100%;" /> </p>
<h4 style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";"><span style="font-weight: bold;"><font size="3">Command line arguments</font></span></h4>
<table style="box-sizing: border-box; border-spacing: 0px; border-collapse: collapse; margin-top: 0px; margin-bottom: 16px; display: block; width: 882px; overflow: auto; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", " Segoe UI Symbol";">
<thead style="box-sizing: border-box;">
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<th style="font-weight: 600; box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Argument</font></th>
<th style="font-weight: 600; box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Description</font></th>
</tr>
</thead>
<tbody style="box-sizing: border-box;">
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">-d,--delay <arg></font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">The refresh delay (in seconds); default is 3 seconds</font></td>
</tr>
<tr style="box-sizing: border-box; background-color: #f6f8fa; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">-h,--help</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Print usage; for help while the tool is running press <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">h</code> key</font></td>
</tr>
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">-m,--mode <arg></font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;">
<p><font size="2">The mode <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">n</code> (Namespace)|<code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">t</code> (Table)|<code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">r</code> (Region)|<code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">s</code> (RegionServer), default is <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">r</code> (Region) Mode</font></p>
</td>
</tr>
</tbody>
</table>
<p> <font size="2"><br />
You can change mode by pressing the <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">m</code> key in the top screen.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/changing_mode.gif" alt="Changing mode" title="Changing mode" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2"><br />
Change the refresh rate by pressing <code style="color: #24292e; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">d</code> key in the top screen.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/changing_refresh_delay.gif" alt="Changing the refresh delay" title="Changing the refresh delay" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2"><br />
You can change the field screen by pressing <code style="font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">f</code> key in the top screen. In the fields screen, you can change the displayed fields by choosing a field and pressing <code style="font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">d</code> key or <code style="font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">space</code> key.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/changing_displayed_fields.gif" alt="Changing the displayed fields" title="Changing the displayed fields" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2"><br />
You can move to the fields screen by pressing <code style="font-size: small; font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">f</code> key in the top screen. In the field screen, you can change the sort field by choosing a field and pressing <code style="font-size: small; font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">s</code> key. Also, you can change the sort order (ascending or descending) by pressing <code style="font-size: small; font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">R</code> key.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/changing_sort_field.gif" alt="Changing the sort field" title="Changing the sort field" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2"><br />
You can move to the fields screen by pressing <code style="font-size: small; font-weight: normal; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">f</code> key in the top screen. In the field screen, you can change the order of the fields.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/changing_order_of_fields.gif" alt="Changing the order of the fields" title="Changing the sort field" style="box-sizing: initial; max-width: 100%;" /> </p>
<h4 style="font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; letter-spacing: normal; margin-top: 24px; margin-bottom: 16px; box-sizing: border-box; line-height: 1.25; color: #24292e;"><font size="3">Filters</font></h4>
<p> <font size="2">You can filter the metric records with the filter feature. We can add filters by pressing <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">o</code> key for ignoring case or <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">O</code> key for case sensitive.</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/adding_filters.gif" alt="Adding filters" title="Adding filters" style="box-sizing: initial; max-width: 100%;" /> </p>
<p> <font size="2">The syntax is as follows:</font> </p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: #f6f8fa; border-radius: 3px; color: #24292e;"><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0px; margin: 0px; background: initial; border-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;"><font size="2"><Field><Operator><Value>
</font></code></pre>
<p> <font size="2">For example, we can add filters like the following:</font> </p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: #f6f8fa; border-radius: 3px; color: #24292e;"><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0px; margin: 0px; background: initial; border-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;"><font size="2">NAMESPACE==default
REQ/S>1000</font></code></pre>
<p> <font size="2">The operators you can specify are as follows:</font> </p>
<table style="box-sizing: border-box; border-spacing: 0px; border-collapse: collapse; margin-top: 0px; margin-bottom: 16px; display: block; width: 882px; overflow: auto; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", " Segoe UI Symbol";">
<thead style="box-sizing: border-box;">
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<th style="font-weight: 600; box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Operator</font></th>
<th style="font-weight: 600; box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Description</font></th>
</tr>
</thead>
<tbody style="box-sizing: border-box;">
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">=</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Partial match</font></td>
</tr>
<tr style="box-sizing: border-box; background-color: #f6f8fa; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">==</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Exact match</font></td>
</tr>
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">></font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Greater than</font></td>
</tr>
<tr style="box-sizing: border-box; background-color: #f6f8fa; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">>=</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Greater than or equal to</font></td>
</tr>
<tr style="box-sizing: border-box; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2"><</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Less than</font></td>
</tr>
<tr style="box-sizing: border-box; background-color: #f6f8fa; border-top: 1px solid #c6cbd1;">
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2"><=</font></td>
<td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid #dfe2e5;"><font size="2">Less than and equal to</font></td>
</tr>
</tbody>
</table>
<p> <font size="2"><br />
You can see the current filters by pressing <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">^o</code> key and clear them by pressing <code style="box-sizing: border-box; font-family: "SFMono- Regular", Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">=</code> key.<br />
</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/showing_and_clearing_filters.gif" alt="Showing and clearing filters" title="Showing and clearing filters" style="box-sizing: initial; max-width: 100%;" /> </p>
<h4 style="font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; letter-spacing: normal; margin-top: 24px; margin-bottom: 16px; box-sizing: border-box; line-height: 1.25; color: #24292e;"><font size="3">Drilling down</font></h4>
<p> <font size="2">You can drill down on a metric record by pressing <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">i</code> key in the top screen. With this feature, you can find hot regions easily in a top-down manner.</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/driling_down.gif" alt="Drilling down" title="Drilling down" style="box-sizing: initial; max-width: 100%;" /> </p>
<h4 style="font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; letter-spacing: normal; margin-top: 24px; margin-bottom: 16px; box-sizing: border-box; line-height: 1.25; color: #24292e;"><font size="3">Help screen</font></h4>
<p> <font size="2">You can see the help screen by pressing <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">h</code> key in the top screen.</font> </p>
<p> <img src="https://hbase.apache.org/hbtop-images/help_screen.gif" alt="Help screen" title="Help screen" style="box-sizing: initial; max-width: 100%;" /> </p>
<h4 style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";"><span style="font-weight: bold;"><font size="3">How hbtop gets the metrics data</font></span></h4>
<p> <font size="2">hbtop gets the metrics from ClusterMetrics which is returned as the result of a call to Admin#getClusterMetrics() on the current HMaster. To add metrics to hbtop, they will need to be exposed via ClusterMetrics.</font></p>
<p><font size="2">Feedback, bugs, and enhancements are all welcome!</font></p></p>
<p>The Reference Guide - hbtop:</p>
<p><a href="https://hbase.apache.org/book.html#hbtop">https://hbase.apache.org/book.html#hbtop</a> </p></p>