releases/0.8.0/docs/administration.html

---
title: Apache Kudu (incubating) Administration
layout: default
active_nav: docs
last_updated: 'Last updated 2016-04-11 08:31:49 PDT'
---
<!--

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->


<div class="container">
  <div class="row">
    <div class="col-md-9">

<h1>Apache Kudu (incubating) Administration</h1>
      <div id="preamble">
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Kudu is easier to manage with <a href="http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/cloudera-manager.htm">Cloudera Manager</a>
than in a standalone installation. See Cloudera&#8217;s
<a href="http://www.cloudera.com/content/cloudera/en/downloads/betas/kudu/0-5-0.html">Kudu documentation</a>
for more details about using Kudu with Cloudera Manager.
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_starting_and_stopping_kudu_processes"><a class="link" href="#_starting_and_stopping_kudu_processes">Starting and Stopping Kudu Processes</a></h2>
<div class="sectionbody">
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Start Kudu services using the following commands:</p>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-bash" data-lang="bash">$ sudo service kudu-master start
$ sudo service kudu-tserver start</code></pre>
</div>
</div>
</li>
<li>
<p>To stop Kudu services, use the following commands:</p>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-bash" data-lang="bash">$ sudo service kudu-master stop
$ sudo service kudu-tserver stop</code></pre>
</div>
</div>
</li>
</ol>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_kudu_web_interfaces"><a class="link" href="#_kudu_web_interfaces">Kudu Web Interfaces</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu tablet servers and masters expose useful operational information on a built-in web interface,</p>
</div>
<div class="sect2">
<h3 id="_kudu_master_web_interface"><a class="link" href="#_kudu_master_web_interface">Kudu Master Web Interface</a></h3>
<div class="paragraph">
<p>Kudu master processes serve their web interface on port 8051. The interface exposes several pages
with information about the cluster state:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A list of tablet servers, their host names, and the time of their last heartbeat.</p>
</li>
<li>
<p>A list of tables, including schema and tablet location information for each.</p>
</li>
<li>
<p>SQL code which you can paste into Impala Shell to add an existing table to Impala&#8217;s list of known data sources.</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="_kudu_tablet_server_web_interface"><a class="link" href="#_kudu_tablet_server_web_interface">Kudu Tablet Server Web Interface</a></h3>
<div class="paragraph">
<p>Each tablet server serves a web interface on port 8050. The interface exposes information
about each tablet hosted on the server, its current state, and debugging information
about maintenance background operations.</p>
</div>
</div>
<div class="sect2">
<h3 id="_common_web_interface_pages"><a class="link" href="#_common_web_interface_pages">Common Web Interface Pages</a></h3>
<div class="paragraph">
<p>Both Kudu masters and tablet servers expose a common set of information via their web interfaces:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>HTTP access to server logs.</p>
</li>
<li>
<p>an <code>/rpcz</code> endpoint which lists currently running RPCs via JSON.</p>
</li>
<li>
<p>pages giving an overview and detailed information on the memory usage of different
components of the process.</p>
</li>
<li>
<p>information on the current set of configuration flags.</p>
</li>
<li>
<p>information on the currently running threads and their resource consumption.</p>
</li>
<li>
<p>a JSON endpoint exposing metrics about the server.</p>
</li>
<li>
<p>information on the deployed version number of the daemon.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These interfaces are linked from the landing page of each daemon&#8217;s web UI.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_kudu_metrics"><a class="link" href="#_kudu_metrics">Kudu Metrics</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Kudu daemons expose a large number of metrics. Some metrics are associated with an entire
server process, whereas others are associated with a particular tablet replica.</p>
</div>
<div class="sect2">
<h3 id="_listing_available_metrics"><a class="link" href="#_listing_available_metrics">Listing available metrics</a></h3>
<div class="paragraph">
<p>The full set of available metrics for a Kudu server can be dumped via a special command
line flag:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-bash" data-lang="bash">$ kudu-tserver --dump_metrics_json
$ kudu-master --dump_metrics_json</code></pre>
</div>
</div>
<div class="paragraph">
<p>This will output a large JSON document. Each metric indicates its name, label, description,
units, and type. Because the output is JSON-formatted, this information can easily be
parsed and fed into other tooling which collects metrics from Kudu servers.</p>
</div>
</div>
<div class="sect2">
<h3 id="_collecting_metrics_via_http"><a class="link" href="#_collecting_metrics_via_http">Collecting metrics via HTTP</a></h3>
<div class="paragraph">
<p>Metrics can be collected from a server process via its HTTP interface by visiting
<code>/metrics</code>. The output of this page is JSON for easy parsing by monitoring services.
This endpoint accepts several <code>GET</code> parameters in its query string:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/metrics?metrics=&lt;substring1&gt;,&lt;substring2&gt;,&#8230;&#8203;</code> - limits the returned metrics to those which contain
at least one of the provided substrings. The substrings also match entity names, so this
may be used to collect metrics for a specific tablet.</p>
</li>
<li>
<p><code>/metrics?include_schema=1</code> - includes metrics schema information such as unit, description,
and label in the JSON output. This information is typically elided to save space.</p>
</li>
<li>
<p><code>/metrics?compact=1</code> - eliminates unnecessary whitespace from the resulting JSON, which can decrease
bandwidth when fetching this page from a remote host.</p>
</li>
<li>
<p><code>/metrics?include_raw_histograms=1</code> - include the raw buckets and values for histogram metrics,
enabling accurate aggregation of percentile metrics over time and across hosts.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For example:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-bash" data-lang="bash">$ curl -s 'http://example-ts:8050/metrics?include_schema=1&amp;metrics=connections_accepted'</code></pre>
</div>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-json" data-lang="json">[
    {
        "type": "server",
        "id": "kudu.tabletserver",
        "attributes": {},
        "metrics": [
            {
                "name": "rpc_connections_accepted",
                "label": "RPC Connections Accepted",
                "type": "counter",
                "unit": "connections",
                "description": "Number of incoming TCP connections made to the RPC server",
                "value": 92
            }
        ]
    }
]

[source,bash]</code></pre>
</div>
</div>
<div class="paragraph">
<p>$ curl -s 'http://example-ts:8050/metrics?metrics=log_append_latency'</p>
</div>
<div class="listingblock">
<div class="content">
<pre>[source,json]</pre>
</div>
</div>
<div class="paragraph">
<p>[
    {
        "type": "tablet",
        "id": "c0ebf9fef1b847e2a83c7bd35c2056b1",
        "attributes": {
            "table_name": "lineitem",
            "partition": <span class="menuseq"><span class="menu">hash buckets: (55), range: [(&lt;start</span>&#160;&#9656; <span class="submenu">), (&lt;end</span>&#160;&#9656; <span class="menuitem">))</span></span>,
            "table_id": ""
        },
        "metrics": [
            {
                "name": "log_append_latency",
                "total_count": 7498,
                "min": 4,
                "mean": 69.3649,
                "percentile_75": 29,
                "percentile_95": 38,
                "percentile_99": 45,
                "percentile_99_9": 95,
                "percentile_99_99": 167,
                "max": 367244,
                "total_sum": 520098
            }
        ]
    }
]</p>
</div>
<div class="listingblock">
<div class="content">
<pre>NOTE: All histograms and counters are measured since the server start time, and are not reset upon collection.

=== Collecting metrics to a log

Kudu may be configured to periodically dump all of its metrics to a local log file using the
`--metrics_log_interval_ms` flag. Set this flag to the interval at which metrics should be written
to a log file.

The metrics log will be written to the same directory as the other Kudu log files, with the same
naming format. After any metrics log file reaches 64MB uncompressed, the log will be rolled and
the previous file will be gzip-compressed.

The log file generated has three space-separated fields. The first field is the word
`metrics`. The second field is the current timestamp in microseconds since the Unix epoch.
The third is the current value of all metrics on the server, using a compact JSON encoding.
The encoding is the same as the metrics fetched via HTTP described above.

WARNING: Although metrics logging automatically rolls and compresses previous log files, it does
not remove old ones. Since metrics logging can use significant amounts of disk space,
consider setting up a system utility to monitor space in the log directory and archive or
delete old segments.</pre>
</div>
</div>
</div>
</div>
</div>
    </div>
    <div class="col-md-3">

  <div id="toc" data-spy="affix" data-offset-top="70">
  <ul>

      <li>

          <a href="introduction.html">Introducing Kudu</a> 
      </li> 
      <li>

          <a href="release_notes.html">Kudu Release Notes</a> 
      </li> 
      <li>

          <a href="quickstart.html">Getting Started with Kudu</a> 
      </li> 
      <li>

          <a href="installation.html">Installation Guide</a> 
      </li> 
      <li>

          <a href="configuration.html">Configuring Kudu</a> 
      </li> 
      <li>

          <a href="kudu_impala_integration.html">Using Impala with Kudu</a> 
      </li> 
      <li>
<span class="active-toc">Administering Kudu</span>
            <ul class="sectlevel1">
<li><a href="#_starting_and_stopping_kudu_processes">Starting and Stopping Kudu Processes</a></li>
<li><a href="#_kudu_web_interfaces">Kudu Web Interfaces</a>
<ul class="sectlevel2">
<li><a href="#_kudu_master_web_interface">Kudu Master Web Interface</a></li>
<li><a href="#_kudu_tablet_server_web_interface">Kudu Tablet Server Web Interface</a></li>
<li><a href="#_common_web_interface_pages">Common Web Interface Pages</a></li>
</ul>
</li>
<li><a href="#_kudu_metrics">Kudu Metrics</a>
<ul class="sectlevel2">
<li><a href="#_listing_available_metrics">Listing available metrics</a></li>
<li><a href="#_collecting_metrics_via_http">Collecting metrics via HTTP</a></li>
</ul>
</li>
</ul> 
      </li> 
      <li>

          <a href="troubleshooting.html">Troubleshooting Kudu</a> 
      </li> 
      <li>

          <a href="developing.html">Developing Applications with Kudu</a> 
      </li> 
      <li>

          <a href="schema_design.html">Kudu Schema Design</a> 
      </li> 
      <li>

          <a href="transaction_semantics.html">Kudu Transaction Semantics</a> 
      </li> 
      <li>

          <a href="contributing.html">Contributing to Kudu</a> 
      </li> 
      <li>

          <a href="style_guide.html">Kudu Documentation Style Guide</a> 
      </li> 
      <li>

          <a href="configuration_reference.html">Kudu Configuration Reference</a> 
      </li> 
  </ul>
  </div>
    </div>
  </div>
</div>