examples/python/dstat-kudu/README.md

<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Kudu + dstat + Impala

This is an example program that shows how to use the Kudu API in Python to load data into
a new / existing Kudu table generated by an external program.

## Prerequisites

Make sure you have the Kudu client library installed and the kudu Python bindings are
available. If you have the Kudu client library and Python bindings in a special place,
you'll need to set the environment variables:

   LD_LIBRARY_PATH
   PYTHONPATH

To the according directories. In addition you'll need the dstat program, it should be
available from your typical package repository.

## Usage

In this case the `dstat` program is used to generate data about the system load and pipe
this data into a named pipe that is then read and pipe to the python program.

To execute this script simply run:

    python kudu_dstat.py

This will create a table assuming that you have a kudu-master running locally. You can use
the Web UI to access some information about the table using the following link:
http://localhost:8051. The program will run until it is terminated via C-c.

To drop the table in Kudu and start fresh start the program with:

    python kudu_dstat.py drop

To query the data via Impala, create a new Kudu table in Impala using the
following command in the impala-shell.

    CREATE EXTERNAL TABLE dstat (
    `ts` BIGINT,
    `usr` FLOAT,
    `sys` FLOAT,
    `idl` FLOAT,
    `wai` FLOAT,
    `hiq` FLOAT,
    `siq` FLOAT,
    `read` FLOAT,
    `writ` FLOAT,
    `recv` FLOAT,
    `send` FLOAT,
    `in` FLOAT,
    `out` FLOAT,
    `int` FLOAT,
    `csw` FLOAT
    )
    TBLPROPERTIES(
      'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',
      'kudu.table_name' = 'dstat',
      'kudu.master_addresses' = '127.0.0.1:7051',
      'kudu.key_columns' = 'ts'
    );

Now you can query your local system's load using:


    -- How many rows are stored right now?
    select count(*) from dstat;

    -- Average load in 10s windows
    select (ts - ts % 10 ) as mod_ts, avg(usr), avg(sys), avg(idl) from dstat group by mod_ts order by mod_ts