Google Git
Sign in
apache / metron / apache-metron-0.4.0-release / . / site / current-book / metron-docker / index.html
blob: a87945b2c0590d000fa2f5b02c89a9349c6dbc97 [file] [log] [blame]
<!DOCTYPE html>
<!--
| Generated by Apache Maven Doxia at 2017-02-23
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="Date-Revision-yyyymmdd" content="20170223" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron &#x2013; Metron Docker</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
<link rel="stylesheet" href="../css/site.css" />
<link rel="stylesheet" href="../css/print.css" media="print" />
<script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
</head>
<body class="topBarDisabled">
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
<a href="http://metron.incubator.apache.org/" id="bannerLeft">
<img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
</a>
</div>
<div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
<img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
</a>
</div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
<li class="">
<a href="http://www.apache.org" class="externalLink" title="Apache">
Apache</a>
</li>
<li class="divider ">/</li>
<li class="">
<a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
Metron-Incubating</a>
</li>
<li class="divider ">/</li>
<li class="">
<a href="../index.html" title="Documentation">
Documentation</a>
</li>
<li class="divider ">/</li>
<li class="">Metron Docker</li>
<li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
<li id="projectVersion" class="pull-right">Version: 0.3.1</li>
</ul>
</div>
<div class="row-fluid">
<div id="leftColumn" class="span3">
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
<li>
<a href="../index.html" title="Metron">
<i class="icon-chevron-down"></i>
Metron</a>
<ul class="nav nav-list">
<li>
<a href="../Upgrading.html" title="Upgrading">
<i class="none"></i>
Upgrading</a>
</li>
<li>
<a href="../metron-analytics/index.html" title="Analytics">
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
<li>
<a href="../metron-deployment/index.html" title="Deployment">
<i class="icon-chevron-right"></i>
Deployment</a>
</li>
<li class="active">
<a href="#"><i class="none"></i>Docker</a>
</li>
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
</ul>
</li>
</ul>
<hr class="divider" />
<div id="poweredBy">
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
</a>
</div>
</div>
</div>
<div id="bodyColumn" class="span9" >
<h1>Metron Docker</h1>
<p><a name="Metron_Docker"></a></p>
<p>Metron Docker is a <a class="externalLink" href="https://docs.docker.com/compose/overview/">Docker Compose</a> application that is intended for development and integration testing of Metron. Use this instead of Vagrant when:</p>
<ul>
<li>You want an environment that can be built and spun up quickly</li>
<li>You need to frequently rebuild and restart services</li>
<li>You only need to test, troubleshoot or develop against a subset of services</li>
</ul>
<p>Metron Docker includes these images that have been customized for Metron:</p>
<ul>
<li>Kafka (with Zookeeper)</li>
<li>HBase</li>
<li>Storm (with all topologies deployed)</li>
<li>Elasticsearch</li>
<li>Kibana</li>
</ul>
<div class="section">
<h2><a name="Setup"></a>Setup</h2>
<p>Install <a class="externalLink" href="https://docs.docker.com/docker-for-mac/">Docker for Mac</a> or <a class="externalLink" href="https://docs.docker.com/docker-for-windows/">Docker for Windows</a>. The following versions have been tested:</p>
<ul>
<li>Docker version 1.12.0</li>
<li>docker-machine version 0.8.0</li>
<li>docker-compose version 1.8.0</li>
</ul>
<p>Build Metron from the top level directory with:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_HOME
$ mvn clean install -DskipTests
</pre></div></div>
<p>You are welcome to use an existing Docker host but we prefer one with more resources. You can create one of those with this script:</p>
<div class="source">
<div class="source">
<pre>$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker
$ cd $METRON_DOCKER_HOME &amp;&amp; ./scripts/create-docker-machine.sh
</pre></div></div>
<p>This will create a host called &#x201c;metron-machine&#x201d;. Anytime you want to run Docker commands against this host, make sure you run this first to set the Docker environment variables:</p>
<div class="source">
<div class="source">
<pre>$ eval &quot;$(docker-machine env metron-machine)&quot;
</pre></div></div>
<p>If you wish to use a local docker-engine install, please set an environment variable BROKER_IP_ADDR to the IP address of your host machine. This cannot be the loopback address.</p></div>
<div class="section">
<h2><a name="Usage"></a>Usage</h2>
<p>Navigate to the compose application root:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_DOCKER_HOME/compose/
</pre></div></div>
<p>The Metron Docker environment lifecycle is controlled by the <a class="externalLink" href="https://docs.docker.com/compose/reference/overview/">docker-compose</a> command. The service names can be found in the docker-compose.yml file. For example, to build and start the environment run this command:</p>
<div class="source">
<div class="source">
<pre>$ eval &quot;$(docker-machine env metron-machine)&quot;
$ docker-compose up -d
</pre></div></div>
<p>After all services have started list the containers and ensure their status is &#x2018;Up&#x2019;:</p>
<div class="source">
<div class="source">
<pre>$ docker ps --format 'table {{.Names}}\t{{.Status}}'
NAMES STATUS
metron_storm_1 Up 5 minutes
metron_hbase_1 Up 5 minutes
metron_kibana_1 Up 5 minutes
metron_kafkazk_1 Up 5 minutes
metron_elasticsearch_1 Up 5 minutes
</pre></div></div>
<p>Various services are exposed through http on the Docker host. Get the host ip from the URL property:</p>
<div class="source">
<div class="source">
<pre>$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
metron-machine * virtualbox Running tcp://192.168.99.100:2376 v1.12.5
</pre></div></div>
<p>Then, assuming a host ip of <tt>192.168.99.100</tt>, the UIs and APIs are available at:</p>
<ul>
<li>Storm - <a class="externalLink" href="http://192.168.99.100:8080/">http://192.168.99.100:8080/</a></li>
<li>HBase - <a class="externalLink" href="http://192.168.99.100:16010/">http://192.168.99.100:16010/</a></li>
<li>Elasticsearch - <a class="externalLink" href="http://192.168.99.100:9200/">http://192.168.99.100:9200/</a></li>
<li>Kibana - <a class="externalLink" href="http://192.168.99.100:5601/">http://192.168.99.100:5601/</a></li>
</ul>
<p>The Storm logs can be useful when troubleshooting topologies. They can be found on the Storm container in <tt>/usr/share/apache-storm/logs</tt>.</p>
<p>When done using the machine, shut it down with:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose down
</pre></div></div></div>
<div class="section">
<h2><a name="Examples"></a>Examples</h2>
<ul>
<li><a href="#Deploy_a_new_parser_class">Deploy a new parser class</a></li>
<li><a href="#Connect_to_a_container">Connect to a container</a></li>
<li><a href="create-a-sensor-from-sample-data/index.html">Create a sensor from sample data</a></li>
<li><a href="upload-configs-to-zookeeper/index.html">Upload configs to Zookeeper</a></li>
<li><a href="manage-a-topology/index.html">Manage a topology</a></li>
<li><a href="run-sensor-data-end-to-end/index.html">Run sensor data end to end</a></li>
</ul>
<div class="section">
<h3><a name="Deploy_a_new_parser_class"></a>Deploy a new parser class</h3>
<p>After adding a new parser to metron-parsers, build Metron from the top level directory:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_HOME
$ mvn clean install -DskipTests
</pre></div></div>
<p>Then run these commands to redeploy the parsers to the Storm image:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_DOCKER_HOME/compose
$ docker-compose down
$ docker-compose build storm
$ docker-compose up -d
</pre></div></div></div>
<div class="section">
<h3><a name="Connect_to_a_container"></a>Connect to a container</h3>
<p>Suppose there is a problem with Kafka and the logs are needed for further investigation. Run this command to connect and explore the running Kafka container:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_DOCKER_HOME/compose
$ docker-compose exec kafkazk bash
</pre></div></div></div>
<div class="section">
<h3><a name="Create_a_sensor_from_sample_data"></a>Create a sensor from sample data</h3>
<p>A tool for producing test data in Kafka is included with the Kafka/Zookeeper image. It loops through lines in a test data file and outputs them to Kafka at the desired frequency. Create a test data file in <tt>./kafkazk/data/</tt> and rebuild the Kafka/Zookeeper image:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_DOCKER_HOME/compose
$ printf 'first test data\nsecond test data\nthird test data\n' &gt; ./kafkazk/data/TestData.txt
$ docker-compose down
$ docker-compose build kafkazk
$ docker-compose up -d
</pre></div></div>
<p>This will deploy the test data file to the Kafka/Zookeeper container. Now that data can be streamed to a Kafka topic:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose exec kafkazk ./bin/produce-data.sh
Usage: produce-data.sh data_path topic [message_delay_in_seconds]
# Stream data in TestData.txt to the 'test' Kafka topic at a frequency of 5 seconds (default is 1 second)
$ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5
</pre></div></div>
<p>The Kafka/Zookeeper image comes with sample Bro and Squid data:</p>
<div class="source">
<div class="source">
<pre># Stream Bro test data every 1 second
$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
# Stream Squid test data every 0.1 seconds
$ docker-compose exec kafkazk ./bin/produce-data.sh /data/SquidExampleOutput.txt squid 0.1
</pre></div></div></div>
<div class="section">
<h3><a name="Upload_configs_to_Zookeeper"></a>Upload configs to Zookeeper</h3>
<p>Parser configs and a global config configured for this Docker environment are included with the Kafka/Zookeeper image. Load them with:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose exec kafkazk bash
# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i $METRON_HOME/config/zookeeper
# exit
</pre></div></div>
<p>Dump out the configs with:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose exec kafkazk bash
# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
# exit
</pre></div></div></div>
<div class="section">
<h3><a name="Manage_a_topology"></a>Manage a topology</h3>
<p>The Storm image comes with a script to easily start parser topologies:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm ./bin/start_docker_parser_topology.sh sensor_name
</pre></div></div>
<p>The enrichment topology can be started with:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm ./bin/start_enrichment_topology.sh
</pre></div></div>
<p>The indexing topology can be started with:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm ./bin/start_elasticsearch_topology.sh
</pre></div></div>
<p>Topologies can be stopped using the Storm CLI. For example, stop the enrichment topology with:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm storm kill enrichments -w 0
</pre></div></div></div>
<div class="section">
<h3><a name="Run_sensor_data_end_to_end"></a>Run sensor data end to end</h3>
<p>First ensure configs were uploaded as described in the previous example. Then start a sensor and leave it running:</p>
<div class="source">
<div class="source">
<pre>$ cd $METRON_DOCKER_HOME/compose
$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
</pre></div></div>
<p>Open a separate console session and verify the sensor is running by consuming a message from Kafka:</p>
<div class="source">
<div class="source">
<pre>$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker
$ cd $METRON_DOCKER_HOME/compose
$ docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
</pre></div></div>
<p>A new message should be printed every second. Now kill the consumer and start the Bro parser topology:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose exec storm ./bin/start_docker_parser_topology.sh bro
</pre></div></div>
<p>Bro data should be flowing through the bro parser topology and into the Kafka enrichments topic. The enrichments topic should be created automatically:</p>
<div class="source">
<div class="source">
<pre>$ docker-compose exec kafkazk ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
bro
enrichments
indexing
</pre></div></div>
<p>Verify parsed Bro data is in the Kafka enrichments topic:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic enrichments
</pre></div></div>
<p>Now start the enrichment topology:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm ./bin/start_enrichment_topology.sh
</pre></div></div>
<p>Parsed Bro data should be flowing through the enrichment topology and into the Kafka indexing topic. Verify enriched Bro data is in the Kafka indexing topic:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic indexing
</pre></div></div>
<p>Now start the indexing topology:</p>
<div class="source">
<div class="source">
<pre>docker-compose exec storm ./bin/start_elasticsearch_topology.sh
</pre></div></div>
<p>Enriched Bro data should now be present in the Elasticsearch container:</p>
<div class="source">
<div class="source">
<pre>$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
metron-machine * virtualbox Running tcp://192.168.99.100:2376 v1.12.5
$ curl -XGET http://192.168.99.100:9200/_cat/indices?v
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 1 1 1 0 3.1kb 3.1kb
yellow open bro_index_2016.12.19.18 5 1 180 0 475kb 475kb
</pre></div></div></div></div>
</div>
</div>
</div>
<hr/>
<footer>
<div class="container-fluid">
<div class="row span12">Copyright &copy; 2017.
All Rights Reserved.
</div>
</div>
</footer>
</body>
</html>