Metron Docker is a Docker Compose application that is intended for development and integration testing of Metron. Use this instead of Vagrant when:
Metron Docker includes these images that have been customized for Metron:
Install Docker for Mac or Docker for Windows. The following versions have been tested:
Build Metron from the top level directory with:
$ cd $METRON_HOME $ mvn clean install -DskipTests
You are welcome to use an existing Docker host but we prefer one with more resources. You can create one of those with this script:
$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker $ cd $METRON_DOCKER_HOME && ./scripts/create-docker-machine.sh
This will create a host called “metron-machine”. Anytime you want to run Docker commands against this host, make sure you run this first to set the Docker environment variables:
$ eval "$(docker-machine env metron-machine)"
If you wish to use a local docker-engine install, please set an environment variable BROKER_IP_ADDR to the IP address of your host machine. This cannot be the loopback address.
Navigate to the compose application root:
$ cd $METRON_DOCKER_HOME/compose/
The Metron Docker environment lifecycle is controlled by the docker-compose command. The service names can be found in the docker-compose.yml file. For example, to build and start the environment run this command:
$ eval "$(docker-machine env metron-machine)" $ docker-compose up -d
After all services have started list the containers and ensure their status is ‘Up’:
$ docker ps --format 'table {{.Names}}\t{{.Status}}' NAMES STATUS metron_storm_1 Up 5 minutes metron_hbase_1 Up 5 minutes metron_kibana_1 Up 5 minutes metron_kafkazk_1 Up 5 minutes metron_elasticsearch_1 Up 5 minutes
Various services are exposed through http on the Docker host. Get the host ip from the URL property:
$ docker-machine ls NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS metron-machine * virtualbox Running tcp://192.168.99.100:2376 v1.12.5
Then, assuming a host ip of 192.168.99.100
, the UIs and APIs are available at:
The Storm logs can be useful when troubleshooting topologies. They can be found on the Storm container in /usr/share/apache-storm/logs
.
When done using the machine, shut it down with:
$ docker-compose down
After adding a new parser to metron-parsers, build Metron from the top level directory:
$ cd $METRON_HOME $ mvn clean install -DskipTests
Then run these commands to redeploy the parsers to the Storm image:
$ cd $METRON_DOCKER_HOME/compose $ docker-compose down $ docker-compose build storm $ docker-compose up -d
Suppose there is a problem with Kafka and the logs are needed for further investigation. Run this command to connect and explore the running Kafka container:
$ cd $METRON_DOCKER_HOME/compose $ docker-compose exec kafkazk bash
A tool for producing test data in Kafka is included with the Kafka/Zookeeper image. It loops through lines in a test data file and outputs them to Kafka at the desired frequency. Create a test data file in ./kafkazk/data/
and rebuild the Kafka/Zookeeper image:
$ cd $METRON_DOCKER_HOME/compose $ printf 'first test data\nsecond test data\nthird test data\n' > ./kafkazk/data/TestData.txt $ docker-compose down $ docker-compose build kafkazk $ docker-compose up -d
This will deploy the test data file to the Kafka/Zookeeper container. Now that data can be streamed to a Kafka topic:
$ docker-compose exec kafkazk ./bin/produce-data.sh Usage: produce-data.sh data_path topic [message_delay_in_seconds] # Stream data in TestData.txt to the 'test' Kafka topic at a frequency of 5 seconds (default is 1 second) $ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5
The Kafka/Zookeeper image comes with sample Bro and Squid data:
# Stream Bro test data every 1 second $ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro # Stream Squid test data every 0.1 seconds $ docker-compose exec kafkazk ./bin/produce-data.sh /data/SquidExampleOutput.txt squid 0.1
Parser configs and a global config configured for this Docker environment are included with the Kafka/Zookeeper image. Load them with:
$ docker-compose exec kafkazk bash # $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i $METRON_HOME/config/zookeeper # exit
Dump out the configs with:
$ docker-compose exec kafkazk bash # $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m DUMP # exit
The Storm image comes with a script to easily start parser topologies:
docker-compose exec storm ./bin/start_docker_parser_topology.sh sensor_name
The enrichment topology can be started with:
docker-compose exec storm ./bin/start_enrichment_topology.sh
The indexing topology can be started with:
docker-compose exec storm ./bin/start_elasticsearch_topology.sh
Topologies can be stopped using the Storm CLI. For example, stop the enrichment topology with:
docker-compose exec storm storm kill enrichments -w 0
First ensure configs were uploaded as described in the previous example. Then start a sensor and leave it running:
$ cd $METRON_DOCKER_HOME/compose $ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
Open a separate console session and verify the sensor is running by consuming a message from Kafka:
$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker $ cd $METRON_DOCKER_HOME/compose $ docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
A new message should be printed every second. Now kill the consumer and start the Bro parser topology:
$ docker-compose exec storm ./bin/start_docker_parser_topology.sh bro
Bro data should be flowing through the bro parser topology and into the Kafka enrichments topic. The enrichments topic should be created automatically:
$ docker-compose exec kafkazk ./bin/kafka-topics.sh --zookeeper localhost:2181 --list bro enrichments indexing
Verify parsed Bro data is in the Kafka enrichments topic:
docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic enrichments
Now start the enrichment topology:
docker-compose exec storm ./bin/start_enrichment_topology.sh
Parsed Bro data should be flowing through the enrichment topology and into the Kafka indexing topic. Verify enriched Bro data is in the Kafka indexing topic:
docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic indexing
Now start the indexing topology:
docker-compose exec storm ./bin/start_elasticsearch_topology.sh
Enriched Bro data should now be present in the Elasticsearch container:
$ docker-machine ls NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS metron-machine * virtualbox Running tcp://192.168.99.100:2376 v1.12.5 $ curl -XGET http://192.168.99.100:9200/_cat/indices?v health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana 1 1 1 0 3.1kb 3.1kb yellow open bro_index_2016.12.19.18 5 1 180 0 475kb 475kb