commit	e7f2081db09c931b8353a289e6319971681e43ea	[log] [tgz]
author	mattf-horton <mfoley@hortonworks.com>	Wed Apr 05 09:42:40 2017 -0400
committer	cstella <cestella@gmail.com>	Wed Apr 05 09:42:40 2017 -0400
tree	d52db3a9f04e818bc410bd1ddc383a76e6ba44a5
parent	1aa805c1cb6b1cd05362432e1ed1c943a9e8ab99 [diff]

tree: d52db3a9f04e818bc410bd1ddc383a76e6ba44a5

README.md

Bro Logging Output to Kafka

A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.

Installation

Install librdkafka (https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4. In order to support interacting with a kerberized kafka, you will need libsasl2 installed

# curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
# cd librdkafka-0.9.4/
# ./configure --enable-sasl
# make
# sudo make install

Then compile this Bro plugin using the following commands.

# ./configure --bro-dist=$BRO_SRC
# make
# sudo make install

Run the following command to ensure that the plugin was installed successfully.

# bro -N Bro::Kafka
Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)

Activation

The easiest way to enable Kafka output is to load the plugin's logs-to-kafka.bro script. If you are using BroControl, the following lines added to local.bro will activate it.

@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:9092"
);

This example will send all HTTP, DNS, and Conn logs to a Kafka broker running on the localhost to a topic called bro. Any configuration value accepted by librdkafka can be added to the kafka_conf configuration table.

Settings

`kafka_conf`

The global configuration settings for Kafka. These values are passed through directly to librdkafka. Any valid librdkafka settings can be defined in this table. The full set of valid librdkafka settings are available here.

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:9092",
    ["client.id"] = "bro"
);

`topic_name`

The name of the topic in Kafka where all Bro logs will be sent to.

redef Kafka::topic_name = "bro";

`max_wait_on_shutdown`

The maximum number of milliseconds that the plugin will wait for any backlog of queued messages to be sent to Kafka before forced shutdown.

redef Kafka::max_wait_on_shutdown = 3000;

`tag_json`

If true, a log stream identifier is appended to each JSON-formatted message. For example, a Conn::LOG message will look like { 'conn' : { ... }}.

redef Kafka::tag_json = T;

`debug`

A comma separated list of debug contexts in librdkafka which you want to enable. The available contexts are:

generic
broker
topic
metadata
queue
msg
protocol
cgrp
security
fetch
feature
all

Kerberos

This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set.

SASL

If you are using SASL as a security protocol for kafka, then you must have libsasl or libsasl2 installed. You can tell if sasl is enabled by running the following from the directory in which you have build librdkafka:

examples/rdkafka_example -X builtin.features
builtin.features = gzip,snappy,ssl,sasl,regex

Producer Config

As stated above, you can configure the producer kafka configs in ${BRO_HOME}/share/bro/site/local.bro. There are a few configs necessary to set, which are described here. For an environment where the following is true:

The broker is node1:6667
This kafka is using SASL_PLAINTEXT as the security protocol
The keytab used is the metron keytab
The service principal for metron is metron@EXAMPLE.COM

The kafka topic bro has been given permission for the metron user to write:

# login using the metron user 
kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro

The following is how the ${BRO_HOME}/share/bro/site/local.bro looks:

@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667"
                               , ["security.protocol"] = "SASL_PLAINTEXT"
                               , ["sasl.kerberos.keytab"] = "/etc/security/keytabs/metron.headless.keytab"
                               , ["sasl.kerberos.principal"] = "metron@EXAMPLE.COM"
                               );