A Bro Log filter that sends log data to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the log data generated by Bro.
Install librdkafka, a native client library for Kafka.
Download and compile Bro from source code. Replace version 2.4.1 with the most recent stable release. See the INSTALL documentation within the Bro release for more details on prerequisites and compilation.
curl https://www.bro.org/downloads/release/bro-2.4.1.tar.gz | tar -xvz cd bro-2.4.1 ./configure make sudo make install export BRO_SRC=`pwd`
Download and compile this Kafka plugin for Bro.
./configure --bro-dist=$BRO_SRC make sudo make install
Optional: Instead of running make install as above, after the plugin has been compiled, the build directory can be added to the BRO_PLUGIN_PATH. Bro will automatically load any plugins that it finds at this location.
export BRO_PLUGIN_PATH="/home/the-dude/bro-plugin-kafka/build/"
Run the following command to ensure that the plugin was installed successfully. The ‘bro’ executable must be in your PATH.
bro -N Metron::Kafka
After installing the plugin, create a bro script that will activate it. In most cases, this can be placed in the site/local file. In most cases this file is located at $BRO_HOME/share/bro/site/local.bro.
This following example Bro script will direct all HTTP, DNS, and Conn logs to a Kafka broker running locally to a topic called bro.
@load Metron/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); redef Kafka::kafka_broker_list = "localhost:9092"; redef Kafka::topic_name = "bro";
Next, simply start Bro. If you are starting Bro with broctl after updating local.bro you will need to run install and then start.
$ /usr/local/bro/bin/broctl Welcome to BroControl 1.4 Type "help" for help. [BroControl] > install removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating standalone-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... [BroControl] > start starting bro ...
The plugin does not interfere with the existing file-based logging. To validate proper functioning compare the data being received at Kafka to the data in the local log file.