| |
| =============================== |
| Writing Logging Output to Kafka |
| =============================== |
| |
| A log writer that sends logging output to Kafka. This provides a convenient |
| means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to |
| process the data generated by Bro. |
| |
| .. contents:: |
| |
| Installation |
| ------------ |
| |
| Install librdkafka (https://github.com/edenhill/librdkafka), a native client |
| library for Kafka. This plugin has been tested against the latest release of |
| librdkafka, which at the time of this writing is v0.8.6. |
| |
| # curl -L https://github.com/edenhill/librdkafka/archive/0.8.6.tar.gz | tar xvz |
| # cd librdkafka-0.8.6/ |
| # ./configure |
| # make |
| # sudo make install |
| |
| Then compile this Bro plugin using the following commands. |
| |
| # ./configure --bro-dist=$BRO_SRC |
| # make |
| # sudo make install |
| |
| Run the following command to ensure that the plugin was installed successfully. |
| |
| # bro -N Bro::Kafka |
| Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1) |
| |
| Activation |
| ---------- |
| |
| The easiest way to enable Kafka output is to load the plugin's |
| ``logs-to-kafka.bro`` script. If you are using BroControl, the following lines |
| added to local.bro will activate it. |
| |
| .. console:: |
| |
| @load Bro/Kafka/logs-to-kafka.bro |
| redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); |
| redef Kafka::topic_name = "bro"; |
| redef Kafka::kafka_conf = table( |
| ["metadata.broker.list"] = "localhost:9092" |
| ); |
| |
| This example will send all HTTP, DNS, and Conn logs to a Kafka broker running on |
| the localhost to a topic called ``bro``. Any configuration value accepted by |
| librdkafka can be added to the ``kafka_conf`` configuration table. |
| |
| Settings |
| -------- |
| |
| ``kafka_conf`` |
| |
| The global configuration settings for Kafka. These values are passed through |
| directly to librdkafka. Any valid librdkafka settings can be defined in this |
| table. |
| |
| .. console:: |
| |
| redef Kafka::kafka_conf = table( |
| ["metadata.broker.list"] = "localhost:9092", |
| ["client.id"] = "bro" |
| ); |
| |
| ``topic_name`` |
| |
| The name of the topic in Kafka where all Bro logs will be sent to. |
| |
| .. console:: |
| |
| redef Kafka::topic_name = "bro"; |
| |
| ``max_wait_on_shutdown`` |
| |
| The maximum number of milliseconds that the plugin will wait for any backlog of |
| queued messages to be sent to Kafka before forced shutdown. |
| |
| .. console:: |
| |
| redef Kafka::max_wait_on_shutdown = 3000; |
| |
| ``tag_json`` |
| |
| If true, a log stream identifier is appended to each JSON-formatted message. For |
| example, a Conn::LOG message will look like ``{ 'conn' : { ... }}``. |
| |
| .. console:: |
| |
| redef Kafka::tag_json = T; |