
===============================
Writing Logging Output to Kafka
===============================

A log writer that sends logging output to Kafka.  This provides a convenient
means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to
process the data generated by Bro.

.. contents::

Installation
------------

Install librdkafka (https://github.com/edenhill/librdkafka), a native client
library for Kafka.  This plugin has been tested against the latest release of
librdkafka, which at the time of this writing is v0.8.6.

    # curl -L https://github.com/edenhill/librdkafka/archive/0.8.6.tar.gz | tar xvz
    # cd librdkafka-0.8.6/
    # ./configure
    # make
    # sudo make install

Then compile this Bro plugin using the following commands.

    # ./configure --bro-dist=$BRO_SRC
    # make
    # sudo make install

Run the following command to ensure that the plugin was installed successfully.

    # bro -N Bro::Kafka
    Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)

Activation
----------

The easiest way to enable Kafka output is to load the plugin's
``logs-to-kafka.bro`` script.  If you are using BroControl, the following lines
added to local.bro will activate it.

.. console::

    @load Bro/Kafka/logs-to-kafka.bro
    redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
    redef Kafka::topic_name = "bro";
    redef Kafka::kafka_conf = table(
        ["metadata.broker.list"] = "localhost:9092"
    );

This example will send all HTTP, DNS, and Conn logs to a Kafka broker running on
the localhost to a topic called ``bro``. Any configuration value accepted by
librdkafka can be added to the ``kafka_conf`` configuration table.

Settings
--------

``kafka_conf``

The global configuration settings for Kafka.  These values are passed through
directly to librdkafka.  Any valid librdkafka settings can be defined in this
table.

.. console::

    redef Kafka::kafka_conf = table(
        ["metadata.broker.list"] = "localhost:9092",
        ["client.id"] = "bro"
    );

``topic_name``

The name of the topic in Kafka where all Bro logs will be sent to.

.. console::

    redef Kafka::topic_name = "bro";

``max_wait_on_shutdown``

The maximum number of milliseconds that the plugin will wait for any backlog of
queued messages to be sent to Kafka before forced shutdown.

.. console::

    redef Kafka::max_wait_on_shutdown = 3000;

``tag_json``

If true, a log stream identifier is appended to each JSON-formatted message. For
example, a Conn::LOG message will look like ``{ 'conn' : { ... }}``.

.. console::

    redef Kafka::tag_json = T;