commit | abbbc9b8e049fa281dde1ff6ab159d194dc10d4a | [log] [tgz] |
---|---|---|
author | ottobackwards <ottobackwards@gmail.com> | Tue Oct 01 14:56:01 2019 -0400 |
committer | otto <otto@apache.org> | Tue Oct 01 14:56:01 2019 -0400 |
tree | 8ab94fb929d4ed12b829c91f50ea902fa2bdb982 | |
parent | e70500cd178e70dfe49c6f00c3d42f4b8eed2fea [diff] |
METRON-2269 Cannot run Docker tests if src is not a git repo (ottobackwards) closes apache/metron-bro-plugin-kafka#37
A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.
This software is a part of the Apache Metron project which integrates a variety of open source, big data technologies to offer a platform to detect and respond to cyber threats at-scale.
bro-pkg
Installationbro-pkg
is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to activate and configure the plugin after your installation.
Install librdkafka, a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.11.5.
In order to use this plugin within a kerberized Kafka environment, you will also need libsasl2
installed and will need to pass --enable-sasl
to the configure
script.
$ curl -L https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz | tar xvz $ cd librdkafka-0.11.5/ $ ./configure --enable-sasl $ make $ sudo make install
Configure bro-pkg
by following the quickstart guide here.
Install the plugin using bro-pkg install
.
$ bro-pkg install apache/metron-bro-plugin-kafka --version master The following packages will be INSTALLED: bro/apache/metron-bro-plugin-kafka (master) Verify the following REQUIRED external dependencies: (Ensure their installation on all relevant systems before proceeding): from bro/apache/metron-bro-plugin-kafka (master): librdkafka ~0.11.5 Proceed? [Y/n] bro/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib] Saved answers to config file: /home/jonzeolla/.bro-pkg/config Running unit tests for "bro/apache/metron-bro-plugin-kafka" all 10 tests successful Installing "bro/apache/metron-bro-plugin-kafka"........ Installed "bro/apache/metron-bro-plugin-kafka" (master) Loaded "bro/apache/metron-bro-plugin-kafka"
Run the following command to ensure that the plugin was installed successfully.
$ bro -N Apache::Kafka Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)
Manually installing the plugin should only occur in situations where installing and configuring bro-pkg
is not reasonable, such as in a docker container. If you are running bro in an environment where you do not have Internet connectivity, investigate bundles or creating an internal package source.
These instructions could also be helpful if you were interested in distributing this as a package (such as a deb or rpm).
Install librdkafka, a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.11.5.
In order to use this plugin within a kerberized Kafka environment, you will also need libsasl2
installed and will need to pass --enable-sasl
to the configure
script.
$ curl -L https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz | tar xvz $ cd librdkafka-0.11.5/ $ ./configure --enable-sasl $ make $ sudo make install
Build the plugin using the following commands.
$ ./configure --bro-dist=$BRO_SRC $ make $ sudo make install
Run the following command to ensure that the plugin was installed successfully.
$ bro -N Apache::Kafka Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)
The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your local.bro
file (for example, /usr/share/bro/site/local.bro
) as shown to demonstrate the example.
The goal in this example is to send all HTTP and DNS records to a Kafka topic named bro
.
kafka_conf
configuration table.topic_name
will default to send all records to a single Kafka topic called ‘bro’.logs_to_send
will send the HTTP and DNS records to the brokers specified in your Kafka::kafka_conf
.@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "server1.example.com:9092,server2.example.com:9092" );
This plugin has the ability send all active logs to the “bro” kafka topic with the following configuration.
@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::send_all_active_logs = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:9092" );
You can also specify a blacklist of bro logs to ensure they aren't being sent to kafka regardless of the Kafka::send_all_active_logs
and Kafka::logs_to_send
configurations. In this example, we will send all of the enabled logs except for the Conn log.
@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::send_all_active_logs = T; redef Kafka::logs_to_exclude = set(Conn::LOG); redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:9092" );
It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named http
and all DNS records to a separate Kafka topic named dns
.
topic_name
value must be set to an empty string.$path
value of Bro's Log Writer mechanism is used to define the topic name.$config
configuration table.@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::topic_name = ""; redef Kafka::tag_json = T; event bro_init() &priority=-10 { # handles HTTP local http_filter: Log::Filter = [ $name = "kafka-http", $writer = Log::WRITER_KAFKAWRITER, $config = table( ["metadata.broker.list"] = "localhost:9092" ), $path = "http" ]; Log::add_filter(HTTP::LOG, http_filter); # handles DNS local dns_filter: Log::Filter = [ $name = "kafka-dns", $writer = Log::WRITER_KAFKAWRITER, $config = table( ["metadata.broker.list"] = "localhost:9092" ), $path = "dns" ]; Log::add_filter(DNS::LOG, dns_filter); }
You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn't support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are multiple ways to approach this). In this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration, such that:
bro
topic.http
, dns
, or conn
), by setting Kafka::tag_json
to true.@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::tag_json = T; event bro_init() &priority=-10 { # handles HTTP Log::add_filter(HTTP::LOG, [ $name = "kafka-http", $writer = Log::WRITER_KAFKAWRITER, $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, $config = table( ["metadata.broker.list"] = "localhost:9092" ) ]); # handles DNS Log::add_filter(DNS::LOG, [ $name = "kafka-dns", $writer = Log::WRITER_KAFKAWRITER, $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, $config = table( ["metadata.broker.list"] = "localhost:9092" ) ]); # handles Conn Log::add_filter(Conn::LOG, [ $name = "kafka-conn", $writer = Log::WRITER_KAFKAWRITER, $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, $config = table( ["metadata.broker.list"] = "localhost:9092" ) ]); }
logs_to_send
is mutually exclusive with $pred
, thus for each log you want to set $pred
on, you must individually setup a Log::add_filter
and refrain from including that log in logs_to_send
.is_v6_subnet()
bro function in your predicate, as of their 2.5 release, however the above example should work on bro 2.4 and newer, which has been the focus of the kafka plugin.You are able to send a single bro log to multiple different kafka topics in the same kafka cluster by overriding the default topic (configured with Kafka::topic_name
) by creating a custom bro Log::Filter
. In this example, the DHCP, RADIUS, and DNS logs are sent to the “bro” topic; the RADIUS log is duplicated to the “shew_bro_radius” topic; and the DHCP log is duplicated to the “shew_bro_dhcp” topic.
@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "server1.example.com:9092,server2.example.com:9092" ); redef Kafka::tag_json = T; event bro_init() &priority=-10 { # Send RADIUS to the shew_bro_radius topic local shew_radius_filter: Log::Filter = [ $name = "kafka-radius-shew", $writer = Log::WRITER_KAFKAWRITER, $path = "shew_bro_radius" $config = table(["topic_name"] = "shew_bro_radius") ]; Log::add_filter(RADIUS::LOG, shew_radius_filter); # Send DHCP to the shew_bro_dhcp topic local shew_dhcp_filter: Log::Filter = [ $name = "kafka-dhcp-shew", $writer = Log::WRITER_KAFKAWRITER, $path = "shew_bro_dhcp" $config = table(["topic_name"] = "shew_bro_dhcp") ]; Log::add_filter(DHCP::LOG, shew_dhcp_filter); }
Note: Because Kafka::tag_json
is set to True in this example, the value of $path
is used as the tag for each Log::Filter
. If you were to add a log filter with the same $path
as an existing filter, Bro will append “-N”, where N is an integer starting at 2, to the end of the log path so that each filter has its own unique log path. For instance, the second instance of conn
would become conn-2
.
logs_to_send
A set of logs to send to kafka.
redef Kafka::logs_to_send = set(Conn::LOG, DHCP::LOG);
send_all_active_logs
If true, all active logs will be sent to kafka other than those specified in logs_to_exclude
.
redef Kafka::send_all_active_logs = T;
logs_to_exclude
A set of logs to exclude from being sent to kafka.
redef Kafka::logs_to_exclude = set(Conn::LOG, DNS::LOG);
topic_name
The name of the topic in Kafka where all Bro logs will be sent to.
redef Kafka::topic_name = "bro";
kafka_conf
The global configuration settings for Kafka. These values are passed through directly to librdkafka. Any valid librdkafka settings can be defined in this table. The full set of valid librdkafka settings are available here.
redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:9092", ["client.id"] = "bro" );
tag_json
If true, a log stream identifier is appended to each JSON-formatted message. For example, a Conn::LOG message will look like { 'conn' : { ... }}
.
redef Kafka::tag_json = T;
json_timestamps
Uses Ascii log writer for timestamp format. Default is JSON::TS_EPOCH
. Other options are JSON::TS_MILLIS
and JSON::TS_ISO8601
.
redef Kafka::json_timestamps = JSON::TS_ISO8601;
max_wait_on_shutdown
The maximum number of milliseconds that the plugin will wait for any backlog of queued messages to be sent to Kafka before forced shutdown.
redef Kafka::max_wait_on_shutdown = 3000;
debug
A comma separated list of debug contexts in librdkafka which you want to enable. The available contexts are:
This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set.
If you are using SASL as a security protocol for kafka, then you must have libsasl or libsasl2 installed. You can tell if sasl is enabled by running the following from the directory in which you have build librdkafka:
examples/rdkafka_example -X builtin.features builtin.features = gzip,snappy,ssl,sasl,regex
As stated above, you can configure the producer kafka configs in ${BRO_HOME}/share/bro/site/local.bro
. There are a few configs necessary to set, which are described here. For an environment where the following is true:
node1:6667
SASL_PLAINTEXT
as the security protocolmetron
keytabmetron
is metron@EXAMPLE.COM
The kafka topic bro
has been given permission for the metron
user to write:
# login using the metron user kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM ${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
The following is how the ${BRO_HOME}/share/bro/site/local.bro
looks:
@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667" , ["security.protocol"] = "SASL_PLAINTEXT" , ["sasl.kerberos.keytab"] = "/etc/security/keytabs/metron.headless.keytab" , ["sasl.kerberos.principal"] = "metron@EXAMPLE.COM" );
If you are interested in contributing to this plugin, please see the Apache Metron CONTRIBUTING.md.