Google Git
Sign in
apache / metron / 410993daf60dfe898a973fa8a9906ddbffd83208 / . / current-book / metron-sensors / pycapa / index.html
blob: 84ec3d5d48bbc17825b3a23fa4a4d88abaa57e35 [file] [log] [blame]
<!DOCTYPE html>
<!--
| Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/pycapa/index.md at 2019-05-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="Date-Revision-yyyymmdd" content="20190514" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron &#x2013; Pycapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
<script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
<script type="text/javascript">
$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
</script>
</head>
<body class="topBarDisabled">
<div class="container-fluid">
<div id="banner">
<div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
<div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
<li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Pycapa</li>
<li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li>
<li id="projectVersion" class="pull-right">Version: 0.7.1</li>
</ul>
</div>
<div class="row-fluid">
<div id="leftColumn" class="span2">
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
<li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
<ul class="nav nav-list">
<li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
<li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
<li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
<li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
<li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
<li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li>
<li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
<li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-down"></span>Sensors</a>
<ul class="nav nav-list">
<li><a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"><span class="none"></span>Fastcapa</a></li>
<li class="active"><a href="#"><span class="none"></span>Pycapa</a></li>
</ul>
</li>
<li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
<li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
<li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
<li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
</ul>
</li>
</ul>
<hr />
<div id="poweredBy">
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
</div>
</div>
</div>
<div id="bodyColumn" class="span10" >
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div class="section">
<h2><a name="Pycapa"></a>Pycapa</h2>
<ul>
<li><a href="#Overview">Overview</a></li>
<li><a href="#Installation">Installation</a>
<ul>
<li><a href="#Centos_7">Centos 7</a></li>
<li><a href="#Centos_6">Centos 6</a></li>
</ul>
</li>
<li><a href="#Usage">Usage</a>
<ul>
<li><a href="#Parameters">Parameters</a></li>
<li><a href="#Examples">Examples</a></li>
<li><a href="#Kerberos">Kerberos</a></li>
</ul>
</li>
<li><a href="#FAQs">FAQs</a></li>
</ul></div>
<div class="section">
<h2><a name="Overview"></a>Overview</h2>
<p>Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/metron">Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor ongoing activity.</p></div>
<div class="section">
<h2><a name="Installation"></a>Installation</h2>
<p>General notes on the installation of Pycapa.</p>
<ul>
<li>Python 2.7 is required.</li>
<li>The following package dependencies are required and can be installed automatically with <tt>pip</tt>. The requirements are installed as part of step 4
<ul>
<li><a class="externalLink" href="https://github.com/confluentinc/confluent-kafka-python">confluent-kafka-python</a></li>
<li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy">pcapy</a></li>
</ul>
</li>
</ul>
<div class="section">
<h3><a name="Centos_7"></a>Centos 7</h3>
<ul>
<li>These instructions can be used directly on CentOS 7+.</li>
<li>Other Linux distributions that come with Python 2.7 can use these instructions with some minor modifications.</li>
</ul>
<ol style="list-style-type: decimal">
<li>
<p>Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.</p>
<div>
<div>
<pre class="source">yum -y install &quot;@Development tools&quot; python-devel libpcap-devel
</pre></div></div>
</li>
<li>
<p>Install Librdkafka at your chosen $PREFIX.</p>
<div>
<div>
<pre class="source">export PREFIX=/usr
wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
cd librdkafka-0.11.5/
./configure --prefix=$PREFIX
make
make install
</pre></div></div>
</li>
<li>
<p>Add Librdkafka to the dynamic library load path.</p>
<div>
<div>
<pre class="source">echo &quot;$PREFIX/lib&quot; &gt;&gt; /etc/ld.so.conf.d/pycapa.conf
ldconfig -v
</pre></div></div>
</li>
<li>
<p>Install Pycapa. This assumes that you already have the Metron source code on the host.</p>
<div>
<div>
<pre class="source">cd metron/metron-sensors/pycapa
pip install -r requirements.txt
python setup.py install
</pre></div></div>
</li>
</ol></div>
<div class="section">
<h3><a name="Centos_6"></a>Centos 6</h3>
<ul>
<li>These instructions can be used directly on CentOS 6 - useful for developers using the Full Dev Vagrant test box.</li>
<li>Older distributions, like CentOS 6, that come with Python 2.6 installed, should install Python 2.7 within a virtual environment and then run Pycapa from within the virtual environment.</li>
</ul>
<ol style="list-style-type: decimal">
<li>
<p>Set up a couple environment variables.</p>
<div>
<div>
<pre class="source">PYCAPA_HOME=/opt/pycapa
PYTHON27_HOME=/opt/rh/python27/root
</pre></div></div>
</li>
<li>
<p>Install required packages.</p>
<div>
<div>
<pre class="source">for item in epel-release centos-release-scl &quot;@Development tools&quot; python27 python27-scldevel python27-python-virtualenv libpcap-devel libselinux-python; do yum install -y $item; done
</pre></div></div>
</li>
<li>
<p>Setup Pycapa directory.</p>
<div>
<div>
<pre class="source">mkdir $PYCAPA_HOME &amp;&amp; chmod 755 $PYCAPA_HOME
</pre></div></div>
</li>
<li>
<p>Create the virtualenv.</p>
<div>
<div>
<pre class="source">export LD_LIBRARY_PATH=&quot;/opt/rh/python27/root/usr/lib64&quot;
cd $PYCAPA_HOME
${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
</pre></div></div>
</li>
<li>
<p>Install Librdkafka at your chosen $PREFIX.</p>
<div>
<div>
<pre class="source">export PREFIX=/usr
wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
cd librdkafka-0.11.5/
./configure --prefix=$PREFIX
make
make install
</pre></div></div>
</li>
<li>
<p>Add Librdkafka to the dynamic library load path.</p>
<div>
<div>
<pre class="source">echo &quot;$PREFIX/lib&quot; &gt;&gt; /etc/ld.so.conf.d/pycapa.conf
ldconfig -v
</pre></div></div>
</li>
<li>
<p>Copy the Pycapa source files from the Metron project to your chosen $PYCAPA_HOME (e.g. <tt>/opt/pycapa</tt>). You should have pycapa source files in <tt>/opt/pycapa/pycapa</tt>.</p>
<div>
<div>
<pre class="source">scp -r metron-sensors/pycapa root@node1:$PYCAPA_HOME
</pre></div></div>
</li>
<li>
<p>Install Pycapa using the <tt>pycapa-venv</tt> virtualenv you created earlier.</p>
<div>
<div>
<pre class="source">cd ${PYCAPA_HOME}/pycapa
# activate the virtualenv
source ${PYCAPA_HOME}/pycapa-venv/bin/activate
pip install -r requirements.txt
python setup.py install
</pre></div></div>
</li>
<li>
<p>Special notes on running pycapa on Centos 6. You should run it using the virtualenv.</p>
<div>
<div>
<pre class="source">cd ${PYCAPA_HOME}/pycapa-venv/bin
pycapa --producer --kafka-topic pcap --interface eth1 --kafka-broker $BROKERLIST
</pre></div></div>
</li>
</ol>
<p><b>Note:</b> To deactivate your virtualenv, simply type &#x201c;deactivate&#x201d; and hit enter.</p></div></div>
<div class="section">
<h2><a name="Usage"></a>Usage</h2>
<p>Pycapa has two primary runtime modes.</p>
<ul>
<li>
<p><b>Producer Mode</b>: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.</p>
</li>
<li>
<p><b>Consumer Mode</b>: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor activity.</p>
</li>
</ul>
<div class="section">
<h3><a name="Parameters"></a>Parameters</h3>
<div>
<div>
<pre class="source">$ pycapa --help
usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC]
[-o {begin,end,stored}] [-i NETWORK_IFACE] [-m MAX_PACKETS]
[-pp PRETTY_PRINT] [-ll LOG_LEVEL] [-X KAFKA_CONFIGS]
[-s SNAPLEN]
optional arguments:
-h, --help show this help message and exit
-p, --producer sniff packets and send to kafka
-c, --consumer read packets from kafka
-k KAFKA_BROKERS, --kafka-broker KAFKA_BROKERS
kafka broker(s) as host:port
-t KAFKA_TOPIC, --kafka-topic KAFKA_TOPIC
kafka topic
-o {begin,end,stored}, --kafka-offset {begin,end,stored}
kafka offset to consume from; default=end
-i NETWORK_IFACE, --interface NETWORK_IFACE
network interface to listen on
-m MAX_PACKETS, --max-packets MAX_PACKETS
stop after this number of packets
-pp PRETTY_PRINT, --pretty-print PRETTY_PRINT
pretty print every X packets
-ll LOG_LEVEL, --log-level LOG_LEVEL
set the log level; DEBUG, INFO, WARN
-X KAFKA_CONFIGS define a kafka client parameter; key=value
-s SNAPLEN, --snaplen SNAPLEN
capture only the first X bytes of each packet;
default=65535
</pre></div></div>
</div>
<div class="section">
<h3><a name="Examples"></a>Examples</h3>
<div class="section">
<h4><a name="Example_1"></a>Example 1</h4>
<p>Capture 10 packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost:9092</tt>. The process will not terminate until all messages have been delivered to Kafka.</p>
<div>
<div>
<pre class="source">$ pycapa --producer \
--interface eth0 \
--kafka-broker localhost:9092 \
--kafka-topic pcap \
--max-packets 10
INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
INFO:root:Starting packet capture
INFO:root:Waiting for '6' message(s) to flush
INFO:root:'10' packet(s) in, '10' packet(s) out
</pre></div></div>
</div>
<div class="section">
<h4><a name="Example_2"></a>Example 2</h4>
<p>Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.</p>
<div>
<div>
<pre class="source">$ pycapa --producer \
--interface en0 \
--kafka-broker localhost:9092 \
--kafka-topic pcap
INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
INFO:root:Starting packet capture
^C
INFO:root:Clean shutdown process started
INFO:root:Waiting for '2' message(s) to flush
INFO:root:'21' packet(s) in, '21' packet(s) out
</pre></div></div>
</div>
<div class="section">
<h4><a name="Example_3"></a>Example 3</h4>
<p>While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.</p>
<div>
<div>
<pre class="source">$ pycapa --producer \
--interface eth0 \
--kafka-broker localhost:9092 \
--kafka-topic pcap \
--pretty-print 5
INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'UAWINMBDNQEH'}
INFO:root:Starting packet capture
Packet received[5]
Packet delivered[5]: date=2017-05-08 14:48:54.474031 topic=pcap partition=0 offset=29086 len=42
Packet received[10]
Packet received[15]
Packet delivered[10]: date=2017-05-08 14:48:58.879710 topic=pcap partition=0 offset=0 len=187
Packet delivered[15]: date=2017-05-08 14:48:59.633127 topic=pcap partition=0 offset=0 len=43
Packet received[20]
Packet delivered[20]: date=2017-05-08 14:49:01.949628 topic=pcap partition=0 offset=29101 len=134
Packet received[25]
^C
INFO:root:Clean shutdown process started
Packet delivered[25]: date=2017-05-08 14:49:03.589940 topic=pcap partition=0 offset=0 len=142
INFO:root:Waiting for '1' message(s) to flush
INFO:root:'27' packet(s) in, '27' packet(s) out
</pre></div></div>
</div>
<div class="section">
<h4><a name="Example_4"></a>Example 4</h4>
<p>Consume 10 packets and create a libpcap-compliant pcap file.</p>
<div>
<div>
<pre class="source"> $ pycapa --consumer \
--kafka-broker localhost:9092 \
--kafka-topic pcap \
--max-packets 10 \
&gt; out.pcap
$ tshark -r out.pcap
1 0.000000 199.193.204.147 &#x2192; 192.168.0.3 TLSv1.2 151 Application Data
2 0.000005 199.193.204.147 &#x2192; 192.168.0.3 TLSv1.2 1191 Application Data
3 0.000088 192.168.0.3 &#x2192; 199.193.204.147 TCP 66 54788 &#x2192; 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612
4 0.000089 192.168.0.3 &#x2192; 199.193.204.147 TCP 66 54788 &#x2192; 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612
5 0.948788 192.168.0.3 &#x2192; 192.30.253.125 TCP 54 54671 &#x2192; 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0
6 1.005175 192.30.253.125 &#x2192; 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 &#x2192; 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339
7 1.636312 fe80::1286:8cff:fe0e:65df &#x2192; ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
8 2.253052 192.175.27.112 &#x2192; 192.168.0.3 TLSv1.2 928 Application Data
9 2.253140 192.168.0.3 &#x2192; 192.175.27.112 TCP 66 55078 &#x2192; 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238
10 2.494769 192.168.0.3 &#x2192; 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, &quot;QM&quot; question
</pre></div></div>
</div>
<div class="section">
<h4><a name="Example_5"></a>Example 5</h4>
<p>Consume 10 packets from the Kafka topic <tt>pcap</tt> running on <tt>localhost:9092</tt>, then pipe those into Wireshark for DPI.</p>
<div>
<div>
<pre class="source">$ pycapa --consumer \
--kafka-broker localhost:9092 \
--kafka-topic pcap \
--max-packets 10 \
| tshark -i -
Capturing on 'Standard input'
1 0.000000 ArrisGro_0e:65:df &#x2192; Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
2 0.000044 Apple_bf:0d:43 &#x2192; ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
3 0.203495 fe80::1286:8cff:fe0e:65df &#x2192; ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
4 2.031988 192.168.0.3 &#x2192; 96.27.183.249 TCP 54 55110 &#x2192; 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
5 2.035816 192.30.253.125 &#x2192; 192.168.0.3 TLSv1.2 97 Application Data
6 2.035892 192.168.0.3 &#x2192; 192.30.253.125 TCP 66 54671 &#x2192; 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
7 2.035994 192.168.0.3 &#x2192; 192.30.253.125 TLSv1.2 101 Application Data
8 2.053866 96.27.183.249 &#x2192; 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 &#x2192; 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
9 2.083872 192.30.253.125 &#x2192; 192.168.0.3 TCP 66 443 &#x2192; 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
10 3.173189 fe80::1286:8cff:fe0e:65df &#x2192; ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
10 packets captured
</pre></div></div>
</div></div>
<div class="section">
<h3><a name="Kerberos"></a>Kerberos</h3>
<p>The probe can be used in a Kerberized environment. The Python client README (<a class="externalLink" href="https://github.com/confluentinc/confluent-kafka-python">https://github.com/confluentinc/confluent-kafka-python</a>) has an important note for Kerberos case that the pre-built Linux wheels do NOT contain SASL Kerberos support. You will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p>
<ul>
<li>The Kafka broker is at <tt>kafka1:6667</tt></li>
<li>Zookeeper is at <tt>zookeeper1:2181</tt></li>
<li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li>
<li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li>
<li>The service principal is <tt>metron@EXAMPLE.COM</tt></li>
</ul>
<ol style="list-style-type: decimal">
<li>
<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p>
<div>
<div>
<pre class="source"> yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
</pre></div></div>
</li>
<li>
<p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>) and install at your chosen $PREFIX.</p>
<div>
<div>
<pre class="source">wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz
cd librdkafka-0.11.5/
./configure --prefix=$PREFIX --enable-sasl
make
make install
</pre></div></div>
</li>
<li>
<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p>
<div>
<div>
<pre class="source">$ examples/rdkafka_example -X builtin.features
builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins
</pre></div></div>
</li>
<li>The source install of confluent-kafka.
<p>If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed</p>
<div>
<div>
<pre class="source"> pip uninstall -y confluent-kafka
</pre></div></div>
<div>
<div>
<pre class="source"> pip install --no-binary :all: confluent-kafka
</pre></div></div>
</li>
<li>
<p>Grant access to your Kafka topic. In this example the topic is simply named <tt>pcap</tt>.</p>
<div>
<div>
<pre class="source">${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=zookeeper1:2181 \
--add \
--allow-principal User:metron \
--topic pcap
${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=zookeeper1:2181 \
--add \
--allow-principal User:metron \
--group pycapa
</pre></div></div>
</li>
<li>
<p>Use Pycapa as you normally would, but append the following three additional parameters</p>
<ul>
<li><tt>security.protocol</tt></li>
<li><tt>sasl.kerberos.keytab</tt></li>
<li><tt>sasl.kerberos.principal</tt>
<div>
<div>
<pre class="source">$ pycapa --producer \
--interface eth0 \
--kafka-broker kafka1:6667 \
--kafka-topic pcap --max-packets 10 \
-X security.protocol=SASL_PLAINTEXT \
-X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless .keytab \
-X sasl.kerberos.principal=metron-metron@METRONEXAMPLE.COM
INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-metron@METRONEXAMPLE.COM', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'}
INFO:root:Starting packet capture
INFO:root:Waiting for '1' message(s) to flush
INFO:root:'10' packet(s) in, '10' packet(s) out
</pre></div></div>
</li>
</ul>
</li>
</ol></div></div>
<div class="section">
<h2><a name="FAQs"></a>FAQs</h2>
<div class="section">
<h3><a name="How_do_I_get_more_logs.3F"></a>How do I get more logs?</h3>
<p>Use the following two command-line arguments to get detailed logging.</p>
<div>
<div>
<pre class="source">-X debug=all --log-level DEBUG
</pre></div></div>
</div>
<div class="section">
<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled.2C_why_do_I_get_an_error_like_.E2.80.9CNo_such_configuration_property:_.E2.80.98sasl.kerberos.principal.E2.80.99.E2.80.9D.3F"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like &#x201c;No such configuration property: &#x2018;sasl.kerberos.principal&#x2019;&#x201d;?</h3>
<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md">here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p>
<p>Bottom Line: Make sure that Pycapa is running against a version of Librdkafka with SASL support enabled.</p></div></div>
</div>
</div>
</div>
<hr/>
<footer>
<div class="container-fluid">
<div class="row-fluid">
© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
and the Apache Metron project logo are trademarks of The Apache Software Foundation.
</div>
</div>
</footer>
</body>
</html>