| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-pcap-backend/index.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Metron PCAP Backend</title> |
| <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../css/site.css" /> |
| <link rel="stylesheet" href="../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Metron PCAP Backend</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> |
| <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> |
| <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> |
| <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> |
| <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> |
| <li><a href="../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li> |
| <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> |
| <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> |
| <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> |
| <li><a href="../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-right"></span>Parsing</a></li> |
| <li class="active"><a href="#"><span class="none"></span>Pcap-backend</a></li> |
| <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> |
| <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> |
| </ul> |
| </li> |
| <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Metron PCAP Backend</h1> |
| <p><a name="Metron_PCAP_Backend"></a></p> |
| <p>The purpose of the Metron PCAP backend is to create a storm topology capable of rapidly ingesting raw packet capture data directly into HDFS from Kafka.</p> |
| <ul> |
| |
| <li><a href="#the-sensors-feeding-kafka">Sensors</a></li> |
| <li><a href="#the-pcap-topology">PCAP Topology</a></li> |
| <li><a href="#the-files-on-hdfs">HDFS Files</a></li> |
| <li><a href="#Configuration">Configuration</a></li> |
| <li><a href="#Starting_the_Topology">Starting the Topology</a></li> |
| <li><a href="#Utilities">Utilities</a> |
| <ul> |
| |
| <li><a href="#Inspector_Utility">Inspector Utility</a></li> |
| <li><a href="#Query_Filter_Utility">Query Filter Utility</a></li> |
| </ul> |
| </li> |
| <li><a href="#Performance_Tuning">Performance Tuning</a></li> |
| </ul> |
| <div class="section"> |
| <h2><a name="The_Sensors_Feeding_Kafka"></a>The Sensors Feeding Kafka</h2> |
| <p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p> |
| <ul> |
| |
| <li>The pycapa <a href="../../metron-sensors/pycapa/index.html">tool</a> aimed at low-volume packet capture</li> |
| <li>The <a class="externalLink" href="http://dpdk.org/">DPDK</a> based <a href="../../metron-sensors/fastcapa/index.html">tool</a> aimed at high-volume packet capture</li> |
| </ul> |
| <p>Both of these sensors feed kafka raw packet data directly into Kafka. The format of the record structure that this component expects is the following:</p> |
| <ul> |
| |
| <li>A key which is the byte representation of a 64-bit <tt>unsigned long</tt> representing a time-unit since the unix epoch</li> |
| <li>A value which is the raw packet data without header (either global pcap header or packet header)</li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="The_PCAP_Topology"></a>The PCAP Topology</h2> |
| <p>The structure of the topology is extremely simple. In fact, it is a spout-only topology. The <tt>Storm Kafka</tt> spout is used but extended to allow a callback to be used rather than having a separate bolt.</p> |
| <p>The following happens as part of this spout for each packet:</p> |
| <ul> |
| |
| <li>A custom <tt>Scheme</tt> is used which attaches the appropriate headers to the packet (both global and packet headers) using the timestamp in the key and the raw packet data in the value.</li> |
| <li>A callback is called which appends the packet data to a sequence file in HDFS.</li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="The_Files_on_HDFS"></a>The Files on HDFS</h2> |
| <p>The sequence files on HDFS fit the following pattern: <tt>$BASE_PATH/pcap_$TOPIC_$TS_$PARTITION_$UUID</tt></p> |
| <p>where</p> |
| <ul> |
| |
| <li><tt>BASE_PATH</tt> is the base path to where pcap data is stored in HDFS</li> |
| <li><tt>TOPIC</tt> is the kafka topic</li> |
| <li><tt>TS</tt> is the timestamp, in nanoseconds since the unix epoch</li> |
| <li><tt>PARTITION</tt> is the kafka partition</li> |
| <li><tt>UUID</tt> the UUID for the storm worker</li> |
| </ul> |
| <p>These files contain a set of packet data with headers on them in sequence files.</p></div> |
| <div class="section"> |
| <h2><a name="Configuration"></a>Configuration</h2> |
| <p>The configuration properties for PCAP sensor is managed via Ambari at Services -> Metron -> Config -> PCAP tab. Note that changes to PCAP sensor config properties via Ambari requires restarting the Metron PCAP service.</p> |
| <p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/pcap.properties</tt> and the possible options are as follows:</p> |
| <ul> |
| |
| <li><tt>spout.kafka.topic.pcap</tt> : The kafka topic to listen to</li> |
| <li><tt>storm.auto.credentials</tt> : The kerberos ticket renewal. If running on a kerberized cluster, this should be <tt>['org.apache.storm.security.auth.kerberos.AutoTGT']</tt></li> |
| <li><tt>kafka.security.protocol</tt> : The security protocol to use for kafka. This should be <tt>PLAINTEXT</tt> for a non-kerberized cluster and probably <tt>SASL_PLAINTEXT</tt> for a kerberized cluster.</li> |
| <li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li> |
| <li><tt>kafka.pcap.start</tt> : One of <tt>EARLIEST</tt>, <tt>LATEST</tt>, <tt>UNCOMMITTED_EARLIEST</tt>, <tt>UNCOMMITTED_LATEST</tt> representing where to start listening on the queue.</li> |
| <li><tt>kafka.pcap.numPackets</tt> : The number of packets to keep in one file.</li> |
| <li><tt>kafka.pcap.maxTimeMS</tt> : The number of packets to keep in one file in terms of duration (in milliseconds). For instance, you may only want to keep an hour’s worth of packets in a given file.</li> |
| <li><tt>kafka.pcap.ts_scheme</tt> : One of <tt>FROM_KEY</tt> or <tt>FROM_VALUE</tt>. You really only want <tt>FROM_KEY</tt> as that fits the current tooling. <tt>FROM_VALUE</tt> assumes that fully headerized packets are coming in on the value, which is legacy.</li> |
| <li><tt>kafka.pcap.out</tt> : The directory in HDFS to store the packet capture data</li> |
| <li><tt>kafka.pcap.ts_granularity</tt> : The granularity of timing used in the timestamps. One of <tt>MILLISECONDS</tt>, <tt>MICROSECONDS</tt>, or <tt>NANOSECONDS</tt> representing milliseconds, microseconds or nanoseconds since the unix epoch (respectively).</li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="Starting_the_Topology"></a>Starting the Topology</h2> |
| <p>To assist in starting the topology, a utility script which takes no arguments has been created to make this very simple. Simply, execute <tt>$METRON_HOME/bin/start_pcap_topology.sh</tt>.</p></div> |
| <div class="section"> |
| <h2><a name="Utilities"></a>Utilities</h2> |
| <div class="section"> |
| <h3><a name="Inspector_Utility"></a>Inspector Utility</h3> |
| <p>In order to ensure that data can be read back out, a utility, <tt>$METRON_HOME/bin/pcap_inspector.sh</tt> has been created to read portions of the sequence files.</p> |
| |
| <div> |
| <div> |
| <pre class="source">usage: PcapInspector |
| -h,--help Generate Help screen |
| -i,--input <SEQ_FILE> Input sequence file on HDFS |
| -n,--num_packets <N> Number of packets to dump |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Query_Filter_Utility"></a>Query Filter Utility</h3> |
| <p>This tool exposes the two methods for filtering PCAP data via a command line tool:</p> |
| <ul> |
| |
| <li>fixed</li> |
| <li>query (via Stellar)</li> |
| </ul> |
| <p>The tool is executed via</p> |
| |
| <div> |
| <div> |
| <pre class="source">${metron_home}/bin/pcap_query.sh [fixed|query] |
| </pre></div></div> |
| |
| <div class="section"> |
| <h4><a name="Usage"></a>Usage</h4> |
| |
| <div> |
| <div> |
| <pre class="source">usage: Fixed filter options |
| -bop,--base_output_path <arg> Query result output path. Default is |
| '/tmp' |
| -bp,--base_path <arg> Base PCAP data path. Default is |
| '/apps/metron/pcap' |
| -da,--ip_dst_addr <arg> Destination IP address |
| -df,--date_format <arg> Date format to use for parsing start_time |
| and end_time. Default is to use time in |
| millis since the epoch. |
| -dp,--ip_dst_port <arg> Destination port |
| -pf,--packet_filter <arg> Packet filter regex |
| -et,--end_time <arg> Packet end time range. Default is current |
| system time. |
| -nr,--num_reducers <arg> The number of reducers to use. Default |
| is 10. |
| -h,--help Display help |
| -ps,--print_status Print the status of the job as it runs |
| -ir,--include_reverse Indicates if filter should check swapped |
| src/dest addresses and IPs |
| -p,--protocol <arg> IP Protocol |
| -sa,--ip_src_addr <arg> Source IP address |
| -sp,--ip_src_port <arg> Source port |
| -st,--start_time <arg> (required) Packet start time range. |
| -yq,--yarn_queue <arg> Yarn queue this job will be submitted to |
| </pre></div></div> |
| |
| <div> |
| <div> |
| <pre class="source">usage: Query filter options |
| -bop,--base_output_path <arg> Query result output path. Default is |
| '/tmp' |
| -bp,--base_path <arg> Base PCAP data path. Default is |
| '/apps/metron/pcap' |
| -df,--date_format <arg> Date format to use for parsing start_time |
| and end_time. Default is to use time in |
| millis since the epoch. |
| -et,--end_time <arg> Packet end time range. Default is current |
| system time. |
| -nr,--num_reducers <arg> The number of reducers to use. Default |
| is 10. |
| -h,--help Display help |
| -ps,--print_status Print the status of the job as it runs |
| -q,--query <arg> Query string to use as a filter |
| -st,--start_time <arg> (required) Packet start time range. |
| -yq,--yarn_queue <arg> Yarn queue this job will be submitted to |
| </pre></div></div> |
| |
| <p>The Query filter’s <tt>--query</tt> argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:</p> |
| <ul> |
| |
| <li><tt>packet</tt> : The packet data (a <tt>byte[]</tt>)</li> |
| <li><tt>ip_src_addr</tt> : The source address for the packet (a <tt>String</tt>)</li> |
| <li><tt>ip_src_port</tt> : The source port for the packet (an <tt>Integer</tt>)</li> |
| <li><tt>ip_dst_addr</tt> : The destination address for the packet (a <tt>String</tt>)</li> |
| <li><tt>ip_dst_port</tt> : The destination port for the packet (an <tt>Integer</tt>)</li> |
| </ul></div> |
| <div class="section"> |
| <h4><a name="Binary_Regex"></a>Binary Regex</h4> |
| <p>Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:</p> |
| <ul> |
| |
| <li>The <tt>-pf</tt> or <tt>--packet_filter</tt> options for the fixed query filter</li> |
| <li>The <tt>BYTEARRAY_MATCHER(pattern, data)</tt> Stellar function. The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via the<tt>packet</tt> variable in Stellar.</li> |
| </ul> |
| <p>The format of this regular expression is described <a class="externalLink" href="https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md">here</a>.</p></div></div></div> |
| <div class="section"> |
| <h2><a name="Performance_Tuning"></a>Performance Tuning</h2> |
| <p>The PCAP topology is extremely lightweight and functions as a Spout-only topology. In order to tune the topology, users currently must specify a combination of properties in pcap.properties as well as configuration in the pcap remote.yaml flux file itself. Tuning the number of partitions in your Kafka topic will have a dramatic impact on performance as well. We ran data into Kafka at 1.1 Gbps and our tests resulted in configuring 128 partitions for our kakfa topic along with the following settings in pcap.properties and remote.yaml (unrelated properties for performance have been removed):</p> |
| <div class="section"> |
| <h3><a name="pcap.properties_file"></a>pcap.properties file</h3> |
| |
| <div> |
| <div> |
| <pre class="source">spout.kafka.topic.pcap=pcap |
| storm.topology.workers=16 |
| kafka.spout.parallelism=128 |
| kafka.pcap.numPackets=1000000000 |
| kafka.pcap.maxTimeMS=0 |
| hdfs.replication=1 |
| hdfs.sync.every=10000 |
| </pre></div></div> |
| |
| <p>You’ll notice that the number of kakfa partitions equals the spout parallelism, and this is no coincidence. The ordering guarantees for a partition in Kafka enforces that you may have no more consumers than 1 per topic. Any additional parallelism will leave you with dormant threads consuming resources but performing no additional work. For our cluster with 4 Storm Supervisors, we found 16 workers to provide optimal throughput as well. We were largely IO bound rather than CPU bound with the incoming PCAP data.</p></div> |
| <div class="section"> |
| <h3><a name="remote.yaml"></a>remote.yaml</h3> |
| <p>In the flux file, we introduced the following configuration:</p> |
| |
| <div> |
| <div> |
| <pre class="source">name: "pcap" |
| config: |
| topology.workers: ${storm.topology.workers} |
| topology.worker.childopts: ${topology.worker.childopts} |
| topology.auto-credentials: ${storm.auto.credentials} |
| topology.ackers.executors: 0 |
| components: |
| |
| # Any kafka props for the producer go here. |
| - id: "kafkaProps" |
| className: "java.util.HashMap" |
| configMethods: |
| - name: "put" |
| args: |
| - "value.deserializer" |
| - "org.apache.kafka.common.serialization.ByteArrayDeserializer" |
| - name: "put" |
| args: |
| - "key.deserializer" |
| - "org.apache.kafka.common.serialization.ByteArrayDeserializer" |
| - name: "put" |
| args: |
| - "group.id" |
| - "pcap" |
| - name: "put" |
| args: |
| - "security.protocol" |
| - "${kafka.security.protocol}" |
| - name: "put" |
| args: |
| - "poll.timeout.ms" |
| - 100 |
| - name: "put" |
| args: |
| - "offset.commit.period.ms" |
| - 30000 |
| - name: "put" |
| args: |
| - "session.timeout.ms" |
| - 30000 |
| - name: "put" |
| args: |
| - "max.uncommitted.offsets" |
| - 200000000 |
| - name: "put" |
| args: |
| - "max.poll.interval.ms" |
| - 10 |
| - name: "put" |
| args: |
| - "max.poll.records" |
| - 200000 |
| - name: "put" |
| args: |
| - "receive.buffer.bytes" |
| - 431072 |
| - name: "put" |
| args: |
| - "max.partition.fetch.bytes" |
| - 8097152 |
| |
| - id: "hdfsProps" |
| className: "java.util.HashMap" |
| configMethods: |
| - name: "put" |
| args: |
| - "io.file.buffer.size" |
| - 1000000 |
| - name: "put" |
| args: |
| - "dfs.blocksize" |
| - 1073741824 |
| |
| - id: "kafkaConfig" |
| className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder" |
| constructorArgs: |
| - ref: "kafkaProps" |
| # topic name |
| - "${spout.kafka.topic.pcap}" |
| - "${kafka.zk}" |
| configMethods: |
| - name: "setFirstPollOffsetStrategy" |
| args: |
| # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST |
| - ${kafka.pcap.start} |
| |
| - id: "writerConfig" |
| className: "org.apache.metron.spout.pcap.HDFSWriterConfig" |
| configMethods: |
| - name: "withOutputPath" |
| args: |
| - "${kafka.pcap.out}" |
| - name: "withNumPackets" |
| args: |
| - ${kafka.pcap.numPackets} |
| - name: "withMaxTimeMS" |
| args: |
| - ${kafka.pcap.maxTimeMS} |
| - name: "withZookeeperQuorum" |
| args: |
| - "${kafka.zk}" |
| - name: "withSyncEvery" |
| args: |
| - ${hdfs.sync.every} |
| - name: "withReplicationFactor" |
| args: |
| - ${hdfs.replication} |
| - name: "withHDFSConfig" |
| args: |
| - ref: "hdfsProps" |
| - name: "withDeserializer" |
| args: |
| - "${kafka.pcap.ts_scheme}" |
| - "${kafka.pcap.ts_granularity}" |
| spouts: |
| - id: "kafkaSpout" |
| className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout" |
| parallelism: ${kafka.spout.parallelism} |
| constructorArgs: |
| - ref: "kafkaConfig" |
| - ref: "writerConfig" |
| |
| </pre></div></div> |
| |
| <div class="section"> |
| <h4><a name="Flux_Changes_Introduced"></a>Flux Changes Introduced</h4> |
| <div class="section"> |
| <h5><a name="Topology_Configuration"></a>Topology Configuration</h5> |
| <p>The only change here is <tt>topology.ackers.executors: 0</tt>, which disables Storm tuple acking for maximum throughput.</p></div> |
| <div class="section"> |
| <h5><a name="Kafka_configuration"></a>Kafka configuration</h5> |
| |
| <div> |
| <div> |
| <pre class="source">poll.timeout.ms |
| offset.commit.period.ms |
| session.timeout.ms |
| max.uncommitted.offsets |
| max.poll.interval.ms |
| max.poll.records |
| receive.buffer.bytes |
| max.partition.fetch.bytes |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h5><a name="Writer_Configuration"></a>Writer Configuration</h5> |
| <p>This is a combination of settings for the HDFSWriter (see pcap.properties values above) as well as HDFS.</p> |
| <p><b>HDFS config</b></p> |
| <p>Component config HashMap with the following properties:</p> |
| |
| <div> |
| <div> |
| <pre class="source">io.file.buffer.size |
| dfs.blocksize |
| </pre></div></div> |
| |
| <p><b>Writer config</b></p> |
| <p>References the HDFS props component specified above.</p> |
| |
| <div> |
| <div> |
| <pre class="source"> - name: "withHDFSConfig" |
| args: |
| - ref: "hdfsProps" |
| </pre></div></div></div></div></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
