current-book/metron-platform/metron-enrichment/metron-enrichment-storm/index.html - metron - Git at Google

 <!DOCTYPE html>
 <!--
  | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/metron-enrichment-storm/index.md at 2019-05-14
  | Rendered using Apache Maven Fluido Skin 1.7
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
     <meta name="Date-Revision-yyyymmdd" content="20190514" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Metron &#x2013; Enrichment</title>
     <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
     <link rel="stylesheet" href="../../../css/site.css" />
     <link rel="stylesheet" href="../../../css/print.css" media="print" />
     <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script>
 <script type="text/javascript">
               $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
             </script>
   </head>
   <body class="topBarDisabled">
     <div class="container-fluid">
       <div id="banner">
         <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../images/metron-logo.png"  alt="Apache Metron" width="148px" height="48px"/></a></div>
         <div class="pull-right"></div>
         <div class="clear"><hr/></div>
       </div>

       <div id="breadcrumbs">
         <ul class="breadcrumb">
       <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
       <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
       <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
     <li class="active ">Enrichment</li>
         <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li>
           <li id="projectVersion" class="pull-right">Version: 0.7.1</li>
         </ul>
       </div>
       <div class="row-fluid">
         <div id="leftColumn" class="span2">
           <div class="well sidebar-nav">
     <ul class="nav nav-list">
       <li class="nav-header">User Documentation</li>
     <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
     <ul class="nav nav-list">
     <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
     <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
     <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
     <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
     <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
     <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
     <li><a href="../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li>
     <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a>
     <ul class="nav nav-list">
     <li><a href="../../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li>
     <li><a href="../../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li>
     <li><a href="../../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li>
     <li><a href="../../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
     <li><a href="../../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-down"></span>Enrichment</a>
     <ul class="nav nav-list">
     <li><a href="../../../metron-platform/metron-enrichment/metron-enrichment-common/index.html" title="Enrichment-common"><span class="none"></span>Enrichment-common</a></li>
     <li class="active"><a href="#"><span class="icon-chevron-down"></span>Enrichment-storm</a>
     <ul class="nav nav-list">
     <li><a href="../../../metron-platform/metron-enrichment/metron-enrichment-storm/Performance.html" title="Performance"><span class="none"></span>Performance</a></li>
     </ul>
 </li>
     </ul>
 </li>
     <li><a href="../../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li>
     <li><a href="../../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li>
     <li><a href="../../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li>
     <li><a href="../../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li>
     <li><a href="../../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-right"></span>Parsing</a></li>
     <li><a href="../../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
     <li><a href="../../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li>
     <li><a href="../../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li>
     </ul>
 </li>
     <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
     <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
     <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
     <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
     <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
     </ul>
 </li>
 </ul>
           <hr />
           <div id="poweredBy">
             <div class="clear"></div>
             <div class="clear"></div>
             <div class="clear"></div>
             <div class="clear"></div>
 <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a>
             </div>
           </div>
         </div>
         <div id="bodyColumn"  class="span10" >
 <!--
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
 <h1>Enrichment</h1>
 <p><a name="Enrichment"></a></p>
 <div class="section">
 <h2><a name="Introduction"></a>Introduction</h2>
 <p>This module holds code for enrichments running in a Storm topology.</p></div>
 <div class="section">
 <h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2>
 <p><img src="../../../images/unified_enrichment_arch.svg" alt="Unified Architecture" /></p>
 <div class="section">
 <h3><a name="Unified_Enrichment_Topology"></a>Unified Enrichment Topology</h3>
 <p>The unified enrichment topology uses data parallelism as opposed to the deprecated split/join topology&#x2019;s task parallelism. This architecture uses a worker pool to fully enrich any message within a worker.  This results in</p>
 <ul>

 <li>Fewer bolts in the topology</li>
 <li>Each bolt fully operates on a message.</li>
 <li>Fewer network hops</li>
 </ul>
 <p>This architecture is fully backwards compatible with the old split-join topology; the only difference is how the enrichment will operate on each message (in one bolt where the split/join is done in a threadpool as opposed to split across multiple bolts).</p>
 <div class="section">
 <h4><a name="Configuring_It"></a>Configuring It</h4>
 <p>There are two parameters which you might want to tune in this topology. Both of them are topology configuration adjustable in the flux file <tt>$METRON_HOME/config/flux/enrichment/remote-unified.yaml</tt>:</p>
 <ul>

 <li><tt>metron.threadpool.size</tt> : The size of the threadpool.  This can take a number or a multiple of the number of cores (e.g. <tt>5C</tt> to 5 times the number of cores).  The default is <tt>2C</tt>.</li>
 <li><tt>metron.threadpool.type</tt> : The type of threadpool. (note: descriptions taken from <a class="externalLink" href="https://zeroturnaround.com/rebellabs/fixedthreadpool-cachedthreadpool-or-forkjoinpool-picking-correct-java-executors-for-background-tasks/">here</a>).
 <ul>

 <li><tt>FIXED</tt> is a fixed threadpool of size <tt>n</tt>. <tt>n</tt> threads will process tasks at the time, when the pool is saturated, new tasks will get added to a queue without a limit on size. Good for CPU intensive tasks.  This is the default.</li>
 <li><tt>WORK_STEALING</tt> is a work stealing threadpool.  This will create and shut down threads dynamically to accommodate the required parallelism level. It also tries to reduce the contention on the task queue, so can be really good in heavily loaded environments. Also good when your tasks create more tasks for the executor, like recursive tasks.</li>
 </ul>
 </li>
 </ul>
 <p>In order to configure the parallelism for the enrichment bolt and threat intel bolt, the configurations will be taken from the respective join bolt parallelism.  When proper ambari support for this is added, we will add its own property.</p></div></div>
 <div class="section">
 <h3><a name="Split-Join_Enrichment_Topology"></a>Split-Join Enrichment Topology</h3>
 <p>The now-deprecated split/join topology is also available and performs enrichments in parallel. This poses some issues in terms of ease of tuning and reasoning about performance.</p>
 <p><img src="../../../images/enrichment_arch.png" alt="Architecture" /></p>
 <div class="section">
 <h4><a name="Using_It"></a>Using It</h4>
 <p>In order to use the older, deprecated topology, you will need to</p>
 <ul>

 <li>Edit <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> and adjust it to use <tt>remote-splitjoin.yaml</tt> instead of <tt>remote-unified.yaml</tt></li>
 <li>Restart the enrichment topology.</li>
 </ul></div></div></div>
 <div class="section">
 <h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2>
 <p>The configuration for the <tt>enrichment</tt> topology, the topology primarily responsible for enrichment and threat intelligence enrichment, is defined by JSON documents stored in zookeeper.</p>
 <p>See <a href="../metron-enrichment-common/index.html#Enrichment_Configuration">Enrichment Configuration</a></p>
 <p><a name="Example_Enrichment_via_Stellar"></a></p>
 <h1>Example Enrichment via Stellar</h1>
 <p>Let&#x2019;s walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.</p></div>
 <div class="section">
 <h2><a name="Install_Prerequisites"></a>Install Prerequisites</h2>
 <p>Now let&#x2019;s install some prerequisites:</p>
 <ul>

 <li>Squid client via <tt>yum install squid</tt></li>
 <li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></li>
 </ul>
 <p>Start Squid via <tt>service squid start</tt></p></div>
 <div class="section">
 <h2><a name="Adjust_Enrichment_Configurations_for_Squid_to_Call_Stellar"></a>Adjust Enrichment Configurations for Squid to Call Stellar</h2>
 <p>Let&#x2019;s adjust the configurations for the Squid topology to annotate the messages using some Stellar functions.</p>
 <ul>

 <li>Edit the squid enrichment configuration at <tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will not exist, so create a new one) to add some new fields based on stellar queries:</li>
 </ul>

 <div>
 <div>
 <pre class="source">{
   &quot;enrichment&quot; : {
     &quot;fieldMap&quot;: {
       &quot;stellar&quot; : {
         &quot;config&quot; : {
           &quot;numeric&quot; : {
                       &quot;foo&quot;: &quot;1 + 1&quot;
                       }
           ,&quot;ALL_CAPS&quot; : &quot;TO_UPPER(source.type)&quot;
         }
       }
      }
   },
   &quot;threatIntel&quot; : {
     &quot;fieldMap&quot;:{
      &quot;stellar&quot; : {
         &quot;config&quot; : {
           &quot;bar&quot; : &quot;TO_UPPER(source.type)&quot;
         }
       }
     },
     &quot;triageConfig&quot; : {
     }
   }
 }
 </pre></div></div>

 <p>We have added the following fields as part of the enrichment phase of the enrichment topology:</p>
 <ul>

 <li><tt>foo</tt> ==  2</li>
 <li><tt>ALL_CAPS</tt> == SQUID</li>
 </ul>
 <p>We have added the following as part of the threat intel:</p>
 <ul>

 <li><tt>bar</tt> == SQUID</li>
 </ul>
 <p>Please note that foo and ALL_CAPS will be applied in separate workers due to them being in separate groups.</p>
 <ul>

 <li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z node1:2181</tt></li>
 <li>Make the Squid topic in kafka via <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1</tt></li>
 </ul></div>
 <div class="section">
 <h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send Data</h2>
 <p>Now we need to start the topologies and send some data:</p>
 <ul>

 <li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid</tt></li>
 <li>Generate some data via the squid client:
 <ul>

 <li><tt>squidclient http://yahoo.com</tt></li>
 <li><tt>squidclient http://cnn.com</tt></li>
 </ul>
 </li>
 <li>Send the data to kafka via <tt>cat /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid</tt></li>
 <li>Browse the data in elasticsearch via the ES Head plugin @ <a class="externalLink" href="http://node1:9200/_plugin/head/">http://node1:9200/_plugin/head/</a> and verify that in the squid index you have two documents</li>
 <li>Ensure that the documents have new fields <tt>foo</tt>, <tt>bar</tt> and <tt>ALL_CAPS</tt> with values as described above.</li>
 </ul>
 <p>Note that we could have used any Stellar statements here, including calling out to HBase via <tt>ENRICHMENT_GET</tt> and <tt>ENRICHMENT_EXISTS</tt> or even calling a machine learning model via <a href="../../../metron-analytics/metron-maas-service/index.html">Model as a Service</a>.</p></div>
         </div>
       </div>
     </div>
     <hr/>
     <footer>
       <div class="container-fluid">
         <div class="row-fluid">
 Â© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
             and the Apache Metron project logo are trademarks of The Apache Software Foundation.
         </div>
       </div>
     </footer>
   </body>
 </html>
	<!DOCTYPE html>
	<!--
	\| Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/metron-enrichment-storm/index.md at 2019-05-14
	\| Rendered using Apache Maven Fluido Skin 1.7
	-->
	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
	<meta charset="UTF-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1.0" />
	<meta name="Date-Revision-yyyymmdd" content="20190514" />
	<meta http-equiv="Content-Language" content="en" />
	<title>Metron – Enrichment</title>
	<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
	<link rel="stylesheet" href="../../../css/site.css" />
	<link rel="stylesheet" href="../../../css/print.css" media="print" />
	<script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script>
	<script type="text/javascript">
	$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
	</script>
	</head>
	<body class="topBarDisabled">
	<div class="container-fluid">
	<div id="banner">
	<div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
	<div class="pull-right"></div>
	<div class="clear"><hr/></div>
	</div>

	<div id="breadcrumbs">
	<ul class="breadcrumb">
	<li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
	<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
	<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
	<li class="active ">Enrichment</li>
	<li id="publishDate" class="pull-right"><span class="divider">\|</span> Last Published: 2019-05-14</li>
	<li id="projectVersion" class="pull-right">Version: 0.7.1</li>
	</ul>
	</div>
	<div class="row-fluid">
	<div id="leftColumn" class="span2">
	<div class="well sidebar-nav">
	<ul class="nav nav-list">
	<li class="nav-header">User Documentation</li>
	<li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
	<ul class="nav nav-list">
	<li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
	<li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
	<li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
	<li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
	<li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
	<li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
	<li><a href="../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li>
	<li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a>
	<ul class="nav nav-list">
	<li><a href="../../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li>
	<li><a href="../../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li>
	<li><a href="../../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li>
	<li><a href="../../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
	<li><a href="../../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-down"></span>Enrichment</a>
	<ul class="nav nav-list">
	<li><a href="../../../metron-platform/metron-enrichment/metron-enrichment-common/index.html" title="Enrichment-common"><span class="none"></span>Enrichment-common</a></li>
	<li class="active"><a href="#"><span class="icon-chevron-down"></span>Enrichment-storm</a>
	<ul class="nav nav-list">
	<li><a href="../../../metron-platform/metron-enrichment/metron-enrichment-storm/Performance.html" title="Performance"><span class="none"></span>Performance</a></li>
	</ul>
	</li>
	</ul>
	</li>
	<li><a href="../../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li>
	<li><a href="../../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li>
	<li><a href="../../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li>
	<li><a href="../../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li>
	<li><a href="../../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-right"></span>Parsing</a></li>
	<li><a href="../../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
	<li><a href="../../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li>
	<li><a href="../../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li>
	</ul>
	</li>
	<li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
	<li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
	<li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
	<li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
	<li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
	</ul>
	</li>
	</ul>
	<hr />
	<div id="poweredBy">
	<div class="clear"></div>
	<div class="clear"></div>
	<div class="clear"></div>
	<div class="clear"></div>
	<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a>
	</div>
	</div>
	</div>
	<div id="bodyColumn" class="span10" >
	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->
	<h1>Enrichment</h1>
	<p><a name="Enrichment"></a></p>
	<div class="section">
	<h2><a name="Introduction"></a>Introduction</h2>
	<p>This module holds code for enrichments running in a Storm topology.</p></div>
	<div class="section">
	<h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2>
	<p><img src="../../../images/unified_enrichment_arch.svg" alt="Unified Architecture" /></p>
	<div class="section">
	<h3><a name="Unified_Enrichment_Topology"></a>Unified Enrichment Topology</h3>
	<p>The unified enrichment topology uses data parallelism as opposed to the deprecated split/join topology’s task parallelism. This architecture uses a worker pool to fully enrich any message within a worker. This results in</p>
	<ul>

	<li>Fewer bolts in the topology</li>
	<li>Each bolt fully operates on a message.</li>
	<li>Fewer network hops</li>
	</ul>
	<p>This architecture is fully backwards compatible with the old split-join topology; the only difference is how the enrichment will operate on each message (in one bolt where the split/join is done in a threadpool as opposed to split across multiple bolts).</p>
	<div class="section">
	<h4><a name="Configuring_It"></a>Configuring It</h4>
	<p>There are two parameters which you might want to tune in this topology. Both of them are topology configuration adjustable in the flux file <tt>$METRON_HOME/config/flux/enrichment/remote-unified.yaml</tt>:</p>
	<ul>

	<li><tt>metron.threadpool.size</tt> : The size of the threadpool. This can take a number or a multiple of the number of cores (e.g. <tt>5C</tt> to 5 times the number of cores). The default is <tt>2C</tt>.</li>
	<li><tt>metron.threadpool.type</tt> : The type of threadpool. (note: descriptions taken from <a class="externalLink" href="https://zeroturnaround.com/rebellabs/fixedthreadpool-cachedthreadpool-or-forkjoinpool-picking-correct-java-executors-for-background-tasks/">here</a>).
	<ul>

	<li><tt>FIXED</tt> is a fixed threadpool of size <tt>n</tt>. <tt>n</tt> threads will process tasks at the time, when the pool is saturated, new tasks will get added to a queue without a limit on size. Good for CPU intensive tasks. This is the default.</li>
	<li><tt>WORK_STEALING</tt> is a work stealing threadpool. This will create and shut down threads dynamically to accommodate the required parallelism level. It also tries to reduce the contention on the task queue, so can be really good in heavily loaded environments. Also good when your tasks create more tasks for the executor, like recursive tasks.</li>
	</ul>
	</li>
	</ul>
	<p>In order to configure the parallelism for the enrichment bolt and threat intel bolt, the configurations will be taken from the respective join bolt parallelism. When proper ambari support for this is added, we will add its own property.</p></div></div>
	<div class="section">
	<h3><a name="Split-Join_Enrichment_Topology"></a>Split-Join Enrichment Topology</h3>
	<p>The now-deprecated split/join topology is also available and performs enrichments in parallel. This poses some issues in terms of ease of tuning and reasoning about performance.</p>
	<p><img src="../../../images/enrichment_arch.png" alt="Architecture" /></p>
	<div class="section">
	<h4><a name="Using_It"></a>Using It</h4>
	<p>In order to use the older, deprecated topology, you will need to</p>
	<ul>

	<li>Edit <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> and adjust it to use <tt>remote-splitjoin.yaml</tt> instead of <tt>remote-unified.yaml</tt></li>
	<li>Restart the enrichment topology.</li>
	</ul></div></div></div>
	<div class="section">
	<h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2>
	<p>The configuration for the <tt>enrichment</tt> topology, the topology primarily responsible for enrichment and threat intelligence enrichment, is defined by JSON documents stored in zookeeper.</p>
	<p>See <a href="../metron-enrichment-common/index.html#Enrichment_Configuration">Enrichment Configuration</a></p>
	<p><a name="Example_Enrichment_via_Stellar"></a></p>
	<h1>Example Enrichment via Stellar</h1>
	<p>Let’s walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.</p></div>
	<div class="section">
	<h2><a name="Install_Prerequisites"></a>Install Prerequisites</h2>
	<p>Now let’s install some prerequisites:</p>
	<ul>

	<li>Squid client via <tt>yum install squid</tt></li>
	<li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></li>
	</ul>
	<p>Start Squid via <tt>service squid start</tt></p></div>
	<div class="section">
	<h2><a name="Adjust_Enrichment_Configurations_for_Squid_to_Call_Stellar"></a>Adjust Enrichment Configurations for Squid to Call Stellar</h2>
	<p>Let’s adjust the configurations for the Squid topology to annotate the messages using some Stellar functions.</p>
	<ul>

	<li>Edit the squid enrichment configuration at <tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will not exist, so create a new one) to add some new fields based on stellar queries:</li>
	</ul>

	<div>
	<div>
	<pre class="source">{
	"enrichment" : {
	"fieldMap": {
	"stellar" : {
	"config" : {
	"numeric" : {
	"foo": "1 + 1"
	}
	,"ALL_CAPS" : "TO_UPPER(source.type)"
	}
	}
	}
	},
	"threatIntel" : {
	"fieldMap":{
	"stellar" : {
	"config" : {
	"bar" : "TO_UPPER(source.type)"
	}
	}
	},
	"triageConfig" : {
	}
	}
	}
	</pre></div></div>

	<p>We have added the following fields as part of the enrichment phase of the enrichment topology:</p>
	<ul>

	<li><tt>foo</tt> == 2</li>
	<li><tt>ALL_CAPS</tt> == SQUID</li>
	</ul>
	<p>We have added the following as part of the threat intel:</p>
	<ul>

	<li><tt>bar</tt> == SQUID</li>
	</ul>
	<p>Please note that foo and ALL_CAPS will be applied in separate workers due to them being in separate groups.</p>
	<ul>

	<li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z node1:2181</tt></li>
	<li>Make the Squid topic in kafka via <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1</tt></li>
	</ul></div>
	<div class="section">
	<h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send Data</h2>
	<p>Now we need to start the topologies and send some data:</p>
	<ul>

	<li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid</tt></li>
	<li>Generate some data via the squid client:
	<ul>

	<li><tt>squidclient http://yahoo.com</tt></li>
	<li><tt>squidclient http://cnn.com</tt></li>
	</ul>
	</li>
	<li>Send the data to kafka via <tt>cat /var/log/squid/access.log \| /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid</tt></li>
	<li>Browse the data in elasticsearch via the ES Head plugin @ <a class="externalLink" href="http://node1:9200/_plugin/head/">http://node1:9200/_plugin/head/</a> and verify that in the squid index you have two documents</li>
	<li>Ensure that the documents have new fields <tt>foo</tt>, <tt>bar</tt> and <tt>ALL_CAPS</tt> with values as described above.</li>
	</ul>
	<p>Note that we could have used any Stellar statements here, including calling out to HBase via <tt>ENRICHMENT_GET</tt> and <tt>ENRICHMENT_EXISTS</tt> or even calling a machine learning model via <a href="../../../metron-analytics/metron-maas-service/index.html">Model as a Service</a>.</p></div>
	</div>
	</div>
	</div>
	<hr/>
	<footer>
	<div class="container-fluid">
	<div class="row-fluid">
	Â© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
	and the Apache Metron project logo are trademarks of The Apache Software Foundation.
	</div>
	</div>
	</footer>
	</body>
	</html>