METRON-1937: Update public web site to point at 0.7.0 new release
diff --git a/current-book/CONTRIBUTING.html b/current-book/CONTRIBUTING.html
index bdc25b3..1517586 100644
--- a/current-book/CONTRIBUTING.html
+++ b/current-book/CONTRIBUTING.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/CONTRIBUTING.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/CONTRIBUTING.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – How To Contribute</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">How To Contribute</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/Upgrading.html b/current-book/Upgrading.html
index 884deb6..db9c0dd 100644
--- a/current-book/Upgrading.html
+++ b/current-book/Upgrading.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/Upgrading.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/Upgrading.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Upgrading</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Upgrading</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -93,6 +93,29 @@
<p><a name="Upgrading"></a></p>
<p>This document constitutes a per-version listing of changes of configuration which are non-backwards compatible.</p>
<div class="section">
+<h2><a name="a0.6.0_to_0.7.0"></a>0.6.0 to 0.7.0</h2>
+<div class="section">
+<h3><a name="METRON-1834:_Migrate_Elasticsearch_from_TransportClient_to_new_Java_REST_API"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-1834">METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API</a></h3>
+<p>The Elasticsearch Java client has now been migrated from TransportClient to the new Java REST client. The motivation for this change is that TransportClient will be deprecated in Elasticsearch 7.0 and removed entirely in 8.0. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html">ES Java API </a> for more details. The primary client-facing change for upgrades will be the new properties for configuring the new client. An explanation of the new properties as well as a mapping from the old properties to the new can be found in <a href="metron-platform/metron-elasticsearch/index.html#Properties">metron-elasticsearch</a> under <tt>es.client.settings</tt>.</p></div>
+<div class="section">
+<h3><a name="METRON-1855:_Make_unified_enrichment_topology_the_default_and_deprecate_split-join"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-1855">METRON-1855: Make unified enrichment topology the default and deprecate split-join</a></h3>
+<p>The unified enrichment topology will be the new default in this release, and the split-join enrichment topology is now considered deprecated. If you wish to keep the deprecated split-join enrichment topology, you will need to make the following changes:</p>
+<ul>
+
+<li>In Ambari > Metron > Config > Enrichment set the enrichment_topology setting to “Split-Join”</li>
+<li>If running <tt>start_enrichment_topology.sh</tt> manually, pass in the parameters to start the Split-Join topology as follows
+
+<div>
+<div>
+<pre class="source">$METRON_HOME/bin/start_enrichment_topology.sh --remote $METRON_HOME/flux/enrichment/remote-splitjoin.yaml --filter $METRON_HOME/config/enrichment-splitjoin.properties
+</pre></div></div>
+</li>
+<li>
+
+<p>Restart the enrichment topology</p>
+</li>
+</ul></div></div>
+<div class="section">
<h2><a name="a0.4.2_to_0.5.0"></a>0.4.2 to 0.5.0</h2>
<div class="section">
<h3><a name="METRON-941:_native_PaloAlto_parser_corrupts_message_when_having_a_comma_in_the_payload"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-941">METRON-941: native PaloAlto parser corrupts message when having a comma in the payload</a></h3>
diff --git a/current-book/index.html b/current-book/index.html
index f54bd47..cf4cc2d 100644
--- a/current-book/index.html
+++ b/current-book/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Apache Metron</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Apache Metron</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -195,7 +195,7 @@
<li><a href="metron-platform/metron-common/index.html">Stellar</a> : A custom data transformation language that is used throughout metron from simple field transformation to expressing triage rules.</li>
<li><a href="metron-analytics/metron-maas-service/index.html">Model as a Service</a> : A Yarn application which can deploy machine learning and statistical models onto the cluster along with the associated Stellar functions to be able to call out to them in a scalable manner.</li>
<li><a href="metron-platform/metron-data-management/index.html">Data management</a> : A set of data management utilities aimed at getting data into HBase in a format which will allow data flowing through metron to be enriched with the results. Contains integrations with threat intelligence feeds exposed via TAXII as well as simple flat file structures.</li>
-<li><a href="metron-analytics/metron-profiler/index.html">Profiler</a> : A feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</li>
+<li><a href="metron-analytics/metron-profiler-common/index.html">Profiler</a> : A feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</li>
</ul>
<p><a name="Notes_on_Adding_a_New_Sensor"></a></p>
<h1>Notes on Adding a New Sensor</h1>
diff --git a/current-book/metron-analytics/index.html b/current-book/metron-analytics/index.html
index e35ba6d..77dbf24 100644
--- a/current-book/metron-analytics/index.html
+++ b/current-book/metron-analytics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Analytics</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Analytics</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -48,8 +48,11 @@
<li class="active"><a href="#"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li><a href="../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
- <li><a href="../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li>
<li><a href="../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li><a href="../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
</ul>
</li>
diff --git a/current-book/metron-analytics/metron-maas-service/index.html b/current-book/metron-analytics/metron-maas-service/index.html
index 47b0774..99cfb00 100644
--- a/current-book/metron-analytics/metron-maas-service/index.html
+++ b/current-book/metron-analytics/metron-maas-service/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-maas-service/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-maas-service/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Model Management Infrastructure</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Model Management Infrastructure</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -48,8 +48,11 @@
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li class="active"><a href="#"><span class="none"></span>Maas-service</a></li>
- <li><a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li>
<li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
</ul>
</li>
diff --git a/current-book/metron-analytics/metron-profiler-client/index.html b/current-book/metron-analytics/metron-profiler-client/index.html
index 487ab31..1445e79 100644
--- a/current-book/metron-analytics/metron-profiler-client/index.html
+++ b/current-book/metron-analytics/metron-profiler-client/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-client/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-client/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler Client</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Profiler Client</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -48,8 +48,11 @@
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
- <li><a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li>
<li class="active"><a href="#"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
</ul>
</li>
diff --git a/current-book/metron-analytics/metron-profiler-common/index.html b/current-book/metron-analytics/metron-profiler-common/index.html
new file mode 100644
index 0000000..f2c802e
--- /dev/null
+++ b/current-book/metron-analytics/metron-profiler-common/index.html
@@ -0,0 +1,562 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-common/index.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Metron Profiler</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
+ <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Metron Profiler</h1>
+<p><a name="Metron_Profiler"></a></p>
+<ul>
+
+<li><a href="#Introduction">Introduction</a></li>
+<li><a href="#Getting_Started">Getting Started</a></li>
+<li><a href="#Profiles">Profiles</a></li>
+<li><a href="#Examples">Examples</a></li>
+</ul>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</p>
+<p>This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows. A summary statistic is applied to the data received within a given window. Collecting these values across many windows result in a time series that is useful for analysis.</p>
+<p>Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language.</p>
+<p>There are three separate ports of the Profiler that share this common code base.</p>
+<ul>
+
+<li>The <a href="../metron-profiler-storm/index.html">Storm Profiler</a> builds low-latency profiles over streaming data sets.</li>
+<li>The <a href="../metron-profiler-spark/index.html">Spark Profiler</a> backfills profiles using archived telemetry.</li>
+<li>The <a href="../metron-profiler-repl/index.html">REPL Profiler</a> allows profiles to be tested and debugged within the Stellar REPL.</li>
+</ul></div>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p><a href="../metron-profiler-repl/index.html#getting-started">Create a profile</a> using the Stellar REPL. Validate your profile using mock data, then apply real, live data.</p>
+</li>
+<li>
+
+<p><a href="../metron-profiler-spark/index.html#getting-started">Backfill your profile</a> using archived telemetry to see how your profile behaves over time.</p>
+</li>
+<li>
+
+<p><a href="../metron-profiler-storm/index.html#getting-started">Deploy your profile</a> to Storm to maintain a low-latency profile over a streaming data set.</p>
+</li>
+<li>
+
+<p><a href="../metron-profiler-client/index.html">Retrieve your profile data</a> using the Stellar API so that you can build enrichments, alert on abnormalities.</p>
+</li>
+<li>
+
+<p>Explore more ways to create <a href="#more-examples">profiles</a>.</p>
+</li>
+</ol></div>
+<div class="section">
+<h2><a name="Profiles"></a>Profiles</h2>
+<p>Let’s start with a simple example. The following profile maintains a count of the number of telemetry messages for each IP source address. A counter is initialized to 0, then incremented each time a message is received for a given IP source address. At regular intervals the count is flushed and stored. Over time this results in a time series describing the amount of telemetry received for each IP source address.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "foreach": "ip_src_addr",
+ "init": {
+ "count": "0"
+ },
+ "update": {
+ "count": "count + 1"
+ },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div>
+
+<p>A profile definition contains two fields; only one of which is required.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ { "profile": "one", ... },
+ { "profile": "two", ... }
+ ],
+ "timestampField": "timestamp"
+}
+</pre></div></div>
+
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Name </th>
+<th> </th>
+<th> Description</th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> <a href="#profiles">profiles</a> </td>
+<td> Required </td>
+<td> A list of zero or more Profile definitions.</td></tr>
+<tr class="a">
+<td> <a href="#timestampField">timestampField</a> </td>
+<td> Optional </td>
+<td> Indicates whether processing time or event time should be used. By default, processing time is enabled.</td></tr>
+</tbody>
+</table>
+<div class="section">
+<div class="section">
+<h4><a name="profiles"></a><tt>profiles</tt></h4>
+<p><i>Required</i></p>
+<p>A list of zero or more Profile definitions.</p></div>
+<div class="section">
+<h4><a name="timestampField"></a><tt>timestampField</tt></h4>
+<p><i>Optional</i></p>
+<p>Indicates whether processing time or event time is used. By default, processing time is enabled.</p>
+<div class="section">
+<h5><a name="Processing_Time"></a>Processing Time</h5>
+<p>By default, no <tt>timestampField</tt> is defined. In this case, the Profiler uses system time when generating profiles. This means that the profiles are generated based on when the data has been processed by the Profiler. This is also known as ‘processing time’.</p>
+<p>This is the simplest mode of operation, but has some draw backs. If the Profiler is consuming live data and all is well, the processing and event times will likely remain similar and consistent. If processing time diverges from event time, then the Profiler will generate skewed profiles.</p>
+<p>There are a few scenarios that might cause skewed profiles when using processing time. For example when a system has undergone a scheduled maintenance window and is restarted, a high volume of messages will need to be processed by the Profiler. The output of the Profiler might indicate an increase in activity during this time, although no change in activity actually occurred on the target network. The same situation could occur if an upstream system which provides telemetry undergoes an outage.</p>
+<p><a href="#Event_Time">Event Time</a> can be used to mitigate these problems.</p></div>
+<div class="section">
+<h5><a name="Event_Time"></a>Event Time</h5>
+<p>Alternatively, a <tt>timestampField</tt> can be defined. This must be the name of a field contained within the telemetry processed by the Profiler. The Profiler will extract and use the timestamp contained within this field.</p>
+<ul>
+
+<li>
+
+<p>If a message does not contain this field, it will be dropped.</p>
+</li>
+<li>
+
+<p>The field must contain a timestamp in epoch milliseconds expressed as either a numeric or string. Otherwise, the message will be dropped.</p>
+</li>
+<li>
+
+<p>The Profiler will use the same field across all telemetry sources and for all profiles.</p>
+</li>
+<li>
+
+<p>Be aware of clock skew across telemetry sources. If your profile is processing telemetry from multiple sources where the clock differs significantly, the Profiler may assume that some of those messages are late and will be ignored. Adjusting the <a href="#profiler.window.duration"><tt>profiler.window.duration</tt></a> and <a href="#profiler.window.lag"><tt>profiler.window.lag</tt></a> can help accommodate skewed clocks.</p>
+</li>
+</ul></div></div></div>
+<div class="section">
+<h3><a name="Profiles"></a>Profiles</h3>
+<p>A profile definition requires a JSON-formatted set of elements, many of which can contain Stellar code. The specification contains the following elements. (For the impatient, skip ahead to the <a href="#Examples">Examples</a>.)</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Name </th>
+<th> </th>
+<th> Description</th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> <a href="#profile">profile</a> </td>
+<td> Required </td>
+<td> Unique name identifying the profile.</td></tr>
+<tr class="a">
+<td> <a href="#foreach">foreach</a> </td>
+<td> Required </td>
+<td> A separate profile is maintained “for each” of these.</td></tr>
+<tr class="b">
+<td> <a href="#onlyif">onlyif</a> </td>
+<td> Optional </td>
+<td> Boolean expression that determines if a message should be applied to the profile.</td></tr>
+<tr class="a">
+<td> <a href="#groupBy">groupBy</a> </td>
+<td> Optional </td>
+<td> One or more Stellar expressions used to group the profile measurements when persisted.</td></tr>
+<tr class="b">
+<td> <a href="#init">init</a> </td>
+<td> Optional </td>
+<td> One or more expressions executed at the start of a window period.</td></tr>
+<tr class="a">
+<td> <a href="#update">update</a> </td>
+<td> Required </td>
+<td> One or more expressions executed when a message is applied to the profile.</td></tr>
+<tr class="b">
+<td> <a href="#result">result</a> </td>
+<td> Required </td>
+<td> Stellar expressions that are executed when the window period expires.</td></tr>
+<tr class="a">
+<td> <a href="#expires">expires</a> </td>
+<td> Optional </td>
+<td> Profile data is purged after this period of time, specified in days.</td></tr>
+</tbody>
+</table>
+<div class="section">
+<h4><a name="profile"></a><tt>profile</tt></h4>
+<p><i>Required</i></p>
+<p>A unique name identifying the profile. The field is treated as a string.</p></div>
+<div class="section">
+<h4><a name="foreach"></a><tt>foreach</tt></h4>
+<p><i>Required</i></p>
+<p>A separate profile is maintained ‘for each’ of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name.</p>
+<p>For example, if <tt>ip_src_addr</tt> then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc.</p></div>
+<div class="section">
+<h4><a name="onlyif"></a><tt>onlyif</tt></h4>
+<p><i>Optional</i></p>
+<p>An expression that determines if a message should be applied to the profile. A Stellar expression that returns a Boolean is expected. A message is only applied to a profile if this expression is true. This allows a profile to filter the messages that get applied to it.</p></div>
+<div class="section">
+<h4><a name="groupBy"></a><tt>groupBy</tt></h4>
+<p><i>Optional</i></p>
+<p>One or more Stellar expressions used to group the profile measurements when persisted. This can be used to sort the Profile data to allow for a contiguous scan when accessing subsets of the data. This is also one way to deal with calendar effects. For example, where activity on a weekday can be very different from a weekend.</p>
+<p>A common use case would be grouping by day of week. This allows a contiguous scan to access all profile data for Mondays only. Using the following definition would achieve this.</p>
+
+<div>
+<div>
+<pre class="source">"groupBy": [ "DAY_OF_WEEK(start)" ]
+</pre></div></div>
+
+<p>The expression can reference any of these variables.</p>
+<ul>
+
+<li>Any variable defined by the profile in its <tt>init</tt> or <tt>update</tt> expressions.</li>
+<li><tt>profile</tt> The name of the profile.</li>
+<li><tt>entity</tt> The name of the entity being profiled.</li>
+<li><tt>start</tt> The start time of the profile period in epoch milliseconds.</li>
+<li><tt>end</tt> The end time of the profile period in epoch milliseconds.</li>
+<li><tt>duration</tt> The duration of the profile period in milliseconds.</li>
+<li><tt>result</tt> The result of executing the <tt>result</tt> expression.</li>
+</ul></div>
+<div class="section">
+<h4><a name="init"></a><tt>init</tt></h4>
+<p><i>Optional</i></p>
+<p>One or more expressions executed at the start of a window period. A map is expected where the key is the variable name and the value is a Stellar expression. The map can contain zero or more variable:expression pairs. At the start of each window period, each expression is executed once and stored in the given variable. Note that constant init values such as “0” must be in quotes regardless of their type, as the init value must be a string to be executed by Stellar.</p>
+
+<div>
+<div>
+<pre class="source">"init": {
+ "var1": "0",
+ "var2": "1"
+}
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="update"></a><tt>update</tt></h4>
+<p><i>Required</i></p>
+<p>One or more expressions executed when a message is applied to the profile. A map is expected where the key is the variable name and the value is a Stellar expression. The map can include 0 or more variables/expressions. When each message is applied to the profile, the expression is executed and stored in a variable with the given name.</p>
+
+<div>
+<div>
+<pre class="source">"update": {
+ "var1": "var1 + 1",
+ "var2": "var2 + 1"
+}
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="result"></a><tt>result</tt></h4>
+<p><i>Required</i></p>
+<p>Stellar expressions that are executed when the window period expires. The expressions are expected to summarize the messages that were applied to the profile over the window period. In the most basic form a single result is persisted for later retrieval.</p>
+
+<div>
+<div>
+<pre class="source">"result": "var1 + var2"
+</pre></div></div>
+
+<p>For more advanced use cases, a profile can generate two types of results. A profile can define one or both of these result types at the same time.</p>
+<ul>
+
+<li><tt>profile</tt>: A required expression that defines a value that is persisted for later retrieval.</li>
+<li><tt>triage</tt>: An optional expression that defines values that are accessible within the Threat Triage process.</li>
+</ul>
+<p><b>profile</b></p>
+<p>A required Stellar expression that results in a value that is persisted in the profile store for later retrieval. The expression can result in any object that is Kryo serializable. These values can be retrieved for later use with the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
+
+<div>
+<div>
+<pre class="source">"result": {
+ "profile": "2 + 2"
+}
+</pre></div></div>
+
+<p>An alternative, simplified form is also acceptable.</p>
+
+<div>
+<div>
+<pre class="source">"result": "2 + 2"
+</pre></div></div>
+
+<p><b>triage</b></p>
+<p>An optional map of one or more Stellar expressions. The value of each expression is made available to the Threat Triage process under the given name. Each expression must result in a either a primitive type, like an integer, long, or short, or a String. All other types will result in an error.</p>
+<p>In the following example, three values, the minimum, the maximum and the mean are appended to a message. This message is consumed by Metron, like other sources of telemetry, and each of these values are accessible from within the Threat Triage process using the given field names; <tt>min</tt>, <tt>max</tt>, and <tt>mean</tt>.</p>
+
+<div>
+<div>
+<pre class="source">"result": {
+ "triage": {
+ "min": "STATS_MIN(stats)",
+ "max": "STATS_MAX(stats)",
+ "mean": "STATS_MEAN(stats)"
+ }
+}
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="expires"></a><tt>expires</tt></h4>
+<p><i>Optional</i></p>
+<p>A numeric value that defines how many days the profile data is retained. After this time, the data expires and is no longer accessible. If no value is defined, the data does not expire.</p>
+<p>The REPL can be a powerful tool for developing profiles. Read all about <a href="../metron-profiler-client/index.html#developing_profiles">Developing Profiles</a>.</p></div></div></div>
+<div class="section">
+<h2><a name="Examples"></a>Examples</h2>
+<p>The following examples are intended to highlight the functionality provided by the Profiler. Try out these examples easily in the Stellar REPL as described in the <a href="#Getting_Started">Getting Started</a> section.</p>
+<div class="section">
+<h3><a name="Example_1"></a>Example 1</h3>
+<p>This example captures the ratio of DNS traffic to HTTP traffic for each host. The following configuration would be used to generate this profile.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ {
+ "profile": "dns-to-http-by-source",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'DNS' or protocol == 'HTTP'",
+ "init": {
+ "num_dns": 1.0,
+ "num_http": 1.0
+ },
+ "update": {
+ "num_dns": "num_dns + (if protocol == 'DNS' then 1 else 0)",
+ "num_http": "num_http + (if protocol == 'HTTP' then 1 else 0)"
+ },
+ "result": "num_dns / num_http"
+ }
+ ]
+}
+</pre></div></div>
+
+<p>This creates a profile…</p>
+<ul>
+
+<li>Named ‘dns-to-http-by-source’</li>
+<li>That for each IP source address</li>
+<li>Only if the ‘protocol’ field equals ‘HTTP’ or ‘DNS’</li>
+<li>Accumulates the number of DNS requests</li>
+<li>Accumulates the number of HTTP requests</li>
+<li>Returns the ratio of these as the result</li>
+</ul></div>
+<div class="section">
+<h3><a name="Example_2"></a>Example 2</h3>
+<p>This example captures the average of the <tt>length</tt> field for HTTP traffic. The following profile could be used.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ {
+ "profile": "avg-http-length",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'HTTP'",
+ "update": { "s": "STATS_ADD(s, length)" },
+ "result": "STATS_MEAN(s)"
+ }
+ ]
+}
+</pre></div></div>
+
+<p>This creates a profile…</p>
+<ul>
+
+<li>Named ‘avg-http-length’</li>
+<li>That for each IP source address</li>
+<li>Only if the ‘protocol’ field is ‘HTTP’</li>
+<li>Captures the <tt>length</tt> field</li>
+<li>Calculates the average as the result</li>
+</ul>
+<p>It is important to note that the Profiler can persist any serializable Object, not just numeric values. Instead of storing the actual mean, the profile could store a statistical sketch of the lengths. This summary can then be used at a later time to calculate the mean, min, max, percentiles, or any other sensible metric. This provides a much greater degree of flexibility. The following Stellar REPL session shows how you might do this.</p>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Retrieve the last 30 minutes of profile measurements for a specific host.</p>
+
+<div>
+<div>
+<pre class="source">$ source /etc/default/metron
+$ bin/stellar -z $ZOOKEEPER
+
+[Stellar]>>> stats := PROFILE_GET( "example4", "10.0.0.1", PROFILE_FIXED(30, "MINUTES"))
+[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9, ...]
+</pre></div></div>
+</li>
+<li>
+
+<p>Calculate different summary metrics using the same profile data.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> aStat := GET_FIRST(stats)
+org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9
+
+[Stellar]>>> STATS_MEAN(aStat)
+15979.0625
+
+[Stellar]>>> STATS_PERCENTILE(aStat, 90)
+30310.958
+</pre></div></div>
+</li>
+<li>
+
+<p>Merge all of the profile measurements over the past 30 minutes into a single sketch and calculate the 90th percentile.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> merged := STATS_MERGE( stats)
+
+[Stellar]>>> STATS_PERCENTILE(merged, 90)
+29810.992
+</pre></div></div>
+</li>
+</ol>
+<p>More information on accessing profile data can be found in the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
+<p>More information on using the <a href="../metron-statistics/index.html"><tt>STATS_*</tt> functions</a>.</p></div>
+<div class="section">
+<h3><a name="Example_3"></a>Example 3</h3>
+<p>This profile captures the vertex degree of a host. If you view network communication as a directed graph, the in and out degree of each host can distinguish behaviors. Anomalies can serve as an indicator of compromise. For example, you might find clients normally have an out-degree >> in-degree, whereas a server might be the opposite.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ {
+ "profile": "in-degrees",
+ "onlyif": "source.type == 'yaf'",
+ "foreach": "ip_dst_addr",
+ "update": { "in": "HLLP_ADD(in, ip_src_addr)" },
+ "result": "HLLP_CARDINALITY(in)"
+ },
+ {
+ "profile": "out-degrees",
+ "onlyif": "source.type == 'yaf'",
+ "foreach": "ip_src_addr",
+ "update": { "out": "HLLP_ADD(out, ip_dst_addr)" },
+ "result": "HLLP_CARDINALITY(out)"
+ }
+ ]
+}
+</pre></div></div>
+
+<p>This creates a profile…</p>
+<ul>
+
+<li>Named ‘in-degrees’</li>
+<li>That for each IP destination address</li>
+<li>Captures the IP source address</li>
+<li>Then calculates the cardinality; the number of unique IPs this host has interacted with</li>
+</ul>
+<p>The second profile calculates the out-degree.</p></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-analytics/metron-profiler-repl/index.html b/current-book/metron-analytics/metron-profiler-repl/index.html
new file mode 100644
index 0000000..eafb1dc
--- /dev/null
+++ b/current-book/metron-analytics/metron-profiler-repl/index.html
@@ -0,0 +1,339 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-repl/index.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler for the Stellar REPL</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Metron Profiler for the Stellar REPL</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
+ <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Metron Profiler for the Stellar REPL</h1>
+<p><a name="Metron_Profiler_for_the_Stellar_REPL"></a></p>
+<p>This project allows profiles to be executed within the Stellar REPL. This is a port of the Profiler to the Stellar REPL that allows profiles to be tested and debugged within a controlled environment.</p>
+<ul>
+
+<li><a href="#Introduction">Introduction</a></li>
+<li><a href="#Getting_Started">Getting Started</a></li>
+<li><a href="#Installation">Installation</a></li>
+</ul>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>Creating and refining profiles is an iterative process. Iterating against a live stream of data is slow, difficult and error prone. Running the Profiler in the Stellar REPL provides a controlled and isolated execution environment to create, refine and troubleshoot profiles.</p>
+<p>For an introduction to the Profiler, see the <a href="../metron-profiler-common/index.html">Profiler README</a>.</p></div>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>This section describes how to get started using the Profiler in the Stellar REPL. This outlines a useful workflow for creating, testing, and deploying profiles.</p>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Launch the Stellar REPL.</p>
+
+<div>
+<div>
+<pre class="source">[root@node1 ~]# $METRON_HOME/bin/stellar
+Stellar, Go!
+[Stellar]>>>
+</pre></div></div>
+</li>
+<li>
+
+<p>The following functions should be accessible. Documentation is also provided for each function; for example by executing<tt>?PROFILER_FLUSH</tt>.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> %functions PROFILER
+PROFILER_APPLY, PROFILER_FLUSH, PROFILER_INIT
+</pre></div></div>
+</li>
+<li>
+
+<p>Create a simple <tt>hello-world</tt> profile that will count the number of messages for each <tt>ip_src_addr</tt>. The <tt>SHELL_EDIT</tt> function will open an editor in which you can copy/paste the following Profiler configuration.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> conf := SHELL_EDIT()
+[Stellar]>>> conf
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Create the profile execution environment.</p>
+<p>The Profiler will output the number of profiles that have been defined, the number of messages that have been applied and the number of routes that have been followed.</p>
+<p>While the idea of a profile and message may be well understood, a route may need further explanation. A route is defined when a message is applied to a specific profile.</p>
+<ul>
+
+<li>If a message is not needed by any profile, then there are no routes.</li>
+<li>If a message is needed by one profile, then one route has been followed.</li>
+<li>If a message is needed by two profiles, then two routes have been followed.</li>
+</ul>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> profiler := PROFILER_INIT(conf)
+[Stellar]>>> profiler
+Profiler{1 profile(s), 0 messages(s), 0 route(s)}
+</pre></div></div>
+</li>
+<li>
+
+<p>Create a message that mimics the telemetry that your profile will consume.</p>
+<p>This message can be as simple or complex as you like. For the <tt>hello-world</tt> profile, all you need is a message containing an <tt>ip_src_addr</tt> field.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> msg := SHELL_EDIT()
+[Stellar]>>> msg
+{
+ "ip_src_addr": "10.0.0.1"
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Apply the message to your Profiler, as many times as you like.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler)
+Profiler{1 profile(s), 1 messages(s), 1 route(s)}
+</pre></div></div>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler)
+Profiler{1 profile(s), 2 messages(s), 2 route(s)}
+</pre></div></div>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler)
+Profiler{1 profile(s), 3 messages(s), 3 route(s)}
+</pre></div></div>
+</li>
+<li>
+
+<p>Flush the Profiler.</p>
+<p>A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of Profile Measurements. Each measurement is a map containing detailed information about the profile data that has been generated. The <tt>value</tt> field is what is written to HBase when running the Profiler in either Storm or Spark.</p>
+<p>There will always be one measurement for each [profile, entity] pair. This profile simply counts the number of messages by IP source address. Notice that the value is ‘3’ for the entity ‘10.0.0.1’ as we applied 3 messages with an ‘ip_src_addr’ of ’10.0.0.1’.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> values := PROFILER_FLUSH(profiler)
+[Stellar]>>> values
+[{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
+profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
+</pre></div></div>
+</li>
+<li>
+
+<p>In addition to testing with mock data, you can also apply real, live telemetry to your profile. This can be useful to test your profile against the complexities that exist in real data.</p>
+<p>This example extracts 10 messages of live, enriched telemetry to test your profile(s).</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> msgs := KAFKA_GET("indexing", 10)
+[Stellar]>>> LENGTH(msgs)
+10
+</pre></div></div>
+
+<p>Now apply those 10 messages to your profile.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(msgs, profiler)
+ Profiler{1 profile(s), 10 messages(s), 10 route(s)}
+</pre></div></div>
+</li>
+<li>
+
+<p>After you are satisfied with your profile, the next step is to deploy the profile against the live stream of telemetry being capture by Metron. This involves deploying the profile to either the <a href="../metron-profiler-storm/index.html">Storm Profiler</a> or the <a href="../metron-profiler-spark/index.html">Spark Profiler</a>.</p>
+</li>
+</ol></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<p>This package is installed automatically when installing Metron using the Ambari MPack.</p>
+<p>This package can also be installed manually by using either the RPM or DEB package.</p>
+<div class="section">
+<h3><a name="Build_the_RPM"></a>Build the RPM</h3>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Build Metron.</p>
+
+<div>
+<div>
+<pre class="source">mvn clean package -DskipTests -T2C
+</pre></div></div>
+</li>
+<li>
+
+<p>Build the RPMs.</p>
+
+<div>
+<div>
+<pre class="source">cd metron-deployment/
+mvn clean package -Pbuild-rpms
+</pre></div></div>
+</li>
+<li>
+
+<p>Retrieve the package.</p>
+
+<div>
+<div>
+<pre class="source">find ./ -name "metron-profiler-repl*.rpm"
+</pre></div></div>
+</li>
+</ol></div>
+<div class="section">
+<h3><a name="Build_the_DEB"></a>Build the DEB</h3>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Build Metron.</p>
+
+<div>
+<div>
+<pre class="source">mvn clean package -DskipTests -T2C
+</pre></div></div>
+</li>
+<li>
+
+<p>Build the DEBs.</p>
+
+<div>
+<div>
+<pre class="source">cd metron-deployment/
+mvn clean package -Pbuild-debs
+</pre></div></div>
+</li>
+<li>
+
+<p>Retrieve the package.</p>
+
+<div>
+<div>
+<pre class="source">find ./ -name "metron-profiler-repl*.deb"
+</pre></div></div>
+</li>
+</ol></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-analytics/metron-profiler-spark/index.html b/current-book/metron-analytics/metron-profiler-spark/index.html
new file mode 100644
index 0000000..80eb19c
--- /dev/null
+++ b/current-book/metron-analytics/metron-profiler-spark/index.html
@@ -0,0 +1,539 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-spark/index.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler for Spark</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Metron Profiler for Spark</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
+ <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Metron Profiler for Spark</h1>
+<p><a name="Metron_Profiler_for_Spark"></a></p>
+<p>This project allows profiles to be executed using <a class="externalLink" href="https://spark.apache.org">Apache Spark</a>. This is a port of the Profiler to Spark that allows you to backfill profiles using archived telemetry.</p>
+<ul>
+
+<li><a href="#Introduction">Introduction</a></li>
+<li><a href="#Getting_Started">Getting Started</a></li>
+<li><a href="#Installation">Installation</a></li>
+<li><a href="#Running_the_Profiler">Running the Profiler</a></li>
+<li><a href="#Configuring_the_Profiler">Configuring the Profiler</a></li>
+</ul>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>Using the <a href="../metron-profiler-storm/index.html">Streaming Profiler</a> in <a class="externalLink" href="http://storm.apache.org">Apache Storm</a> allows you to create profiles based on the stream of telemetry being captured, enriched, triaged, and indexed by Metron. This does not allow you to create a profile based on telemetry that was captured in the past.</p>
+<p>There are many cases where you might want to produce a profile from telemetry in the past. This is referred to as profile seeding or backfilling.</p>
+<ul>
+
+<li>
+
+<p>As a Security Data Scientist, I want to understand the historical behaviors and trends of a profile so that I can determine if the profile has predictive value for model building.</p>
+</li>
+<li>
+
+<p>As a Security Platform Engineer, I want to generate a profile using archived telemetry when I deploy a new model to production so that models depending on that profile can function on day 1.</p>
+</li>
+</ul>
+<p>The Batch Profiler running in <a class="externalLink" href="https://spark.apache.org">Apache Spark</a> allows you to seed a profile using archived telemetry.</p>
+<p>The portion of a profile produced by the Batch Profiler should be indistinguishable from the portion created by the Streaming Profiler. Consumers of the profile should not care how the profile was generated. Using the Streaming Profiler together with the Batch Profiler allows you to create a complete profile over a wide range of time.</p>
+<p>For an introduction to the Profiler, see the <a href="../metron-profiler-common/index.html">Profiler README</a>.</p></div>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Create a profile definition by editing <tt>$METRON_HOME/config/zookeeper/profiler.json</tt> as follows.</p>
+
+<div>
+<div>
+<pre class="source">cat $METRON_HOME/config/zookeeper/profiler.json
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "foreach": "'global'",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ],
+ "timestampField": "timestamp"
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Ensure that you have archived telemetry available for the Batch Profiler to consume. By default, Metron will store this in HDFS at <tt>/apps/metron/indexing/indexed/*/*</tt>.</p>
+
+<div>
+<div>
+<pre class="source">hdfs dfs -cat /apps/metron/indexing/indexed/*/* | wc -l
+</pre></div></div>
+</li>
+<li>
+
+<p>Review the Batch Profiler’s properties located at <tt>$METRON_HOME/config/batch-profiler.properties</tt>. See <a href="#Configuring_the_Profiler">Configuring the Profiler</a> for more information on these properties.</p>
+</li>
+<li>
+
+<p>You may want to edit the log4j properties that sits in your config directory in <tt>${SPARK_HOME}</tt> or create one. It may be helpful to turn on <tt>DEBUG</tt> logging for the Profiler by adding the following line.</p>
+
+<div>
+<div>
+<pre class="source"> log4j.logger.org.apache.metron.profiler.spark=DEBUG
+</pre></div></div>
+</li>
+<li>
+
+<p>Run the Batch Profiler.</p>
+
+<div>
+<div>
+<pre class="source">source /etc/default/metron
+cd $METRON_HOME
+$METRON_HOME/bin/start_batch_profiler.sh
+</pre></div></div>
+</li>
+<li>
+
+<p>Query for the profile data using the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
+</li>
+</ol></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<p>The Batch Profiler package is installed automatically when installing Metron using the Ambari MPack. See the following notes when installing the Batch Profiler without the Ambari MPack.</p>
+<div class="section">
+<h3><a name="Prerequisites"></a>Prerequisites</h3>
+<p>The Batch Profiler requires Spark version 2.3.0+.</p>
+<div class="section">
+<h4><a name="Build_the_RPM"></a>Build the RPM</h4>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Build Metron.</p>
+
+<div>
+<div>
+<pre class="source">mvn clean package -DskipTests -T2C
+</pre></div></div>
+</li>
+<li>
+
+<p>Build the RPMs.</p>
+
+<div>
+<div>
+<pre class="source">cd metron-deployment/
+mvn clean package -Pbuild-rpms
+</pre></div></div>
+</li>
+<li>
+
+<p>Retrieve the package.</p>
+
+<div>
+<div>
+<pre class="source">find ./ -name "metron-profiler-spark*.rpm"
+</pre></div></div>
+</li>
+</ol></div>
+<div class="section">
+<h4><a name="Build_the_DEB"></a>Build the DEB</h4>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Build Metron.</p>
+
+<div>
+<div>
+<pre class="source">mvn clean package -DskipTests -T2C
+</pre></div></div>
+</li>
+<li>
+
+<p>Build the DEBs.</p>
+
+<div>
+<div>
+<pre class="source">cd metron-deployment/
+mvn clean package -Pbuild-debs
+</pre></div></div>
+</li>
+<li>
+
+<p>Retrieve the package.</p>
+
+<div>
+<div>
+<pre class="source">find ./ -name "metron-profiler-spark*.deb"
+</pre></div></div>
+</li>
+</ol></div></div></div>
+<div class="section">
+<h2><a name="Running_the_Profiler"></a>Running the Profiler</h2>
+<ul>
+
+<li><a href="#Usage">Usage</a></li>
+<li><a href="#Advanced_Usage">Advanced Usage</a></li>
+<li><a href="#Spark_Execution">Spark Execution</a></li>
+<li><a href="#Kerberos">Kerberos</a></li>
+<li><a href="#Input_Formats">Input Formats</a></li>
+</ul>
+<div class="section">
+<h3><a name="Usage"></a>Usage</h3>
+<p>A script located at <tt>$METRON_HOME/bin/start_batch_profiler.sh</tt> has been provided to simplify running the Batch Profiler. This script makes the following assumptions.</p>
+<ul>
+
+<li>
+
+<p>The script builds the profiles defined in <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>.</p>
+</li>
+<li>
+
+<p>The properties defined in <tt>$METRON_HOME/config/batch-profiler.properties</tt> are passed to both the Profiler and Spark. You can define both Spark and Profiler properties in this same file.</p>
+</li>
+<li>
+
+<p>The script assumes that Spark is installed at <tt>/usr/hdp/current/spark2-client</tt>. This can be overridden if you define an environment variable called <tt>SPARK_HOME</tt> prior to executing the script.</p>
+</li>
+</ul></div>
+<div class="section">
+<h3><a name="Advanced_Usage"></a>Advanced Usage</h3>
+<p>The Batch Profiler may also be started using <tt>spark-submit</tt> as follows. See the Spark Documentation for more information about <a class="externalLink" href="https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit"><tt>spark-submit</tt></a>.</p>
+
+<div>
+<div>
+<pre class="source">${SPARK_HOME}/bin/spark-submit \
+ --class org.apache.metron.profiler.spark.cli.BatchProfilerCLI \
+ --properties-file ${SPARK_PROPS_FILE} \
+ ${METRON_HOME}/lib/metron-profiler-spark-*.jar \
+ --config ${PROFILER_PROPS_FILE} \
+ --profiles ${PROFILES_FILE}
+</pre></div></div>
+
+<p>The Batch Profiler accepts the following arguments when run from the command line as shown above. All arguments following the Profiler jar are passed to the Profiler. All argument preceeding the Profiler jar are passed to Spark.</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Argument </th>
+<th> Description</th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> <a href="#a--profiles"><tt>-p</tt>, <tt>--profiles</tt></a> </td>
+<td> Path to the profile definitions.</td></tr>
+<tr class="a">
+<td> <a href="#a--config"><tt>-c</tt>, <tt>--config</tt></a> </td>
+<td> Path to the profiler properties file.</td></tr>
+<tr class="b">
+<td> <a href="#a--globals"><tt>-g</tt>, <tt>--globals</tt></a> </td>
+<td> Path to the Stellar global config file.</td></tr>
+<tr class="a">
+<td> <a href="#a--reader"><tt>-r</tt>, <tt>--reader</tt></a> </td>
+<td> Path to properties for the DataFrameReader.</td></tr>
+<tr class="b">
+<td> <tt>-h</tt>, <tt>--help</tt> </td>
+<td> Print the help text.</td></tr>
+</tbody>
+</table>
+<div class="section">
+<h4><a name="a--profiles"></a><tt>--profiles</tt></h4>
+<p>The path to a file containing the profile definition in JSON.</p></div>
+<div class="section">
+<h4><a name="a--config"></a><tt>--config</tt></h4>
+<p>The path to a file containing key-value properties for the Profiler. This file would contain the properties described under <a href="#Configuring_the_Profiler">Configuring the Profiler</a>.</p></div>
+<div class="section">
+<h4><a name="a--globals"></a><tt>--globals</tt></h4>
+<p>The path to a file containing key-value properties that define the global properties. This can be used to customize how certain Stellar functions behave during execution.</p></div>
+<div class="section">
+<h4><a name="a--reader"></a><tt>--reader</tt></h4>
+<p>The path to a file containing key-value properties that are passed to the DataFrameReader when reading the input telemetry. This allows additional customization for how the input telemetry is read.</p></div></div>
+<div class="section">
+<h3><a name="Spark_Execution"></a>Spark Execution</h3>
+<p>Spark supports a number of different <a class="externalLink" href="https://spark.apache.org/docs/latest/cluster-overview.html#cluster-manager-types">cluster managers</a>. The underlying cluster manager is transparent to the Profiler. To run the Profiler on a particular cluster manager, it is just a matter of setting the appropriate options as defined in the Spark documentation.</p>
+<div class="section">
+<h4><a name="Local_Mode"></a>Local Mode</h4>
+<p>By default, the Batch Profiler instructs Spark to run in local mode. This will run all of the Spark execution components within a single JVM. This mode is only useful for testing with a limited set of data.</p>
+<p><tt>$METRON_HOME/config/batch-profiler.properties</tt></p>
+
+<div>
+<div>
+<pre class="source">spark.master=local
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="Spark_on_YARN"></a>Spark on YARN</h4>
+<p>To run the Profiler using <a class="externalLink" href="https://spark.apache.org/docs/latest/running-on-yarn.html#running-spark-on-yarn">Spark on YARN</a>, at a minimum edit the value of <tt>spark.master</tt> as shown. In many cases it also makes sense to set the YARN <a class="externalLink" href="https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn">deploy mode</a> to <tt>cluster</tt>.</p>
+<p><tt>$METRON_HOME/config/batch-profiler.properties</tt></p>
+
+<div>
+<div>
+<pre class="source">spark.master=yarn
+spark.submit.deployMode=cluster
+</pre></div></div>
+
+<p>See the Spark documentation for information on how to further control the execution of Spark on YARN. Any of <a class="externalLink" href="http://spark.apache.org/docs/latest/running-on-yarn.html#spark-properties">these properties</a> can be added to the Profiler properties file.</p>
+<p>The following command can be useful to review the logs generated when the Profiler is executed on YARN.</p>
+
+<div>
+<div>
+<pre class="source">yarn logs -applicationId <application-id>
+</pre></div></div>
+</div></div>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>See the Spark documentation for information on running the Batch Profiler in a <a class="externalLink" href="https://spark.apache.org/docs/latest/running-on-yarn.html#running-in-a-secure-cluster">secure, kerberized cluster</a>.</p></div>
+<div class="section">
+<h3><a name="Input_Formats"></a>Input Formats</h3>
+<p>The Profiler can consume archived telemetry stored in a variety of input formats. By default, it is configured to consume the text/json that Metron archives in HDFS. This is often not the best format for archiving telemetry. If you choose a different format, you should be able to configure the Profiler to consume it by doing the following.</p>
+<ol style="list-style-type: decimal">
+
+<li>Edit <a href="#profiler.batch.input.format"><tt>profiler.batch.input.format</tt></a> and <a href="#profiler.batch.input.path"><tt>profiler.batch.input.path</tt></a> as needed. For example, to read ORC you might do the following.
+<p><tt>$METRON_HOME/config/batch-profiler.properties</tt></p>
+
+<div>
+<div>
+<pre class="source"></pre></div></div>
+</li>
+</ol>
+<p>profiler.batch.input.format=org.apache.spark.sql.execution.datasources.orc */*profiler.batch.input.path=<a class="externalLink" href="hdfs://localhost:9000/apps/metron/indexing/orc/">hdfs://localhost:9000/apps/metron/indexing/orc/</a> ```</p>
+<ol style="list-style-type: decimal">
+
+<li>If additional options are required for your input format, then use the <a href="#a--reader"><tt>--reader</tt></a> command-line argument when launching the Batch Profiler as <a href="#advanced-usage">described here</a>.</li>
+</ol>
+<div class="section">
+<h4><a name="Common_Formats"></a>Common Formats</h4>
+<p>The following examples highlight the configuration values needed to read telemetry stored in common formats. These values should be defined in the Profiler properties (see <a href="#a--config"><tt>--config</tt></a>).</p>
+<div class="section">
+<h5><a name="JSON"></a>JSON</h5>
+
+<div>
+<div>
+<pre class="source">profiler.batch.input.reader=json
+profiler.batch.input.path=/path/to/json/
+</pre></div></div>
+</div>
+<div class="section">
+<h5><a name="Apache_ORC"></a><a class="externalLink" href="https://orc.apache.org/">Apache ORC</a></h5>
+
+<div>
+<div>
+<pre class="source">profiler.batch.input.reader=orc
+profiler.batch.input.path=/path/to/orc/
+</pre></div></div>
+</div>
+<div class="section">
+<h5><a name="Apache_Parquet"></a><a class="externalLink" href="http://parquet.apache.org/">Apache Parquet</a></h5>
+
+<div>
+<div>
+<pre class="source">profiler.batch.input.reader=parquet
+profiler.batch.input.path=/path/to/parquet/
+</pre></div></div>
+</div></div></div></div>
+<div class="section">
+<h2><a name="Configuring_the_Profiler"></a>Configuring the Profiler</h2>
+<p>By default, the configuration for the Batch Profiler is stored in the local filesystem at <tt>$METRON_HOME/config/batch-profiler.properties</tt>.</p>
+<p>You can store both settings for the Profiler along with settings for Spark in this same file. Spark will only read settings that start with <tt>spark.</tt>.</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Setting </th>
+<th> Description</th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> <a href="#profiler.batch.input.path"><tt>profiler.batch.input.path</tt></a> </td>
+<td> The path to the input data read by the Batch Profiler.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.batch.input.reader"><tt>profiler.batch.input.reader</tt></a> </td>
+<td> The telemetry reader used to read the input data.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.batch.input.format"><tt>profiler.batch.input.format</tt></a> </td>
+<td> The format of the input data read by the Batch Profiler.</td></tr>
+<tr class="a">
+<td> <a href="#profilerbatchinputend"><tt>profiler.batch.input.begin</tt></a> </td>
+<td> Only messages with a timestamp after this will be profiled.</td></tr>
+<tr class="b">
+<td> <a href="#profilerbatchinputbegin"><tt>profiler.batch.input.end</tt></a> </td>
+<td> Only messages with a timestamp before this will be profiled.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a> </td>
+<td> The duration of each profile period.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a> </td>
+<td> The units used to specify the <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a>.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.hbase.salt.divisor"><tt>profiler.hbase.salt.divisor</tt></a> </td>
+<td> A salt is prepended to the row key to help prevent hot-spotting.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.hbase.table"><tt>profiler.hbase.table</tt></a> </td>
+<td> The name of the HBase table that profiles are written to.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.hbase.column.family"><tt>profiler.hbase.column.family</tt></a> </td>
+<td> The column family used to store profiles.</td></tr>
+</tbody>
+</table>
+<div class="section">
+<h3><a name="profiler.batch.input.path"></a><tt>profiler.batch.input.path</tt></h3>
+<p><i>Default</i>*/*: <a class="externalLink" href="hdfs://localhost:9000/apps/metron/indexing/indexed/">hdfs://localhost:9000/apps/metron/indexing/indexed/</a></p>
+<p>The path to the input data read by the Batch Profiler.</p></div>
+<div class="section">
+<h3><a name="profiler.batch.input.reader"></a><tt>profiler.batch.input.reader</tt></h3>
+<p><i>Default</i>: json</p>
+<p>Defines how the input data is treated when read. The value is not case sensitive so <tt>JSON</tt> and <tt>json</tt> are equivalent.</p>
+<ul>
+
+<li><tt>json</tt>: Read text/json formatted telemetry</li>
+<li><tt>orc</tt>: Read <a class="externalLink" href="https://orc.apache.org/">Apache ORC</a> formatted telemetry</li>
+<li><tt>parquet</tt>: Read <a class="externalLink" href="http://parquet.apache.org/">Apache Parquet</a> formatted telemetry</li>
+<li><tt>text</tt> Consumes input data stored as raw text. Should be defined along with <a href="#profiler.batch.input.format"><tt>profiler.batch.input.format</tt></a>. Only use if the input format is not directly supported like <tt>json</tt>.</li>
+<li><tt>columnar</tt> Consumes input data stored in columnar formats. Should be defined along with <a href="#profiler.batch.input.format"><tt>profiler.batch.input.format</tt></a>. Only use if the input format is not directly supported like <tt>json</tt>.</li>
+</ul>
+<p>See <a href="#Common_Formats">Common Formats</a> for further information.</p></div>
+<div class="section">
+<h3><a name="profiler.batch.input.format"></a><tt>profiler.batch.input.format</tt></h3>
+<p><i>Default</i>: text</p>
+<p>The format of the input data read by the Batch Profiler. This is optional and not required in most cases. For example, this property is not required when <a href="#profiler.batch.input.reader"><tt>profiler.batch.input.reader</tt></a> is <tt>json</tt>, <tt>orc</tt>, or <tt>parquet</tt>.</p></div>
+<div class="section">
+<h3><a name="profiler.batch.input.begin"></a><tt>profiler.batch.input.begin</tt></h3>
+<p><i>Default</i>: undefined; no time constraint</p>
+<p>Only messages with a timestamp equal to or after this will be profiled. The Profiler will only profiles messages with a timestamp in [<tt>profiler.batch.input.begin</tt>, <tt>profiler.batch.input.end</tt>] inclusive.</p>
+<p>By default, no time constraint is defined. The value is expected to follow the <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_INSTANT">ISO-8601 instant format</a>; 2011-12-03T10:15:30Z.</p></div>
+<div class="section">
+<h3><a name="profiler.batch.input.end"></a><tt>profiler.batch.input.end</tt></h3>
+<p><i>Default</i>: undefined; no time constraint</p>
+<p>Only messages with a timestamp before or equal to this will be profiled. The Profiler will only profiles messages with a timestamp in [<tt>profiler.batch.input.begin</tt>, <tt>profiler.batch.input.end</tt>] inclusive.</p>
+<p>By default, no time constraint is defined. The value is expected to follow the <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_INSTANT">ISO-8601 instant format</a>; 2011-12-03T10:15:30Z.</p></div>
+<div class="section">
+<h3><a name="profiler.period.duration"></a><tt>profiler.period.duration</tt></h3>
+<p><i>Default</i>: 15</p>
+<p>The duration of each profile period. This value should be defined along with <a href="#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a>.</p>
+<p><i>Important</i>: To read a profile using the <a href="metron-analytics/metron-profiler-client/index.html">Profiler Client</a>, the Profiler Client’s <tt>profiler.client.period.duration</tt> property must match this value. Otherwise, the Profiler Client will be unable to read the profile data.</p></div>
+<div class="section">
+<h3><a name="profiler.period.duration.units"></a><tt>profiler.period.duration.units</tt></h3>
+<p><i>Default</i>: MINUTES</p>
+<p>The units used to specify the <tt>profiler.period.duration</tt>. This value should be defined along with <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a>.</p>
+<p><i>Important</i>: To read a profile using the Profiler Client, the Profiler Client’s <tt>profiler.client.period.duration.units</tt> property must match this value. Otherwise, the <a href="metron-analytics/metron-profiler-client/index.html">Profiler Client</a> will be unable to read the profile data.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.salt.divisor"></a><tt>profiler.hbase.salt.divisor</tt></h3>
+<p><i>Default</i>: 1000</p>
+<p>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. This constant should be roughly equal to the number of nodes in the Hbase cluster to ensure even distribution of data.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.table"></a><tt>profiler.hbase.table</tt></h3>
+<p><i>Default</i>: profiler</p>
+<p>The name of the HBase table that profile data is written to. The Profiler expects that the table exists and is writable. It will not create the table.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.column.family"></a><tt>profiler.hbase.column.family</tt></h3>
+<p><i>Default</i>: P</p>
+<p>The column family used to store profile data in HBase.</p></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-analytics/metron-profiler-storm/index.html b/current-book/metron-analytics/metron-profiler-storm/index.html
new file mode 100644
index 0000000..9c97ac8
--- /dev/null
+++ b/current-book/metron-analytics/metron-profiler-storm/index.html
@@ -0,0 +1,559 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-storm/index.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler for Storm</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Metron Profiler for Storm</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Profiler-storm</a></li>
+ <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Metron Profiler for Storm</h1>
+<p><a name="Metron_Profiler_for_Storm"></a></p>
+<p>This project allows profiles to be executed using <a class="externalLink" href="https://storm.apache.org">Apache Storm</a>. This is a port of the Profiler to Storm that builds low-latency profiles over streaming data sets.</p>
+<ul>
+
+<li><a href="#Introduction">Introduction</a></li>
+<li><a href="#Getting_Started">Getting Started</a></li>
+<li><a href="#Installation">Installation</a></li>
+<li><a href="#Configuring_the_Profiler">Configuring the Profiler</a></li>
+<li><a href="#Implementation">Implementation</a></li>
+</ul>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</p>
+<p>This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.</p>
+<p>Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles and ensure that the Profiler topology is running.</p>
+<p>For an introduction to the Profiler, see the <a href="../metron-profiler-common/index.html">Profiler README</a>.</p></div>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>This section will describe the steps required to get your first “Hello, World!”" profile running. This assumes that you have a successful Profiler <a href="#Installation">Installation</a> and have it running. You can deploy profiles in two different ways.</p>
+<ul>
+
+<li><a href="#Deploying_Profiles_with_the_Stellar_Shell">Deploying Profiles with the Stellar Shell</a></li>
+<li><a href="#Deploying_Profiles_from_the_Command_Line">Deploying Profiles from the Command Line</a></li>
+</ul>
+<div class="section">
+<h3><a name="Deploying_Profiles_with_the_Stellar_Shell"></a>Deploying Profiles with the Stellar Shell</h3>
+<p>Continuing the previous running example, at this point, you have seen how your profile behaves against real, live telemetry in a controlled execution environment. The next step is to deploy your profile to the live, actively running Profiler topology.</p>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Start the Stellar Shell with the <tt>-z</tt> command line argument so that a connection to Zookeeper is established. This is required when deploying a new profile definition as shown in the steps below.</p>
+
+<div>
+<div>
+<pre class="source">[root@node1 ~]# source /etc/default/metron
+[root@node1 ~]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
+Stellar, Go!
+[Stellar]>>>
+[Stellar]>>> %functions CONFIG CONFIG_GET, CONFIG_PUT
+</pre></div></div>
+</li>
+<li>
+
+<p>If you haven’t already, define your profile.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> conf := SHELL_EDIT()
+[Stellar]>>> conf
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "onlyif": "exists(ip_src_addr)",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Check what is already deployed.</p>
+<p>Pushing a new profile configuration is destructive. It will overwrite any existing configuration. Check what you have out there. Manually merge the existing configuration with your new profile definition.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> existing := CONFIG_GET("PROFILER")
+</pre></div></div>
+</li>
+<li>
+
+<p>Deploy your profile. This will push the configuration to to the live, actively running Profiler topology. This will overwrite any existing profile definitions.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> CONFIG_PUT("PROFILER", conf)
+</pre></div></div>
+</li>
+</ol></div>
+<div class="section">
+<h3><a name="Deploying_Profiles_from_the_Command_Line"></a>Deploying Profiles from the Command Line</h3>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Create the profile definition in a file located at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>. This file will likely not exist, if you have never created Profiles before.</p>
+<p>The following example will create a profile that simply counts the number of messages per <tt>ip_src_addr</tt>.</p>
+
+<div>
+<div>
+<pre class="source">{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "onlyif": "exists(ip_src_addr)",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Upload the profile definition to Zookeeper.</p>
+
+<div>
+<div>
+<pre class="source">$ source /etc/default/metron
+$ cd $METRON_HOME
+$ bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z $ZOOKEEPER
+</pre></div></div>
+
+<p>You can validate this by reading back the Metron configuration from Zookeeper using the same script. The result should look-like the following.</p>
+
+<div>
+<div>
+<pre class="source">$ bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER
+...
+PROFILER Config: profiler
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "onlyif": "exists(ip_src_addr)",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div>
+</li>
+<li>
+
+<p>Ensure that test messages are being sent to the Profiler’s input topic in Kafka. The Profiler will consume messages from the input topic defined in the Profiler’s configuration (see <a href="#Configuring_the_Profiler">Configuring the Profiler</a>). By default this is the <tt>indexing</tt> topic.</p>
+</li>
+<li>
+
+<p>Check the HBase table to validate that the Profiler is writing the profile. Remember that the Profiler is flushing the profile every 15 minutes. You will need to wait at least this long to start seeing profile data in HBase.</p>
+
+<div>
+<div>
+<pre class="source">$ /usr/hdp/current/hbase-client/bin/hbase shell
+hbase(main):001:0> count 'profiler'
+</pre></div></div>
+</li>
+<li>
+
+<p>Use the <a href="../metron-profiler-client/index.html">Profiler Client</a> to read the profile data. The following <tt>PROFILE_GET</tt> command will read the data written by the <tt>hello-world</tt> profile. This assumes that <tt>10.0.0.1</tt> is one of the values for <tt>ip_src_addr</tt> contained within the telemetry consumed by the Profiler.</p>
+
+<div>
+<div>
+<pre class="source">$ source /etc/default/metron
+$ bin/stellar -z $ZOOKEEPER
+[Stellar]>>> PROFILE_GET( "hello-world", "10.0.0.1", PROFILE_FIXED(30, "MINUTES"))
+[451, 448]
+</pre></div></div>
+
+<p>This result indicates that over the past 30 minutes, the Profiler stored two values related to the source IP address “10.0.0.1”. In the first 15 minute period, the IP <tt>10.0.0.1</tt> was seen in 451 telemetry messages. In the second 15 minute period, the same IP was seen in 448 telemetry messages.</p>
+<p>It is assumed that the <tt>PROFILE_GET</tt> client is correctly configured to match the Profile configuration before using it to read that Profile. More information on configuring and using the Profiler client can be found <a href="../metron-profiler-client/index.html">here</a>.</p>
+</li>
+</ol></div></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<p>The Profiler can be installed with either of these two methods.</p>
+<ul>
+
+<li><a href="#Ambari_Installation">Ambari Installation</a></li>
+<li><a href="#Manual_Installation">Manual Installation</a></li>
+</ul>
+<div class="section">
+<h3><a name="Ambari_Installation"></a>Ambari Installation</h3>
+<p>The Metron Profiler is installed automatically when installing Metron using the Ambari MPack. You can skip the <a href="#Installation">Installation</a> section and move ahead to <a href="#Creating_Profiles">Creating Profiles</a> should this be the case.</p></div>
+<div class="section">
+<h3><a name="Manual_Installation"></a>Manual Installation</h3>
+<p>This section will describe the steps necessary to manually install the Profiler on an RPM-based Linux distribution. This assumes that core Metron has already been installed and validated. If you installed Metron using the <a href="#Ambari_MPack">Ambari MPack</a>, then the Profiler has already been installed and you can skip this section.</p>
+<ol style="list-style-type: decimal">
+
+<li>
+
+<p>Build the Metron RPMs (see Building the <a href="../../metron-deployment/index.html#RPMs">RPMs</a>).</p>
+<p>You may have already built the Metron RPMs when core Metron was installed.</p>
+
+<div>
+<div>
+<pre class="source">$ find metron-deployment/ -name "metron-profiler*.rpm"
+metron-deployment//packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.4.1-201707131420.noarch.rpm
+</pre></div></div>
+</li>
+<li>
+
+<p>Copy the Profiler RPM to the installation host.</p>
+<p>The installation host must be the same host on which core Metron was installed. Depending on how you installed Metron, the Profiler RPM might have already been copied to this host with the other Metron RPMs.</p>
+
+<div>
+<div>
+<pre class="source">[root@node1 ~]# find /localrepo/ -name "metron-profiler*.rpm"
+/localrepo/metron-profiler-0.4.1-201707112313.noarch.rpm
+</pre></div></div>
+</li>
+<li>
+
+<p>Install the RPM.</p>
+
+<div>
+<div>
+<pre class="source">[root@node1 ~]# rpm -ivh metron-profiler-*.noarch.rpm
+Preparing... ########################################### [100%]
+ 1:metron-profiler ########################################### [100%]
+</pre></div></div>
+
+<div>
+<div>
+<pre class="source">[root@node1 ~]# rpm -ql metron-profiler
+/usr/metron
+/usr/metron/0.4.2
+/usr/metron/0.4.2/bin
+/usr/metron/0.4.2/bin/start_profiler_topology.sh
+/usr/metron/0.4.2/config
+/usr/metron/0.4.2/config/profiler.properties
+/usr/metron/0.4.2/flux
+/usr/metron/0.4.2/flux/profiler
+/usr/metron/0.4.2/flux/profiler/remote.yaml
+/usr/metron/0.4.2/lib
+/usr/metron/0.4.2/lib/metron-profiler-0.4.2-uber.jar
+</pre></div></div>
+</li>
+<li>
+
+<p>Edit the configuration file located at <tt>$METRON_HOME/config/profiler.properties</tt>.</p>
+
+<div>
+<div>
+<pre class="source">kafka.zk=node1:2181
+kafka.broker=node1:6667
+</pre></div></div>
+
+<ul>
+
+<li>Change <tt>kafka.zk</tt> to refer to Zookeeper in your environment.</li>
+<li>Change <tt>kafka.broker</tt> to refer to a Kafka Broker in your environment.</li>
+</ul>
+</li>
+<li>
+
+<p>Create a table within HBase that will store the profile data. By default, the table is named <tt>profiler</tt> with a column family <tt>P</tt>. The table name and column family must match the Profiler’s configuration (see <a href="#Configuring_the_Profiler">Configuring the Profiler</a>).</p>
+
+<div>
+<div>
+<pre class="source">$ /usr/hdp/current/hbase-client/bin/hbase shell
+hbase(main):001:0> create 'profiler', 'P'
+</pre></div></div>
+</li>
+<li>
+
+<p>Start the Profiler topology.</p>
+
+<div>
+<div>
+<pre class="source">$ cd $METRON_HOME
+$ bin/start_profiler_topology.sh
+</pre></div></div>
+</li>
+</ol>
+<p>At this point the Profiler is running and consuming telemetry messages. We have not defined any profiles yet, so it is not doing anything very useful. The next section walks you through the steps to create your very first “Hello, World!” profile.</p></div></div>
+<div class="section">
+<h2><a name="Configuring_the_Profiler"></a>Configuring the Profiler</h2>
+<p>The Profiler runs as an independent Storm topology. The configuration for the Profiler topology is stored in local filesystem at <tt>$METRON_HOME/config/profiler.properties</tt>. After changing these values, the Profiler topology must be restarted for the changes to take effect.</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Setting </th>
+<th> Description</th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> <a href="#profiler.input.topic"><tt>profiler.input.topic</tt></a> </td>
+<td> The name of the input Kafka topic.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.output.topic"><tt>profiler.output.topic</tt></a> </td>
+<td> The name of the output Kafka topic.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a> </td>
+<td> The duration of each profile period.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a> </td>
+<td> The units used to specify the <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a>.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.window.duration"><tt>profiler.window.duration</tt></a> </td>
+<td> The duration of each profile window.</td></tr>
+<tr class="a">
+<td> <a href="#profilerpwindowdurationunits"><tt>profiler.window.duration.units</tt></a> </td>
+<td> The units used to specify the <a href="#profiler.window.duration"><tt>profiler.window.duration</tt></a>.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.window.lag"><tt>profiler.window.lag</tt></a> </td>
+<td> The maximum time lag for timestamps.</td></tr>
+<tr class="a">
+<td> <a href="#profilerpwindowlagunits"><tt>profiler.window.lag.units</tt></a> </td>
+<td> The units used to specify the <a href="#profiler.window.lag"><tt>profiler.window.lag</tt></a>.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.workers"><tt>profiler.workers</tt></a> </td>
+<td> The number of worker processes for the topology.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.executors"><tt>profiler.executors</tt></a> </td>
+<td> The number of executors to spawn per component.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.ttl"><tt>profiler.ttl</tt></a> </td>
+<td> If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.ttl.units"><tt>profiler.ttl.units</tt></a> </td>
+<td> The units used to specify the <tt>profiler.ttl</tt>.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.hbase.salt.divisor"><tt>profiler.hbase.salt.divisor</tt></a> </td>
+<td> A salt is prepended to the row key to help prevent hot-spotting.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.hbase.table"><tt>profiler.hbase.table</tt></a> </td>
+<td> The name of the HBase table that profiles are written to.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.hbase.column.family"><tt>profiler.hbase.column.family</tt></a> </td>
+<td> The column family used to store profiles.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.hbase.batch"><tt>profiler.hbase.batch</tt></a> </td>
+<td> The number of puts that are written to HBase in a single batch.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.hbase.flush.interval.seconds"><tt>profiler.hbase.flush.interval.seconds</tt></a> </td>
+<td> The maximum number of seconds between batch writes to HBase.</td></tr>
+<tr class="a">
+<td> <a href="#topology.kryo.register"><tt>topology.kryo.register</tt></a> </td>
+<td> Storm will use Kryo serialization for these classes.</td></tr>
+<tr class="b">
+<td> <a href="#profiler.writer.batchSize"><tt>profiler.writer.batchSize</tt></a> </td>
+<td> The number of records to batch when writing to Kakfa.</td></tr>
+<tr class="a">
+<td> <a href="#profiler.writer.batchTimeout"><tt>profiler.writer.batchTimeout</tt></a> </td>
+<td> The timeout in ms for batching when writing to Kakfa.</td></tr>
+</tbody>
+</table>
+<div class="section">
+<h3><a name="profiler.input.topic"></a><tt>profiler.input.topic</tt></h3>
+<p><i>Default</i>: indexing</p>
+<p>The name of the Kafka topic from which to consume data. By default, the Profiler consumes data from the <tt>indexing</tt> topic so that it has access to fully enriched telemetry.</p></div>
+<div class="section">
+<h3><a name="profiler.output.topic"></a><tt>profiler.output.topic</tt></h3>
+<p><i>Default</i>: enrichments</p>
+<p>The name of the Kafka topic to which profile data is written. This property is only applicable to profiles that define the <a href="#result"><tt>result</tt> <tt>triage</tt> field</a>. This allows Profile data to be selectively triaged like any other source of telemetry in Metron.</p></div>
+<div class="section">
+<h3><a name="profiler.period.duration"></a><tt>profiler.period.duration</tt></h3>
+<p><i>Default</i>: 15</p>
+<p>The duration of each profile period. This value should be defined along with <a href="#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a>.</p>
+<p><i>Important</i>: To read a profile using the <a href="metron-analytics/metron-profiler-client/index.html">Profiler Client</a>, the Profiler Client’s <tt>profiler.client.period.duration</tt> property must match this value. Otherwise, the Profiler Client will be unable to read the profile data.</p></div>
+<div class="section">
+<h3><a name="profiler.period.duration.units"></a><tt>profiler.period.duration.units</tt></h3>
+<p><i>Default</i>: MINUTES</p>
+<p>The units used to specify the <tt>profiler.period.duration</tt>. This value should be defined along with <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a>.</p>
+<p><i>Important</i>: To read a profile using the Profiler Client, the Profiler Client’s <tt>profiler.client.period.duration.units</tt> property must match this value. Otherwise, the <a href="metron-analytics/metron-profiler-client/index.html">Profiler Client</a> will be unable to read the profile data.</p></div>
+<div class="section">
+<h3><a name="profiler.window.duration"></a><tt>profiler.window.duration</tt></h3>
+<p><i>Default</i>: 30</p>
+<p>The duration of each profile window. Telemetry that arrives within a slice of time is processed within a single window.</p>
+<p>Many windows of telemetry will be processed during a single profile period. This does not change the output of the Profiler, it only changes how the Profiler processes data. The window defines how much data the Profiler processes in a single pass.</p>
+<p>This value should be defined along with <a href="#profiler.window.duration.units"><tt>profiler.window.duration.units</tt></a>.</p>
+<p>This value must be less than the period duration as defined by <a href="#profiler.period.duration"><tt>profiler.period.duration</tt></a> and <a href="#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a>.</p></div>
+<div class="section">
+<h3><a name="profiler.window.duration.units"></a><tt>profiler.window.duration.units</tt></h3>
+<p><i>Default</i>: SECONDS</p>
+<p>The units used to specify the <tt>profiler.window.duration</tt>. This value should be defined along with <a href="#profiler.window.duration"><tt>profiler.window.duration</tt></a>.</p></div>
+<div class="section">
+<h3><a name="profiler.window.lag"></a><tt>profiler.window.lag</tt></h3>
+<p><i>Default</i>: 1</p>
+<p>The maximum time lag for timestamps. Timestamps cannot arrive out-of-order by more than this amount. This value should be defined along with <a href="#profiler.window.lag.units"><tt>profiler.window.lag.units</tt></a>.</p></div>
+<div class="section">
+<h3><a name="profiler.window.lag.units"></a><tt>profiler.window.lag.units</tt></h3>
+<p><i>Default</i>: SECONDS</p>
+<p>The units used to specify the <tt>profiler.window.lag</tt>. This value should be defined along with <a href="#profiler.window.lag"><tt>profiler.window.lag</tt></a>.</p></div>
+<div class="section">
+<h3><a name="profiler.workers"></a><tt>profiler.workers</tt></h3>
+<p><i>Default</i>: 1</p>
+<p>The number of worker processes to create for the Profiler topology. This property is useful for performance tuning the Profiler.</p></div>
+<div class="section">
+<h3><a name="profiler.executors"></a><tt>profiler.executors</tt></h3>
+<p><i>Default</i>: 0</p>
+<p>The number of executors to spawn per component for the Profiler topology. This property is useful for performance tuning the Profiler.</p></div>
+<div class="section">
+<h3><a name="profiler.ttl"></a><tt>profiler.ttl</tt></h3>
+<p><i>Default</i>: 30</p>
+<p>If a message has not been applied to a Profile in this period of time, the Profile will be terminated and its resources will be cleaned up. This value should be defined along with <a href="#profiler.ttl.units"><tt>profiler.ttl.units</tt></a>.</p>
+<p>This time-to-live does not affect the persisted Profile data in HBase. It only affects the state stored in memory during the execution of the latest profile period. This state will be deleted if the time-to-live is exceeded.</p></div>
+<div class="section">
+<h3><a name="profiler.ttl.units"></a><tt>profiler.ttl.units</tt></h3>
+<p><i>Default</i>: MINUTES</p>
+<p>The units used to specify the <a href="#profiler.ttl"><tt>profiler.ttl</tt></a>.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.salt.divisor"></a><tt>profiler.hbase.salt.divisor</tt></h3>
+<p><i>Default</i>: 1000</p>
+<p>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. This constant should be roughly equal to the number of nodes in the Hbase cluster to ensure even distribution of data.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.table"></a><tt>profiler.hbase.table</tt></h3>
+<p><i>Default</i>: profiler</p>
+<p>The name of the HBase table that profile data is written to. The Profiler expects that the table exists and is writable. It will not create the table.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.column.family"></a><tt>profiler.hbase.column.family</tt></h3>
+<p><i>Default</i>: P</p>
+<p>The column family used to store profile data in HBase.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.batch"></a><tt>profiler.hbase.batch</tt></h3>
+<p><i>Default</i>: 10</p>
+<p>The number of puts that are written to HBase in a single batch.</p></div>
+<div class="section">
+<h3><a name="profiler.hbase.flush.interval.seconds"></a><tt>profiler.hbase.flush.interval.seconds</tt></h3>
+<p><i>Default</i>: 30</p>
+<p>The maximum number of seconds between batch writes to HBase.</p></div>
+<div class="section">
+<h3><a name="topology.kryo.register"></a><tt>topology.kryo.register</tt></h3>
+<p><i>Default</i>:</p>
+
+<div>
+<div>
+<pre class="source">[ org.apache.metron.profiler.ProfileMeasurement, \
+ org.apache.metron.profiler.ProfilePeriod, \
+ org.apache.metron.common.configuration.profiler.ProfileResult, \
+ org.apache.metron.common.configuration.profiler.ProfileResultExpressions, \
+ org.apache.metron.common.configuration.profiler.ProfileTriageExpressions, \
+ org.apache.metron.common.configuration.profiler.ProfilerConfig, \
+ org.apache.metron.common.configuration.profiler.ProfileConfig, \
+ org.json.simple.JSONObject, \
+ java.util.LinkedHashMap, \
+ org.apache.metron.statistics.OnlineStatisticsProvider ]
+</pre></div></div>
+
+<p>Storm will use Kryo serialization for these classes. Kryo serialization is more performant than Java serialization, in most cases.</p>
+<p>For these classes, Storm will uses Kryo’s <tt>FieldSerializer</tt> as defined in the <a class="externalLink" href="http://storm.apache.org/releases/1.1.2/Serialization.html">Storm Serialization docs</a>. For all other classes not in this list, Storm defaults to using Java serialization which is slower and not recommended for a production topology.</p>
+<p>This value should only need altered if you have defined a profile that results in a non-primitive, user-defined type that is not in this list. If the class is not defined in this list, Java serialization will be used and the class must adhere to Java’s serialization requirements.</p>
+<p>The performance of the entire Profiler topology can be negatively impacted if any profile produces results that undergo Java serialization.</p></div></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-analytics/metron-statistics/HLLP.html b/current-book/metron-analytics/metron-statistics/HLLP.html
index 759f04d..e7434db 100644
--- a/current-book/metron-analytics/metron-statistics/HLLP.html
+++ b/current-book/metron-analytics/metron-statistics/HLLP.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-statistics/HLLP.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-statistics/HLLP.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – HyperLogLogPlus</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">HyperLogLogPlus</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -48,8 +48,11 @@
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
- <li><a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li>
<li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-down"></span>Statistics</a>
<ul class="nav nav-list">
<li class="active"><a href="#"><span class="none"></span>HLLP</a></li>
diff --git a/current-book/metron-analytics/metron-statistics/index.html b/current-book/metron-analytics/metron-statistics/index.html
index f378dce..25f92e5 100644
--- a/current-book/metron-analytics/metron-statistics/index.html
+++ b/current-book/metron-analytics/metron-statistics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-statistics/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-statistics/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Statistics and Mathematical Functions</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Statistics and Mathematical Functions</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -48,8 +48,11 @@
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
- <li><a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"><span class="none"></span>Profiler</a></li>
<li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
+ <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li class="active"><a href="#"><span class="icon-chevron-down"></span>Statistics</a>
<ul class="nav nav-list">
<li><a href="../../metron-analytics/metron-statistics/HLLP.html" title="HLLP"><span class="none"></span>HLLP</a></li>
diff --git a/current-book/metron-contrib/metron-docker/index.html b/current-book/metron-contrib/metron-docker/index.html
index 4e6f954..2bd84d2 100644
--- a/current-book/metron-contrib/metron-docker/index.html
+++ b/current-book/metron-contrib/metron-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-contrib/metron-docker/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-contrib/metron-docker/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Docker</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Docker</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-contrib/metron-performance/index.html b/current-book/metron-contrib/metron-performance/index.html
index afc6ba0..675f5c3 100644
--- a/current-book/metron-contrib/metron-performance/index.html
+++ b/current-book/metron-contrib/metron-performance/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-contrib/metron-performance/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-contrib/metron-performance/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Performance Utilities</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Performance Utilities</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/Kerberos-ambari-setup.html b/current-book/metron-deployment/Kerberos-ambari-setup.html
index ddc7e07..7f2541f 100644
--- a/current-book/metron-deployment/Kerberos-ambari-setup.html
+++ b/current-book/metron-deployment/Kerberos-ambari-setup.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/Kerberos-ambari-setup.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/Kerberos-ambari-setup.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Setting Up Kerberos in Vagrant Full Dev</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Setting Up Kerberos in Vagrant Full Dev</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/Kerberos-manual-setup.html b/current-book/metron-deployment/Kerberos-manual-setup.html
index 5fd1b12..89e052b 100644
--- a/current-book/metron-deployment/Kerberos-manual-setup.html
+++ b/current-book/metron-deployment/Kerberos-manual-setup.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/Kerberos-manual-setup.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/Kerberos-manual-setup.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Kerberos Setup</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Kerberos Setup</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -117,7 +117,7 @@
<li><a href="#Start_Metron">Start Metron</a></li>
<li><a href="#Push_Data">Push Data</a></li>
<li><a href="#More_Information">More Information</a></li>
-<li><a href="#x-pack">Elasticseach X-Pack</a></li>
+<li><a href="#X-Pack">Elasticseach X-Pack</a></li>
</ul>
<div class="section">
<h2><a name="Setup"></a>Setup</h2>
@@ -798,17 +798,16 @@
</li>
<li>
-<p>New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below.</p>
-<p>Add the <tt>es.client.settings</tt> to global.json</p>
+<p>New settings have been added to configure the Elasticsearch client.</p>
+<p>Modify the <tt>es.client.settings</tt> key in global.json</p>
<div>
<div>
-<pre class="source">/usr/metron/0.6.0/config/zookeeper/global.json ->
+<pre class="source">$METRON_HOME/config/zookeeper/global.json ->
"es.client.settings" : {
- "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient",
- "es.xpack.username" : "transport_client_user",
- "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password"
+ "xpack.username" : "transport_client_user",
+ "xpack.password.file" : "/apps/metron/elasticsearch/xpack-password"
}
</pre></div></div>
@@ -821,160 +820,6 @@
</li>
<li>
-<p>The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help.</p>
-
-<div>
-<div>
-<pre class="source"><?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software
- Foundation (ASF) under one or more contributor license agreements. See the
- NOTICE file distributed with this work for additional information regarding
- copyright ownership. The ASF licenses this file to You under the Apache License,
- Version 2.0 (the "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software distributed
- under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
- OR CONDITIONS OF ANY KIND, either express or implied. See the License for
- the specific language governing permissions and limitations under the License.
- -->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <groupId>org.elasticsearch</groupId>
- <artifactId>elasticsearch-xpack-shaded</artifactId>
- <name>elasticsearch-xpack-shaded</name>
- <packaging>jar</packaging>
- <version>5.6.2</version>
- <repositories>
- <repository>
- <id>elasticsearch-releases</id>
- <url>https://artifacts.elastic.co/maven</url>
- <releases>
- <enabled>true</enabled>
- </releases>
- <snapshots>
- <enabled>false</enabled>
- </snapshots>
- </repository>
- </repositories>
- <dependencies>
- <dependency>
- <groupId>org.elasticsearch.client</groupId>
- <artifactId>x-pack-transport</artifactId>
- <version>5.6.2</version>
- <exclusions>
- <exclusion>
- <groupId>com.fasterxml.jackson.dataformat</groupId>
- <artifactId>jackson-dataformat-yaml</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.fasterxml.jackson.dataformat</groupId>
- <artifactId>jackson-dataformat-cbor</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion> <!-- this is causing a weird build error if not excluded - Error creating shaded jar: null: IllegalArgumentException -->
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- </dependencies>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <version>2.4.3</version>
- <configuration>
- <createDependencyReducedPom>true</createDependencyReducedPom>
- </configuration>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <filters>
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <relocations>
- <relocation>
- <pattern>io.netty</pattern>
- <shadedPattern>org.apache.metron.io.netty</shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.apache.logging.log4j</pattern>
- <shadedPattern>org.apache.metron.logging.log4j</shadedPattern>
- </relocation>
- </relocations>
- <artifactSet>
- <excludes>
- <exclude>org.slf4j.impl*</exclude>
- <exclude>org.slf4j:slf4j-log4j*</exclude>
- </excludes>
- </artifactSet>
- <transformers>
- <transformer
- implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
- <resources>
- <resource>.yaml</resource>
- <resource>LICENSE.txt</resource>
- <resource>ASL2.0</resource>
- <resource>NOTICE.txt</resource>
- </resources>
- </transformer>
- <transformer
- implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
- <transformer
- implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <mainClass></mainClass>
- </transformer>
- </transformers>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-</project>
-</pre></div></div>
-</li>
-<li>
-
-<p>Once you’ve built the <tt>elasticsearch-xpack-shaded-5.6.2.jar</tt>, it needs to be made available to Storm when you submit the topology. Create a contrib directory for indexing and put the jar file in this directory.</p>
-
-<div>
-<div>
-<pre class="source">/usr/metron/0.6.0/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar
-</pre></div></div>
-</li>
-<li>
-
<p>Now you can restart the Elasticsearch topology. Note, you should perform this step manually, as follows.</p>
<div>
diff --git a/current-book/metron-deployment/amazon-ec2/index.html b/current-book/metron-deployment/amazon-ec2/index.html
index b3bb627..200f02d 100644
--- a/current-book/metron-deployment/amazon-ec2/index.html
+++ b/current-book/metron-deployment/amazon-ec2/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/amazon-ec2/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/amazon-ec2/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Apache Metron on Amazon EC2</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Apache Metron on Amazon EC2</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -122,7 +122,7 @@
<p>The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.</p>
<ul>
-<li>Ansible 2.0.0.2, 2.2.2.0, or 2.5.0</li>
+<li>Ansible 2.4.0+</li>
<li>Python 2.7.11</li>
<li>Maven 3.3.9</li>
</ul>
diff --git a/current-book/metron-deployment/ansible/index.html b/current-book/metron-deployment/ansible/index.html
index c630757..544c186 100644
--- a/current-book/metron-deployment/ansible/index.html
+++ b/current-book/metron-deployment/ansible/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ansible Assets</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Ansible Assets</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/ansible/roles/index.html b/current-book/metron-deployment/ansible/roles/index.html
index 70e9ee6..cd3d63c 100644
--- a/current-book/metron-deployment/ansible/roles/index.html
+++ b/current-book/metron-deployment/ansible/roles/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ansible Roles</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Ansible Roles</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/ansible/roles/opentaxii/index.html b/current-book/metron-deployment/ansible/roles/opentaxii/index.html
index dbdaea0..40a014d 100644
--- a/current-book/metron-deployment/ansible/roles/opentaxii/index.html
+++ b/current-book/metron-deployment/ansible/roles/opentaxii/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/opentaxii/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/opentaxii/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – OpenTAXII</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">OpenTAXII</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/ansible/roles/pcap_replay/index.html b/current-book/metron-deployment/ansible/roles/pcap_replay/index.html
index 1b9e1db..f569f7d 100644
--- a/current-book/metron-deployment/ansible/roles/pcap_replay/index.html
+++ b/current-book/metron-deployment/ansible/roles/pcap_replay/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/pcap_replay/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/pcap_replay/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pcap Replay</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Pcap Replay</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html b/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html
index 92a315b..8b75316 100644
--- a/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html
+++ b/current-book/metron-deployment/ansible/roles/sensor-stubs/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-stubs/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-stubs/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Stubs</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Sensor Stubs</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html b/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html
index 54aed37..9716def 100644
--- a/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html
+++ b/current-book/metron-deployment/ansible/roles/sensor-test-mode/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-test-mode/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/ansible/roles/sensor-test-mode/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Test Mode</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Sensor Test Mode</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/development/centos6/index.html b/current-book/metron-deployment/development/centos6/index.html
index aefc706..efb7a53 100644
--- a/current-book/metron-deployment/development/centos6/index.html
+++ b/current-book/metron-deployment/development/centos6/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/centos6/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/centos6/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron on CentOS 6</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron on CentOS 6</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -119,7 +119,7 @@
<p>The computer used to deploy Apache Metron will need to have the following components installed.</p>
<ul>
-<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> (2.0.0.2, 2.2.2.0, or 2.5.0)</li>
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.4.0+</li>
<li><a class="externalLink" href="https://www.docker.com/community-edition">Docker</a></li>
<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 2.0+</li>
<li><a class="externalLink" href="https://github.com/devopsgroup-io/vagrant-hostmanager">Vagrant Hostmanager Plugin</a></li>
@@ -150,10 +150,9 @@
<div>
<div>
-<pre class="source">brew cask install vagrant virtualbox docker
+<pre class="source">brew cask install vagrant virtualbox docker ansible
brew cask install caskroom/versions/java8
brew install maven@3.3 git
-pip install ansible==2.2.2.0
vagrant plugin install vagrant-hostmanager
open /Applications/Docker.app
</pre></div></div>
@@ -209,14 +208,35 @@
<pre class="source">vagrant --ansible-tags="sensor-stubs" provision
</pre></div></div>
-<p>Tags are listed in the playbooks, some frequently used tags:</p>
+<p>Tags are listed in the playbooks. Here are some frequently used tags:</p>
<ul>
<li><tt>hdp-install</tt> - Install HDP</li>
<li><tt>hdp-deploy</tt> - Deploy and Start HDP Services (will start all Hadoop Services)</li>
-<li><tt>sensors</tt> - Deploy and start the sensors.</li>
+<li><tt>sensors</tt> - Deploy the sensors (see <a href="#Sensors">Sensors</a> for more details regarding this tag)</li>
<li><tt>sensor-stubs</tt> - Deploy and start the sensor stubs.</li>
-</ul></div></div>
+</ul>
+<div class="section">
+<h4><a name="Sensors"></a>Sensors</h4>
+<p>By default, the Metron development environment uses sensor stubs to mimic the behavior of the full sensors. This is done because the full sensors take a significant amount of time and CPU to build, install, and run.</p>
+<p>From time to time you may want to install the full sensors for testing (see the specifics of what that means <a href="../../ansible/playbooks/sensor_install.yml">here</a>). This can be done by running the following command:</p>
+
+<div>
+<div>
+<pre class="source">vagrant --ansible-skip-tags="sensor-stubs" up
+</pre></div></div>
+
+<p>This will skip only the <tt>sensor-stubs</tt> tag, allowing the ansible roles with the <tt>sensors</tt> tag to be run. This provisions the full sensors in a ‘testing mode’ so that they are more active, and thus more useful for testing (more details on that <a href="../../ansible/roles/sensor-test-mode/index.html">here</a>). <b>However</b>, when vagrant completes the sensors will NOT be running. In order to start the sensors and simulate traffic through them (which will create a fair amount of load on your test system), complete the below steps:</p>
+
+<div>
+<div>
+<pre class="source">vagrant ssh
+sudo su -
+service pcap-replay restart
+service yaf restart
+service snortd restart
+service snort-producer restart
+</pre></div></div></div></div></div>
</div>
</div>
</div>
diff --git a/current-book/metron-deployment/development/fastcapa/index.html b/current-book/metron-deployment/development/fastcapa/index.html
index c209ef5..ca9a216 100644
--- a/current-book/metron-deployment/development/fastcapa/index.html
+++ b/current-book/metron-deployment/development/fastcapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/fastcapa/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/fastcapa/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa Test Environment</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Fastcapa Test Environment</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/development/index.html b/current-book/metron-deployment/development/index.html
index 6b899ae..7152a3f 100644
--- a/current-book/metron-deployment/development/index.html
+++ b/current-book/metron-deployment/development/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Development Environments</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Development Environments</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -121,6 +121,53 @@
<div class="section">
<h2><a name="Vagrant_Cachier_recommendations"></a>Vagrant Cachier recommendations</h2>
<p>The development boxes are designed to be spun up and destroyed on a regular basis as part of the development cycle. In order to avoid the overhead of re-downloading many of the heavy platform dependencies, Vagrant can use the <a class="externalLink" href="http://fgrehm.viewdocs.io/vagrant-cachier/">vagrant-cachier</a> plugin to store package caches between builds. If the plugin has been installed to your vagrant it will be used, and packages will be cached in ~/.vagrant/cache.</p></div>
+<div class="section">
+<h2><a name="Knox_Demo_LDAP"></a>Knox Demo LDAP</h2>
+<p>The development environment can be set up to authenticate against Knox’s demo LDAP.</p>
+<p>A couple notes</p>
+<ul>
+
+<li>A custom LDIF file is used to setup users. This is to get the roles and passwords setup correctly.</li>
+<li>The demo LDAP uses plaintext passwords with no encryption prefix (e.g. {SSHA}).</li>
+<li>You may need or want to shut down any or all of the topologies. This is optional, but clears some room</li>
+</ul>
+<p>To setup this up, start full dev.</p>
+<ul>
+
+<li>
+
+<p>In Ambari, add the Knox service (Actions -> +Add Service). Accept all defaults and let it install. The configs that will be set how we need by default are:</p>
+<ul>
+
+<li>LDAP URL = <a class="externalLink" href="ldap://localhost:33389">ldap://localhost:33389</a></li>
+<li>User dn pattern = uid={0},ou=people,dc=hadoop,dc=apache,dc=org</li>
+<li>LDAP user searchbase = ou=people,dc=hadoop,dc=apache,dc=org</li>
+<li>Group Search Base = ou=groups,dc=hadoop,dc=apache,dc=org</li>
+<li>Group Search Filter = member={0}</li>
+<li>User Base DN = uid=admin,ou=people,dc=hadoop,dc=apache,dc=org</li>
+<li>User Search Filter is empty</li>
+<li>User password attribute = userPassword</li>
+<li>LDAP group role attribute = cn</li>
+<li>Bind User = uid=admin,ou=people,dc=hadoop,dc=apache,dc=org</li>
+<li>LDAP Truststore is empty</li>
+<li>LDAP Truststore Password is empty</li>
+</ul>
+</li>
+<li>
+
+<p>In the Knox configuration, go to “Advanced users-ldif”. We have a custom ldif file “knox-demo-ldap.ldif” in “metron-deployment/development” that contains a customized variant of the users and groups defined here. Replace the default ldif configuration with the contents of “knox-demo-ldap.ldif”</p>
+</li>
+<li>Start the Demo LDAP (In Knox, "Service Actions -> Start Demo LDAP)</li>
+<li>In Metron’s configs, we’re going to make two changes
+<ul>
+
+<li>Set “LDAP Enabled” to “On”</li>
+<li>In Security, set “Bind user password” to match the admin user’s password from the ldif file (admin-password).</li>
+</ul>
+</li>
+<li>Restart the REST application</li>
+</ul>
+<p>Now, when you go to Swagger or the UIs, you should be able to give a user and password. “admin” will have the roles ROLE_ADMIN and ROLE_USER, which can be verified via the “/whoami/roles” endpoint in Swagger. Similarly, there is a user “sam” that only has ROLE_USER. A third user, “tom” has neither role.</p></div>
</div>
</div>
</div>
diff --git a/current-book/metron-deployment/development/ubuntu14/index.html b/current-book/metron-deployment/development/ubuntu14/index.html
index 90962dc..3ddc804 100644
--- a/current-book/metron-deployment/development/ubuntu14/index.html
+++ b/current-book/metron-deployment/development/ubuntu14/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/ubuntu14/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/development/ubuntu14/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron on Ubuntu 14</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron on Ubuntu 14</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -119,7 +119,7 @@
<p>The computer used to deploy Apache Metron will need to have the following components installed.</p>
<ul>
-<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> (2.0.0.2, 2.2.2.0, or 2.5.0)</li>
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.4.0+</li>
<li><a class="externalLink" href="https://www.docker.com/community-edition">Docker</a></li>
<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 2.0+</li>
<li><a class="externalLink" href="https://github.com/devopsgroup-io/vagrant-hostmanager">Vagrant Hostmanager Plugin</a></li>
@@ -150,10 +150,9 @@
<div>
<div>
-<pre class="source">brew cask install vagrant virtualbox docker
+<pre class="source">brew cask install vagrant virtualbox docker ansible
brew cask install caskroom/versions/java8
brew install maven@3.3 git
-pip install ansible==2.2.2.0
vagrant plugin install vagrant-hostmanager
open /Applications/Docker.app
</pre></div></div>
@@ -209,14 +208,35 @@
<pre class="source">vagrant --ansible-tags="sensor-stubs" provision
</pre></div></div>
-<p>Tags are listed in the playbooks, some frequently used tags:</p>
+<p>Tags are listed in the playbooks. Here are some frequently used tags:</p>
<ul>
<li><tt>hdp-install</tt> - Install HDP</li>
<li><tt>hdp-deploy</tt> - Deploy and Start HDP Services (will start all Hadoop Services)</li>
-<li><tt>sensors</tt> - Deploy and start the sensors.</li>
+<li><tt>sensors</tt> - Deploy the sensors (see <a href="#Sensors">Sensors</a> for more details regarding this tag)</li>
<li><tt>sensor-stubs</tt> - Deploy and start the sensor stubs.</li>
-</ul></div></div>
+</ul>
+<div class="section">
+<h4><a name="Sensors"></a>Sensors</h4>
+<p>By default, the Metron development environment uses sensor stubs to mimic the behavior of the full sensors. This is done because the full sensors take a significant amount of time and CPU to build, install, and run.</p>
+<p>From time to time you may want to install the full sensors for testing (see the specifics of what that means <a href="../../ansible/playbooks/sensor_install.yml">here</a>). This can be done by running the following command:</p>
+
+<div>
+<div>
+<pre class="source">vagrant --ansible-skip-tags="sensor-stubs" up
+</pre></div></div>
+
+<p>This will skip only the <tt>sensor-stubs</tt> tag, allowing the ansible roles with the <tt>sensors</tt> tag to be run. This provisions the full sensors in a ‘testing mode’ so that they are more active, and thus more useful for testing (more details on that <a href="../../ansible/roles/sensor-test-mode/index.html">here</a>). <b>However</b>, when vagrant completes the sensors will NOT be running. In order to start the sensors and simulate traffic through them (which will create a fair amount of load on your test system), complete the below steps:</p>
+
+<div>
+<div>
+<pre class="source">vagrant ssh
+sudo su -
+service pcap-replay restart
+service yaf restart
+service snortd restart
+service snort-producer restart
+</pre></div></div></div></div></div>
</div>
</div>
</div>
diff --git a/current-book/metron-deployment/index.html b/current-book/metron-deployment/index.html
index a4acbe2..5a28ee4 100644
--- a/current-book/metron-deployment/index.html
+++ b/current-book/metron-deployment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/other-examples/index.html b/current-book/metron-deployment/other-examples/index.html
index 8e2e91a..d18a58b 100644
--- a/current-book/metron-deployment/other-examples/index.html
+++ b/current-book/metron-deployment/other-examples/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Other Example Deployments</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Other Example Deployments</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html b/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
index e9f7820..56622a5 100644
--- a/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
+++ b/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html b/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html
index 59620a6..93b2902 100644
--- a/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html
+++ b/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/elasticsearch-mpack/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/elasticsearch-mpack/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/ambari/index.html b/current-book/metron-deployment/packaging/ambari/index.html
index 847308f..51336b6 100644
--- a/current-book/metron-deployment/packaging/ambari/index.html
+++ b/current-book/metron-deployment/packaging/ambari/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ambari Management Pack Development</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Ambari Management Pack Development</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html b/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html
index f002768..d5fdf0b 100644
--- a/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html
+++ b/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
index 299cff1..3689ddc 100644
--- a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
+++ b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/ansible-docker/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/ansible-docker/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/docker/deb-docker/index.html b/current-book/metron-deployment/packaging/docker/deb-docker/index.html
index 18722a0..3b3cc79 100644
--- a/current-book/metron-deployment/packaging/docker/deb-docker/index.html
+++ b/current-book/metron-deployment/packaging/docker/deb-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/deb-docker/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/deb-docker/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
index 81c07ca..6c7e9a1 100644
--- a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
+++ b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/rpm-docker/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/docker/rpm-docker/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-deployment/packaging/packer-build/index.html b/current-book/metron-deployment/packaging/packer-build/index.html
index 15da9ef..706fe21 100644
--- a/current-book/metron-deployment/packaging/packer-build/index.html
+++ b/current-book/metron-deployment/packaging/packer-build/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/packer-build/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/packer-build/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Build Metron Images</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Build Metron Images</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-interface/metron-alerts/index.html b/current-book/metron-interface/metron-alerts/index.html
index afe4f1c..a150adb 100644
--- a/current-book/metron-interface/metron-alerts/index.html
+++ b/current-book/metron-interface/metron-alerts/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-alerts/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-alerts/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active "></li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-interface/metron-config/index.html b/current-book/metron-interface/metron-config/index.html
index 83bc940..6b781f4 100644
--- a/current-book/metron-interface/metron-config/index.html
+++ b/current-book/metron-interface/metron-config/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-config/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-config/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Management UI</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Management UI</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-interface/metron-rest/index.html b/current-book/metron-interface/metron-rest/index.html
index dd5b4b4..54a8861 100644
--- a/current-book/metron-interface/metron-rest/index.html
+++ b/current-book/metron-interface/metron-rest/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-rest/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-interface/metron-rest/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron REST</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron REST</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -396,7 +396,48 @@
METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
</pre></div></div>
-</div></div>
+</div>
+<div class="section">
+<h3><a name="LDAP"></a>LDAP</h3>
+<p>Metron REST can be configured to use LDAP for authentication and roles. Configuration can be performed via Ambari in the “Security” tab.</p>
+<p>Configuration will default to matching Knox’s Demo LDAP for convenience. This should only be used for development purposes. Manual instructions for setting up demo LDAP and finalizing configuration (e.g. setting up the user LDIF file) can be found in the <a href="../../metron-deployment/development/index.html#knox-demo-ldap">Development README</a>.</p>
+<div class="section">
+<h4><a name="LDAPS"></a>LDAPS</h4>
+<p>There is configuration to provide a path to a truststore with SSL certificates and provide a password. Users should import certificates as needed to appropriate truststores. An example of doing this is:</p>
+
+<div>
+<div>
+<pre class="source">keytool -import -alias <alias> -file <certificate> -keystore <keystore_file> -storepass <password>
+</pre></div></div>
+</div>
+<div class="section">
+<h4><a name="Roles"></a>Roles</h4>
+<p>Roles used by Metron are ROLE_ADMIN and ROLE_USER. Metron will use a property in a group containing the appropriate role to construct this.</p>
+<p>For example, our ldif file could create this group:</p>
+
+<div>
+<div>
+<pre class="source">dn: cn=admin,ou=groups,dc=hadoop,dc=apache,dc=org
+objectclass:top
+objectclass: groupofnames
+cn: admin
+description:admin group
+member: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
+</pre></div></div>
+
+<p>If we are using “cn” as our role attribute, Metron will give the “admin” user the role “ROLE_ADMIN”.</p>
+<p>Similarly, we could give a user “sam” ROLE_USER with the following group:</p>
+
+<div>
+<div>
+<pre class="source">dn: cn=user,ou=groups,dc=hadoop,dc=apache,dc=org
+objectclass:top
+objectclass: groupofnames
+cn: user
+description: user group
+member: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
+</pre></div></div>
+</div></div></div>
<div class="section">
<h2><a name="Spring_Profiles"></a>Spring Profiles</h2>
<p>The REST application comes with a few <a class="externalLink" href="http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html">Spring Profiles</a> to aid in testing and development.</p>
@@ -434,7 +475,7 @@
<p>The REST application exposes endpoints for querying Pcap data. For more information about filtering options see <a href="../../metron-platform/metron-pcap-backend/index.html#Query_Filter_Utility">Query Filter Utility</a>.</p>
<p>There is an endpoint available that will return Pcap data in <a class="externalLink" href="https://wiki.wireshark.org/PDML">PDML</a> format. <a class="externalLink" href="https://www.wireshark.org/">Wireshark</a> must be installed for this feature to work. Installing wireshark in CentOS can be done with <tt>yum -y install wireshark</tt>.</p>
<p>The REST application uses a Java Process object to call out to the <tt>pcap_to_pdml.sh</tt> script. This script is installed at <tt>$METRON_HOME/bin/pcap_to_pdml.sh</tt> by default. Out of the box it is a simple wrapper around the tshark command to transform raw pcap data to PDML. However it can be extended to do additional processing as long as the expected input/output is maintained. REST will supply the script with raw pcap data through standard in and expects PDML data serialized as XML.</p>
-<p>Pcap query jobs can be configured for submission to a YARN queue. This setting is exposed as the Spring property <tt>pcap.yarn.queue</tt>. If configured, the REST application will set the <tt>mapreduce.job.queuename</tt> Hadoop property to that value. It is highly recommended that a dedicated YARN queue be created and configured for Pcap queries to prevent a job from consuming too many cluster resources. More information about setting up YARN queues can be found <a class="externalLink" href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setting_up_queues">here</a>.</p>
+<p>Pcap query jobs can be configured for submission to a YARN queue. This setting is exposed as the Spring property <tt>pcap.yarn.queue</tt> and can be set in the PCAP tab under Metron service -> Configs in Ambari. If configured, the REST application will set the <tt>mapreduce.job.queuename</tt> Hadoop property to that value. It is highly recommended that a dedicated YARN queue be created and configured for Pcap queries to prevent a job from consuming too many cluster resources. More information about setting up YARN queues can be found <a class="externalLink" href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setting_up_queues">here</a>.</p>
<p>Pcap query results are stored in HDFS. The location of query results when run through the REST app is determined by a couple factors. The root of Pcap query results defaults to <tt>/apps/metron/pcap/output</tt> but can be changed with the Spring property <tt>pcap.final.output.path</tt>. Assuming the default Pcap query output directory, the path to a result page will follow this pattern:</p>
<div>
@@ -444,7 +485,7 @@
<p>Over time Pcap query results will accumulate in HDFS. Currently these results are not cleaned up automatically so cluster administrators should be aware of this and monitor them. It is highly recommended that a process be put in place to periodically delete files and directories under the Pcap query results root.</p>
<p>Users should also be mindful of date ranges used in queries so they don’t produce result sets that are too large. Currently there are no limits enforced on date ranges.</p>
-<p>Queries can also be configured on a global level for setting the number of results per page via a Spring property <tt>pcap.page.size</tt>. By default, this value is set to 10 pcaps per page, but you may choose to set this value higher based on observing frequenetly-run query result sizes. This setting works in conjunction with the property for setting finalizer threadpool size when optimizing query performance.</p>
+<p>Queries can also be configured on a global level for setting the number of results per page via a Spring property <tt>pcap.page.size</tt>. This property can be set in the PCAP tab under Metron service -> Configs, in Ambari. By default, this value is set to 10 pcaps per page, but you may choose to set this value higher based on observing frequenetly-run query result sizes. This setting works in conjunction with the property for setting finalizer threadpool size when optimizing query performance.</p>
<p>Pcap query jobs have a finalization routine that writes their results out to HDFS in pages. Depending on the size of your pcaps, the number or results typically returned, page sizing (described above), and available CPU cores for running your REST application, your performance can be improved by adjusting the number of files that can be written to HDFS in parallel. To this end, there is a threadpool used for this finalization step that can be configured to use a specified number of threads. This setting is exposed as the Spring property <tt>pcap.finalizer.threadpool.size</tt>. A default value of “1” is used if not specified by the user. Generally speaking, you should see a performance gain when this value is set to anything higher than 1. A sizeable increase in performance can be achieved, especially for larger numbers of files of smaller size, by increasing the number of threads. It should be noted that this property is parsed as a String to allow for more complex parallelism values. In addition to normal integer values, you can specify a multiple of the number of cores. If it’s a string and ends with “C”, then strip the C and treat it as an integral multiple of the number of cores. If it’s a string and does not end with a C, then treat it as a number in string form.</p></div>
<div class="section">
<h2><a name="API"></a>API</h2>
@@ -627,8 +668,10 @@
<tr class="b">
<td> <a href="#PATCH_apiv1updatepatch"> <tt>PATCH /api/v1/update/patch</tt></a></td></tr>
<tr class="a">
-<td> <a href="#PUT_apiv1updatereplace"> <tt>PUT /api/v1/update/replace</tt></a></td></tr>
+<td> <a href="#put-apiv1updateaddcomment"> <tt>POST /api/v1/update/add/comment</tt></a></td></tr>
<tr class="b">
+<td> <a href="#put-apiv1updateremovecomment"> <tt>POST /api/v1/update/remove/comment</tt></a></td></tr>
+<tr class="a">
<td> <a href="#GET_apiv1user"> <tt>GET /api/v1/user</tt></a></td></tr>
</tbody>
</table>
@@ -2055,45 +2098,38 @@
</li>
</ul></div>
<div class="section">
-<h3><a name="PUT_.2Fapi.2Fv1.2Fupdate.2Freplace"></a><tt>PUT /api/v1/update/replace</tt></h3>
+<h3><a name="POST_.2Fapi.2Fv1.2Fupdate.2Fadd.2Fcomment"></a><tt>POST /api/v1/update/add/comment</tt></h3>
<ul>
-<li>Description: Replace a document</li>
+<li>Description: Add a comment to an alert</li>
<li>Input:
<ul>
-<li>request - Replacement request
-<ul>
-
-<li>guid - The Patch UUID</li>
-<li>sensorType - The sensor type</li>
-<li>replacement - A Map representing the replaced document</li>
-</ul>
-</li>
-<li>Example replacing a <tt>bro</tt> message with guid of <tt>000-000-0000</tt> :
-
-<div>
-<div>
-<pre class="source">{
- "guid" : "000-000-0000",
- "sensorType" : "bro",
- "replacement" : {
- "source:type": "bro",
- "guid" : "bro_index_2017.01.01.01:1",
- "ip_src_addr":"192.168.1.2",
- "ip_src_port": 8009,
- "timestamp":200,
- "rejected":false
- }
-}
-</pre></div></div>
-</li>
+<li>request - Comment add request</li>
</ul>
</li>
<li>Returns:
<ul>
-<li>200 - Current user</li>
+<li>200 - Returns the complete alert document with comments added.</li>
+</ul>
+</li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_.2Fapi.2Fv1.2Fupdate.2Fremove.2Fcomment"></a><tt>POST /api/v1/update/remove/comment</tt></h3>
+<ul>
+
+<li>Description: Remove a comment from an alert</li>
+<li>Input:
+<ul>
+
+<li>request - Comment remove request</li>
+</ul>
+</li>
+<li>Returns:
+<ul>
+
+<li>200 - Returns the complete alert document with comments removed.</li>
</ul>
</li>
</ul></div>
diff --git a/current-book/metron-platform/Performance-tuning-guide.html b/current-book/metron-platform/Performance-tuning-guide.html
index 3298d07..42461be 100644
--- a/current-book/metron-platform/Performance-tuning-guide.html
+++ b/current-book/metron-platform/Performance-tuning-guide.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/Performance-tuning-guide.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/Performance-tuning-guide.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Performance Tuning Guide</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Performance Tuning Guide</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -351,8 +351,9 @@
</tbody>
</table>
<p><b>Enrichment</b></p>
+<p><b>Note</b> These recommendations are based on the deprecated split-join enrichment topology. See <a href="metron-enrichment/Performance.html">Enrichment Performance</a> for tuning recommendations for the new default unified enrichment topology.</p>
<p>This is a mapping of the various performance tuning properties for enrichments and how they are materialized.</p>
-<p>Flux file found here - $METRON_HOME/flux/enrichment/remote.yaml</p>
+<p>Flux file found here - $METRON_HOME/flux/enrichment/remote-splitjoin.yaml</p>
<p><i>Note 1:</i> Changes to Flux file properties that are managed by Ambari will render Ambari unable to further manage the property.</p>
<p><i>Note 2:</i> Many of these settings will be irrelevant in the alternate non-split-join topology</p>
<table border="0" class="table table-striped">
@@ -782,7 +783,7 @@
<div>
<div>
-<pre class="source">/usr/metron/0.6.0/bin/start_parser_topology.sh \
+<pre class="source">/usr/metron/0.7.0/bin/start_parser_topology.sh \
-e ~metron/.storm/storm-bro.config \
-esc ~/.storm/spout-bro.config \
-k $BROKERLIST \
@@ -831,6 +832,7 @@
</div>
<div class="section">
<h3><a name="Enrichment_Tuning"></a>Enrichment Tuning</h3>
+<p><b>Note</b> These tuning suggestions are based on the deprecated split-join topology.</p>
<p>We landed on the same number of partitions for enrichemnt and indexing as we did for bro - 48.</p>
<p>For configuring Storm, there is a flux file and properties file that we modified. Here are the settings we changed for bro in Flux. Note that the main Metron-specific option we’ve changed to accomodate the desired rate of data throughput is max cache size in the join bolts. More information on Flux can be found here - <a class="externalLink" href="http://storm.apache.org/releases/1.0.1/flux.html">http://storm.apache.org/releases/1.0.1/flux.html</a></p>
<p><b>General storm settings</b></p>
@@ -967,7 +969,7 @@
export STORM_UI=http://node1:8744
export ELASTIC=http://node1:9200
export ZOOKEEPER=node1:2181
-export METRON_VERSION=0.6.0
+export METRON_VERSION=0.7.0
export METRON_HOME=/usr/metron/${METRON_VERSION}
</pre></div></div>
diff --git a/current-book/metron-platform/index.html b/current-book/metron-platform/index.html
index 640dda4..11d0324 100644
--- a/current-book/metron-platform/index.html
+++ b/current-book/metron-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Current Build</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Current Build</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -105,7 +105,7 @@
-->
<h1>Current Build</h1>
<p><a name="Current_Build"></a></p>
-<p>The latest build of metron-platform is 0.6.0.</p>
+<p>The latest build of metron-platform is 0.7.0.</p>
<p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p>
<p><a name="metron-platform"></a></p>
<h1>metron-platform</h1>
diff --git a/current-book/metron-platform/metron-common/index.html b/current-book/metron-platform/metron-common/index.html
index 17189bf..baadb3f 100644
--- a/current-book/metron-platform/metron-common/index.html
+++ b/current-book/metron-platform/metron-common/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-common/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-common/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Contents</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Contents</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -173,114 +173,119 @@
<th> Property Name </th>
<th> Subsystem </th>
<th> Type </th>
-<th> Ambari Property </th></tr>
+<th> Ambari Property </th></tr>
</thead><tbody>
<tr class="b">
<td> <a href="../metron-elasticsearch/index.html#es.clustername"><tt>es.clustername</tt></a> </td>
<td> Indexing </td>
<td> String </td>
-<td> <tt>es_cluster_name</tt> </td></tr>
+<td> <tt>es_cluster_name</tt> </td></tr>
<tr class="a">
<td> <a href="../metron-elasticsearch/index.html#es.ip"><tt>es.ip</tt></a> </td>
<td> Indexing </td>
<td> String </td>
-<td> <tt>es_hosts</tt> </td></tr>
+<td> <tt>es_hosts</tt> </td></tr>
<tr class="b">
<td> <a href="../metron-elasticsearch/index.html#es.port"><tt>es.port</tt></a> </td>
<td> Indexing </td>
<td> String </td>
-<td> <tt>es_port</tt> </td></tr>
+<td> <tt>es_port</tt> </td></tr>
<tr class="a">
<td> <a href="../metron-elasticsearch/index.html#es.date.format"><tt>es.date.format</tt></a> </td>
<td> Indexing </td>
<td> String </td>
-<td> <tt>es_date_format</tt> </td></tr>
+<td> <tt>es_date_format</tt> </td></tr>
<tr class="b">
+<td> <a href="../metron-elasticsearch/index.html#es.client.settings"><tt>es.client.settings</tt></a> </td>
+<td> Indexing </td>
+<td> Object </td>
+<td> N/A </td></tr>
+<tr class="a">
<td> <a href="#validation-framework"><tt>fieldValidations</tt></a> </td>
<td> Parsing </td>
<td> Object </td>
-<td> N/A </td></tr>
-<tr class="a">
+<td> N/A </td></tr>
+<tr class="b">
<td> <a href="../metron-parsers/index.html#parser.error.topic"><tt>parser.error.topic</tt></a> </td>
<td> Parsing </td>
<td> String </td>
-<td> N/A </td></tr>
-<tr class="b">
+<td> N/A </td></tr>
+<tr class="a">
<td> <a href="../../metron-stellar/stellar-common/index.html#stellar.function.paths"><tt>stellar.function.paths</tt></a> </td>
<td> Stellar </td>
<td> CSV String </td>
-<td> N/A </td></tr>
-<tr class="a">
+<td> N/A </td></tr>
+<tr class="b">
<td> <a href="../../metron-stellar/stellar-common/index.html#stellarfunctionresolverincludesexcludes"><tt>stellar.function.resolver.includes</tt></a> </td>
<td> Stellar </td>
<td> CSV String </td>
-<td> N/A </td></tr>
-<tr class="b">
+<td> N/A </td></tr>
+<tr class="a">
<td> <a href="../../metron-stellar/stellar-common/index.html#stellarfunctionresolverincludesexcludes"><tt>stellar.function.resolver.excludes</tt></a> </td>
<td> Stellar </td>
<td> CSV String </td>
-<td> N/A </td></tr>
-<tr class="a">
+<td> N/A </td></tr>
+<tr class="b">
<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.period.duration"><tt>profiler.period.duration</tt></a> </td>
<td> Profiler </td>
<td> Integer </td>
-<td> <tt>profiler_period_duration</tt> </td></tr>
-<tr class="b">
+<td> <tt>profiler_period_duration</tt> </td></tr>
+<tr class="a">
<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a> </td>
<td> Profiler </td>
<td> String </td>
-<td> <tt>profiler_period_units</tt> </td></tr>
-<tr class="a">
+<td> <tt>profiler_period_units</tt> </td></tr>
+<tr class="b">
<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.writer.batchSize"><tt>profiler.writer.batchSize</tt></a> </td>
<td> Profiler </td>
<td> Integer </td>
-<td> N/A </td></tr>
-<tr class="b">
+<td> N/A </td></tr>
+<tr class="a">
<td> <a href="../../metron-analytics/metron-profiler/index.html#profiler.writer.batchTimeout"><tt>profiler.writer.batchTimeout</tt></a> </td>
<td> Profiler </td>
<td> Integer </td>
-<td> N/A </td></tr>
-<tr class="a">
+<td> N/A </td></tr>
+<tr class="b">
<td> <a href="../metron-indexing/index.html#update.hbase.table"><tt>update.hbase.table</tt></a> </td>
<td> REST/Indexing </td>
<td> String </td>
-<td> <tt>update_hbase_table</tt> </td></tr>
-<tr class="b">
+<td> <tt>update_hbase_table</tt> </td></tr>
+<tr class="a">
<td> <a href="../metron-indexing/index.html#update.hbase.cf"><tt>update.hbase.cf</tt></a> </td>
<td> REST/Indexing </td>
<td> String </td>
-<td> <tt>update_hbase_cf</tt> </td></tr>
-<tr class="a">
+<td> <tt>update_hbase_cf</tt> </td></tr>
+<tr class="b">
<td> <a href="../metron-enrichment/index.html#geo.hdfs.file"><tt>geo.hdfs.file</tt></a> </td>
<td> Enrichment </td>
<td> String </td>
-<td> <tt>geo_hdfs_file</tt> </td></tr>
-<tr class="b">
+<td> <tt>geo_hdfs_file</tt> </td></tr>
+<tr class="a">
<td> <a href="../metron-enrichment/index.html#enrichment.writer.batchSize"><tt>enrichment.writer.batchSize</tt></a> </td>
<td> Enrichment </td>
<td> Integer </td>
-<td> N/A </td></tr>
-<tr class="a">
+<td> N/A </td></tr>
+<tr class="b">
<td> <a href="../metron-enrichment/index.html#enrichment.writer.batchTimeout"><tt>enrichment.writer.batchTimeout</tt></a> </td>
<td> Enrichment </td>
<td> Integer </td>
-<td> N/A </td></tr>
-<tr class="b">
+<td> N/A </td></tr>
+<tr class="a">
<td> <a href="../metron-enrichment/index.html#geo.hdfs.file"><tt>geo.hdfs.file</tt></a> </td>
<td> Enrichment </td>
<td> String </td>
-<td> <tt>geo_hdfs_file</tt> </td></tr>
-<tr class="a">
+<td> <tt>geo_hdfs_file</tt> </td></tr>
+<tr class="b">
<td> <a href="../../metron-interface/metron-alerts/index.html#source.type.field"><tt>source.type.field</tt></a> </td>
<td> UI </td>
<td> String </td>
-<td> <tt>source_type_field</tt> </td></tr>
-<tr class="b">
+<td> <tt>source_type_field</tt> </td></tr>
+<tr class="a">
<td> <a href="../../metron-interface/metron-alerts/index.html#threat.triage.score.field"><tt>threat.triage.score.field</tt></a> </td>
<td> UI </td>
<td> String </td>
-<td> <tt>threat_triage_score_field</tt> </td></tr>
+<td> <tt>threat_triage_score_field</tt> </td></tr>
</tbody>
</table>
<div class="section">
@@ -634,7 +639,9 @@
Metron home directory
-p DIRECTORY, --hdp_home=DIRECTORY
HDP home directory
-</pre></div></div></div></div>
+</pre></div></div>
+
+<p>`</p></div></div>
</div>
</div>
</div>
diff --git a/current-book/metron-platform/metron-data-management/index.html b/current-book/metron-platform/metron-data-management/index.html
index 8fb49e0..5f2195c 100644
--- a/current-book/metron-platform/metron-data-management/index.html
+++ b/current-book/metron-platform/metron-data-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-data-management/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-data-management/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Resource Data Management</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Resource Data Management</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-elasticsearch/index.html b/current-book/metron-platform/metron-elasticsearch/index.html
index a9bf4da..24ee901 100644
--- a/current-book/metron-platform/metron-elasticsearch/index.html
+++ b/current-book/metron-platform/metron-elasticsearch/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-elasticsearch/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-elasticsearch/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Elasticsearch in Metron</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Elasticsearch in Metron</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -142,7 +142,146 @@
<div class="section">
<h3><a name="es.date.format"></a><tt>es.date.format</tt></h3>
<p>The date format to use when constructing the indices. For every message, the date format will be applied to the current time and that will become the last part of the index name where the message is written to.</p>
-<p>For instance, an <tt>es.date.format</tt> of <tt>yyyy.MM.dd.HH</tt> would have the consequence that the indices would roll hourly, whereas an <tt>es.date.format</tt> of <tt>yyyy.MM.dd</tt> would have the consequence that the indices would roll daily.</p></div></div>
+<p>For instance, an <tt>es.date.format</tt> of <tt>yyyy.MM.dd.HH</tt> would have the consequence that the indices would roll hourly, whereas an <tt>es.date.format</tt> of <tt>yyyy.MM.dd</tt> would have the consequence that the indices would roll daily.</p></div>
+<div class="section">
+<h3><a name="es.client.settings"></a><tt>es.client.settings</tt></h3>
+<p>This field in global config allows you to specify Elasticsearch REST client options. These are used in conjunction with the previously mentioned Elasticsearch properties when setting up client connections to an Elasticsearch cluster. The available properties should be supplied as an object map. Current available options are as follows:</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Property Name </th>
+<th> Type </th>
+<th> Required? </th>
+<th> Default Value </th>
+<th> Description </th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> connection.timeout.millis </td>
+<td> Integer </td>
+<td> No </td>
+<td> 1000 </td>
+<td> Sets connection timeout. </td></tr>
+<tr class="a">
+<td> socket.timeout.millis </td>
+<td> Integer </td>
+<td> No </td>
+<td> 30000 </td>
+<td> Sets socket timeout. </td></tr>
+<tr class="b">
+<td> max.retry.timeout.millis </td>
+<td> Integer </td>
+<td> No </td>
+<td> 30000 </td>
+<td> Sets the maximum timeout (in milliseconds) to honour in case of multiple retries of the same request. </td></tr>
+<tr class="a">
+<td> num.client.connection.threads </td>
+<td> Integer </td>
+<td> No </td>
+<td> 1 </td>
+<td> Number of worker threads used by the connection manager. Defaults to Runtime.getRuntime().availableProcessors(). </td></tr>
+<tr class="b">
+<td> xpack.username </td>
+<td> String </td>
+<td> No </td>
+<td> null </td>
+<td> X-Pack username. </td></tr>
+<tr class="a">
+<td> xpack.password.file </td>
+<td> String </td>
+<td> No </td>
+<td> null </td>
+<td> 1-line HDFS file where the X-Pack password is set. </td></tr>
+<tr class="b">
+<td> ssl.enabled </td>
+<td> Boolean </td>
+<td> No </td>
+<td> false </td>
+<td> Turn on SSL connections. </td></tr>
+<tr class="a">
+<td> keystore.type </td>
+<td> String </td>
+<td> No </td>
+<td> “jks” </td>
+<td> Allows you to specify a keytstore type. See <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyStore">https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyStore</a> for more details. </td></tr>
+<tr class="b">
+<td> keystore.path </td>
+<td> String </td>
+<td> No </td>
+<td> null </td>
+<td> Path to the Trust Store that holds your Elasticsearch certificate authorities and certificate. </td></tr>
+<tr class="a">
+<td> keystore.password.file </td>
+<td> String </td>
+<td> No </td>
+<td> null </td>
+<td> 1-line HDFS file where the keystore password is set. </td></tr>
+</tbody>
+</table>
+<p><b>Note:</b> The migration from Elasticsearch’s TransportClient to the Java REST client has resulted in some existing properties to change. Below is a mapping of the old properties to the new ones:</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Old Property Name </th>
+<th> New Property Name </th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> client.transport.ping_timeout </td>
+<td> n/a </td></tr>
+<tr class="a">
+<td> n/a </td>
+<td> connection.timeout.millis </td></tr>
+<tr class="b">
+<td> n/a </td>
+<td> socket.timeout.millis </td></tr>
+<tr class="a">
+<td> n/a </td>
+<td> max.retry.timeout.millis </td></tr>
+<tr class="b">
+<td> n/a </td>
+<td> num.client.connection.threads </td></tr>
+<tr class="a">
+<td> es.client.class </td>
+<td> n/a </td></tr>
+<tr class="b">
+<td> es.xpack.username </td>
+<td> xpack.username </td></tr>
+<tr class="a">
+<td> es.xpack.password.file </td>
+<td> xpack.password.file </td></tr>
+<tr class="b">
+<td> xpack.security.transport.ssl.enabled </td>
+<td> ssl.enabled </td></tr>
+<tr class="a">
+<td> xpack.ssl.key </td>
+<td> n/a </td></tr>
+<tr class="b">
+<td> xpack.ssl.certificate </td>
+<td> n/a </td></tr>
+<tr class="a">
+<td> xpack.ssl.certificate_authorities </td>
+<td> n/a </td></tr>
+<tr class="b">
+<td> n/a </td>
+<td> keystore.type </td></tr>
+<tr class="a">
+<td> keystore.path </td>
+<td> keystore.path </td></tr>
+<tr class="b">
+<td> n/a </td>
+<td> keystore.password.file </td></tr>
+</tbody>
+</table>
+<p><b>Notes:</b></p>
+<ul>
+
+<li>The transport client implementation provides for a ‘xpack.security.user’ property, however we never used this property directly. Rather, in order to secure the password we used custom properties for user/pass. These properties have been carried over as <tt>xpack.username</tt> and <tt>xpack.password.file</tt>.</li>
+<li>See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/_common_configuration.html">https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/_common_configuration.html</a> for more specifics on the new client properties.</li>
+<li>Other notes on JSSE - <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html">https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html</a></li>
+</ul></div></div>
<div class="section">
<h2><a name="Upgrading_to_5.6.2"></a>Upgrading to 5.6.2</h2>
<p>Users should be prepared to re-index when migrating from Elasticsearch 2.3.3 to 5.6.2. There are a number of template changes, most notably around string type handling, that may cause issues when upgrading.</p>
@@ -403,10 +542,31 @@
<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html</a></li>
<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html</a></li>
<li><a class="externalLink" href="https://www.elastic.co/blog/strings-are-dead-long-live-strings">https://www.elastic.co/blog/strings-are-dead-long-live-strings</a></li>
-</ul></div></div></div></div>
+</ul></div></div></div>
+<div class="section">
+<h3><a name="Metron_Properties"></a>Metron Properties</h3>
+<p>Metron depends on some internal fields being defined in sensor templates. A field is defined in Elasticsearch by adding an entry to the <tt>properties</tt> section of the template:</p>
+
+<div>
+<div>
+<pre class="source">"properties": {
+ "metron_field": {
+ "type": "keyword"
+ }
+}
+</pre></div></div>
+
+<p>The following is a list of properties that need to be defined along with their type:</p>
+<ul>
+
+<li>source:type - keyword</li>
+<li>alert_status - keyword</li>
+<li>metron_alert - nested</li>
+</ul></div></div>
<div class="section">
<h2><a name="Using_Metron_with_Elasticsearch_5.6.2"></a>Using Metron with Elasticsearch 5.6.2</h2>
-<p>There is a requirement that all sensors templates have a nested <tt>metron_alert</tt> field defined. This field is a dummy field. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields">Ignoring Unmapped Fields</a> for more information</p>
+<p>Although infrequent, sometimes an internal field is added in Metron and existing templates must be updated. The following steps outlines how to do this, using <tt>metron_alert</tt> as an example.</p>
+<p>With the addition of the meta alert feature, there is a requirement that all sensors templates have a nested <tt>metron_alert</tt> field defined. This field is a dummy field. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields">Ignoring Unmapped Fields</a> for more information</p>
<p>Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service’s logs.</p>
<p>Exception seen:</p>
diff --git a/current-book/metron-platform/metron-enrichment/Performance.html b/current-book/metron-platform/metron-enrichment/Performance.html
index dccd5b1..b50bbc1 100644
--- a/current-book/metron-platform/metron-enrichment/Performance.html
+++ b/current-book/metron-platform/metron-enrichment/Performance.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/Performance.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/Performance.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Enrichment Performance</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Enrichment Performance</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-enrichment/index.html b/current-book/metron-platform/metron-enrichment/index.html
index 0747140..a6d9697 100644
--- a/current-book/metron-platform/metron-enrichment/index.html
+++ b/current-book/metron-platform/metron-enrichment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-enrichment/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Enrichment</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Enrichment</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -121,27 +121,17 @@
</ul></div>
<div class="section">
<h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2>
-<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p>
+<p><img src="../../images/unified_enrichment_arch.svg" alt="Unified Architecture" /></p>
<div class="section">
<h3><a name="Unified_Enrichment_Topology"></a>Unified Enrichment Topology</h3>
-<p>There is an experimental unified enrichment topology which is shipped. Currently the architecture, as described above, has a split/join in order to perform enrichments in parallel. This poses some issues in terms of ease of tuning and reasoning about performance.</p>
-<p>In order to deal with these issues, there is an alternative enrichment topology which uses data parallelism as opposed to the split/join task parallelism. This architecture uses a worker pool to fully enrich any message within a worker. This results in</p>
+<p>The unified enrichment topology uses data parallelism as opposed to the deprecated split/join topology’s task parallelism. This architecture uses a worker pool to fully enrich any message within a worker. This results in</p>
<ul>
<li>Fewer bolts in the topology</li>
<li>Each bolt fully operates on a message.</li>
<li>Fewer network hops</li>
</ul>
-<p><img src="../../images/unified_enrichment_arch.svg" alt="Unified Architecture" /></p>
-<p>This architecture is fully backwards compatible; the only difference is how the enrichment will operate on each message (in one bolt where the split/join is done in a threadpool as opposed to split across multiple bolts).</p>
-<div class="section">
-<h4><a name="Using_It"></a>Using It</h4>
-<p>In order to use this, you will need to</p>
-<ul>
-
-<li>Edit <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> and adjust it to use <tt>remote-unified.yaml</tt> instead of <tt>remote.yaml</tt></li>
-<li>Restart the enrichment topology.</li>
-</ul></div>
+<p>This architecture is fully backwards compatible with the old split-join topology; the only difference is how the enrichment will operate on each message (in one bolt where the split/join is done in a threadpool as opposed to split across multiple bolts).</p>
<div class="section">
<h4><a name="Configuring_It"></a>Configuring It</h4>
<p>There are two parameters which you might want to tune in this topology. Both of them are topology configuration adjustable in the flux file <tt>$METRON_HOME/config/flux/enrichment/remote-unified.yaml</tt>:</p>
@@ -156,7 +146,19 @@
</ul>
</li>
</ul>
-<p>In order to configure the parallelism for the enrichment bolt and threat intel bolt, the configurations will be taken from the respective join bolt parallelism. When proper ambari support for this is added, we will add its own property.</p></div></div></div>
+<p>In order to configure the parallelism for the enrichment bolt and threat intel bolt, the configurations will be taken from the respective join bolt parallelism. When proper ambari support for this is added, we will add its own property.</p></div></div>
+<div class="section">
+<h3><a name="Split-Join_Enrichment_Topology"></a>Split-Join Enrichment Topology</h3>
+<p>The now-deprecated split/join topology is also available and performs enrichments in parallel. This poses some issues in terms of ease of tuning and reasoning about performance.</p>
+<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p>
+<div class="section">
+<h4><a name="Using_It"></a>Using It</h4>
+<p>In order to use the older, deprecated topology, you will need to</p>
+<ul>
+
+<li>Edit <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> and adjust it to use <tt>remote-splitjoin.yaml</tt> instead of <tt>remote-unified.yaml</tt></li>
+<li>Restart the enrichment topology.</li>
+</ul></div></div></div>
<div class="section">
<h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2>
<p>The configuration for the <tt>enrichment</tt> topology, the topology primarily responsible for enrichment and threat intelligence enrichment, is defined by JSON documents stored in zookeeper.</p>
diff --git a/current-book/metron-platform/metron-indexing/index.html b/current-book/metron-platform/metron-indexing/index.html
index 0057389..5763be7 100644
--- a/current-book/metron-platform/metron-indexing/index.html
+++ b/current-book/metron-platform/metron-indexing/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-indexing/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-indexing/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Indexing</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Indexing</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-job/index.html b/current-book/metron-platform/metron-job/index.html
index 43a8cc2..0fb4b7d 100644
--- a/current-book/metron-platform/metron-job/index.html
+++ b/current-book/metron-platform/metron-job/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-job/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-job/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Job</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Job</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-management/index.html b/current-book/metron-platform/metron-management/index.html
index 0c8109c..ce18f57 100644
--- a/current-book/metron-platform/metron-management/index.html
+++ b/current-book/metron-platform/metron-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-management/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-management/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Stellar REPL Management Utilities</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Stellar REPL Management Utilities</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-parsers/3rdPartyParser.html b/current-book/metron-platform/metron-parsers/3rdPartyParser.html
index 15798ba..75b1a76 100644
--- a/current-book/metron-platform/metron-parsers/3rdPartyParser.html
+++ b/current-book/metron-platform/metron-parsers/3rdPartyParser.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/3rdPartyParser.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/3rdPartyParser.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Custom Metron Parsers</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Custom Metron Parsers</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -66,7 +66,9 @@
<ul class="nav nav-list">
<li class="active"><a href="#"><span class="none"></span>3rdPartyParser</a></li>
<li><a href="../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li>
+ <li><a href="../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li>
<li><a href="../../metron-platform/metron-parsers/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li>
</ul>
</li>
<li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
diff --git a/current-book/metron-platform/metron-parsers/ParserChaining.html b/current-book/metron-platform/metron-parsers/ParserChaining.html
index aec793f..880d8c4 100644
--- a/current-book/metron-platform/metron-parsers/ParserChaining.html
+++ b/current-book/metron-platform/metron-parsers/ParserChaining.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/ParserChaining.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/ParserChaining.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parser Chaining</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Parser Chaining</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -66,7 +66,9 @@
<ul class="nav nav-list">
<li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
<li class="active"><a href="#"><span class="none"></span>ParserChaining</a></li>
+ <li><a href="../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li>
<li><a href="../../metron-platform/metron-parsers/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li>
</ul>
</li>
<li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
@@ -129,6 +131,25 @@
<li>Route the unfolded data to the appropriate kafka topic associated with the enveloped sensor data</li>
</ul>
<p>Because the data emitted from the routing parser is just like any data emitted from any other parser, in that it is a JSON blob like any data emitted from any parser, we will need to adjust the downstream parsers to extract the enveloped data from the JSON blob and treat it as the data to parse.</p>
+<p><a name="Aggregated_Parsers_with_Parser_Chaining"></a></p>
+<h1>Aggregated Parsers with Parser Chaining</h1>
+<p>Chained parsers can be run as aggregated parsers. These parsers continue to use the sensor specific Kafka topics, and do not do internal routing to the appropriate sensor.</p>
+<p>Say, there were three sensors (<tt>bro</tt>, <tt>snort</tt> and <tt>yaf</tt>). Instead of creating a topology per sensor, all 3 can be run in a single aggregated parser. It is also possible to aggregate a subset of these parsers (e.g. run <tt>bro</tt> as it’s own topology, and aggregate the other 2).</p>
+<p>The step to start an aggregated parsers then becomes</p>
+
+<div>
+<div>
+<pre class="source">$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s bro,snort,yaf
+</pre></div></div>
+
+<p>which will result in a single storm topology named <tt>bro__snort__yaf</tt> to run.</p>
+<p>Aggregated parsers can be specified using the Ambari Metron config as well under Services -> Metron -> Configs -> ‘Parsers’ tab -> ‘Metron Parsers’ field. The grouping is configured by enclosing the desired parsers in double quotes.</p>
+<p>Some examples of specifying aggregated parsers are as follows:</p>
+<ul>
+
+<li>“bro,snort,yaf” –> Will start a single topology named <tt>bro__snort__yaf</tt></li>
+<li>“ciscopixA,ciscopixB”,yaf,“squid,ciscopixC” –> Will start three topologies viz. <tt>ciscopixA__ciscopixB</tt>, <tt>yaf</tt> and <tt>squid__ciscopixC</tt></li>
+</ul>
<p><a name="Architecting_a_Parser_Chaining_Solution_in_Metron"></a></p>
<h1>Architecting a Parser Chaining Solution in Metron</h1>
<p>Currently the approach to fulfill this requirement involves a couple knobs in the Parser infrastructure for Metron.</p>
diff --git a/current-book/metron-platform/metron-parsers/index.html b/current-book/metron-platform/metron-parsers/index.html
index 02fea64..15440a1 100644
--- a/current-book/metron-platform/metron-parsers/index.html
+++ b/current-book/metron-platform/metron-parsers/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parsers</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Parsers</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -66,7 +66,9 @@
<ul class="nav nav-list">
<li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
<li><a href="../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li>
+ <li><a href="../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li>
<li><a href="../../metron-platform/metron-parsers/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li>
</ul>
</li>
<li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
@@ -127,10 +129,12 @@
<li><tt>grokPath</tt> : The path in HDFS (or in the Jar) to the grok statement</li>
<li><tt>patternLabel</tt> : The pattern label to use from the grok statement</li>
+<li><tt>multiLine</tt> : The raw data passed in should be handled as a long with multiple lines, with each line to be parsed separately. This setting’s valid values are ‘true’ or ‘false’. The default if unset is ‘false’. When set the parser will handle multiple lines with successfully processed lines emitted normally, and lines with errors sent to the error topic.</li>
<li><tt>timestampField</tt> : The field to use for timestamp</li>
<li><tt>timeFields</tt> : A list of fields to be treated as time</li>
<li><tt>dateFormat</tt> : The date format to use to parse the time fields</li>
<li><tt>timezone</tt> : The timezone to use. <tt>UTC</tt> is default.</li>
+<li>The Grok parser supports either 1 line to parse per incoming message, or incoming messages with multiple log lines, and will produce a json message per line</li>
</ul>
</li>
<li>CSV Parser: <tt>org.apache.metron.parsers.csv.CSVParser</tt> with possible <tt>parserConfig</tt> entries of
@@ -658,21 +662,23 @@
<p>Parser adapters are loaded dynamically in each Metron topology. They are defined in the Parser Config (defined above) JSON file in Zookeeper.</p>
<div class="section">
<h3><a name="Java_Parser_Adapters"></a>Java Parser Adapters</h3>
-<p>Java parser adapters are indended for higher-velocity topologies and are not easily changed or extended. As the adoption of Metron continues we plan on extending our library of Java adapters to process more log formats. As of this moment the Java adapters included with Metron are:</p>
+<p>Java parser adapters are intended for higher-velocity topologies and are not easily changed or extended. As the adoption of Metron continues we plan on extending our library of Java adapters to process more log formats. As of this moment the Java adapters included with Metron are:</p>
<ul>
<li>org.apache.metron.parsers.ise.BasicIseParser : Parse ISE messages</li>
<li>org.apache.metron.parsers.bro.BasicBroParser : Parse Bro messages</li>
<li>org.apache.metron.parsers.sourcefire.BasicSourcefireParser : Parse Sourcefire messages</li>
<li>org.apache.metron.parsers.lancope.BasicLancopeParser : Parse Lancope messages</li>
+<li>org.apache.metron.parsers.syslog.Syslog5424Parser : Parse Syslog RFC 5424 messages</li>
</ul></div>
<div class="section">
<h3><a name="Grok_Parser_Adapters"></a>Grok Parser Adapters</h3>
-<p>Grok parser adapters are designed primarly for someone who is not a Java coder for quickly standing up a parser adapter for lower velocity topologies. Grok relies on Regex for message parsing, which is much slower than purpose-built Java parsers, but is more extensible. Grok parsers are defined via a config file and the topplogy does not need to be recombiled in order to make changes to them. An example of a Grok perser is:</p>
+<p>Grok parser adapters are designed primarily for someone who is not a Java coder for quickly standing up a parser adapter for lower velocity topologies. Grok relies on Regex for message parsing, which is much slower than purpose-built Java parsers, but is more extensible. Grok parsers are defined via a config file and the topplogy does not need to be recompiled in order to make changes to them. Example of a Grok parsers are:</p>
<ul>
-<li>org.apache.metron.parsers.GrokParser</li>
+<li>org.apache.metron.parsers.GrokParser and org.apache.metron.parsers.websphere.GrokWebSphereParser</li>
</ul>
+<p>Parsers that derive from GrokParser typically allow the GrokParser to parse the messages, and then override the methods for postParse to do further parsing. When this is the case, and the Parser has not overridden <tt>parse(byte[])</tt> or <tt>parseResultOptional(byte[])</tt> these parsers will gain support for treating byte[] input as multiple lines, with each line parsed as a separate message ( and returned as such). This is enabled by using the <tt>"multiline":"true"</tt> Parser configuration option.</p>
<p>For more information on the Grok project please refer to the following link:</p>
<p><a class="externalLink" href="https://github.com/thekrakken/java-grok">https://github.com/thekrakken/java-grok</a></p>
<p><a name="Starting_the_Parser_Topology"></a></p>
diff --git a/current-book/metron-platform/metron-parsers/message-parser-implementation-notes.html b/current-book/metron-platform/metron-parsers/message-parser-implementation-notes.html
new file mode 100644
index 0000000..d288c9c
--- /dev/null
+++ b/current-book/metron-platform/metron-parsers/message-parser-implementation-notes.html
@@ -0,0 +1,180 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/message-parser-implementation-notes.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – implementation notes</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active "> implementation notes</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
+ <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li>
+ <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li>
+ <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li>
+ <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
+ <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li>
+ <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li>
+ <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li>
+ <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li>
+ <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-down"></span>Parsers</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
+ <li><a href="../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li>
+ <li class="active"><a href="#"><span class="none"></span>message-parser-implementation-notes</a></li>
+ <li><a href="../../metron-platform/metron-parsers/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
+ <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li>
+ <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1><tt>MessageParser</tt> implementation notes</h1>
+<p><a name="MessageParser_implementation_notes"></a></p>
+<ol style="list-style-type: decimal">
+
+<li>Supporting multiple JSONObject returns from a single byte[] The original <tt>MessageParser</tt> interface supported parsing a message and returning a <tt>List<JSONObject></tt>. Therefore explicitly supporting multiple messages from one input. While this is fine, it only allows for the complete failure of a message for any reason. There can only be one exception thrown. This means that if there <i>are</i> multiple messages in the buffer, any one failure will necessarily fail all of them. To improve on this situation, a new method was added to the <tt>MessageParser</tt> interface (with a default implementation), that introduces a return type to provide not only the JSONObjects produced, but also a <tt>Map</tt> of messages -> throwable.</li>
+</ol>
+<p>To support this in your parser, you should:</p>
+<ul>
+
+<li>Implement the new method</li>
+</ul>
+
+<div>
+<div>
+<pre class="source"> @Override
+ public Optional<MessageParserResult<JSONObject>> parseOptionalResult(byte[] rawMessage)
+</pre></div></div>
+
+<ul>
+
+<li>Implement the original <tt>List<JSONObject> parse(byte[] message)</tt> to delegate to that method such as below:</li>
+</ul>
+
+<div>
+<div>
+<pre class="source"> @Override
+ public List<JSONObject> parse(byte[] rawMessage) {
+ Optional<MessageParserResult<JSONObject>> resultOptional = parseOptionalResult(rawMessage);
+ if (!resultOptional.isPresent()) {
+ return Collections.EMPTY_LIST;
+ }
+ Map<Object,Throwable> errors = resultOptional.get().getMessageThrowables();
+ if (!errors.isEmpty()) {
+ throw new RuntimeException(errors.entrySet().iterator().next().getValue());
+ }
+
+ return resultOptional.get().getMessages();
+ }
+</pre></div></div>
+
+<ul>
+
+<li>
+
+<p>You <i>may</i> want to govern treating the incoming buffer as multiline or not by adding a configuration option for your parser, such as <tt>"multiline":"true"|"false"</tt></p>
+</li>
+<li>
+
+<p>See the org.apache.metron.parsers.GrokParser for an example of this implementation.</p>
+</li>
+</ul>
+<p>The Metron system itself will call the new <tt>parseOptionalResult</tt> method during processing. The default implementation in the interface handles backwards compatability with previous implementations.</p>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-platform/metron-parsers/parser-testing.html b/current-book/metron-platform/metron-parsers/parser-testing.html
index 7ae4693..18cf3ba 100644
--- a/current-book/metron-platform/metron-parsers/parser-testing.html
+++ b/current-book/metron-platform/metron-parsers/parser-testing.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/parser-testing.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/parser-testing.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parser Contribution and Testing</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Parser Contribution and Testing</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -66,7 +66,9 @@
<ul class="nav nav-list">
<li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
<li><a href="../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li>
+ <li><a href="../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li>
<li class="active"><a href="#"><span class="none"></span>parser-testing</a></li>
+ <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li>
</ul>
</li>
<li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
diff --git a/current-book/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html b/current-book/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html
new file mode 100644
index 0000000..744a1ef
--- /dev/null
+++ b/current-book/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html
@@ -0,0 +1,156 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.md at 2018-12-14
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – BasicPaloAltoFirewallParser</title>
+ <link rel="stylesheet" href="../../../../../../../../../../css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="../../../../../../../../../../css/site.css" />
+ <link rel="stylesheet" href="../../../../../../../../../../css/print.css" media="print" />
+ <script type="text/javascript" src="../../../../../../../../../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../../../../../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../../../../../../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">BasicPaloAltoFirewallParser</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
+ </ul>
+ </div>
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../../../../../../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../../../../../../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../../../../../../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
+ <li><a href="../../../../../../../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
+ <li><a href="../../../../../../../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../../../../../../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../../../../../../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../../../../../../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../../../../../../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../../../../../../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a>
+ <ul class="nav nav-list">
+ <li><a href="../../../../../../../../../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-down"></span>Parsers</a>
+ <ul class="nav nav-list">
+ <li><a href="../../../../../../../../../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-parsers/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ <li class="active"><a href="#"><span class="none"></span>Paloalto</a></li>
+ </ul>
+</li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li>
+ <li><a href="../../../../../../../../../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li>
+ </ul>
+</li>
+ <li><a href="../../../../../../../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../../../../../../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../../../../../../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../../../../../../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../../../../../../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../../../../../../../images/logos/maven-feather.png" /></a>
+ </div>
+ </div>
+ </div>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>BasicPaloAltoFirewallParser</h1>
+<p><a name="BasicPaloAltoFirewallParser"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The parser is able to parse logs generated by Palo Alto firewall devices powered by Pan OS. The supported log format is CSV.</p>
+<p>The supported log types and versions</p>
+<table border="0" class="table table-striped">
+<thead>
+
+<tr class="a">
+<th> Log type </th>
+<th> Pan OS version </th></tr>
+</thead><tbody>
+
+<tr class="b">
+<td> Traffic </td>
+<td> 6.0, 6.1, 7.0, 8.0 </td></tr>
+<tr class="a">
+<td> Threat </td>
+<td> 6.0, 6.1, 7.0, 8.0 </td></tr>
+<tr class="b">
+<td> Config </td>
+<td> 6.1, 7.0, 8.0 </td></tr>
+<tr class="a">
+<td> System </td>
+<td> 6.1, 7.0, 8.0 </td></tr>
+</tbody>
+</table></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </footer>
+ </body>
+</html>
diff --git a/current-book/metron-platform/metron-pcap-backend/index.html b/current-book/metron-platform/metron-pcap-backend/index.html
index e2a68ac..9c3278a 100644
--- a/current-book/metron-platform/metron-pcap-backend/index.html
+++ b/current-book/metron-platform/metron-pcap-backend/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-pcap-backend/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-pcap-backend/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron PCAP Backend</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron PCAP Backend</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -161,6 +161,7 @@
<p>These files contain a set of packet data with headers on them in sequence files.</p></div>
<div class="section">
<h2><a name="Configuration"></a>Configuration</h2>
+<p>The configuration properties for PCAP sensor is managed via Ambari at Services -> Metron -> Config -> PCAP tab. Note that changes to PCAP sensor config properties via Ambari requires restarting the Metron PCAP service.</p>
<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/pcap.properties</tt> and the possible options are as follows:</p>
<ul>
diff --git a/current-book/metron-platform/metron-solr/index.html b/current-book/metron-platform/metron-solr/index.html
index ba9ad37..f0d8362 100644
--- a/current-book/metron-platform/metron-solr/index.html
+++ b/current-book/metron-platform/metron-solr/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-solr/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-solr/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Solr in Metron</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Solr in Metron</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-platform/metron-writer/index.html b/current-book/metron-platform/metron-writer/index.html
index dee53f7..1469d6f 100644
--- a/current-book/metron-platform/metron-writer/index.html
+++ b/current-book/metron-platform/metron-writer/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-writer/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-writer/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Writer</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Writer</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-sensors/fastcapa/index.html b/current-book/metron-sensors/fastcapa/index.html
index 0044051..73f176d 100644
--- a/current-book/metron-sensors/fastcapa/index.html
+++ b/current-book/metron-sensors/fastcapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/fastcapa/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/fastcapa/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Fastcapa</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-sensors/index.html b/current-book/metron-sensors/index.html
index afb4ef5..b28fe48 100644
--- a/current-book/metron-sensors/index.html
+++ b/current-book/metron-sensors/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Sensors</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Sensors</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-sensors/pycapa/index.html b/current-book/metron-sensors/pycapa/index.html
index 57c0231..e0a3045 100644
--- a/current-book/metron-sensors/pycapa/index.html
+++ b/current-book/metron-sensors/pycapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/pycapa/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/pycapa/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pycapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Pycapa</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-stellar/stellar-3rd-party-example/index.html b/current-book/metron-stellar/stellar-3rd-party-example/index.html
index 88d9e48..34501bc 100644
--- a/current-book/metron-stellar/stellar-3rd-party-example/index.html
+++ b/current-book/metron-stellar/stellar-3rd-party-example/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-3rd-party-example/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-3rd-party-example/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Introduction</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Introduction</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-stellar/stellar-common/3rdPartyStellar.html b/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
index 0ba1ead..bcee9dc 100644
--- a/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
+++ b/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/3rdPartyStellar.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/3rdPartyStellar.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Custom Stellar Functions</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Custom Stellar Functions</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/metron-stellar/stellar-common/index.html b/current-book/metron-stellar/stellar-common/index.html
index 5d80355..ee93071 100644
--- a/current-book/metron-stellar/stellar-common/index.html
+++ b/current-book/metron-stellar/stellar-common/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Stellar Language</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Stellar Language</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -113,6 +113,16 @@
</ul>
</li>
<li><a href="#Stellar_Configuration">Stellar Configuration</a></li>
+<li><a href="#Stellar_REST_Client">Stellar REST Client</a>
+<ul>
+
+<li><a href="#Configuration">Configuration</a></li>
+<li><a href="#Security">Security</a></li>
+<li><a href="#Examples">Examples</a></li>
+<li><a href="#Latency">Latency</a></li>
+<li><a href="#Response_Handling">Response Handling</a></li>
+</ul>
+</li>
</ul>
<div class="section">
<h2><a name="Introduction"></a>Introduction</h2>
@@ -343,163 +353,167 @@
<tr class="a">
<td> <a href="#COUNT_MATCHES"> <tt>COUNT_MATCHES</tt></a> </td></tr>
<tr class="b">
+<td> <a href="#DATE_FORMAT"> <tt>DATE_FORMAT</tt></a></td></tr>
+<tr class="a">
<td> <a href="#DAY_OF_MONTH"> <tt>DAY_OF_MONTH</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#DAY_OF_WEEK"> <tt>DAY_OF_WEEK</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#DAY_OF_YEAR"> <tt>DAY_OF_YEAR</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#DECODE"> <tt>DECODE</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#DOMAIN_REMOVE_SUBDOMAINS"> <tt>DOMAIN_REMOVE_SUBDOMAINS</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#DOMAIN_REMOVE_TLD"> <tt>DOMAIN_REMOVE_TLD</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#DOMAIN_TO_TLD"> <tt>DOMAIN_TO_TLD</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#ENCODE"> <tt>ENCODE</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#ENDS_WITH"> <tt>ENDS_WITH</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#ENRICHMENT_EXISTS"> <tt>ENRICHMENT_EXISTS</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#ENRICHMENT_GET"> <tt>ENRICHMENT_GET</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#EXP"> <tt>EXP</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#FILL_LEFT"> <tt>FILL_LEFT</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#FILL_RIGHT"> <tt>FILL_RIGHT</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#FILTER"> <tt>FILTER</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#FLOOR"> <tt>FLOOR</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#FORMAT"> <tt>FORMAT</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#GEOHASH_CENTROID"> <tt>GEOHASH_CENTROID</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GEOHASH_DIST"> <tt>GEOHASH_DIST</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#GEOHASH_FROM_LATLONG"> <tt>GEOHASH_FROM_LATLONG</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GEOHASH_FROM_LOC"> <tt>GEOHASH_FROM_LOC</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#GEOHASH_MAX_DIST"> <tt>GEOHASH_MAX_DIST</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GEOHASH_TO_LATLONG"> <tt>GEOHASH_TO_LATLONG</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#GET"> <tt>GET</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#HASH"> <tt>HASH</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_ADD"> <tt>HLLP_ADD</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="../../metron-analytics/metron-statistics/index.html#IT_ENTROPY"> <tt>IT_ENTROPY</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#IS_ENCODING"> <tt>IS_ENCODING</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IS_IP"> <tt>IS_IP</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#IS_NAN"> <tt>IS_NAN</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#IS_URL"> <tt>IS_URL</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#JOIN"> <tt>JOIN</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#LENGTH"> <tt>LENGTH</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#LOG2"> <tt>LOG2</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#LOG10"> <tt>LOG10</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#LN"> <tt>LN</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MAP"> <tt>MAP</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MAP_GET"> <tt>MAP_GET</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MAX"> <tt>MAX</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MIN"> <tt>MIN</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MONTH"> <tt>MONTH</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MULTISET_ADD"> <tt>MULTISET_ADD</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MULTISET_INIT"> <tt>MULTISET_INIT</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MULTISET_MERGE"> <tt>MULTISET_MERGE</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#MULTISET_REMOVE"> <tt>MULTISET_REMOVE</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#MULTISET_TO_SET"> <tt>MULTISET_TO_SET</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#OBJECT_GET"> <tt>OBJECT_GET</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#PREPEND_IF_MISSING"> <tt>PREPEND_IF_MISSING</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#PROFILE_GET"> <tt>PROFILE_GET</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#PROFILE_FIXED"> <tt>PROFILE_FIXED</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#PROFILE_WINDOW"> <tt>PROFILE_WINDOW</tt></a> </td></tr>
-<tr class="b">
+<tr class="a">
<td> <a href="#PROTOCOL_TO_NAME"> <tt>PROTOCOL_TO_NAME</tt></a> </td></tr>
-<tr class="a">
+<tr class="b">
<td> <a href="#REDUCE"> <tt>REDUCE</tt></a> </td></tr>
-<tr class="b">
-<td> <a href="#REGEXP_MATCH"> <tt>REGEXP_MATCH</tt></a> </td></tr>
<tr class="a">
-<td> <a href="#REGEXP_GROUP_VAL"> <tt>REGEXP_GROUP_VAL</tt></a> </td></tr>
+<td> <a href="#REGEXP_MATCH"> <tt>REGEXP_MATCH</tt></a> </td></tr>
<tr class="b">
-<td> <a href="#REGEXP_REPLACE"> <tt>REGEXP_REPLACE</tt></a> </td></tr>
+<td> <a href="#REGEXP_GROUP_VAL"> <tt>REGEXP_GROUP_VAL</tt></a> </td></tr>
+<tr class="a">
+<td> <a href="#REGEXP_REPLACE"> <tt>REGEXP_REPLACE</tt></a></td></tr>
+<tr class="b">
+<td> <a href="#REST_GET"> <tt>REST_GET</tt></a></td></tr>
<tr class="a">
<td> <a href="#ROUND"> <tt>ROUND</tt></a> </td></tr>
<tr class="b">
@@ -767,6 +781,21 @@
<li>Returns: the number of non-overlapping occurrences, 0 if either CharSequence is null.</li>
</ul></div>
<div class="section">
+<h3><a name="DATE_FORMAT"></a><tt>DATE_FORMAT</tt></h3>
+<ul>
+
+<li>Description: Takes an epoch timestamp and converts it to a date format.</li>
+<li>Input:
+<ul>
+
+<li>format - DateTime format as a String.</li>
+<li>timestampField - Optional epoch time in Long format. Defaults to now.</li>
+<li>timezone - Optional timezone in String format.</li>
+</ul>
+</li>
+<li>Returns: Formatted date.</li>
+</ul></div>
+<div class="section">
<h3><a name="DAY_OF_MONTH"></a><tt>DAY_OF_MONTH</tt></h3>
<ul>
@@ -1824,6 +1853,20 @@
<li>Returns: The modified input string with replaced values</li>
</ul></div>
<div class="section">
+<h3><a name="REST_GET"></a><tt>REST_GET</tt></h3>
+<ul>
+
+<li>Description: Performs a REST GET request and parses the JSON results into a map.</li>
+<li>Input:
+<ul>
+
+<li>url - URI to the REST service</li>
+<li>rest_config - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li>
+</ul>
+</li>
+<li>Returns: JSON results as a Map</li>
+</ul></div>
+<div class="section">
<h3><a name="ROUND"></a><tt>ROUND</tt></h3>
<ul>
@@ -2685,7 +2728,7 @@
<div>
<div>
-<pre class="source">metron-stellar/stellar-common/target/stellar-common-0.6.0-stand-alone.tar.gz
+<pre class="source">metron-stellar/stellar-common/target/stellar-common-0.7.0-stand-alone.tar.gz
</pre></div></div>
<p>When unpacked, the following structure will be created:</p>
@@ -2696,7 +2739,7 @@
├── bin
│   └── stellar
└── lib
- └── stellar-common-0.6.0-uber.jar
+ └── stellar-common-0.7.0-uber.jar
</pre></div></div>
<p>To run the Stellar Shell run the following from the directory you unpacked to:</p>
@@ -2807,7 +2850,91 @@
...
"stellar.function.resolver.includes" : "org.apache.metron.*,com.myorg.stellar.*"
}
-</pre></div></div></div>
+</pre></div></div>
+</div>
+<div class="section">
+<h2><a name="Stellar_REST_Client"></a>Stellar REST Client</h2>
+<p>Stellar provides a REST Client with the <tt>REST_GET</tt> function. This function depends on the Apache HttComponents library for executing Http requests. The syntax is:</p>
+
+<div>
+<div>
+<pre class="source">REST_GET( uri , optional config )
+</pre></div></div>
+
+<div class="section">
+<h3><a name="Configuration"></a>Configuration</h3>
+<p>The second argument is an optional Map of settings. The following settings are available:</p>
+<ul>
+
+<li>basic.auth.user - User name for basic authentication.</li>
+<li>basic.auth.password.path - Path to the basic authentication password file stored in HDFS.</li>
+<li>proxy.host - Proxy host.</li>
+<li>proxy.port - Proxy port.</li>
+<li>proxy.basic.auth.user - User name for proxy basic authentication.</li>
+<li>proxy.basic.auth.password.path - Path to the proxy basic authentication password file stored in HDFS.</li>
+<li>timeout - Stellar enforced hard timeout for the total request time. Defaults to 1000 ms. HttpClient timeouts alone are insufficient to guarantee the hard timeout.</li>
+<li>connect.timeout - Connect timeout exposed by the HttpClient object.</li>
+<li>connection.request.timeout - Connection request timeout exposed by the HttpClient object.</li>
+<li>socket.timeout - Socket timeout exposed by the HttpClient object.</li>
+<li>response.codes.allowed - A list of response codes that are allowed. All others will be treated as errors. Defaults to <tt>200</tt>.</li>
+<li>empty.content.override - The default value that will be returned on a successful request with empty content. Defaults to null.</li>
+<li>error.value.override - The default value that will be returned on an error. Defaults to null.</li>
+<li>pooling.max.total - The maximum number of connections in the connection pool.</li>
+<li>pooling.default.max.per.route - The default maximum number of connections per route in the connection pool.</li>
+</ul>
+<p>This Map of settings can also be stored in the global config <tt>stellar.rest.settings</tt> property. For example, to configure basic authentication settings you would add this property to the global config:</p>
+
+<div>
+<div>
+<pre class="source">{
+ "stellar.rest.settings": {
+ "basic.auth.user": "user",
+ "basic.auth.password.path": "/password/path"
+ }
+}
+</pre></div></div>
+
+<p>Any settings passed into the expression will take precedence over the global config settings. The global config settings will take precedence over the defaults.</p>
+<p>For security purposes, passwords are read from a file in HDFS. Passwords are read as is including any new lines or spaces. Be careful not to include these in the file unless they are specifically part of the password.</p></div>
+<div class="section">
+<h3><a name="Security"></a>Security</h3>
+<p>At this time, only basic authentication is supported.</p></div>
+<div class="section">
+<h3><a name="Examples"></a>Examples</h3>
+<p>Perform a simple GET request with no authentication:</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> REST_GET('http://httpbin.org/get')
+{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get}
+</pre></div></div>
+
+<p>Perform a GET request using basic authentication:</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> config := {'basic.auth.user': 'user', 'basic.auth.password.path': '/password/path'}
+{basic.auth.user=user, basic.auth.password.path=/password/path}
+[Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd', config)
+{authenticated=true, user=user}
+</pre></div></div>
+
+<p>Perform a GET request using a proxy:</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> config := {'proxy.host': 'node1', 'proxy.port': 3128, 'proxy.basic.auth.user': 'user', 'proxy.basic.auth.password.path': '/proxy/password/path'}
+{proxy.basic.auth.password.path=/proxy/password/path, proxy.port=3128, proxy.host=node1, proxy.basic.auth.user=user}
+[Stellar]>>> REST_GET('http://httpbin.org/get', config)
+{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get}
+</pre></div></div>
+</div>
+<div class="section">
+<h3><a name="Latency"></a>Latency</h3>
+<p>Performing a REST request will introduce latency in a streaming pipeline. Therefore this function should only be used for low volume telemetries that are unlikely to be affected by higher latency operations. The <tt>timeout</tt> setting can be used to guarantee that requests complete within the configured time.</p></div>
+<div class="section">
+<h3><a name="Response_Handling"></a>Response Handling</h3>
+<p>In cases of Http errors, timeouts, etc this function will log the error and return null. Only a status code of <tt>200</tt> is considered successful by default but this can be changed with the <tt>response.codes.allowed</tt> setting. Values returned on errors or emtpy content can be changed from the default value of null using the <tt>error.value.override</tt> and <tt>empty.content.override</tt> respectively.</p></div></div>
</div>
</div>
</div>
diff --git a/current-book/metron-stellar/stellar-zeppelin/index.html b/current-book/metron-stellar/stellar-zeppelin/index.html
index e856d23..2f3a3f6 100644
--- a/current-book/metron-stellar/stellar-zeppelin/index.html
+++ b/current-book/metron-stellar/stellar-zeppelin/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-zeppelin/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-zeppelin/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Stellar Interpreter for Apache Zeppelin</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Stellar Interpreter for Apache Zeppelin</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
@@ -132,11 +132,11 @@
<li>
<p>Use Zeppelin’s installation utility to install the Stellar Interpreter.</p>
-<p>If Zeppelin was already installed, make sure that it is stopped before running this command. Update the version, ‘0.6.0’ in the example below, to whatever is appropriate for your environment.</p>
+<p>If Zeppelin was already installed, make sure that it is stopped before running this command. Update the version, ‘0.7.0’ in the example below, to whatever is appropriate for your environment.</p>
<div>
<div>
-<pre class="source">bin/install-interpreter.sh --name stellar --artifact org.apache.metron:stellar-zeppelin:0.6.0
+<pre class="source">bin/install-interpreter.sh --name stellar --artifact org.apache.metron:stellar-zeppelin:0.7.0
</pre></div></div>
</li>
<li>
@@ -208,7 +208,7 @@
<div>
<div>
-<pre class="source">org.apache.metron:metron-statistics:0.6.0
+<pre class="source">org.apache.metron:metron-statistics:0.7.0
</pre></div></div>
</li>
<li>
diff --git a/current-book/use-cases/forensic_clustering/index.html b/current-book/use-cases/forensic_clustering/index.html
index a4f0b6a..4d0774d 100644
--- a/current-book/use-cases/forensic_clustering/index.html
+++ b/current-book/use-cases/forensic_clustering/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/forensic_clustering/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/forensic_clustering/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Problem Statement</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Problem Statement</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/use-cases/geographic_login_outliers/index.html b/current-book/use-cases/geographic_login_outliers/index.html
index 38e81b9..97a713d 100644
--- a/current-book/use-cases/geographic_login_outliers/index.html
+++ b/current-book/use-cases/geographic_login_outliers/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/geographic_login_outliers/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/geographic_login_outliers/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Problem Statement</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Problem Statement</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/use-cases/index.html b/current-book/use-cases/index.html
index 1fcb738..24b3d92 100644
--- a/current-book/use-cases/index.html
+++ b/current-book/use-cases/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Worked Examples</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Worked Examples</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/use-cases/parser_chaining/index.html b/current-book/use-cases/parser_chaining/index.html
index 5d3a9cc..f9667c5 100644
--- a/current-book/use-cases/parser_chaining/index.html
+++ b/current-book/use-cases/parser_chaining/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/parser_chaining/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/parser_chaining/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Problem Statement</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Problem Statement</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/current-book/use-cases/typosquat_detection/index.html b/current-book/use-cases/typosquat_detection/index.html
index ec188ff..4ec0f7e 100644
--- a/current-book/use-cases/typosquat_detection/index.html
+++ b/current-book/use-cases/typosquat_detection/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/typosquat_detection/index.md at 2018-09-13
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/use-cases/typosquat_detection/index.md at 2018-12-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180913" />
+ <meta name="Date-Revision-yyyymmdd" content="20181214" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Problem Statement</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
@@ -32,8 +32,8 @@
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Problem Statement</li>
- <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-13</li>
- <li id="projectVersion" class="pull-right">Version: 0.6.0</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li>
+ <li id="projectVersion" class="pull-right">Version: 0.7.0</li>
</ul>
</div>
<div class="row-fluid">
diff --git a/documentation/index.html b/documentation/index.html
index b24ec37..dc2a42d 100644
--- a/documentation/index.html
+++ b/documentation/index.html
@@ -211,21 +211,21 @@
<section class="events-section info-block darken" id="releases">
<div class="wrapper downloads">
<div class="text-center">
- <h4>Current Release: 0.6.0</h4>
+ <h4>Current Release: 0.7.0</h4>
</div>
<div class="content-960 hover-btn text-center">
<ul>
<li>
- <a href="http://www.apache.org/dyn/closer.cgi/metron/0.6.0/apache-metron-0.6.0.tar.gz" target="new">
- apache-metron-0.6.0.tar.gz
+ <a href="http://www.apache.org/dyn/closer.cgi/metron/0.7.0/apache-metron-0.7.0.tar.gz" target="new">
+ apache-metron-0.7.0.tar.gz
</a>
- [ <a href="https://www.apache.org/dist/metron/0.6.0/apache-metron-0.6.0.tar.gz.asc" target="new">
+ [ <a href="https://www.apache.org/dist/metron/0.7.0/apache-metron-0.7.0.tar.gz.asc" target="new">
PGP
</a>]
- [ <a href="https://www.apache.org/dist/metron/0.6.0/apache-metron-0.6.0.tar.gz.sha256" target="new">
+ [ <a href="https://www.apache.org/dist/metron/0.7.0/apache-metron-0.7.0.tar.gz.sha256" target="new">
SHA256
</a>]
- [ <a href="https://www.apache.org/dist/metron/0.6.0/apache-metron-0.6.0.tar.gz.sha512" target="new">
+ [ <a href="https://www.apache.org/dist/metron/0.7.0/apache-metron-0.7.0.tar.gz.sha512" target="new">
SHA512
</a>]
[ <a href="/current-book/index.html" target="new"> Book </a>]
diff --git a/feed.xml b/feed.xml
index 0ff3fff..fa0a69a 100644
--- a/feed.xml
+++ b/feed.xml
@@ -5,9 +5,9 @@
<description></description>
<link>http://localhost:4000/</link>
<atom:link href="http://localhost:4000/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Thu, 11 Oct 2018 08:44:01 -0400</pubDate>
- <lastBuildDate>Thu, 11 Oct 2018 08:44:01 -0400</lastBuildDate>
- <generator>Jekyll v3.8.4</generator>
+ <pubDate>Mon, 17 Dec 2018 09:15:48 -0500</pubDate>
+ <lastBuildDate>Mon, 17 Dec 2018 09:15:48 -0500</lastBuildDate>
+ <generator>Jekyll v3.8.3</generator>
<item>
<title>Debo Dutta joins PPMC</title>
