Google Git
Sign in
apache / metron / 410993daf60dfe898a973fa8a9906ddbffd83208 / . / current-book / metron-analytics / metron-profiler-common / index.html
blob: 59ea1a17bf7fa67dc57a0624935dcd9568b7bc83 [file] [log] [blame]
<!DOCTYPE html>
<!--
| Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-common/index.md at 2019-05-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="Date-Revision-yyyymmdd" content="20190514" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron &#x2013; Metron Profiler</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
<script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
<script type="text/javascript">
$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
</script>
</head>
<body class="topBarDisabled">
<div class="container-fluid">
<div id="banner">
<div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
<div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
<li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Metron Profiler</li>
<li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li>
<li id="projectVersion" class="pull-right">Version: 0.7.1</li>
</ul>
</div>
<div class="row-fluid">
<div id="leftColumn" class="span2">
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
<li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
<ul class="nav nav-list">
<li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
<li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a>
<ul class="nav nav-list">
<li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li>
<li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li>
<li class="active"><a href="#"><span class="none"></span>Profiler-common</a></li>
<li><a href="../../metron-analytics/metron-profiler-repl/index.html" title="Profiler-repl"><span class="none"></span>Profiler-repl</a></li>
<li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li>
<li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li>
<li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
</ul>
</li>
<li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
<li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
<li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
<li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li>
<li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li>
<li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
<li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
<li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
<li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
<li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
</ul>
</li>
</ul>
<hr />
<div id="poweredBy">
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
</div>
</div>
</div>
<div id="bodyColumn" class="span10" >
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<h1>Metron Profiler</h1>
<p><a name="Metron_Profiler"></a></p>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#Getting_Started">Getting Started</a></li>
<li><a href="#Profiles">Profiles</a></li>
<li><a href="#Examples">Examples</a></li>
</ul>
<div class="section">
<h2><a name="Introduction"></a>Introduction</h2>
<p>The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</p>
<p>This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows. A summary statistic is applied to the data received within a given window. Collecting these values across many windows result in a time series that is useful for analysis.</p>
<p>Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language.</p>
<p>There are three separate ports of the Profiler that share this common code base.</p>
<ul>
<li>The <a href="../metron-profiler-storm/index.html">Storm Profiler</a> builds low-latency profiles over streaming data sets.</li>
<li>The <a href="../metron-profiler-spark/index.html">Spark Profiler</a> backfills profiles using archived telemetry.</li>
<li>The <a href="../metron-profiler-repl/index.html">REPL Profiler</a> allows profiles to be tested and debugged within the Stellar REPL.</li>
</ul></div>
<div class="section">
<h2><a name="Getting_Started"></a>Getting Started</h2>
<ol style="list-style-type: decimal">
<li>
<p><a href="../metron-profiler-repl/index.html#getting-started">Create a profile</a> using the Stellar REPL. Validate your profile using mock data, then apply real, live data.</p>
</li>
<li>
<p><a href="../metron-profiler-spark/index.html#getting-started">Backfill your profile</a> using archived telemetry to see how your profile behaves over time.</p>
</li>
<li>
<p><a href="../metron-profiler-storm/index.html#getting-started">Deploy your profile</a> to Storm to maintain a low-latency profile over a streaming data set.</p>
</li>
<li>
<p><a href="../metron-profiler-client/index.html">Retrieve your profile data</a> using the Stellar API so that you can build enrichments, alert on abnormalities.</p>
</li>
<li>
<p>Explore more ways to create <a href="#more-examples">profiles</a>.</p>
</li>
</ol></div>
<div class="section">
<h2><a name="Profiles"></a>Profiles</h2>
<p>Let&#x2019;s start with a simple example. The following profile maintains a count of the number of telemetry messages for each IP source address. A counter is initialized to 0, then incremented each time a message is received for a given IP source address. At regular intervals the count is flushed and stored. Over time this results in a time series describing the amount of telemetry received for each IP source address.</p>
<div>
<div>
<pre class="source">{
&quot;profiles&quot;: [
{
&quot;profile&quot;: &quot;hello-world&quot;,
&quot;foreach&quot;: &quot;ip_src_addr&quot;,
&quot;init&quot;: {
&quot;count&quot;: &quot;0&quot;
},
&quot;update&quot;: {
&quot;count&quot;: &quot;count + 1&quot;
},
&quot;result&quot;: &quot;count&quot;
}
]
}
</pre></div></div>
<p>A profile definition contains two fields; only one of which is required.</p>
<div>
<div>
<pre class="source">{
&quot;profiles&quot;: [
{ &quot;profile&quot;: &quot;one&quot;, ... },
{ &quot;profile&quot;: &quot;two&quot;, ... }
],
&quot;timestampField&quot;: &quot;timestamp&quot;
}
</pre></div></div>
<table border="0" class="table table-striped">
<thead>
<tr class="a">
<th> Name </th>
<th> </th>
<th> Description</th></tr>
</thead><tbody>
<tr class="b">
<td> <a href="#profiles">profiles</a> </td>
<td> Required </td>
<td> A list of zero or more Profile definitions.</td></tr>
<tr class="a">
<td> <a href="#timestampField">timestampField</a> </td>
<td> Optional </td>
<td> Indicates whether processing time or event time should be used. By default, processing time is enabled.</td></tr>
</tbody>
</table>
<div class="section">
<div class="section">
<h4><a name="profiles"></a><tt>profiles</tt></h4>
<p><i>Required</i></p>
<p>A list of zero or more Profile definitions.</p></div>
<div class="section">
<h4><a name="timestampField"></a><tt>timestampField</tt></h4>
<p><i>Optional</i></p>
<p>Indicates whether processing time or event time is used. By default, processing time is enabled.</p>
<div class="section">
<h5><a name="Processing_Time"></a>Processing Time</h5>
<p>By default, no <tt>timestampField</tt> is defined. In this case, the Profiler uses system time when generating profiles. This means that the profiles are generated based on when the data has been processed by the Profiler. This is also known as &#x2018;processing time&#x2019;.</p>
<p>This is the simplest mode of operation, but has some draw backs. If the Profiler is consuming live data and all is well, the processing and event times will likely remain similar and consistent. If processing time diverges from event time, then the Profiler will generate skewed profiles.</p>
<p>There are a few scenarios that might cause skewed profiles when using processing time. For example when a system has undergone a scheduled maintenance window and is restarted, a high volume of messages will need to be processed by the Profiler. The output of the Profiler might indicate an increase in activity during this time, although no change in activity actually occurred on the target network. The same situation could occur if an upstream system which provides telemetry undergoes an outage.</p>
<p><a href="#Event_Time">Event Time</a> can be used to mitigate these problems.</p></div>
<div class="section">
<h5><a name="Event_Time"></a>Event Time</h5>
<p>Alternatively, a <tt>timestampField</tt> can be defined. This must be the name of a field contained within the telemetry processed by the Profiler. The Profiler will extract and use the timestamp contained within this field.</p>
<ul>
<li>
<p>If a message does not contain this field, it will be dropped.</p>
</li>
<li>
<p>The field must contain a timestamp in epoch milliseconds expressed as either a numeric or string. Otherwise, the message will be dropped.</p>
</li>
<li>
<p>The Profiler will use the same field across all telemetry sources and for all profiles.</p>
</li>
<li>
<p>Be aware of clock skew across telemetry sources. If your profile is processing telemetry from multiple sources where the clock differs significantly, the Profiler may assume that some of those messages are late and will be ignored. Adjusting the <a href="#profiler.window.duration"><tt>profiler.window.duration</tt></a> and <a href="#profiler.window.lag"><tt>profiler.window.lag</tt></a> can help accommodate skewed clocks.</p>
</li>
</ul></div></div></div>
<div class="section">
<h3><a name="Profiles"></a>Profiles</h3>
<p>A profile definition requires a JSON-formatted set of elements, many of which can contain Stellar code. The specification contains the following elements. (For the impatient, skip ahead to the <a href="#Examples">Examples</a>.)</p>
<table border="0" class="table table-striped">
<thead>
<tr class="a">
<th> Name </th>
<th> </th>
<th> Description</th></tr>
</thead><tbody>
<tr class="b">
<td> <a href="#profile">profile</a> </td>
<td> Required </td>
<td> Unique name identifying the profile.</td></tr>
<tr class="a">
<td> <a href="#foreach">foreach</a> </td>
<td> Required </td>
<td> A separate profile is maintained &#x201c;for each&#x201d; of these.</td></tr>
<tr class="b">
<td> <a href="#onlyif">onlyif</a> </td>
<td> Optional </td>
<td> Boolean expression that determines if a message should be applied to the profile.</td></tr>
<tr class="a">
<td> <a href="#groupBy">groupBy</a> </td>
<td> Optional </td>
<td> One or more Stellar expressions used to group the profile measurements when persisted.</td></tr>
<tr class="b">
<td> <a href="#init">init</a> </td>
<td> Optional </td>
<td> One or more expressions executed at the start of a window period.</td></tr>
<tr class="a">
<td> <a href="#update">update</a> </td>
<td> Required </td>
<td> One or more expressions executed when a message is applied to the profile.</td></tr>
<tr class="b">
<td> <a href="#result">result</a> </td>
<td> Required </td>
<td> Stellar expressions that are executed when the window period expires.</td></tr>
<tr class="a">
<td> <a href="#expires">expires</a> </td>
<td> Optional </td>
<td> Profile data is purged after this period of time, specified in days.</td></tr>
</tbody>
</table>
<div class="section">
<h4><a name="profile"></a><tt>profile</tt></h4>
<p><i>Required</i></p>
<p>A unique name identifying the profile. The field is treated as a string.</p></div>
<div class="section">
<h4><a name="foreach"></a><tt>foreach</tt></h4>
<p><i>Required</i></p>
<p>A separate profile is maintained &#x2018;for each&#x2019; of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name.</p>
<p>For example, if <tt>ip_src_addr</tt> then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc.</p></div>
<div class="section">
<h4><a name="onlyif"></a><tt>onlyif</tt></h4>
<p><i>Optional</i></p>
<p>An expression that determines if a message should be applied to the profile. A Stellar expression that returns a Boolean is expected. A message is only applied to a profile if this expression is true. This allows a profile to filter the messages that get applied to it.</p></div>
<div class="section">
<h4><a name="groupBy"></a><tt>groupBy</tt></h4>
<p><i>Optional</i></p>
<p>One or more Stellar expressions used to group the profile measurements when persisted. This can be used to sort the Profile data to allow for a contiguous scan when accessing subsets of the data. This is also one way to deal with calendar effects. For example, where activity on a weekday can be very different from a weekend.</p>
<p>A common use case would be grouping by day of week. This allows a contiguous scan to access all profile data for Mondays only. Using the following definition would achieve this.</p>
<div>
<div>
<pre class="source">&quot;groupBy&quot;: [ &quot;DAY_OF_WEEK(start)&quot; ]
</pre></div></div>
<p>The expression can reference any of these variables.</p>
<ul>
<li>Any variable defined by the profile in its <tt>init</tt> or <tt>update</tt> expressions.</li>
<li><tt>profile</tt> The name of the profile.</li>
<li><tt>entity</tt> The name of the entity being profiled.</li>
<li><tt>start</tt> The start time of the profile period in epoch milliseconds.</li>
<li><tt>end</tt> The end time of the profile period in epoch milliseconds.</li>
<li><tt>duration</tt> The duration of the profile period in milliseconds.</li>
<li><tt>result</tt> The result of executing the <tt>result</tt> expression.</li>
</ul></div>
<div class="section">
<h4><a name="init"></a><tt>init</tt></h4>
<p><i>Optional</i></p>
<p>One or more expressions executed at the start of a window period. A map is expected where the key is the variable name and the value is a Stellar expression. The map can contain zero or more variable:expression pairs. At the start of each window period, each expression is executed once and stored in the given variable. Note that constant init values such as &#x201c;0&#x201d; must be in quotes regardless of their type, as the init value must be a string to be executed by Stellar.</p>
<div>
<div>
<pre class="source">&quot;init&quot;: {
&quot;var1&quot;: &quot;0&quot;,
&quot;var2&quot;: &quot;1&quot;
}
</pre></div></div>
</div>
<div class="section">
<h4><a name="update"></a><tt>update</tt></h4>
<p><i>Required</i></p>
<p>One or more expressions executed when a message is applied to the profile. A map is expected where the key is the variable name and the value is a Stellar expression. The map can include 0 or more variables/expressions. When each message is applied to the profile, the expression is executed and stored in a variable with the given name.</p>
<div>
<div>
<pre class="source">&quot;update&quot;: {
&quot;var1&quot;: &quot;var1 + 1&quot;,
&quot;var2&quot;: &quot;var2 + 1&quot;
}
</pre></div></div>
</div>
<div class="section">
<h4><a name="result"></a><tt>result</tt></h4>
<p><i>Required</i></p>
<p>Stellar expressions that are executed when the window period expires. The expressions are expected to summarize the messages that were applied to the profile over the window period. In the most basic form a single result is persisted for later retrieval.</p>
<div>
<div>
<pre class="source">&quot;result&quot;: &quot;var1 + var2&quot;
</pre></div></div>
<p>For more advanced use cases, a profile can generate two types of results. A profile can define one or both of these result types at the same time.</p>
<ul>
<li><tt>profile</tt>: A required expression that defines a value that is persisted for later retrieval.</li>
<li><tt>triage</tt>: An optional expression that defines values that are accessible within the Threat Triage process.</li>
</ul>
<p><b>profile</b></p>
<p>A required Stellar expression that results in a value that is persisted in the profile store for later retrieval. The expression can result in any object that is Kryo serializable. These values can be retrieved for later use with the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
<div>
<div>
<pre class="source">&quot;result&quot;: {
&quot;profile&quot;: &quot;2 + 2&quot;
}
</pre></div></div>
<p>An alternative, simplified form is also acceptable.</p>
<div>
<div>
<pre class="source">&quot;result&quot;: &quot;2 + 2&quot;
</pre></div></div>
<p><b>triage</b></p>
<p>An optional map of one or more Stellar expressions. The value of each expression is made available to the Threat Triage process under the given name. Each expression must result in a either a primitive type, like an integer, long, or short, or a String. All other types will result in an error.</p>
<p>In the following example, three values, the minimum, the maximum and the mean are appended to a message. This message is consumed by Metron, like other sources of telemetry, and each of these values are accessible from within the Threat Triage process using the given field names; <tt>min</tt>, <tt>max</tt>, and <tt>mean</tt>.</p>
<div>
<div>
<pre class="source">&quot;result&quot;: {
&quot;triage&quot;: {
&quot;min&quot;: &quot;STATS_MIN(stats)&quot;,
&quot;max&quot;: &quot;STATS_MAX(stats)&quot;,
&quot;mean&quot;: &quot;STATS_MEAN(stats)&quot;
}
}
</pre></div></div>
</div>
<div class="section">
<h4><a name="expires"></a><tt>expires</tt></h4>
<p><i>Optional</i></p>
<p>A numeric value that defines how many days the profile data is retained. After this time, the data expires and is no longer accessible. If no value is defined, the data does not expire.</p>
<p>The REPL can be a powerful tool for developing profiles. Read all about <a href="../metron-profiler-client/index.html#developing_profiles">Developing Profiles</a>.</p></div></div></div>
<div class="section">
<h2><a name="Examples"></a>Examples</h2>
<p>The following examples are intended to highlight the functionality provided by the Profiler. Try out these examples easily in the Stellar REPL as described in the <a href="#Getting_Started">Getting Started</a> section.</p>
<div class="section">
<h3><a name="Example_1"></a>Example 1</h3>
<p>This example captures the ratio of DNS traffic to HTTP traffic for each host. The following configuration would be used to generate this profile.</p>
<div>
<div>
<pre class="source">{
&quot;profiles&quot;: [
{
&quot;profile&quot;: &quot;dns-to-http-by-source&quot;,
&quot;foreach&quot;: &quot;ip_src_addr&quot;,
&quot;onlyif&quot;: &quot;protocol == 'DNS' or protocol == 'HTTP'&quot;,
&quot;init&quot;: {
&quot;num_dns&quot;: 1.0,
&quot;num_http&quot;: 1.0
},
&quot;update&quot;: {
&quot;num_dns&quot;: &quot;num_dns + (if protocol == 'DNS' then 1 else 0)&quot;,
&quot;num_http&quot;: &quot;num_http + (if protocol == 'HTTP' then 1 else 0)&quot;
},
&quot;result&quot;: &quot;num_dns / num_http&quot;
}
]
}
</pre></div></div>
<p>This creates a profile&#x2026;</p>
<ul>
<li>Named &#x2018;dns-to-http-by-source&#x2019;</li>
<li>That for each IP source address</li>
<li>Only if the &#x2018;protocol&#x2019; field equals &#x2018;HTTP&#x2019; or &#x2018;DNS&#x2019;</li>
<li>Accumulates the number of DNS requests</li>
<li>Accumulates the number of HTTP requests</li>
<li>Returns the ratio of these as the result</li>
</ul></div>
<div class="section">
<h3><a name="Example_2"></a>Example 2</h3>
<p>This example captures the average of the <tt>length</tt> field for HTTP traffic. The following profile could be used.</p>
<div>
<div>
<pre class="source">{
&quot;profiles&quot;: [
{
&quot;profile&quot;: &quot;avg-http-length&quot;,
&quot;foreach&quot;: &quot;ip_src_addr&quot;,
&quot;onlyif&quot;: &quot;protocol == 'HTTP'&quot;,
&quot;update&quot;: { &quot;s&quot;: &quot;STATS_ADD(s, length)&quot; },
&quot;result&quot;: &quot;STATS_MEAN(s)&quot;
}
]
}
</pre></div></div>
<p>This creates a profile&#x2026;</p>
<ul>
<li>Named &#x2018;avg-http-length&#x2019;</li>
<li>That for each IP source address</li>
<li>Only if the &#x2018;protocol&#x2019; field is &#x2018;HTTP&#x2019;</li>
<li>Captures the <tt>length</tt> field</li>
<li>Calculates the average as the result</li>
</ul>
<p>It is important to note that the Profiler can persist any serializable Object, not just numeric values. Instead of storing the actual mean, the profile could store a statistical sketch of the lengths. This summary can then be used at a later time to calculate the mean, min, max, percentiles, or any other sensible metric. This provides a much greater degree of flexibility. The following Stellar REPL session shows how you might do this.</p>
<ol style="list-style-type: decimal">
<li>
<p>Retrieve the last 30 minutes of profile measurements for a specific host.</p>
<div>
<div>
<pre class="source">$ source /etc/default/metron
$ bin/stellar -z $ZOOKEEPER
[Stellar]&gt;&gt;&gt; stats := PROFILE_GET( &quot;example4&quot;, &quot;10.0.0.1&quot;, PROFILE_FIXED(30, &quot;MINUTES&quot;))
[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9, ...]
</pre></div></div>
</li>
<li>
<p>Calculate different summary metrics using the same profile data.</p>
<div>
<div>
<pre class="source">[Stellar]&gt;&gt;&gt; aStat := GET_FIRST(stats)
org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9
[Stellar]&gt;&gt;&gt; STATS_MEAN(aStat)
15979.0625
[Stellar]&gt;&gt;&gt; STATS_PERCENTILE(aStat, 90)
30310.958
</pre></div></div>
</li>
<li>
<p>Merge all of the profile measurements over the past 30 minutes into a single sketch and calculate the 90th percentile.</p>
<div>
<div>
<pre class="source">[Stellar]&gt;&gt;&gt; merged := STATS_MERGE( stats)
[Stellar]&gt;&gt;&gt; STATS_PERCENTILE(merged, 90)
29810.992
</pre></div></div>
</li>
</ol>
<p>More information on accessing profile data can be found in the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
<p>More information on using the <a href="../metron-statistics/index.html"><tt>STATS_*</tt> functions</a>.</p></div>
<div class="section">
<h3><a name="Example_3"></a>Example 3</h3>
<p>This profile captures the vertex degree of a host. If you view network communication as a directed graph, the in and out degree of each host can distinguish behaviors. Anomalies can serve as an indicator of compromise. For example, you might find clients normally have an out-degree &gt;&gt; in-degree, whereas a server might be the opposite.</p>
<div>
<div>
<pre class="source">{
&quot;profiles&quot;: [
{
&quot;profile&quot;: &quot;in-degrees&quot;,
&quot;onlyif&quot;: &quot;source.type == 'yaf'&quot;,
&quot;foreach&quot;: &quot;ip_dst_addr&quot;,
&quot;update&quot;: { &quot;in&quot;: &quot;HLLP_ADD(in, ip_src_addr)&quot; },
&quot;result&quot;: &quot;HLLP_CARDINALITY(in)&quot;
},
{
&quot;profile&quot;: &quot;out-degrees&quot;,
&quot;onlyif&quot;: &quot;source.type == 'yaf'&quot;,
&quot;foreach&quot;: &quot;ip_src_addr&quot;,
&quot;update&quot;: { &quot;out&quot;: &quot;HLLP_ADD(out, ip_dst_addr)&quot; },
&quot;result&quot;: &quot;HLLP_CARDINALITY(out)&quot;
}
]
}
</pre></div></div>
<p>This creates a profile&#x2026;</p>
<ul>
<li>Named &#x2018;in-degrees&#x2019;</li>
<li>That for each IP destination address</li>
<li>Captures the IP source address</li>
<li>Then calculates the cardinality; the number of unique IPs this host has interacted with</li>
</ul>
<p>The second profile calculates the out-degree.</p></div></div>
</div>
</div>
</div>
<hr/>
<footer>
<div class="container-fluid">
<div class="row-fluid">
© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
and the Apache Metron project logo are trademarks of The Apache Software Foundation.
</div>
</div>
</footer>
</body>
</html>