| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-analytics/metron-profiler-repl/index.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Metron Profiler for the Stellar REPL</title> |
| <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../css/site.css" /> |
| <link rel="stylesheet" href="../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Metron Profiler for the Stellar REPL</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-down"></span>Analytics</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"><span class="none"></span>Maas-service</a></li> |
| <li><a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"><span class="none"></span>Profiler-client</a></li> |
| <li><a href="../../metron-analytics/metron-profiler-common/index.html" title="Profiler-common"><span class="none"></span>Profiler-common</a></li> |
| <li class="active"><a href="#"><span class="none"></span>Profiler-repl</a></li> |
| <li><a href="../../metron-analytics/metron-profiler-spark/index.html" title="Profiler-spark"><span class="none"></span>Profiler-spark</a></li> |
| <li><a href="../../metron-analytics/metron-profiler-storm/index.html" title="Profiler-storm"><span class="none"></span>Profiler-storm</a></li> |
| <li><a href="../../metron-analytics/metron-statistics/index.html" title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li> |
| </ul> |
| </li> |
| <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> |
| <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Metron Profiler for the Stellar REPL</h1> |
| <p><a name="Metron_Profiler_for_the_Stellar_REPL"></a></p> |
| <p>This project allows profiles to be executed within the Stellar REPL. This is a port of the Profiler to the Stellar REPL that allows profiles to be tested and debugged within a controlled environment.</p> |
| <ul> |
| |
| <li><a href="#Introduction">Introduction</a></li> |
| <li><a href="#Getting_Started">Getting Started</a></li> |
| <li><a href="#Installation">Installation</a></li> |
| </ul> |
| <div class="section"> |
| <h2><a name="Introduction"></a>Introduction</h2> |
| <p>Creating and refining profiles is an iterative process. Iterating against a live stream of data is slow, difficult and error prone. Running the Profiler in the Stellar REPL provides a controlled and isolated execution environment to create, refine and troubleshoot profiles.</p> |
| <p>For an introduction to the Profiler, see the <a href="../metron-profiler-common/index.html">Profiler README</a>.</p></div> |
| <div class="section"> |
| <h2><a name="Getting_Started"></a>Getting Started</h2> |
| <p>This section describes how to get started using the Profiler in the Stellar REPL. This outlines a useful workflow for creating, testing, and deploying profiles.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Launch the Stellar REPL.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[root@node1 ~]# $METRON_HOME/bin/stellar |
| Stellar, Go! |
| [Stellar]>>> |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>The following functions should be accessible. Documentation is also provided for each function; for example by executing<tt>?PROFILER_FLUSH</tt>.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %functions PROFILER |
| PROFILER_APPLY, PROFILER_FLUSH, PROFILER_INIT |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Create a simple <tt>hello-world</tt> profile that will count the number of messages for each <tt>ip_src_addr</tt>. The <tt>SHELL_EDIT</tt> function will open an editor in which you can copy/paste the following Profiler configuration.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> conf := SHELL_EDIT() |
| [Stellar]>>> conf |
| { |
| "profiles": [ |
| { |
| "profile": "hello-world", |
| "foreach": "ip_src_addr", |
| "init": { "count": "0" }, |
| "update": { "count": "count + 1" }, |
| "result": "count" |
| } |
| ] |
| } |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Create the profile execution environment.</p> |
| <p>The Profiler will output the number of profiles that have been defined, the number of messages that have been applied and the number of routes that have been followed.</p> |
| <p>While the idea of a profile and message may be well understood, a route may need further explanation. A route is defined when a message is applied to a specific profile.</p> |
| <ul> |
| |
| <li>If a message is not needed by any profile, then there are no routes.</li> |
| <li>If a message is needed by one profile, then one route has been followed.</li> |
| <li>If a message is needed by two profiles, then two routes have been followed.</li> |
| </ul> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> profiler := PROFILER_INIT(conf) |
| [Stellar]>>> profiler |
| Profiler{1 profile(s), 0 messages(s), 0 route(s)} |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Create a message that mimics the telemetry that your profile will consume.</p> |
| <p>This message can be as simple or complex as you like. For the <tt>hello-world</tt> profile, all you need is a message containing an <tt>ip_src_addr</tt> field.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> msg := SHELL_EDIT() |
| [Stellar]>>> msg |
| { |
| "ip_src_addr": "10.0.0.1" |
| } |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Apply the message to your Profiler, as many times as you like.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler) |
| Profiler{1 profile(s), 1 messages(s), 1 route(s)} |
| </pre></div></div> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler) |
| Profiler{1 profile(s), 2 messages(s), 2 route(s)} |
| </pre></div></div> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> PROFILER_APPLY(msg, profiler) |
| Profiler{1 profile(s), 3 messages(s), 3 route(s)} |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Flush the Profiler.</p> |
| <p>A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of Profile Measurements. Each measurement is a map containing detailed information about the profile data that has been generated. The <tt>value</tt> field is what is written to HBase when running the Profiler in either Storm or Spark.</p> |
| <p>There will always be one measurement for each [profile, entity] pair. This profile simply counts the number of messages by IP source address. Notice that the value is ‘3’ for the entity ‘10.0.0.1’ as we applied 3 messages with an ‘ip_src_addr’ of ’10.0.0.1’.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> values := PROFILER_FLUSH(profiler) |
| [Stellar]>>> values |
| [{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000}, |
| profile=hello-world, groups=[], value=3, entity=10.0.0.1}] |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>In addition to testing with mock data, you can also apply real, live telemetry to your profile. This can be useful to test your profile against the complexities that exist in real data.</p> |
| <p>This example extracts 10 messages of live, enriched telemetry to test your profile(s).</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> msgs := KAFKA_GET("indexing", 10) |
| [Stellar]>>> LENGTH(msgs) |
| 10 |
| </pre></div></div> |
| |
| <p>Now apply those 10 messages to your profile.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> PROFILER_APPLY(msgs, profiler) |
| Profiler{1 profile(s), 10 messages(s), 10 route(s)} |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>After you are satisfied with your profile, the next step is to deploy the profile against the live stream of telemetry being capture by Metron. This involves deploying the profile to either the <a href="../metron-profiler-storm/index.html">Storm Profiler</a> or the <a href="../metron-profiler-spark/index.html">Spark Profiler</a>.</p> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h2><a name="Installation"></a>Installation</h2> |
| <p>This package is installed automatically when installing Metron using the Ambari MPack.</p> |
| <p>This package can also be installed manually by using either the RPM or DEB package.</p> |
| <div class="section"> |
| <h3><a name="Build_the_RPM"></a>Build the RPM</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Build Metron.</p> |
| |
| <div> |
| <div> |
| <pre class="source">mvn clean package -DskipTests -T2C |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Build the RPMs.</p> |
| |
| <div> |
| <div> |
| <pre class="source">cd metron-deployment/ |
| mvn clean package -Pbuild-rpms |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Retrieve the package.</p> |
| |
| <div> |
| <div> |
| <pre class="source">find ./ -name "metron-profiler-repl*.rpm" |
| </pre></div></div> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Build_the_DEB"></a>Build the DEB</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Build Metron.</p> |
| |
| <div> |
| <div> |
| <pre class="source">mvn clean package -DskipTests -T2C |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Build the DEBs.</p> |
| |
| <div> |
| <div> |
| <pre class="source">cd metron-deployment/ |
| mvn clean package -Pbuild-debs |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Retrieve the package.</p> |
| |
| <div> |
| <div> |
| <pre class="source">find ./ -name "metron-profiler-repl*.deb" |
| </pre></div></div> |
| </li> |
| </ol></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
