| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/index.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Stellar Language</title> |
| <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../css/site.css" /> |
| <link rel="stylesheet" href="../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Stellar Language</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> |
| <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li class="active"><a href="#"><span class="icon-chevron-down"></span>Stellar-common</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../metron-stellar/stellar-common/3rdPartyStellar.html" title="3rdPartyStellar"><span class="none"></span>3rdPartyStellar</a></li> |
| </ul> |
| </li> |
| <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Stellar Language</h1> |
| <p><a name="Stellar_Language"></a></p> |
| <p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p> |
| <ul> |
| |
| <li><a href="#Introduction">Introduction</a></li> |
| <li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li> |
| <li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li> |
| <li><a href="#Stellar_Shell">Stellar Shell</a> |
| <ul> |
| |
| <li><a href="#Getting_Started">Getting Started</a></li> |
| <li><a href="#Command_Line_Options">Command Line Options</a></li> |
| <li><a href="#Variable_Assignment">Variable Assignment</a></li> |
| <li><a href="#Magic_Commands">Magic Commands</a></li> |
| <li><a href="#Advanced_Usage">Advanced Usage</a></li> |
| <li><a href="#Implementation">Implementation</a></li> |
| </ul> |
| </li> |
| <li><a href="#Stellar_Configuration">Stellar Configuration</a></li> |
| <li><a href="#Stellar_REST_Client">Stellar REST Client</a> |
| <ul> |
| |
| <li><a href="#Configuration">Configuration</a></li> |
| <li><a href="#Security">Security</a></li> |
| <li><a href="#Examples">Examples</a></li> |
| <li><a href="#Latency">Latency</a></li> |
| <li><a href="#Response_Handling">Response Handling</a></li> |
| </ul> |
| </li> |
| </ul> |
| <div class="section"> |
| <h2><a name="Introduction"></a>Introduction</h2> |
| <p>The Stellar language supports the following:</p> |
| <ul> |
| |
| <li>Referencing fields in the enriched JSON</li> |
| <li>Referencing all fields in the enriched JSON via the <tt>_</tt> reserved variable name.</li> |
| <li>String literals are quoted with either <tt>'</tt> or <tt>"</tt></li> |
| <li>String literals support escaping for <tt>'</tt>, <tt>"</tt>, <tt>\t</tt>, <tt>\r</tt>, <tt>\n</tt>, and backslash |
| <ul> |
| |
| <li>The literal <tt>'\'foo\''</tt> would represent <tt>'foo'</tt></li> |
| <li>The literal <tt>"\"foo\""</tt> would represent <tt>"foo"</tt></li> |
| <li>The literal <tt>'foo \\ bar'</tt> would represent <tt>foo \ bar</tt></li> |
| </ul> |
| </li> |
| <li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt></li> |
| <li>Simple arithmetic operations: <tt>*</tt>, <tt>/</tt>, <tt>+</tt>, <tt>-</tt> on real numbers or integers</li> |
| <li>Simple comparison operations <tt><</tt>, <tt>></tt>, <tt><=</tt>, <tt>>=</tt></li> |
| <li>Simple equality comparison operations <tt>==</tt>, <tt>!=</tt></li> |
| <li>if/then/else comparisons (i.e. <tt>if var1 < 10 then 'less than 10' else '10 or more'</tt>)</li> |
| <li>Simple match evaluations (i.e. <tt>match{ var1 < 10 => 'warn', var1 >= 10 => 'critical', default => 'info'}</tt></li> |
| <li>Determining whether a field exists (via <tt>exists</tt>)</li> |
| <li>An <tt>in</tt> operator that works like the <tt>in</tt> in Python</li> |
| <li>The ability to have parenthesis to make order of operations explicit</li> |
| <li>User defined functions, including Lambda expressions</li> |
| </ul> |
| <div class="section"> |
| <h3><a name="Boolean_Expressions"></a>Boolean Expressions</h3> |
| <p>Variables may be used in boolean expressions and variables which are not explicitly boolean may be interpreted as booleans subject to the following rules:</p> |
| <ul> |
| |
| <li>Similar to python and javascript, empty collections (e.g. <tt>[]</tt>) will be interpreted as <tt>false</tt></li> |
| <li>Similar to python and javascript, missing variables will be interpreted as <tt>false</tt></li> |
| <li>Variables set to <tt>null</tt> will be interpreted as <tt>false</tt></li> |
| </ul> |
| <p>Otherwise, boolean variables will be interpreted as their values reflect.</p></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h3> |
| <p>The following keywords need to be single quote escaped in order to be used in Stellar expressions:</p> |
| <table border="0" class="table table-striped"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="center"> </th> |
| <th align="center"> </th> |
| <th align="center"> </th> |
| <th align="center"> </th> |
| <th align="center"> </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="center"> not </td> |
| <td align="center"> else </td> |
| <td align="center"> exists </td> |
| <td align="center"> if </td> |
| <td align="center"> then </td></tr> |
| <tr class="a"> |
| <td align="center"> and </td> |
| <td align="center"> or </td> |
| <td align="center"> in </td> |
| <td align="center"> NaN </td> |
| <td align="center"> match </td></tr> |
| <tr class="b"> |
| <td align="center"> default </td> |
| <td align="center"> == </td> |
| <td align="center"> != </td> |
| <td align="center"> <= </td> |
| <td align="center"> > </td></tr> |
| <tr class="a"> |
| <td align="center"> >= </td> |
| <td align="center"> + </td> |
| <td align="center"> - </td> |
| <td align="center"> < </td> |
| <td align="center"> ? </td></tr> |
| <tr class="b"> |
| <td align="center"> * </td> |
| <td align="center"> / </td> |
| <td align="center"> , </td> |
| <td align="center"> { </td> |
| <td align="center"> } </td></tr> |
| <tr class="a"> |
| <td align="center"> => </td> |
| <td align="center"> </td> |
| <td align="center"> </td> |
| <td align="center"> </td> |
| <td align="center"> </td></tr> |
| </tbody> |
| </table> |
| <p>Using parens such as: “foo” : “<ok>” requires escaping; “foo”: “'<ok>'”</p></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Inclusion_Checks_.28in_and_not_in.29"></a>Stellar Language Inclusion Checks (<tt>in</tt> and <tt>not in</tt>)</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li><tt>in</tt> supports string contains. e.g. <tt>'foo' in 'foobar' == true</tt></li> |
| <li><tt>in</tt> supports collection contains. e.g. <tt>'foo' in [ 'foo', 'bar' ] == true</tt></li> |
| <li><tt>in</tt> supports map key contains. e.g. <tt>'foo' in { 'foo' : 5} == true</tt></li> |
| <li><tt>not in</tt> is the negation of the in expression. e.g. <tt>'grok' not in 'foobar' == true</tt></li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Comparisons_.28.3C.2C_.3C.3D.2C_.3E.2C_.3E.3D.29"></a>Stellar Language Comparisons (<tt><</tt>, <tt><=</tt>, <tt>></tt>, <tt>>=</tt>)</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li>If either side of the comparison is null then return false.</li> |
| <li>If both values being compared implement number then the following: |
| <ul> |
| |
| <li>If either side is a double then get double value from both sides and compare using given operator.</li> |
| <li>Else if either side is a float then get float value from both sides and compare using given operator.</li> |
| <li>Else if either side is a long then get long value from both sides and compare using given operator.</li> |
| <li>Otherwise get the int value from both sides and compare using given operator.</li> |
| </ul> |
| </li> |
| <li>If both sides are of the same type and are comparable then use the compareTo method to compare values.</li> |
| <li>If none of the above are met then an exception is thrown.</li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Equality_Check_.28.3D.3D.2C_.21.3D.29"></a>Stellar Language Equality Check (<tt>==</tt>, <tt>!=</tt>)</h3> |
| <p>Below is how the <tt>==</tt> operator is expected to work:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>If either side of the expression is null then check equality using Java’s <tt>==</tt> expression.</li> |
| <li>Else if both sides of the expression are of Java’s type Number then: |
| <ul> |
| |
| <li>If either side of the expression is a double then use the double value of both sides to test equality.</li> |
| <li>Else if either side of the expression is a float then use the float value of both sides to test equality.</li> |
| <li>Else if either side of the expression is a long then use long value of both sides to test equality.</li> |
| <li>Otherwise use int value of both sides to test equality</li> |
| </ul> |
| </li> |
| <li>Otherwise use equals method compare the left side with the right side.</li> |
| </ol> |
| <p>The <tt>!=</tt> operator is the negation of the above.</p></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h3> |
| <p>Stellar provides the capability to pass lambda expressions to functions which wish to support that layer of indirection. The syntax is:</p> |
| <ul> |
| |
| <li><tt>(named_variables) -> stellar_expression</tt> : Lambda expression with named variables |
| <ul> |
| |
| <li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be be expressed as <tt>(x) -> TO_UPPER(x)</tt>.</li> |
| </ul> |
| </li> |
| <li><tt>var -> stellar_expression</tt> : Lambda expression with a single named variable, <tt>var</tt> |
| <ul> |
| |
| <li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be expressed as <tt>x -> TO_UPPER(x)</tt>. Note, this is more succinct but equivalent to the example directly above.</li> |
| </ul> |
| </li> |
| <li><tt>() -> stellar_expression</tt> : Lambda expression with no named variables. |
| <ul> |
| |
| <li>If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant <tt>false</tt> would be <tt>() -> false</tt></li> |
| </ul> |
| </li> |
| </ul> |
| <p>where</p> |
| <ul> |
| |
| <li><tt>named_variables</tt> is a comma separated list of variables to use in the Stellar expression</li> |
| <li><tt>stellar_expression</tt> is an arbitrary stellar expression</li> |
| </ul> |
| <p>In the core language functions, we support basic functional programming primitives such as</p> |
| <ul> |
| |
| <li><tt>MAP</tt> - Applies a lambda expression over a list of input. For instance <tt>MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )</tt> returns <tt>[ 'FOO', 'BAR' ]</tt></li> |
| <li><tt>FILTER</tt> - Filters a list by a predicate in the form of a lambda expression. For instance <tt>FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )</tt> returns <tt>[ 'foo' ]</tt></li> |
| <li><tt>REDUCE</tt> - Applies a function over a list of input. For instance <tt>REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )</tt> returns <tt>6</tt></li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Stellar_Language_Match_Expression"></a>Stellar Language Match Expression</h3> |
| <p>Stellar provides the capability to write match expressions, which are similar to switch statements commonly found in c like languages.</p> |
| <p>The syntax is:</p> |
| <ul> |
| |
| <li><tt>match{ logical_expression1 => evaluation expression1, logical_expression2 => evaluation_expression2, default => default_expression}</tt></li> |
| </ul> |
| <p>Where:</p> |
| <ul> |
| |
| <li><tt>logical_expression</tt> is a Stellar expression that evaluates to true or false. For instance <tt>var > 0</tt> or <tt>var > 0 AND var2 == 'foo'</tt> or <tt>IF ... THEN ... ELSE</tt></li> |
| <li><tt>evaluation_expression</tt> is a Stellar Expression</li> |
| <li><tt>default</tt> is a required default return value, should no logical expression match</li> |
| </ul> |
| <blockquote> |
| |
| <p>default is required</p> |
| <p>Lambda expressions are supported, but they must be no argument lambdas such as <tt>() -> STATEMENT</tt></p> |
| </blockquote> |
| <ul> |
| |
| <li>Only the first clause that evaluates to true will be executed.</li> |
| </ul></div></div> |
| <div class="section"> |
| <h2><a name="Stellar_Core_Functions"></a>Stellar Core Functions</h2> |
| <table border="0" class="table table-striped"> |
| <thead> |
| |
| <tr class="a"> |
| <th> </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#ABS"> <tt>ABS</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#APPEND_IF_MISSING"> <tt>APPEND_IF_MISSING</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#ASN_GET"> <tt>ASN_GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#BIN"> <tt>BIN</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#BLOOM_ADD"> <tt>BLOOM_ADD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#BLOOM_EXISTS"> <tt>BLOOM_EXISTS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#BLOOM_INIT"> <tt>BLOOM_INIT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#BLOOM_MERGE"> <tt>BLOOM_MERGE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#CEILING"> <tt>CEILING</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#COS"> <tt>COS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#CHOP"> <tt>CHOP</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#CHOMP"> <tt>CHOMP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#COUNT_MATCHES"> <tt>COUNT_MATCHES</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#DATE_FORMAT"> <tt>DATE_FORMAT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#DAY_OF_MONTH"> <tt>DAY_OF_MONTH</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#DAY_OF_WEEK"> <tt>DAY_OF_WEEK</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#DAY_OF_YEAR"> <tt>DAY_OF_YEAR</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#DECODE"> <tt>DECODE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#DOMAIN_REMOVE_SUBDOMAINS"> <tt>DOMAIN_REMOVE_SUBDOMAINS</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#DOMAIN_REMOVE_TLD"> <tt>DOMAIN_REMOVE_TLD</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#DOMAIN_TO_TLD"> <tt>DOMAIN_TO_TLD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#ENCODE"> <tt>ENCODE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#ENDS_WITH"> <tt>ENDS_WITH</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#ENRICHMENT_EXISTS"> <tt>ENRICHMENT_EXISTS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#ENRICHMENT_GET"> <tt>ENRICHMENT_GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#EXP"> <tt>EXP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#FILL_LEFT"> <tt>FILL_LEFT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#FILL_RIGHT"> <tt>FILL_RIGHT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#FILTER"> <tt>FILTER</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#FLOOR"> <tt>FLOOR</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#FORMAT"> <tt>FORMAT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#GEOHASH_CENTROID"> <tt>GEOHASH_CENTROID</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GEOHASH_DIST"> <tt>GEOHASH_DIST</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#GEOHASH_FROM_LATLONG"> <tt>GEOHASH_FROM_LATLONG</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GEOHASH_FROM_LOC"> <tt>GEOHASH_FROM_LOC</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#GEOHASH_MAX_DIST"> <tt>GEOHASH_MAX_DIST</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GEOHASH_TO_LATLONG"> <tt>GEOHASH_TO_LATLONG</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#GET"> <tt>GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#HASH"> <tt>HASH</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_ADD"> <tt>HLLP_ADD</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#IT_ENTROPY"> <tt>IT_ENTROPY</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#IS_ENCODING"> <tt>IS_ENCODING</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IS_IP"> <tt>IS_IP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#IS_NAN"> <tt>IS_NAN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#IS_URL"> <tt>IS_URL</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#JOIN"> <tt>JOIN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#LENGTH"> <tt>LENGTH</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#LOG2"> <tt>LOG2</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#LOG10"> <tt>LOG10</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#LN"> <tt>LN</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MAP"> <tt>MAP</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MAP_GET"> <tt>MAP_GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MAP_MERGE"> <tt>MAP_MERGE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MAP_PUT"> <tt>MAP_PUT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MAX"> <tt>MAX</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MIN"> <tt>MIN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MONTH"> <tt>MONTH</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MULTISET_ADD"> <tt>MULTISET_ADD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MULTISET_INIT"> <tt>MULTISET_INIT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MULTISET_MERGE"> <tt>MULTISET_MERGE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#MULTISET_REMOVE"> <tt>MULTISET_REMOVE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#MULTISET_TO_SET"> <tt>MULTISET_TO_SET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#OBJECT_GET"> <tt>OBJECT_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#PREPEND_IF_MISSING"> <tt>PREPEND_IF_MISSING</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#PROFILE_GET"> <tt>PROFILE_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#PROFILE_VERBOSE"> <tt>PROFILE_VERBOSE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#PROFILE_FIXED"> <tt>PROFILE_FIXED</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#PROFILE_WINDOW"> <tt>PROFILE_WINDOW</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#PROTOCOL_TO_NAME"> <tt>PROTOCOL_TO_NAME</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#REDUCE"> <tt>REDUCE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#REGEXP_MATCH"> <tt>REGEXP_MATCH</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#REGEXP_GROUP_VAL"> <tt>REGEXP_GROUP_VAL</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#REGEXP_REPLACE"> <tt>REGEXP_REPLACE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#REST_GET"> <tt>REST_GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#ROUND"> <tt>ROUND</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#SAMPLE_ADD"> <tt>SAMPLE_ADD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#SAMPLE_GET"> <tt>SAMPLE_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#SAMPLE_INIT"> <tt>SAMPLE_INIT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#SAMPLE_MERGE"> <tt>SAMPLE_MERGE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SET_ADD"> <tt>SET_ADD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SET_INIT"> <tt>SET_INIT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SET_MERGE"> <tt>SET_MERGE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SET_REMOVE"> <tt>SET_REMOVE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SIN"> <tt>SIN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SHELL_EDIT"> <tt>SHELL_EDIT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SHELL_GET_EXPRESSION"> <tt>SHELL_GET_EXPRESSION</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SHELL_LIST_VARS"> <tt>SHELL_LIST_VARS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SHELL_MAP2TABLE"> <tt>SHELL_MAP2TABLE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SHELL_VARS2MAP"> <tt>SHELL_VARS2MAP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SPLIT"> <tt>SPLIT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SQRT"> <tt>SQRT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#STARTS_WITH"> <tt>STARTS_WITH</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_ADD"> <tt>STATS_ADD</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_BIN"> <tt>STATS_BIN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_COUNT"> <tt>STATS_COUNT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_GEOMETRIC_MEAN"> <tt>STATS_GEOMETRIC_MEAN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_INIT"> <tt>STATS_INIT</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_KURTOSIS"> <tt>STATS_KURTOSIS</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_MAX"> <tt>STATS_MAX</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_MEAN"> <tt>STATS_MEAN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_MERGE"> <tt>STATS_MERGE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_MIN"> <tt>STATS_MIN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_PERCENTILE"> <tt>STATS_PERCENTILE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_POPULATION_VARIANCE"> <tt>STATS_POPULATION_VARIANCE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_QUADRATIC_MEAN"> <tt>STATS_QUADRATIC_MEAN</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_SD"> <tt>STATS_SD</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_SKEWNESS"> <tt>STATS_SKEWNESS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_SUM"> <tt>STATS_SUM</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_SUM_LOGS"> <tt>STATS_SUM_LOGS</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_SUM_SQUARES"> <tt>STATS_SUM_SQUARES</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="../../metron-analytics/metron-statistics/index.html#STATS_VARIANCE"> <tt>STATS_VARIANCE</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#STRING_ENTROPY"> <tt>STRING_ENTROPY</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SUBSTRING"> <tt>SUBSTRING</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#SYSTEM_ENV_GET"> <tt>SYSTEM_ENV_GET</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#SYSTEM_PROPERTY_GET"> <tt>SYSTEM_PROPERTY_GET</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TAN"> <tt>TAN</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TLSH_DIST"> <tt>TLSH_DIST</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_DOUBLE"> <tt>TO_DOUBLE</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TO_EPOCH_TIMESTAMP"> <tt>TO_EPOCH_TIMESTAMP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_FLOAT"> <tt>TO_FLOAT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TO_INTEGER"> <tt>TO_INTEGER</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_JSON_LIST"> <tt>TO_JSON_LIST</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TO_JSON_MAP"> <tt>TO_JSON_MAP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_JSON_OBJECT"> <tt>TO_JSON_OBJECT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TO_LONG"> <tt>TO_LONG</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_LOWER"> <tt>TO_LOWER</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TO_STRING"> <tt>TO_STRING</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#TO_UPPER"> <tt>TO_UPPER</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#TRIM"> <tt>TRIM</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#URL_TO_HOST"> <tt>URL_TO_HOST</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#URL_TO_PATH"> <tt>URL_TO_PATH</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#URL_TO_PORT"> <tt>URL_TO_PORT</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#URL_TO_PROTOCOL"> <tt>URL_TO_PROTOCOL</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#WEEK_OF_MONTH"> <tt>WEEK_OF_MONTH</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#WEEK_OF_YEAR"> <tt>WEEK_OF_YEAR</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#YEAR"> <tt>YEAR</tt></a> </td></tr> |
| <tr class="a"> |
| <td> <a href="#ZIP"> <tt>ZIP</tt></a> </td></tr> |
| <tr class="b"> |
| <td> <a href="#ZIP_LONGEST"> <tt>ZIP_LONGEST</tt></a> </td></tr> |
| </tbody> |
| </table> |
| <div class="section"> |
| <h3><a name="APPEND_IF_MISSING"></a><tt>APPEND_IF_MISSING</tt></h3> |
| <ul> |
| |
| <li>Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to be appended.</li> |
| <li>suffix - The string suffix to append to the end of the string.</li> |
| <li>additionalsuffix - Optional - Additional string suffix that is a valid terminator.</li> |
| </ul> |
| </li> |
| <li>Returns: A new String if prefix was prepended, the same string otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ASN_GET"></a><tt>ASN_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Look up an IPV4 address and returns Autonomous System Number information about it</li> |
| <li>Input: |
| <ul> |
| |
| <li>ip - The IPV4 address to lookup</li> |
| <li>fields - Optional list of ASN fields to grab. Options are network, autonomous_system_number, autonomous_system_organization.</li> |
| </ul> |
| </li> |
| <li>Returns: If a Single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="BLOOM_ADD"></a><tt>BLOOM_ADD</tt></h3> |
| <ul> |
| |
| <li>Description: Adds an element to the bloom filter passed in</li> |
| <li>Input: |
| <ul> |
| |
| <li>bloom - The bloom filter</li> |
| <li>value(s) - The value(s) to add</li> |
| </ul> |
| </li> |
| <li>Returns: Bloom Filter</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="BLOOM_EXISTS"></a><tt>BLOOM_EXISTS</tt></h3> |
| <ul> |
| |
| <li>Description: If the bloom filter contains the value</li> |
| <li>Input: |
| <ul> |
| |
| <li>bloom - The bloom filter</li> |
| <li>value - The value to check</li> |
| </ul> |
| </li> |
| <li>Returns: True if the filter might contain the value and false otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="BLOOM_INIT"></a><tt>BLOOM_INIT</tt></h3> |
| <ul> |
| |
| <li>Description: Returns an empty bloom filter</li> |
| <li>Input: |
| <ul> |
| |
| <li>expectedInsertions - The expected insertions</li> |
| <li>falsePositiveRate - The false positive rate you are willing to tolerate</li> |
| </ul> |
| </li> |
| <li>Returns: Bloom Filter</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="BLOOM_MERGE"></a><tt>BLOOM_MERGE</tt></h3> |
| <ul> |
| |
| <li>Description: Returns a merged bloom filter</li> |
| <li>Input: |
| <ul> |
| |
| <li>bloomfilters - A list of bloom filters to merge</li> |
| </ul> |
| </li> |
| <li>Returns: Bloom Filter or null if the list is empty</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="CEILING"></a><tt>CEILING</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the ceiling of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the ceiling of</li> |
| </ul> |
| </li> |
| <li>Returns: The ceiling of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="COS"></a><tt>COS</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the cosine of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the cosine of</li> |
| </ul> |
| </li> |
| <li>Returns: The cosine of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="CHOP"></a><tt>CHOP</tt></h3> |
| <ul> |
| |
| <li>Description: Remove the last character from a String</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - the String to chop last character from, may be null</li> |
| </ul> |
| </li> |
| <li>Returns: String without last character, null if null String input</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="CHOMP"></a><tt>CHOMP</tt></h3> |
| <ul> |
| |
| <li>Description: Removes one newline from end of a String if it’s there, otherwise leave it alone. A newline is “\n”, “\r”, or “\r\n”</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - the String to chomp a newline from, may be null</li> |
| </ul> |
| </li> |
| <li>Returns: String without newline, null if null String input</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="COUNT_MATCHES"></a><tt>COUNT_MATCHES</tt></h3> |
| <ul> |
| |
| <li>Description: Counts how many times the substring appears in the larger string.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - the CharSequence to check, may be null.</li> |
| <li>substring/character - the substring or character to count, may be null.</li> |
| </ul> |
| </li> |
| <li>Returns: the number of non-overlapping occurrences, 0 if either CharSequence is null.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DATE_FORMAT"></a><tt>DATE_FORMAT</tt></h3> |
| <ul> |
| |
| <li>Description: Takes an epoch timestamp and converts it to a date format.</li> |
| <li>Input: |
| <ul> |
| |
| <li>format - DateTime format as a String.</li> |
| <li>timestampField - Optional epoch time in Long format. Defaults to now.</li> |
| <li>timezone - Optional timezone in String format.</li> |
| </ul> |
| </li> |
| <li>Returns: Formatted date.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DAY_OF_MONTH"></a><tt>DAY_OF_MONTH</tt></h3> |
| <ul> |
| |
| <li>Description: The numbered day within the month. The first day within the month has a value of 1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The numbered day within the month.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DAY_OF_WEEK"></a><tt>DAY_OF_WEEK</tt></h3> |
| <ul> |
| |
| <li>Description: The numbered day within the week. The first day of the week, Sunday, has a value of 1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The numbered day within the week.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DAY_OF_YEAR"></a><tt>DAY_OF_YEAR</tt></h3> |
| <ul> |
| |
| <li>Description: The day number within the year. The first day of the year has value of 1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The day number within the year.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DECODE"></a><tt>DECODE</tt></h3> |
| <ul> |
| |
| <li>Description: Decodes the passed string with the provided encoding, which must be one of the encodings returned from <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a></li> |
| <li>Input: |
| <ul> |
| |
| <li>string - the string to decode</li> |
| <li>encoding - the encoding to use, must be one of encodings returned from <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a></li> |
| <li>verify - (optional), true or false to determine if string should be verified as being encoded with the passed encoding</li> |
| </ul> |
| </li> |
| <li>Returns: |
| <ul> |
| |
| <li>The decoded string on success</li> |
| <li>The original string the string cannot be decoded</li> |
| <li>null on usage error</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DOMAIN_REMOVE_SUBDOMAINS"></a><tt>DOMAIN_REMOVE_SUBDOMAINS</tt></h3> |
| <ul> |
| |
| <li>Description: Removes the subdomains from a domain.</li> |
| <li>Input: |
| <ul> |
| |
| <li>domain - Fully qualified domain name</li> |
| </ul> |
| </li> |
| <li>Returns: The domain without the subdomains. (for example, DOMAIN_REMOVE_SUBDOMAINS(‘mail.yahoo.com’) yields ‘yahoo.com’)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DOMAIN_REMOVE_TLD"></a><tt>DOMAIN_REMOVE_TLD</tt></h3> |
| <ul> |
| |
| <li>Description: Removes the top level domain (TLD) suffix from a domain.</li> |
| <li>Input: |
| <ul> |
| |
| <li>domain - Fully qualified domain name</li> |
| </ul> |
| </li> |
| <li>Returns: The domain without the TLD. (for example, DOMAIN_REMOVE_TLD(‘mail.yahoo.co.uk’) yields ‘mail.yahoo’)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="DOMAIN_TO_TLD"></a><tt>DOMAIN_TO_TLD</tt></h3> |
| <ul> |
| |
| <li>Description: Extracts the top level domain from a domain</li> |
| <li>Input: |
| <ul> |
| |
| <li>domain - Fully qualified domain name</li> |
| </ul> |
| </li> |
| <li>Returns: The TLD of the domain. (for example, DOMAIN_TO_TLD(‘mail.yahoo.co.uk’) yields ‘co.uk’)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ENCODE"></a><tt>ENCODE</tt></h3> |
| <ul> |
| |
| <li>Description: Encodes the passed string with the provided encoding, which must be one of the encodings returned from <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a></li> |
| <li>Input: |
| <ul> |
| |
| <li>string - the string to encode</li> |
| <li>encoding - the encoding to use, must be one of encodings returned from <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a></li> |
| </ul> |
| </li> |
| <li>Returns: |
| <ul> |
| |
| <li>The encoded string on success</li> |
| <li>null on error</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ENDS_WITH"></a><tt>ENDS_WITH</tt></h3> |
| <ul> |
| |
| <li>Description: Determines whether a string ends with a specified suffix</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to test</li> |
| <li>suffix - The proposed suffix</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string ends with the specified suffix and false if otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ENRICHMENT_EXISTS"></a><tt>ENRICHMENT_EXISTS</tt></h3> |
| <ul> |
| |
| <li>Description: Interrogates the HBase table holding the simple hbase enrichment data and returns whether the enrichment type and indicator are in the table.</li> |
| <li>Input: |
| <ul> |
| |
| <li>enrichment_type - The enrichment type</li> |
| <li>indicator - The string indicator to look up</li> |
| <li>nosql_table - The NoSQL Table to use</li> |
| <li>column_family - The Column Family to use</li> |
| </ul> |
| </li> |
| <li>Returns: True if the enrichment indicator exists and false otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ENRICHMENT_GET"></a><tt>ENRICHMENT_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Interrogates the HBase table holding the simple hbase enrichment data and retrieves the tabular value associated with the enrichment type and indicator.</li> |
| <li>Input: |
| <ul> |
| |
| <li>enrichment_type - The enrichment type</li> |
| <li>indicator - The string indicator to look up</li> |
| <li>nosql_table - The NoSQL Table to use</li> |
| <li>column_family - The Column Family to use</li> |
| </ul> |
| </li> |
| <li>Returns: A Map associated with the indicator and enrichment type. Empty otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="EXP"></a><tt>EXP</tt></h3> |
| <ul> |
| |
| <li>Description: Returns Euler’s number raised to the power of the argument.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The power to which e is raised.</li> |
| </ul> |
| </li> |
| <li>Returns: Euler’s number raised to the power of the argument.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FILL_LEFT"></a><tt>FILL_LEFT</tt></h3> |
| <ul> |
| |
| <li>Description: Fills or pads a given string with a given character, to a given length on the left</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - string</li> |
| <li>fill - the fill character</li> |
| <li>len - the required length</li> |
| </ul> |
| </li> |
| <li>Returns: the filled string</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FILL_RIGHT"></a><tt>FILL_RIGHT</tt></h3> |
| <ul> |
| |
| <li>Description: Fills or pads a given string with a given character, to a given length on the right</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - string</li> |
| <li>fill - the fill character string</li> |
| <li>len - the required length</li> |
| </ul> |
| </li> |
| <li>Returns: Last element of the list</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FILTER"></a><tt>FILTER</tt></h3> |
| <ul> |
| |
| <li>Description: Applies a filter in the form of a lambda expression to a list. e.g. <tt>FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo')</tt> would yield <tt>[ 'foo']</tt></li> |
| <li>Input: |
| <ul> |
| |
| <li>list - List of arguments.</li> |
| <li>predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean.</li> |
| </ul> |
| </li> |
| <li>Returns: The input list filtered by the predicate.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FLOOR"></a><tt>FLOOR</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the floor of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the floor of</li> |
| </ul> |
| </li> |
| <li>Returns: The floor of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FORMAT"></a><tt>FORMAT</tt></h3> |
| <ul> |
| |
| <li>Description: Returns a formatted string using the specified format string and arguments. Uses Java’s string formatting conventions.</li> |
| <li>Input: |
| <ul> |
| |
| <li>format - string</li> |
| <li>arguments… - object(s)</li> |
| </ul> |
| </li> |
| <li>Returns: A formatted string.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FUZZY_LANGS"></a><tt>FUZZY_LANGS</tt></h3> |
| <ul> |
| |
| <li>Description: Returns a list of IETF BCP 47 available to the system, such as en, fr, de.</li> |
| <li>Returns: A list of IEF BGP 47 language tag strings</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="FUZZY_SCORE"></a><tt>FUZZY_SCORE</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the Fuzzy Score which indicates the similarity score between two strings. One point is given for every matched character. Subsequent matches yield two bonus points. A higher score indicates a higher similarity.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The full term that should be matched against.</li> |
| <li>string - The query that will be matched against a term.</li> |
| <li>string - The IETF BCP 47 language code to use.</li> |
| </ul> |
| </li> |
| <li>Returns: An Integer representing the score.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEO_GET"></a><tt>GEO_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Look up an IPV4 address and returns geographic information about it</li> |
| <li>Input: |
| <ul> |
| |
| <li>ip - The IPV4 address to lookup</li> |
| <li>fields - Optional list of GeoIP fields to grab. Options are locID, country, city postalCode, dmaCode, latitude, longitude, location_point</li> |
| </ul> |
| </li> |
| <li>Returns: If a Single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_CENTROID"></a><tt>GEOHASH_CENTROID</tt></h3> |
| <ul> |
| |
| <li>Description: Compute the centroid (geographic midpoint or center of gravity) of a set of <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohashes</a></li> |
| <li>Input: |
| <ul> |
| |
| <li>hashes - A collection of <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohashes</a> or a map associating geohashes to numeric weights</li> |
| <li>character_precision? - The number of characters to use in the hash. Default is 12</li> |
| </ul> |
| </li> |
| <li>Returns: The geohash of the centroid</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_DIST"></a><tt>GEOHASH_DIST</tt></h3> |
| <ul> |
| |
| <li>Description: Compute the distance between <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohashes</a></li> |
| <li>Input: |
| <ul> |
| |
| <li>hash1 - The first point as a geohash</li> |
| <li>hash2 - The second point as a geohash</li> |
| <li>strategy? - The great circle distance strategy to use. One of <a class="externalLink" href="https://en.wikipedia.org/wiki/Haversine_formula">HAVERSINE</a>, <a class="externalLink" href="https://en.wikipedia.org/wiki/Law_of_cosines#Using_the_distance_formula">LAW_OF_COSINES</a>, or <a class="externalLink" href="https://en.wikipedia.org/wiki/Vincenty%27s_formulae">VICENTY</a>. Haversine is default.</li> |
| </ul> |
| </li> |
| <li>Returns: The distance in kilometers between the hashes.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_FROM_LATLONG"></a><tt>GEOHASH_FROM_LATLONG</tt></h3> |
| <ul> |
| |
| <li>Description: Compute <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a> given a lat/long</li> |
| <li>Input: |
| <ul> |
| |
| <li>latitude - The latitude</li> |
| <li>longitude - The longitude</li> |
| <li>character_precision? - The number of characters to use in the hash. Default is 12</li> |
| </ul> |
| </li> |
| <li>Returns: A <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a> of the lat/long</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_FROM_LOC"></a><tt>GEOHASH_FROM_LOC</tt></h3> |
| <ul> |
| |
| <li>Description: Compute <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a> given a geo enrichment location</li> |
| <li>Input: |
| <ul> |
| |
| <li>map - the latitude and logitude in a map (the output of <a href="#GEO_GET">GEO_GET</a> )</li> |
| <li>longitude - The longitude</li> |
| <li>character_precision? - The number of characters to use in the hash. Default is <tt>12</tt></li> |
| </ul> |
| </li> |
| <li>Returns: A <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a> of the location</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_MAX_DIST"></a><tt>GEOHASH_MAX_DIST</tt></h3> |
| <ul> |
| |
| <li>Description: Compute the maximum distance among a list of <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohashes</a></li> |
| <li>Input: |
| <ul> |
| |
| <li>hashes - A set of <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohashes</a></li> |
| <li>strategy? - The great circle distance strategy to use. One of <a class="externalLink" href="https://en.wikipedia.org/wiki/Haversine_formula">HAVERSINE</a>, <a class="externalLink" href="https://en.wikipedia.org/wiki/Law_of_cosines#Using_the_distance_formula">LAW_OF_COSINES</a>, or <a class="externalLink" href="https://en.wikipedia.org/wiki/Vincenty%27s_formulae">VICENTY</a>. Haversine is default.</li> |
| </ul> |
| </li> |
| <li>Returns: The maximum distance in kilometers between any two locations</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GEOHASH_TO_LATLONG"></a><tt>GEOHASH_TO_LATLONG</tt></h3> |
| <ul> |
| |
| <li>Description: Compute the lat/long of a given <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a></li> |
| <li>Input: |
| <ul> |
| |
| <li>hash - The <a class="externalLink" href="https://en.wikipedia.org/wiki/Geohash">geohash</a></li> |
| </ul> |
| </li> |
| <li>Returns: A map containing the latitude and longitude of the hash (keys “latitude” and “longitude”)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GET"></a><tt>GET</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the i’th element of the list</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - List</li> |
| <li>i - The index (0-based)</li> |
| </ul> |
| </li> |
| <li>Returns: First element of the list</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GET_FIRST"></a><tt>GET_FIRST</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the first element of the list</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - List</li> |
| </ul> |
| </li> |
| <li>Returns: First element of the list</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GET_HASHES_AVAILABLE"></a><tt>GET_HASHES_AVAILABLE</tt></h3> |
| <ul> |
| |
| <li>Description: Will return all available hashing algorithms available to ‘HASH’.</li> |
| <li>Returns: A list containing all supported hashing algorithms.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GET_LAST"></a><tt>GET_LAST</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the last element of the list</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - List</li> |
| </ul> |
| </li> |
| <li>Returns: Last element of the list</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="GET_SUPPORTED_ENCODINGS"></a><tt>GET_SUPPORTED_ENCODINGS</tt></h3> |
| <ul> |
| |
| <li>Description: Returns a list of the encodings that are currently supported.</li> |
| <li>Returns: A List of String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="HASH"></a><tt>HASH</tt></h3> |
| <ul> |
| |
| <li>Description: Hashes a given value using the given hashing algorithm and returns a hex encoded string.</li> |
| <li>Input: |
| <ul> |
| |
| <li>toHash - value to hash.</li> |
| <li>hashType - A valid string representation of a hashing algorithm. See ‘GET_HASHES_AVAILABLE’.</li> |
| <li>config? - Configuration for the hash function in the form of a String to object map. |
| <ul> |
| |
| <li>For forensic hash TLSH (see <a class="externalLink" href="https://github.com/trendmicro/tlsh">https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013): |
| <ul> |
| |
| <li>bucketSize : This defines the size of the hash created. Valid values are 128 (default) or 256 (the former results in a 70 character hash and latter results in 134 characters)</li> |
| <li>checksumBytes : This defines how many bytes are used to capture the checksum. Valid values are 1 (default) and 3</li> |
| <li>force : If true (the default) then a hash can be generated from as few as 50 bytes. If false, then at least 256 bytes are required. Insufficient variation or size in the bytes result in a null being returned.</li> |
| <li>hashes : You can compute a second hash for use in fuzzy clustering TLSH signatures. The number of hashes is the lever to adjust the size of those clusters and "fuzzy" the clusters are. If this is specified, then one or more bins are created based on the specified size and the function will return a Map containing the bins.</li> |
| </ul> |
| </li> |
| <li>For all other hashes: |
| <ul> |
| |
| <li>charset : The character set to use (UTF8 is default).</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Returns = A hex encoded string of a hashed value using the given algorithm. If ‘hashType’ is null then ‘00’, padded to the necessary length, will be returned. If ‘toHash’ is not able to be hashed or ‘hashType’ is null then null is returned.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IN_SUBNET"></a><tt>IN_SUBNET</tt></h3> |
| <ul> |
| |
| <li>Description: Returns true if an IP is within a subnet range.</li> |
| <li>Input: |
| <ul> |
| |
| <li>ip - The IP address in string form</li> |
| <li>cidr+ - One or more IP ranges specified in CIDR notation (for example 192.168.0.0/24)</li> |
| </ul> |
| </li> |
| <li>Returns: True if the IP address is within at least one of the network ranges and false if otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_DATE"></a><tt>IS_DATE</tt></h3> |
| <ul> |
| |
| <li>Description: Determines if the date contained in the string conforms to the specified format.</li> |
| <li>Input: |
| <ul> |
| |
| <li>date - The date in string form</li> |
| <li>format - The format of the date</li> |
| </ul> |
| </li> |
| <li>Returns: True if the date is in the specified format and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_DOMAIN"></a><tt>IS_DOMAIN</tt></h3> |
| <ul> |
| |
| <li>Description: Tests if a string refers to a valid domain name. Domain names are evaluated according to the standards RFC1034 section 3, and RFC1123 section 2.1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>address - The string to test</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string refers to a valid domain name and false if otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_EMAIL"></a><tt>IS_EMAIL</tt></h3> |
| <ul> |
| |
| <li>Description: Tests if a string is a valid email address</li> |
| <li>Input: |
| <ul> |
| |
| <li>address - The string to test</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string is a valid email address and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_EMPTY"></a><tt>IS_EMPTY</tt></h3> |
| <ul> |
| |
| <li>Description: Returns true if string or collection is empty or null and false if otherwise.</li> |
| <li>Input: |
| <ul> |
| |
| <li>Object of string or collection type (for example, list)</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string or collection is empty or null and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_ENCODING"></a><tt>IS_ENCODING</tt></h3> |
| <ul> |
| |
| <li>Description: Returns true if the passed string is encoded in one of the supported encodings and false if otherwise.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to test</li> |
| <li>encoding - The name of the encoding as string. See <a href="#GET_SUPPORTED_ENCODINGS"> <tt>GET_SUPPORTED_ENCODINGS</tt></a>.</li> |
| </ul> |
| </li> |
| <li>Returns: True if the passed string is encoded in one of the supported encodings and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_INTEGER"></a><tt>IS_INTEGER</tt></h3> |
| <ul> |
| |
| <li>Description: Determines whether or not an object is an integer.</li> |
| <li>Input: |
| <ul> |
| |
| <li>x - The object to test</li> |
| </ul> |
| </li> |
| <li>Returns: True if the object can be converted to an integer and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_IP"></a><tt>IS_IP</tt></h3> |
| <ul> |
| |
| <li>Description: Determine if an string is an IP or not.</li> |
| <li>Input: |
| <ul> |
| |
| <li>ip - An object which we wish to test is an ip</li> |
| <li>type (optional) - Object of string or collection type (e.g. list) one of IPV4 or IPV6 or both. The default is IPV4.</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string is an IP and false otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_NAN"></a><tt>IS_NAN</tt></h3> |
| <ul> |
| |
| <li>Description: Evaluates if the passed number is NaN. The number is evaluated as a double.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - number to evaluate"</li> |
| </ul> |
| </li> |
| <li>Returns: True if the number is NaN, false if it is</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="IS_URL"></a><tt>IS_URL</tt></h3> |
| <ul> |
| |
| <li>Description: Tests if a string is a valid URL</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - The string to test</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string is a valid URL and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="JOIN"></a><tt>JOIN</tt></h3> |
| <ul> |
| |
| <li>Description: Joins the components in the list of strings with the specified delimiter.</li> |
| <li>Input: |
| <ul> |
| |
| <li>list - List of strings</li> |
| <li>delim - String delimiter</li> |
| </ul> |
| </li> |
| <li>Returns: String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="KAFKA_GET"></a><tt>KAFKA_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Retrieves messages from a Kafka topic. Subsequent calls will continue retrieving messages sequentially from the original offset.</li> |
| <li>Input: |
| <ul> |
| |
| <li>topic - The name of the Kafka topic.</li> |
| <li>count - The number of Kafka messages to retrieve.</li> |
| <li>config - Optional map of key/values that override any global properties.</li> |
| </ul> |
| </li> |
| <li>Returns: List of String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="KAFKA_PROPS"></a><tt>KAFKA_PROPS</tt></h3> |
| <ul> |
| |
| <li>Description: Retrieves the Kafka properties that are used by other KAFKA_* functions like KAFKA_GET and KAFKA_PUT. The Kafka properties are compiled from a set of default properties, the global properties, and any overrides.</li> |
| <li>Input: |
| <ul> |
| |
| <li>config - An optional map of key/values that override any global properties.</li> |
| </ul> |
| </li> |
| <li>Returns: Map of key/value pairs</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="KAFKA_PUT"></a><tt>KAFKA_PUT</tt></h3> |
| <ul> |
| |
| <li>Description: Sends messages to a Kafka topic.</li> |
| <li>Input: |
| <ul> |
| |
| <li>topic - The name of the Kafka topic.</li> |
| <li>messages - A list of messages to write.</li> |
| <li>config - Optional map of key/values that override any global properties.</li> |
| </ul> |
| </li> |
| <li>Returns: n/a</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="KAFKA_TAIL"></a><tt>KAFKA_TAIL</tt></h3> |
| <ul> |
| |
| <li>Description: etrieves messages from a Kafka topic always starting with the most recent message first.</li> |
| <li>Input: |
| <ul> |
| |
| <li>topic - The name of the Kafka topic.</li> |
| <li>count - The number of Kafka messages to retrieve.</li> |
| <li>config - Optional map of key/values that override any global properties.</li> |
| </ul> |
| </li> |
| <li>Returns: List of String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="LENGTH"></a><tt>LENGTH</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the length of a string or size of a collection. Returns 0 for empty or null Strings</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object of string or collection type (e.g. list)</li> |
| </ul> |
| </li> |
| <li>Returns: Integer</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="LIST_ADD"></a><tt>LIST_ADD</tt></h3> |
| <ul> |
| |
| <li>Description: Adds an element to a list.</li> |
| <li>Input: |
| <ul> |
| |
| <li>list - List to add element to.</li> |
| <li>element - Element to add to list</li> |
| </ul> |
| </li> |
| <li>Returns: Resulting list with the item added at the end.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="LOG2"></a><tt>LOG2</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the log (base <tt>2</tt>) of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the log (base <tt>2</tt>) of</li> |
| </ul> |
| </li> |
| <li>Returns: The log (base <tt>2</tt>) of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="LOG10"></a><tt>LOG10</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the log (base <tt>10</tt>) of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the log (base <tt>10</tt>) of</li> |
| </ul> |
| </li> |
| <li>Returns: The log (base <tt>10</tt>) of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="LN"></a><tt>LN</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the natural log of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the natural log of</li> |
| </ul> |
| </li> |
| <li>Returns: The natural log of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAAS_GET_ENDPOINT"></a><tt>MAAS_GET_ENDPOINT</tt></h3> |
| <ul> |
| |
| <li>Description: Inspects ZooKeeper and returns a map containing the name, version and url for the model referred to by the input parameters.</li> |
| <li>Input: |
| <ul> |
| |
| <li>model_name - The name of the model</li> |
| <li>model_version - The optional version of the model. If the model version is not specified, the most current version is used.</li> |
| </ul> |
| </li> |
| <li>Returns: A map containing the name, version, and url for the REST endpoint (fields named name, version and url). Note that the output of this function is suitable for input into the first argument of MAAS_MODEL_APPLY.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAAS_MODEL_APPLY"></a><tt>MAAS_MODEL_APPLY</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the output of a model deployed via Model as a Service. NOTE: Results are cached locally for 10 minutes.</li> |
| <li>Input: |
| <ul> |
| |
| <li>endpoint - A map containing the name, version, and url for the REST endpoint</li> |
| <li>function - The optional endpoint path; default is ‘apply’</li> |
| <li>model_args - A Dictionary of arguments for the model (these become request params)</li> |
| </ul> |
| </li> |
| <li>Returns: The output of the model deployed as a REST endpoint in Map form. Assumes REST endpoint returns a JSON Map.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAP"></a><tt>MAP</tt></h3> |
| <ul> |
| |
| <li>Description: Applies lambda expression to a list of arguments. e.g. <tt>MAP( [ 'foo', 'bar' ] , (x) -> TO_UPPER(x) )</tt> would yield <tt>[ 'FOO', 'BAR' ]</tt></li> |
| <li>Input: |
| <ul> |
| |
| <li>list - List of arguments.</li> |
| <li>transform_expression - The lambda expression to apply. This expression is assumed to take one argument.</li> |
| </ul> |
| </li> |
| <li>Returns: The input list transformed item-wise by the lambda expression.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAP_EXISTS"></a><tt>MAP_EXISTS</tt></h3> |
| <ul> |
| |
| <li>Description: Checks for existence of a key in a map.</li> |
| <li>Input: |
| <ul> |
| |
| <li>key - The key to check for existence</li> |
| <li>map - The map to check for existence of the key</li> |
| </ul> |
| </li> |
| <li>Returns: True if the key is found in the map and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAP_GET"></a><tt>MAP_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Gets the value associated with a key from a map</li> |
| <li>Input: |
| <ul> |
| |
| <li>key - The key</li> |
| <li>map - The map</li> |
| <li>default - Optionally the default value to return if the key is not in the map.</li> |
| </ul> |
| </li> |
| <li>Returns: The object associated with the key in the map. If no value is associated with the key and default is specified, then default is returned. If no value is associated with the key or default, then null is returned.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAP_MERGE"></a><tt>MAP_MERGE</tt></h3> |
| <ul> |
| |
| <li>Description: Merges a list of maps</li> |
| <li>Input: |
| <ul> |
| |
| <li>maps - A collection of maps to merge. Last entry wins for overlapping keys.</li> |
| </ul> |
| </li> |
| <li>Returns: A Map. null if the list of maps is empty.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAP_PUT"></a><tt>MAP_PUT</tt></h3> |
| <ul> |
| |
| <li>Description: Adds a key/value pair to a map</li> |
| <li>Input: |
| <ul> |
| |
| <li>key - The key</li> |
| <li>value - The value</li> |
| <li>map - The map to perform the put on</li> |
| </ul> |
| </li> |
| <li>Returns: The original map modified with the key/value. If the map argument is null, a new map will be created and returned that contains the provided key and value - note: if the ‘map’ argument is null, only the returned map will be non-null and contain the key/value.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MAX"></a><tt>MAX</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the maximum value of a list of input values. |
| <ul> |
| |
| <li>Input:</li> |
| <li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MAX([10,15L,15.3]) would return 15.3, but MAX([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li> |
| </ul> |
| </li> |
| <li>Returns: The maximum value of the list, or null if the list is empty or the input values were not comparable.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MIN"></a><tt>MIN</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the minimum value of a list of input values. |
| <ul> |
| |
| <li>Input:</li> |
| <li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MIN([10,15L,15.3]) would return 10, but MIN([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li> |
| </ul> |
| </li> |
| <li>Returns: The minimum value of the list, or null if the list is empty or the input values were not comparable.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MONTH"></a><tt>MONTH</tt></h3> |
| <ul> |
| |
| <li>Description: The number representing the month. The first month, January, has a value of 0.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The current month (0-based)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MULTISET_ADD"></a><tt>MULTISET_ADD</tt></h3> |
| <ul> |
| |
| <li>Description: Adds to a multiset, which is a map associating objects to their instance counts.</li> |
| <li>Input: |
| <ul> |
| |
| <li>set - The multiset to add to</li> |
| <li>o - object to add to multiset</li> |
| </ul> |
| </li> |
| <li>Returns: A multiset</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MULTISET_INIT"></a><tt>MULTISET_INIT</tt></h3> |
| <ul> |
| |
| <li>Description: Creates an empty multiset, which is a map associating objects to their instance counts.</li> |
| <li>Input: |
| <ul> |
| |
| <li>input (optional) - An initialization of the multiset</li> |
| </ul> |
| </li> |
| <li>Returns: A multiset</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MULTISET_MERGE"></a><tt>MULTISET_MERGE</tt></h3> |
| <ul> |
| |
| <li>Description: Merges a list of multisets, which is a map associating objects to their instance counts.</li> |
| <li>Input: |
| <ul> |
| |
| <li>sets - A collection of multisets to merge</li> |
| </ul> |
| </li> |
| <li>Returns: A multiset</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MULTISET_REMOVE"></a><tt>MULTISET_REMOVE</tt></h3> |
| <ul> |
| |
| <li>Description: Removes from a multiset, which is a map associating objects to their instance counts.</li> |
| <li>Input: |
| <ul> |
| |
| <li>set - The multiset to add to</li> |
| <li>o - object to remove from multiset</li> |
| </ul> |
| </li> |
| <li>Returns: A multiset</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="MULTISET_TO_SET"></a><tt>MULTISET_TO_SET</tt></h3> |
| <ul> |
| |
| <li>Description: Create a set out of a multiset, which is a map associating objects to their instance counts.</li> |
| <li>Input: |
| <ul> |
| |
| <li>multiset - The multiset to convert.</li> |
| </ul> |
| </li> |
| <li>Returns: The set of objects in the multiset ignoring multiplicity</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="OBJECT_GET"></a><tt>OBJECT_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Retrieve and deserialize a serialized object from HDFS. The cache can be specified via two properties in the global config: “object.cache.size” (default 1000), “object.cache.expiration.minutes” (default 1440). Note, if these are changed in global config, topology restart is required.</li> |
| <li>Input: |
| <ul> |
| |
| <li>path - The path in HDFS to the serialized object</li> |
| </ul> |
| </li> |
| <li>Returns: The deserialized object.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PREPEND_IF_MISSING"></a><tt>PREPEND_IF_MISSING</tt></h3> |
| <ul> |
| |
| <li>Description: Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to be prepended.</li> |
| <li>prefix - The string prefix to prepend to the start of the string.</li> |
| <li>additionalprefix - Optional - Additional string prefix that is valid.</li> |
| </ul> |
| </li> |
| <li>Returns: A new String if prefix was prepended, the same string otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Retrieves a series of values from a stored profile.</li> |
| <li>Input: |
| <ul> |
| |
| <li>profile - The name of the profile.</li> |
| <li>entity - The name of the entity.</li> |
| <li>periods - The list of profile periods to fetch. Use PROFILE_WINDOW or PROFILE_FIXED.</li> |
| <li>groups_list - Optional, must correspond to the ‘groupBy’ list used in profile creation - List (in square brackets) of groupBy values used to filter the profile. Default is the empty list, meaning groupBy was not used when creating the profile.</li> |
| <li>config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li> |
| </ul> |
| </li> |
| <li>Returns: The selected profile measurements.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PROFILE_VERBOSE"></a><tt>PROFILE_VERBOSE</tt></h3> |
| <ul> |
| |
| <li>Description: Retrieves a series of measurements from a stored profile. Returns a map containing the profile name, entity, period id, period start, period end for each profile measurement. Provides a more verbose view of each measurement than PROFILE_GET.</li> |
| <li>Input: |
| <ul> |
| |
| <li>profile - The name of the profile.</li> |
| <li>entity - The name of the entity.</li> |
| <li>periods - The list of profile periods to fetch. Use PROFILE_WINDOW or PROFILE_FIXED.</li> |
| <li>groups - Optional - The groups to retrieve. Must correspond to the ‘groupBy’ used during profile creation. Defaults to an empty list, meaning no groups.</li> |
| </ul> |
| </li> |
| <li>Returns: The selected profile measurements.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PROFILE_FIXED"></a><tt>PROFILE_FIXED</tt></h3> |
| <ul> |
| |
| <li>Description: The profile periods associated with a fixed lookback starting from now</li> |
| <li>Input: |
| <ul> |
| |
| <li>durationAgo - How long ago should values be retrieved from?</li> |
| <li>units - The units of ‘durationAgo’.</li> |
| <li>config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li> |
| </ul> |
| </li> |
| <li>Returns: The selected profile measurement timestamps. These are ProfilePeriod objects.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h3> |
| <ul> |
| |
| <li>Description: The profiler periods associated with a window selector statement from an optional reference timestamp.</li> |
| <li>Input: |
| <ul> |
| |
| <li>windowSelector - The statement specifying the window to select.</li> |
| <li>now - Optional - The timestamp to use for now.</li> |
| <li>config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li> |
| </ul> |
| </li> |
| <li>Returns: The selected profile measurement periods. These are ProfilePeriod objects.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="PROTOCOL_TO_NAME"></a><tt>PROTOCOL_TO_NAME</tt></h3> |
| <ul> |
| |
| <li>Description: Converts the IANA protocol number to the protocol name</li> |
| <li>Input: |
| <ul> |
| |
| <li>IANA Number</li> |
| </ul> |
| </li> |
| <li>Returns: The protocol name associated with the IANA number.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="REDUCE"></a><tt>REDUCE</tt></h3> |
| <ul> |
| |
| <li>Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: <tt>REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y, 0)</tt> would sum the input list, yielding <tt>6</tt>.</li> |
| <li>Input: |
| <ul> |
| |
| <li>list - List of arguments.</li> |
| <li>binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list.</li> |
| <li>initial_value - The initial value to use.</li> |
| </ul> |
| </li> |
| <li>Returns: The reduction of the list.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="REGEXP_MATCH"></a><tt>REGEXP_MATCH</tt></h3> |
| <ul> |
| |
| <li>Description: Determines whether a regex matches a string. If a list of patterns is passed, then the matching is an OR operation</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to test</li> |
| <li>pattern - The proposed regex pattern or a list of patterns</li> |
| </ul> |
| </li> |
| <li>Returns: True if the regex pattern matches the string and false if otherwise.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="REGEXP_GROUP_VAL"></a><tt>REGEXP_GROUP_VAL</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the value of a group in a regex against a string</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to test</li> |
| <li>pattern - The proposed regex pattern</li> |
| <li>group - The integer that selects what group to select, starting at 1</li> |
| </ul> |
| </li> |
| <li>Returns: The value of the group, or null if not matched or no group at index.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="REGEXP_REPLACE"></a><tt>REGEXP_REPLACE</tt></h3> |
| <ul> |
| |
| <li>Description: Replace all occurences of the regex pattern within the string by value</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The input string</li> |
| <li>pattern - The proposed regex pattern</li> |
| <li>value - The value to replace the regex pattern</li> |
| </ul> |
| </li> |
| <li>Returns: The modified input string with replaced values</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="REST_GET"></a><tt>REST_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Performs a REST GET request and parses the JSON results into a map.</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - URI to the REST service</li> |
| <li>rest_config - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li> |
| </ul> |
| </li> |
| <li>Returns: JSON results as a Map</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ROUND"></a><tt>ROUND</tt></h3> |
| <ul> |
| |
| <li>Description: Rounds a number to the nearest integer. This is half-up rounding.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to round</li> |
| </ul> |
| </li> |
| <li>Returns: The nearest integer (based on half-up rounding).</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SET_ADD"></a><tt>SET_ADD</tt></h3> |
| <ul> |
| |
| <li>Description: Adds to a set</li> |
| <li>Input: |
| <ul> |
| |
| <li>set - The set to add to</li> |
| <li>o - object to add to set</li> |
| </ul> |
| </li> |
| <li>Returns: A Set</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SET_INIT"></a><tt>SET_INIT</tt></h3> |
| <ul> |
| |
| <li>Description: Creates an new set</li> |
| <li>Input: |
| <ul> |
| |
| <li>input (optional) - An initialization of the set</li> |
| </ul> |
| </li> |
| <li>Returns: A Set</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SET_MERGE"></a><tt>SET_MERGE</tt></h3> |
| <ul> |
| |
| <li>Description: Merges a list of sets</li> |
| <li>Input: |
| <ul> |
| |
| <li>sets - A collection of sets to merge</li> |
| </ul> |
| </li> |
| <li>Returns: A Set</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SET_REMOVE"></a><tt>SET_REMOVE</tt></h3> |
| <ul> |
| |
| <li>Description: Removes from a set</li> |
| <li>Input: |
| <ul> |
| |
| <li>set - The set to add to</li> |
| <li>o - object to add to set</li> |
| </ul> |
| </li> |
| <li>Returns: A Set</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SHELL_EDIT"></a><tt>SHELL_EDIT</tt></h3> |
| <ul> |
| |
| <li>Description: Open an editor (optionally initialized with text) and return whatever is saved from the editor. The editor to use is pulled from <tt>EDITOR</tt> or <tt>VISUAL</tt> environment variable.</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - (Optional) A string whose content is used to initialize the editor.</li> |
| </ul> |
| </li> |
| <li>Returns: The content that the editor saved after editor exit.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SHELL_GET_EXPRESSION"></a><tt>SHELL_GET_EXPRESSION</tt></h3> |
| <ul> |
| |
| <li>Description: Get a stellar expression from a variable</li> |
| <li>Input: |
| <ul> |
| |
| <li>variable - variable name</li> |
| </ul> |
| </li> |
| <li>Returns: The stellar expression associated with the variable.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SHELL_LIST_VARS"></a><tt>SHELL_LIST_VARS</tt></h3> |
| <ul> |
| |
| <li>Description: Return the variables in a tabular form</li> |
| <li>Input: |
| <ul> |
| |
| <li>wrap : Length of string to wrap the columns</li> |
| </ul> |
| </li> |
| <li>Returns: A tabular representation of the variables.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SHELL_MAP2TABLE"></a><tt>SHELL_MAP2TABLE</tt></h3> |
| <ul> |
| |
| <li>Description: Take a map and return a table</li> |
| <li>Input: |
| <ul> |
| |
| <li>map - Map</li> |
| </ul> |
| </li> |
| <li>Returns: The map in table form</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SHELL_VARS2MAP"></a><tt>SHELL_VARS2MAP</tt></h3> |
| <ul> |
| |
| <li>Description: Take a set of variables and return a map</li> |
| <li>Input: |
| <ul> |
| |
| <li>variables* - variable names to use to create map</li> |
| </ul> |
| </li> |
| <li>Returns: A map associating the variable name with the stellar expression.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SIN"></a><tt>SIN</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the sine of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the sine of</li> |
| </ul> |
| </li> |
| <li>Returns: The sine of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SPLIT"></a><tt>SPLIT</tt></h3> |
| <ul> |
| |
| <li>Description: Splits the string by the delimiter.</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - String to split</li> |
| <li>delim - String delimiter</li> |
| </ul> |
| </li> |
| <li>Returns: List of strings</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SQRT"></a><tt>SQRT</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the square root of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the square root of</li> |
| </ul> |
| </li> |
| <li>Returns: The square root of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="STRING_ENTROPY"></a><tt>STRING_ENTROPY</tt></h3> |
| <ul> |
| |
| <li>Description: Computes the base-2 shannon entropy of a string.</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - String</li> |
| </ul> |
| </li> |
| <li>Returns: The base-2 shannon entropy of the string (<a class="externalLink" href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition</a>). The unit of this is bits.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="STARTS_WITH"></a><tt>STARTS_WITH</tt></h3> |
| <ul> |
| |
| <li>Description: Determines whether a string starts with a prefix</li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The string to test</li> |
| <li>prefix - The proposed prefix</li> |
| </ul> |
| </li> |
| <li>Returns: True if the string starts with the specified prefix and false if otherwise</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SYSTEM_ENV_GET"></a><tt>SYSTEM_ENV_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the value associated with an environment variable</li> |
| <li>Input: |
| <ul> |
| |
| <li>env_var - Environment variable name to get the value for</li> |
| </ul> |
| </li> |
| <li>Returns: String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SYSTEM_PROPERTY_GET"></a><tt>SYSTEM_PROPERTY_GET</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the value associated with a Java system property</li> |
| <li>Input: |
| <ul> |
| |
| <li>key - Property to get the value for</li> |
| </ul> |
| </li> |
| <li>Returns: String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="SUBSTRING"></a><tt>SUBSTRING</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the substring of a string</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - The string to take the substring of</li> |
| <li>start - The starting position (<tt>0</tt>-based and inclusive)</li> |
| <li>end? - The ending position (<tt>0</tt>-based and exclusive)</li> |
| </ul> |
| </li> |
| <li>Returns: The substring of the input</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TAN"></a><tt>TAN</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the tangent of a number.</li> |
| <li>Input: |
| <ul> |
| |
| <li>number - The number to take the tangent of</li> |
| </ul> |
| </li> |
| <li>Returns: The tangent of the number passed in.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TLSH_DIST"></a><tt>TLSH_DIST</tt></h3> |
| <ul> |
| |
| <li>Description: Will return the hamming distance between two TLSH hashes (note: must be computed with the same params). For more information, see <a class="externalLink" href="https://github.com/trendmicro/tlsh">https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013. For a discussion of tradeoffs, see Table II on page 5 of <a class="externalLink" href="https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf">https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf</a></li> |
| <li>Input: |
| <ul> |
| |
| <li>hash1 - The first TLSH hash</li> |
| <li>hash2 - The first TLSH hash</li> |
| <li>includeLength? - Include the length in the distance calculation or not? Returns: An integer representing the distance between hash1 and hash2. The distance is roughly hamming distance, so 0 is very similar.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_DOUBLE"></a><tt>TO_DOUBLE</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to a double precision number</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object of string or numeric type</li> |
| </ul> |
| </li> |
| <li>Returns: Double version of the first argument</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_EPOCH_TIMESTAMP"></a><tt>TO_EPOCH_TIMESTAMP</tt></h3> |
| <ul> |
| |
| <li>Description: Returns the epoch timestamp of the dateTime in the specified format. If the format does not have a timestamp and you wish to assume a given timestamp, you may specify the timezone optionally.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - DateTime in String format</li> |
| <li>format - DateTime format as a String</li> |
| <li>timezone - Optional timezone in String format</li> |
| </ul> |
| </li> |
| <li>Returns: Epoch timestamp</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_FLOAT"></a><tt>TO_FLOAT</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to a float</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object of string or numeric type</li> |
| </ul> |
| </li> |
| <li>Returns: Float version of the first argument</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_INTEGER"></a><tt>TO_INTEGER</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to an integer</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object of string or numeric type</li> |
| </ul> |
| </li> |
| <li>Returns: Integer version of the first argument</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_JSON_LIST"></a><tt>TO_JSON_LIST</tt></h3> |
| <ul> |
| |
| <li>Description: Accepts JSON string as an input and returns a List object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>GET_FIRST( TO_JSON_LIST( '[ "foo", 2]')</tt> would yield <tt>foo</tt></li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The JSON string to be parsed</li> |
| </ul> |
| </li> |
| <li>Returns: A parsed List object</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_JSON_MAP"></a><tt>TO_JSON_MAP</tt></h3> |
| <ul> |
| |
| <li>Description: Accepts JSON string as an input and returns a Map object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_MAP( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The JSON string to be parsed</li> |
| </ul> |
| </li> |
| <li>Returns: A parsed Map object</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_JSON_OBJECT"></a><tt>TO_JSON_OBJECT</tt></h3> |
| <ul> |
| |
| <li>Description: Accepts JSON string as an input and returns a JSON Object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_OBJECT( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li> |
| <li>Input: |
| <ul> |
| |
| <li>string - The JSON string to be parsed</li> |
| </ul> |
| </li> |
| <li>Returns: A parsed JSON object</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_LONG"></a><tt>TO_LONG</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to a long integer</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object of string or numeric type</li> |
| </ul> |
| </li> |
| <li>Returns: Long version of the first argument</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_LOWER"></a><tt>TO_LOWER</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to a lowercase string</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - String</li> |
| </ul> |
| </li> |
| <li>Returns: Lowercase string</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_STRING"></a><tt>TO_STRING</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to a string</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - Object</li> |
| </ul> |
| </li> |
| <li>Returns: String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TO_UPPER"></a><tt>TO_UPPER</tt></h3> |
| <ul> |
| |
| <li>Description: Transforms the first argument to an uppercase string</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - String</li> |
| </ul> |
| </li> |
| <li>Returns: Uppercase string</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="TRIM"></a><tt>TRIM</tt></h3> |
| <ul> |
| |
| <li>Description: Trims whitespace from both sides of a string.</li> |
| <li>Input: |
| <ul> |
| |
| <li>input - String</li> |
| </ul> |
| </li> |
| <li>Returns: String</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="URL_TO_HOST"></a><tt>URL_TO_HOST</tt></h3> |
| <ul> |
| |
| <li>Description: Extract the hostname from a URL.</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - URL in String form</li> |
| </ul> |
| </li> |
| <li>Returns: The hostname from the URL as a String. e.g. URL_TO_HOST(‘<a class="externalLink" href="http://www.yahoo.com/foo">http://www.yahoo.com/foo</a>’) would yield ‘<a href="www.yahoo.com">www.yahoo.com</a>’</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="URL_TO_PATH"></a><tt>URL_TO_PATH</tt></h3> |
| <ul> |
| |
| <li>Description: Extract the path from a URL.</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - URL in String form</li> |
| </ul> |
| </li> |
| <li>Returns: The path from the URL as a String. e.g. URL_TO_PATH(‘<a class="externalLink" href="http://www.yahoo.com/foo">http://www.yahoo.com/foo</a>’) would yield ‘foo’</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="URL_TO_PORT"></a><tt>URL_TO_PORT</tt></h3> |
| <ul> |
| |
| <li>Description: Extract the port from a URL. If the port is not explicitly stated in the URL, then an implicit port is inferred based on the protocol.</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - URL in string form</li> |
| </ul> |
| </li> |
| <li>Returns: The port used in the URL as an integer (for example, URL_TO_PORT(‘<a class="externalLink" href="http://www.yahoo.com/foo">http://www.yahoo.com/foo</a>’) would yield 80)</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="URL_TO_PROTOCOL"></a><tt>URL_TO_PROTOCOL</tt></h3> |
| <ul> |
| |
| <li>Description: Extract the protocol from a URL.</li> |
| <li>Input: |
| <ul> |
| |
| <li>url - URL in String form</li> |
| </ul> |
| </li> |
| <li>Returns: The protocol from the URL as a String. e.g. URL_TO_PROTOCOL(‘<a class="externalLink" href="http://www.yahoo.com/foo">http://www.yahoo.com/foo</a>’) would yield ‘http’</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="WEEK_OF_MONTH"></a><tt>WEEK_OF_MONTH</tt></h3> |
| <ul> |
| |
| <li>Description: The numbered week within the month. The first week within the month has a value of 1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The numbered week within the month.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="WEEK_OF_YEAR"></a><tt>WEEK_OF_YEAR</tt></h3> |
| <ul> |
| |
| <li>Description: The numbered week within the year. The first week in the year has a value of 1.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The numbered week within the year.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="YEAR"></a><tt>YEAR</tt></h3> |
| <ul> |
| |
| <li>Description: The number representing the year.</li> |
| <li>Input: |
| <ul> |
| |
| <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> |
| </ul> |
| </li> |
| <li>Returns: The current year</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ZIP"></a><tt>ZIP</tt></h3> |
| <ul> |
| |
| <li>Description: Zips lists into a single list where the ith element is an list containing the ith items from the constituent lists. See <a class="externalLink" href="https://docs.python.org/3/library/functions.html#zip">python</a> and <a class="externalLink" href="https://en.wikipedia.org/wiki/Convolution_(computer_science)">wikipedia</a> for more context.</li> |
| <li>Input: |
| <ul> |
| |
| <li>list(s) - List(s) to zip.</li> |
| </ul> |
| </li> |
| <li>Returns: The zip of the lists. The returned list is the min size of all the lists. e.g. <tt>ZIP( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4] ]</tt></li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="ZIP_LONGEST"></a><tt>ZIP_LONGEST</tt></h3> |
| <ul> |
| |
| <li>Description: Zips lists into a single list where the ith element is an list containing the ith items from the constituent lists. See <a class="externalLink" href="https://docs.python.org/3/library/itertools.html#itertools.zip_longest">python</a> and <a class="externalLink" href="https://en.wikipedia.org/wiki/Convolution_(computer_science)">wikipedia</a> for more context.</li> |
| <li>Input: |
| <ul> |
| |
| <li>list(s) - List(s) to zip.</li> |
| </ul> |
| </li> |
| <li>Returns: The zip of the lists. The returned list is the max size of all the lists. Empty elements are null e.g. <tt>ZIP_LONGEST( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4], [null, 5] ]</tt></li> |
| </ul> |
| <p>The following is an example query (i.e. a function which returns a boolean) which would be seen possibly in threat triage:</p> |
| <p><tt>IN_SUBNET( ip, '192.168.0.0/24') or ip in [ '10.0.0.1', '10.0.0.2' ] or exists(is_local)</tt></p> |
| <p>This evaluates to true precisely when one of the following is true:</p> |
| <ul> |
| |
| <li>The value of the <tt>ip</tt> field is in the <tt>192.168.0.0/24</tt> subnet</li> |
| <li>The value of the <tt>ip</tt> field is <tt>10.0.0.1</tt> or <tt>10.0.0.2</tt></li> |
| <li>The field <tt>is_local</tt> exists</li> |
| </ul> |
| <p>The following is an example transformation which might be seen in a field transformation:</p> |
| <p><tt>TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC'))</tt></p> |
| <p>For a message with a <tt>timestamp</tt> and <tt>dc</tt> field, we want to set the transform the timestamp to an epoch timestamp given a timezone which we will lookup in a separate map, called <tt>dc2tz</tt>.</p> |
| <p>This will convert the timestamp field to an epoch timestamp based on the</p> |
| <ul> |
| |
| <li>Format <tt>yyyy-MM-dd HH:mm:ss</tt></li> |
| <li>The value in <tt>dc2tz</tt> associated with the value associated with field <tt>dc</tt>, defaulting to <tt>UTC</tt></li> |
| </ul></div></div> |
| <div class="section"> |
| <h2><a name="Stellar_Benchmarks"></a>Stellar Benchmarks</h2> |
| <p>A microbenchmarking utility is included to assist in executing microbenchmarks for Stellar functions. The utility can be executed via maven using the <tt>exec</tt> plugin, like so, from the <tt>metron-common</tt> directory:</p> |
| |
| <div> |
| <div> |
| <pre class="source">mvn -DskipTests clean package && \ |
| mvn exec:java -Dexec.mainClass="org.apache.metron.stellar.common.benchmark.StellarMicrobenchmark" -Dexec.args="..." |
| </pre></div></div> |
| |
| <p>where <tt>exec.args</tt> can be one of the following:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> -e,--expressions <FILE> Stellar expressions |
| -h,--help Generate Help screen |
| -n,--num_times <NUM> Number of times to run per expression (after |
| warmup). Default: 1000 |
| -o,--output <FILE> File to write output. |
| -p,--percentiles <NUM> Percentiles to calculate per run. Default: |
| 50.0,75.0,95.0,99.0 |
| -v,--variables <FILE> File containing a JSON Map of variables to use |
| -w,--warmup <NUM> Number of times for warmup per expression. |
| Default: 100 |
| </pre></div></div> |
| |
| <p>For instance, to run with a set of Stellar expression in file <tt>/tmp/expressions.txt</tt>:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> # simple functions |
| TO_UPPER('casey') |
| TO_LOWER(name) |
| # math functions |
| 1 + 2*(3 + int_num) / 10.0 |
| 1.5 + 2*(3 + double_num) / 10.0 |
| # conditionals |
| if ('foo' in ['foo']) OR one == very_nearly_one then 'one' else 'two' |
| 1 + 2*(3 + int_num) / 10.0 |
| #Network funcs |
| DOMAIN_TO_TLD(domain) |
| DOMAIN_REMOVE_SUBDOMAINS(domain) |
| </pre></div></div> |
| |
| <p>And variables in file <tt>/tmp/variables.json</tt>:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name" : "casey", |
| "int_num" : 1, |
| "double_num" : 17.5, |
| "one" : 1, |
| "very_nearly_one" : 1.000001, |
| "domain" : "www.google.com" |
| } |
| </pre></div></div> |
| |
| <p>Written to file <tt>/tmp/output.txt</tt> would be the following command:</p> |
| |
| <div> |
| <div> |
| <pre class="source">mvn -DskipTests clean package && \ |
| mvn exec:java -Dexec.mainClass="org.apache.metron.stellar.common.benchmark.StellarMicrobenchmark" \ |
| -Dexec.args="-e /tmp/expressions.txt -v /tmp/variables.json -o ./output.json" |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h2><a name="Stellar_Shell"></a>Stellar Shell</h2> |
| <p>The Stellar Shell is a REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting, and learning Stellar. It can also be used as a language-checking resource while interacting with a live Metron cluster.</p> |
| <p>The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate behavior encountered in production, isolate initialization errors, or understand function resolution problems. Because it can be run from the command line on any node with Metron installed, it can help the user understand environmental problems that may be interfering with Stellar running in Storm servers.</p> |
| <p>The shell supports customization via <tt>~/.inputrc</tt> as it is backed by a proper readline implementation.</p> |
| <p>Shell-like operations are supported such as</p> |
| <ul> |
| |
| <li>reverse search via ctrl-r</li> |
| <li>autocomplete of Stellar functions and variables via tab |
| <ul> |
| |
| <li>NOTE: Stellar functions are read via a classpath search which happens in the background. Until that happens, autocomplete will not include function names.</li> |
| </ul> |
| </li> |
| <li>emacs or vi keybindings for edit mode</li> |
| </ul> |
| <p>Note: Stellar classpath configuration from the global config is honored here if the REPL knows about zookeeper.</p> |
| <div class="section"> |
| <h3><a name="Environment_Variables"></a>Environment Variables</h3> |
| <p>When starting the REPL via <tt>$METRON_HOME/bin/stellar</tt> you can specify certain environment variables to customize the experience:</p> |
| <ul> |
| |
| <li><tt>JVMFLAGS</tt> - Arbitrary JVM flags to pass to the <tt>java</tt> command when starting the REPL.</li> |
| <li><tt>CONTRIB</tt> - Directory where jars with Stellar functions can be placed. The default is <tt>$METRON_HOME/contrib</tt>.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Getting_Started"></a>Getting Started</h3> |
| <p>To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.</p> |
| |
| <div> |
| <div> |
| <pre class="source">$ $METRON_HOME/bin/stellar |
| |
| Stellar, Go! |
| {es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} |
| |
| [Stellar]>>> %functions |
| BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, ... |
| |
| [Stellar]>>> ?PROTOCOL_TO_NAME |
| PROTOCOL_TO_NAME |
| desc: Convert the IANA protocol number to the protocol name |
| args: IANA Number |
| ret: The protocol name associated with the IANA number. |
| |
| [Stellar]>>> ip.protocol := 6 |
| 6 |
| [Stellar]>>> PROTOCOL_TO_NAME(ip.protocol) |
| TCP |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Command_Line_Options"></a>Command Line Options</h3> |
| |
| <div> |
| <div> |
| <pre class="source">$ $METRON_HOME/bin/stellar -h |
| usage: stellar |
| -h,--help Print help |
| -irc,--inputrc <arg> File containing the inputrc if not the default |
| ~/.inputrc |
| -v,--variables <arg> File containing a JSON Map of variables |
| -z,--zookeeper <arg> Zookeeper URL fragment in the form [HOSTNAME|IPADDRESS]:PORT |
| -na,--no_ansi Make the input prompt not use ANSI colors. |
| </pre></div></div> |
| |
| <div class="section"> |
| <h4><a name="a-v.2C_--variables"></a><tt>-v, --variables</tt></h4> |
| <p><i>Optional</i></p> |
| <p>Optionally load a JSON map which contains variable assignments. This is intended to give you the ability to save off a message from Metron and work on it via the REPL.</p></div> |
| <div class="section"> |
| <h4><a name="a-z.2C_--zookeeper"></a><tt>-z, --zookeeper</tt></h4> |
| <p><i>Optional</i></p> |
| <p>Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console. If specified, then the classpath may be augmented by the paths specified in the stellar config in the global config.</p> |
| |
| <div> |
| <div> |
| <pre class="source">$ $METRON_HOME/bin/stellar -z node1:2181 |
| Stellar, Go! |
| {es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} |
| [Stellar]>>> |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h3><a name="Variable_Assignment"></a>Variable Assignment</h3> |
| <p>Stellar has no concept of variable assignment. For testing and debugging purposes, it is important to be able to create variables that simulate data contained within incoming messages. The REPL has created a means for a user to perform variable assignment outside of the core Stellar language. This is done via the <tt>:=</tt> operator, such as <tt>foo := 1 + 1</tt> would assign the result of the stellar expression <tt>1 + 1</tt> to the variable <tt>foo</tt>.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> foo := 2 + 2 |
| 4.0 |
| [Stellar]>>> 2 + 2 |
| 4.0 |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Magic_Commands"></a>Magic Commands</h3> |
| <p>The REPL has a set of magic commands that provide the REPL user with information about the Stellar execution environment. The following magic commands are supported.</p> |
| <div class="section"> |
| <h4><a name="a.25functions"></a><tt>%functions</tt></h4> |
| <p>This command lists all functions resolvable in the Stellar environment.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %functions |
| BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, |
| DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, GET, GET_FIRST, |
| GET_LAST, GET_ENCODINGS_LIST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_ENCODING, IS_INTEGER, IS_IP, IS_URL, |
| JOIN, LENGTH, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP_EXISTS, MAP_GET, MONTH, PROTOCOL_TO_NAME, |
| REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, |
| STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, |
| STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, |
| STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, |
| TO_INTEGER, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, |
| URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR |
| </pre></div></div> |
| |
| <p>The list of functions returned can also be filtered by passing an argument. Only the functions containing the argument as a substring will be returned.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %functions NET |
| IN_SUBNET |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h4><a name="a.25vars"></a><tt>%vars</tt></h4> |
| <p>Lists all variables in the Stellar environment.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %vars |
| [Stellar]>>> foo := 2 + 2 |
| 4.0 |
| [Stellar]>>> %vars |
| foo = 4.0 |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h4><a name="a.25globals"></a><tt>%globals</tt></h4> |
| <p>Lists all values that are defined in the global configuration.</p> |
| <p>Most of Metron’s functional components have access to what is called the global configuration. This is a key/value configuration store that can be used to customize Metron. Many Stellar functions accept configuration values from the global configuration. The Stellar Shell also leverages the global configuration for customizing the behavior of many Stellar functions.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %globals |
| {es.clustername=metron, es.ip=node1:9300, es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing, update.hbase.table=metron_update, update.hbase.cf=t} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h4><a name="a.25define"></a><tt>%define</tt></h4> |
| <p>Defines a global configuration value in the current shell session. This value will be forgotten once the session is ended.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %define bootstrap.servers := "node1:6667" |
| node1:6667 |
| [Stellar]>>> %globals |
| {bootstrap.servers=node1:6667} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h4><a name="a.25undefine"></a><tt>%undefine</tt></h4> |
| <p>Undefine a global configuration value in the current shell session. This will not modify the persisted global configuration.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> %undefine bootstrap.servers |
| [Stellar]>>> %globals |
| {} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h4><a name="a.3F.3Cfunction.3E"></a><tt>?<function></tt></h4> |
| <p>Returns formatted documentation of the Stellar function. Provides the description of the function along with the expected arguments.</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> ?BLOOM_ADD |
| BLOOM_ADD |
| desc: Adds an element to the bloom filter passed in |
| args: bloom - The bloom filter, value* - The values to add |
| ret: Bloom Filter |
| [Stellar]>>> ?IS_EMAIL |
| IS_EMAIL |
| desc: Tests if a string is a valid email address |
| args: address - The String to test |
| ret: True if the string is a valid email address and false otherwise. |
| [Stellar]>>> |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h3><a name="Advanced_Usage"></a>Advanced Usage</h3> |
| <p>To run the Stellar Shell directly from the Metron source code, run a command like the following. Ensure that Metron has already been built and installed with <tt>mvn clean install -DskipTests</tt>.</p> |
| |
| <div> |
| <div> |
| <pre class="source">$ mvn exec:java \ |
| -Dexec.mainClass="org.apache.metron.stellar.common.shell.cli.StellarShell" \ |
| -pl metron-platform/metron-enrichment |
| ... |
| Stellar, Go! |
| Please note that functions are loading lazily in the background and will be unavailable until loaded fully. |
| [Stellar]>>> Functions loaded, you may refer to functions now... |
| [Stellar]>>> %functions |
| ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR |
| </pre></div></div> |
| |
| <p>Changing the project passed to the <tt>-pl</tt> argument will define which dependencies are included and ultimately which Stellar functions are available within the shell environment.</p> |
| <p>This can be useful for troubleshooting function resolution problems. The previous example defines which functions are available during Enrichment. For example, to determine which functions are available within the Profiler run the following.</p> |
| |
| <div> |
| <div> |
| <pre class="source"> $ mvn exec:java \ |
| -Dexec.mainClass="org.apache.metron.stellar.common.shell.cli.StellarShell" \ |
| -pl metron-analytics/metron-profiler |
| ... |
| Stellar, Go! |
| Please note that functions are loading lazily in the background and will be unavailable until loaded fully. |
| [Stellar]>>> Functions loaded, you may refer to functions now... |
| %functions |
| ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h2><a name="Stellar_Shell_Stand_Alone"></a>Stellar Shell Stand Alone</h2> |
| <p>The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported operating system.</p> |
| <blockquote> |
| |
| <p>Only the base Stellar functions are available as packaged. Other functions, such as those in metron-profiler and metron-management are not available.</p> |
| </blockquote> |
| |
| <div> |
| <div> |
| <pre class="source">metron-stellar/stellar-common/target/stellar-common-0.7.1-stand-alone.tar.gz |
| </pre></div></div> |
| |
| <p>When unpacked, the following structure will be created:</p> |
| |
| <div> |
| <div> |
| <pre class="source">. |
| ├── bin |
| │   └── stellar |
| └── lib |
| └── stellar-common-0.7.1-uber.jar |
| </pre></div></div> |
| |
| <p>To run the Stellar Shell run the following from the directory you unpacked to:</p> |
| |
| <div> |
| <div> |
| <pre class="source">bin/stellar |
| </pre></div></div> |
| |
| <div> |
| <div> |
| <pre class="source">-> % bin/stellar |
| Stellar, Go! |
| Please note that functions are loading lazily in the background and will be unavailable until loaded fully. |
| [Stellar]>>> Functions loaded, you may refer to functions now... |
| |
| [Stellar]>>> %functions |
| ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST |
| [Stellar]>>> |
| </pre></div></div> |
| |
| <p>By default the shell will have the base Stellar Language commands available. Any jars in the lib directory that contain Stellar functions will also be loaded, and their commands will be available to shell, as long as their dependencies are satisfied.</p> |
| <div class="section"> |
| <h3><a name="Implementation"></a>Implementation</h3> |
| <p>The Stellar Shell can be executed both from the command line and from within a Stellar Notebook. The behavior and underlying implementation of the behavior is exactly the same across these two environments.</p> |
| <div class="section"> |
| <h4><a name="org.apache.metron.stellar.common.shell"></a><tt>org.apache.metron.stellar.common.shell</tt></h4> |
| <p>This package contains classes that are reused across both the CLI and Zeppelin shell environments.</p> |
| <ul> |
| |
| <li> |
| |
| <p><tt>StellarShellExecutor</tt> Executes Stellar in a shell-like environment. Provides the Stellar language extensions like variable assignment, comments, magics, and doc strings that are only accessible in the shell.</p> |
| </li> |
| <li> |
| |
| <p><tt>StellarAutoCompleter</tt> Handles auto-completion for Stellar.</p> |
| </li> |
| <li> |
| |
| <p><tt>StellarExecutorListeners</tt> An event listener that can be notified when variables, functions, and specials are defined. This is how a <tt>StellarAutoCompleter</tt> is notified throughout the life of a shell session.</p> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h4><a name="org.apache.metron.stellar.common.shell.specials"></a><tt>org.apache.metron.stellar.common.shell.specials</tt></h4> |
| <p>All Stellar language extensions are contained within this package.</p> |
| <ul> |
| |
| <li><tt>SpecialCommand</tt> The interface for all special commands. A ‘special command’ is anything that is not directly provided by the Stellar language itself. That includes variable assignment, comments, doc strings, magics, and quit.</li> |
| </ul></div> |
| <div class="section"> |
| <h4><a name="org.apache.metron.stellar.common.shell.cli"></a><tt>org.apache.metron.stellar.common.shell.cli</tt></h4> |
| <p>This package contains classes that are specific to the CLI-driven REPL.</p> |
| <ul> |
| |
| <li><tt>StellarShell</tt> This is the main class that drives the CLI REPL. All functionality not related to the command line interface is performed by the shared logic in <tt>org.apache.metron.stellar.common.shell</tt>.</li> |
| </ul></div> |
| <div class="section"> |
| <h4><a name="org.apache.metron.stellar.zeppelin"></a><tt>org.apache.metron.stellar.zeppelin</tt></h4> |
| <p>This package is contained within the <tt>stellar-zeppelin</tt> project and performs all logic for interfacing with Zeppelin. Again, all functionality not related to Zeppelin is performed by the shared logic in <tt>org.apache.metron.stellar.common.shell</tt>.</p></div></div></div> |
| <div class="section"> |
| <h2><a name="Stellar_Configuration"></a>Stellar Configuration</h2> |
| <p>Stellar can be configured in a variety of ways from the <a href="../../metron-platform/metron-common/index.html#Global_Configuration">Global Configuration</a>. In particular, there are three main configuration parameters around configuring Stellar:</p> |
| <ul> |
| |
| <li><tt>stellar.function.paths</tt></li> |
| <li><tt>stellar.function.resolver.includes</tt></li> |
| <li><tt>stellar.function.resolver.excludes</tt></li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="stellar.function.paths"></a><tt>stellar.function.paths</tt></h2> |
| <p>If specified, Stellar will use a custom classloader which will wrap the context classloader and allow for the resolution of classes stored in jars not shipped with Metron and stored in a variety of mediums:</p> |
| <ul> |
| |
| <li>On HDFS</li> |
| <li>In tar.gz files</li> |
| <li>At http/s locations</li> |
| <li>At ftp locations</li> |
| </ul> |
| <p>This path is a comma separated list of</p> |
| <ul> |
| |
| <li>URIs</li> |
| <li>URIs with a regex pattern ending it for matching within a directory</li> |
| </ul> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| ... |
| "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.2.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar" |
| } |
| </pre></div></div> |
| |
| <p>Please be aware that this classloader does not reload functions dynamically and the classpath specified here in the global config is read on topology start. A change in classpath, to be picked up, would necessitate a topology restart at the moment</p></div> |
| <div class="section"> |
| <h2><a name="stellar.function.resolver..7Bincludes.2Cexcludes.7D"></a><tt>stellar.function.resolver.{includes,excludes}</tt></h2> |
| <p>If specified, this defines one or more regular expressions applied to the classes implementing the Stellar function that specify what should be included when searching for Stellar functions.</p> |
| <ul> |
| |
| <li><tt>stellar.function.resolver.includes</tt> defines the list of classes to include.</li> |
| <li><tt>stellar.function.resolver.excludes</tt> defines the list of classes to exclude.</li> |
| </ul> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| ... |
| "stellar.function.resolver.includes" : "org.apache.metron.*,com.myorg.stellar.*" |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h2><a name="Stellar_REST_Client"></a>Stellar REST Client</h2> |
| <p>Stellar provides a REST Client with the <tt>REST_GET</tt> function. This function depends on the Apache HttComponents library for executing Http requests. The syntax is:</p> |
| |
| <div> |
| <div> |
| <pre class="source">REST_GET( uri , optional config ) |
| </pre></div></div> |
| |
| <div class="section"> |
| <h3><a name="Configuration"></a>Configuration</h3> |
| <p>The second argument is an optional Map of settings. The following settings are available:</p> |
| <ul> |
| |
| <li>basic.auth.user - User name for basic authentication.</li> |
| <li>basic.auth.password.path - Path to the basic authentication password file stored in HDFS.</li> |
| <li>proxy.host - Proxy host.</li> |
| <li>proxy.port - Proxy port.</li> |
| <li>proxy.basic.auth.user - User name for proxy basic authentication.</li> |
| <li>proxy.basic.auth.password.path - Path to the proxy basic authentication password file stored in HDFS.</li> |
| <li>timeout - Stellar enforced hard timeout for the total request time. Defaults to 1000 ms. HttpClient timeouts alone are insufficient to guarantee the hard timeout.</li> |
| <li>connect.timeout - Connect timeout exposed by the HttpClient object.</li> |
| <li>connection.request.timeout - Connection request timeout exposed by the HttpClient object.</li> |
| <li>socket.timeout - Socket timeout exposed by the HttpClient object.</li> |
| <li>response.codes.allowed - A list of response codes that are allowed. All others will be treated as errors. Defaults to <tt>200</tt>.</li> |
| <li>empty.content.override - The default value that will be returned on a successful request with empty content. Defaults to null.</li> |
| <li>error.value.override - The default value that will be returned on an error. Defaults to null.</li> |
| <li>pooling.max.total - The maximum number of connections in the connection pool.</li> |
| <li>pooling.default.max.per.route - The default maximum number of connections per route in the connection pool.</li> |
| </ul> |
| <p>This Map of settings can also be stored in the global config <tt>stellar.rest.settings</tt> property. For example, to configure basic authentication settings you would add this property to the global config:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "stellar.rest.settings": { |
| "basic.auth.user": "user", |
| "basic.auth.password.path": "/password/path" |
| } |
| } |
| </pre></div></div> |
| |
| <p>Any settings passed into the expression will take precedence over the global config settings. The global config settings will take precedence over the defaults.</p> |
| <p>For security purposes, passwords are read from a file in HDFS. Passwords are read as is including any new lines or spaces. Be careful not to include these in the file unless they are specifically part of the password.</p></div> |
| <div class="section"> |
| <h3><a name="Security"></a>Security</h3> |
| <p>At this time, only basic authentication is supported.</p></div> |
| <div class="section"> |
| <h3><a name="Examples"></a>Examples</h3> |
| <p>Perform a simple GET request with no authentication:</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> REST_GET('http://httpbin.org/get') |
| {args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get} |
| </pre></div></div> |
| |
| <p>Perform a GET request using basic authentication:</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> config := {'basic.auth.user': 'user', 'basic.auth.password.path': '/password/path'} |
| {basic.auth.user=user, basic.auth.password.path=/password/path} |
| [Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd', config) |
| {authenticated=true, user=user} |
| </pre></div></div> |
| |
| <p>Perform a GET request using a proxy:</p> |
| |
| <div> |
| <div> |
| <pre class="source">[Stellar]>>> config := {'proxy.host': 'node1', 'proxy.port': 3128, 'proxy.basic.auth.user': 'user', 'proxy.basic.auth.password.path': '/proxy/password/path'} |
| {proxy.basic.auth.password.path=/proxy/password/path, proxy.port=3128, proxy.host=node1, proxy.basic.auth.user=user} |
| [Stellar]>>> REST_GET('http://httpbin.org/get', config) |
| {args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Latency"></a>Latency</h3> |
| <p>Performing a REST request will introduce latency in a streaming pipeline. Therefore this function should only be used for low volume telemetries that are unlikely to be affected by higher latency operations. The <tt>timeout</tt> setting can be used to guarantee that requests complete within the configured time.</p></div> |
| <div class="section"> |
| <h3><a name="Response_Handling"></a>Response Handling</h3> |
| <p>In cases of Http errors, timeouts, etc this function will log the error and return null. Only a status code of <tt>200</tt> is considered successful by default but this can be changed with the <tt>response.codes.allowed</tt> setting. Values returned on errors or emtpy content can be changed from the default value of null using the <tt>error.value.override</tt> and <tt>empty.content.override</tt> respectively.</p></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
