Google Git
Sign in
apache / metron / 410993daf60dfe898a973fa8a9906ddbffd83208 / . / current-book / metron-platform / metron-elasticsearch / index.html
blob: 7a6cc0a663b0bbb3cfe8017cb0a649c0fb04dc39 [file] [log] [blame]
<!DOCTYPE html>
<!--
| Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-elasticsearch/index.md at 2019-05-14
| Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="Date-Revision-yyyymmdd" content="20190514" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron &#x2013; Elasticsearch in Metron</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
<script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script>
<script type="text/javascript">
$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );
</script>
</head>
<body class="topBarDisabled">
<div class="container-fluid">
<div id="banner">
<div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div>
<div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
<li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
<li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li>
<li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li>
<li class="active ">Elasticsearch in Metron</li>
<li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li>
<li id="projectVersion" class="pull-right">Version: 0.7.1</li>
</ul>
</div>
<div class="row-fluid">
<div id="leftColumn" class="span2">
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
<li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a>
<ul class="nav nav-list">
<li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li>
<li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li>
<li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li>
<li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li>
<li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li>
<li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li>
<li><a href="../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li>
<li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a>
<ul class="nav nav-list">
<li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li>
<li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li>
<li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li>
<li class="active"><a href="#"><span class="none"></span>Elasticsearch</a></li>
<li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li>
<li><a href="../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li>
<li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li>
<li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li>
<li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li>
<li><a href="../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-right"></span>Parsing</a></li>
<li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
<li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li>
<li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li>
</ul>
</li>
<li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li>
<li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li>
<li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li>
<li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
<li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li>
</ul>
</li>
</ul>
<hr />
<div id="poweredBy">
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<div class="clear"></div>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a>
</div>
</div>
</div>
<div id="bodyColumn" class="span10" >
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<h1>Elasticsearch in Metron</h1>
<p><a name="Elasticsearch_in_Metron"></a></p>
<div class="section">
<h2><a name="Table_of_Contents"></a>Table of Contents</h2>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#Properties">Properties</a></li>
<li><a href="#Upgrading_from_2.3.3_to_5.6">Upgrading from 2.3.3 to 5.6</a></li>
<li><a href="#Type_Mappings">Type Mappings</a></li>
<li><a href="#Using_Metron_with_Elasticsearch_5.6">Using Metron with Elasticsearch 5.6</a></li>
<li><a href="#Installing_Elasticsearch_Templates">Installing Elasticsearch Templates</a></li>
</ul></div>
<div class="section">
<h2><a name="Introduction"></a>Introduction</h2>
<p>Elasticsearch can be used as the real-time portion of the datastore resulting from <a href="../metron-indexing/index.html">metron-indexing</a>.</p></div>
<div class="section">
<h2><a name="Properties"></a>Properties</h2>
<div class="section">
<h3><a name="es.clustername"></a><tt>es.clustername</tt></h3>
<p>The name of the elasticsearch Cluster. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name">here</a></p></div>
<div class="section">
<h3><a name="es.ip"></a><tt>es.ip</tt></h3>
<p>Specifies the nodes in the elasticsearch cluster to use for writing. The format is one of the following:</p>
<ul>
<li>A hostname or IP address with a port (e.g. <tt>hostname1:1234</tt>), in which case <tt>es.port</tt> is ignored.</li>
<li>A hostname or IP address without a port (e.g. <tt>hostname1</tt>), in which case <tt>es.port</tt> is used.</li>
<li>A string containing a CSV of hostnames without ports (e.g. <tt>hostname1,hostname2,hostname3</tt>) without spaces between. <tt>es.port</tt> is assumed to be the port for each host.</li>
<li>A string containing a CSV of hostnames with ports (e.g. <tt>hostname1:1234,hostname2:1234,hostname3:1234</tt>) without spaces between. <tt>es.port</tt> is ignored.</li>
<li>A list of hostnames with ports (e.g. <tt>[ &quot;hostname1:1234&quot;, &quot;hostname2:1234&quot;]</tt>). Note, <tt>es.port</tt> is NOT used in this construction.</li>
</ul></div>
<div class="section">
<h3><a name="es.port"></a><tt>es.port</tt></h3>
<p>The port for the elasticsearch hosts. This will be used in accordance with the discussion of <tt>es.ip</tt>.</p></div>
<div class="section">
<h3><a name="es.date.format"></a><tt>es.date.format</tt></h3>
<p>The date format to use when constructing the indices. For every message, the date format will be applied to the current time and that will become the last part of the index name where the message is written to.</p>
<p>For instance, an <tt>es.date.format</tt> of <tt>yyyy.MM.dd.HH</tt> would have the consequence that the indices would roll hourly, whereas an <tt>es.date.format</tt> of <tt>yyyy.MM.dd</tt> would have the consequence that the indices would roll daily.</p></div>
<div class="section">
<h3><a name="es.client.settings"></a><tt>es.client.settings</tt></h3>
<p>This field in global config allows you to specify Elasticsearch REST client options. These are used in conjunction with the previously mentioned Elasticsearch properties when setting up client connections to an Elasticsearch cluster. The available properties should be supplied as an object map. Current available options are as follows:</p>
<table border="0" class="table table-striped">
<thead>
<tr class="a">
<th> Property Name </th>
<th> Type </th>
<th> Required? </th>
<th> Default Value </th>
<th> Description </th></tr>
</thead><tbody>
<tr class="b">
<td> connection.timeout.millis </td>
<td> Integer </td>
<td> No </td>
<td> 1000 </td>
<td> Sets connection timeout. </td></tr>
<tr class="a">
<td> socket.timeout.millis </td>
<td> Integer </td>
<td> No </td>
<td> 30000 </td>
<td> Sets socket timeout. </td></tr>
<tr class="b">
<td> max.retry.timeout.millis </td>
<td> Integer </td>
<td> No </td>
<td> 30000 </td>
<td> Sets the maximum timeout (in milliseconds) to honour in case of multiple retries of the same request. </td></tr>
<tr class="a">
<td> num.client.connection.threads </td>
<td> Integer </td>
<td> No </td>
<td> 1 </td>
<td> Number of worker threads used by the connection manager. Defaults to Runtime.getRuntime().availableProcessors(). </td></tr>
<tr class="b">
<td> xpack.username </td>
<td> String </td>
<td> No </td>
<td> null </td>
<td> X-Pack username. </td></tr>
<tr class="a">
<td> xpack.password.file </td>
<td> String </td>
<td> No </td>
<td> null </td>
<td> 1-line HDFS file where the X-Pack password is set. </td></tr>
<tr class="b">
<td> ssl.enabled </td>
<td> Boolean </td>
<td> No </td>
<td> false </td>
<td> Turn on SSL connections. </td></tr>
<tr class="a">
<td> keystore.type </td>
<td> String </td>
<td> No </td>
<td> &#x201c;jks&#x201d; </td>
<td> Allows you to specify a keytstore type. See <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyStore">https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyStore</a> for more details. </td></tr>
<tr class="b">
<td> keystore.path </td>
<td> String </td>
<td> No </td>
<td> null </td>
<td> Path to the Trust Store that holds your Elasticsearch certificate authorities and certificate. </td></tr>
<tr class="a">
<td> keystore.password.file </td>
<td> String </td>
<td> No </td>
<td> null </td>
<td> 1-line HDFS file where the keystore password is set. </td></tr>
</tbody>
</table>
<p><b>Note:</b> The migration from Elasticsearch&#x2019;s TransportClient to the Java REST client has resulted in some existing properties to change. Below is a mapping of the old properties to the new ones:</p>
<table border="0" class="table table-striped">
<thead>
<tr class="a">
<th> Old Property Name </th>
<th> New Property Name </th></tr>
</thead><tbody>
<tr class="b">
<td> client.transport.ping_timeout </td>
<td> n/a </td></tr>
<tr class="a">
<td> n/a </td>
<td> connection.timeout.millis </td></tr>
<tr class="b">
<td> n/a </td>
<td> socket.timeout.millis </td></tr>
<tr class="a">
<td> n/a </td>
<td> max.retry.timeout.millis </td></tr>
<tr class="b">
<td> n/a </td>
<td> num.client.connection.threads </td></tr>
<tr class="a">
<td> es.client.class </td>
<td> n/a </td></tr>
<tr class="b">
<td> es.xpack.username </td>
<td> xpack.username </td></tr>
<tr class="a">
<td> es.xpack.password.file </td>
<td> xpack.password.file </td></tr>
<tr class="b">
<td> xpack.security.transport.ssl.enabled </td>
<td> ssl.enabled </td></tr>
<tr class="a">
<td> xpack.ssl.key </td>
<td> n/a </td></tr>
<tr class="b">
<td> xpack.ssl.certificate </td>
<td> n/a </td></tr>
<tr class="a">
<td> xpack.ssl.certificate_authorities </td>
<td> n/a </td></tr>
<tr class="b">
<td> n/a </td>
<td> keystore.type </td></tr>
<tr class="a">
<td> keystore.path </td>
<td> keystore.path </td></tr>
<tr class="b">
<td> n/a </td>
<td> keystore.password.file </td></tr>
</tbody>
</table>
<p><b>Notes:</b></p>
<ul>
<li>The transport client implementation provides for a &#x2018;xpack.security.user&#x2019; property, however we never used this property directly. Rather, in order to secure the password we used custom properties for user/pass. These properties have been carried over as <tt>xpack.username</tt> and <tt>xpack.password.file</tt>.</li>
<li>See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/_common_configuration.html">https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/_common_configuration.html</a> for more specifics on the new client properties.</li>
<li>Other notes on JSSE - <a class="externalLink" href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html">https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html</a></li>
</ul></div></div>
<div class="section">
<h2><a name="Upgrading_from_2.3.3_to_5.6"></a>Upgrading from 2.3.3 to 5.6</h2>
<p>Users should be prepared to re-index when migrating from Elasticsearch 2.3.3 to 5.6. There are a number of template changes, most notably around string type handling, that may cause issues when upgrading.</p>
<p><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html</a></p>
<p>Be aware that if you add a new string value and want to be able to filter and search on this value from the Alerts UI, you <b>must</b> add a mapping for that type to the appropriate Elasticsearch template. Below is more detail on how to choose the appropriate mapping type for your string value.</p></div>
<div class="section">
<h2><a name="Type_Mappings"></a>Type Mappings</h2>
<p>Type mappings have changed quite a bit from ES 2.x -&gt; 5.x. Here is a brief rundown of the biggest changes. More detailed references from Elasticsearch are provided in the <a href="#Type_Mapping_References">Type Mapping References</a> section below.</p>
<ul>
<li>string fields replaced by text/keyword type</li>
<li>strings have new default mappings as follows
<div>
<div>
<pre class="source">{
&quot;type&quot;: &quot;text&quot;,
&quot;fields&quot;: {
&quot;keyword&quot;: {
&quot;type&quot;: &quot;keyword&quot;,
&quot;ignore_above&quot;: 256
}
}
}
</pre></div></div>
</li>
<li>
<p>There is no longer a <tt>_timestamp</tt> field that you can set &#x201c;enabled&#x201d; on. This field now causes an exception on templates. Replace with an application-created timestamp of &#x201c;date&#x201d; type.</p>
</li>
</ul>
<p>The semantics for string types have changed. In 2.x, you have the concept of index settings as either &#x201c;analyzed&#x201d; or &#x201c;not_analyzed&#x201d; which basically means &#x201c;full text&#x201d; and &#x201c;keyword&#x201d;, respectively. Analyzed text basically means the indexer will split the text using a text analyzer thus allowing you to search on substrings within the original text. &#x201c;New York&#x201d; is split and indexed as two buckets, &#x201c;New&#x201d; and &#x201c;York&#x201d;, so you can search or query for aggregate counts for those terms independently and will match against the individual terms &#x201c;New&#x201d; or &#x201c;York.&#x201d; &#x201c;Keyword&#x201d; means that the original text will not be split/analyzed during indexing and instead treated as a whole unit, i.e. &#x201c;New&#x201d; or &#x201c;York&#x201d; will not match in searches against the document containing &#x201c;New York&#x201d;, but searching on &#x201c;New York&#x201d; as the full city name will. In 5.x language instead of using the &#x201c;index&#x201d; setting, you now set the &#x201c;type&#x201d; to either &#x201c;text&#x201d; for full text, or &#x201c;keyword&#x201d; for keywords.</p>
<p>Below is a table depicting the changes to how String types are now handled.</p>
<table border="0" class="table table-striped">
<tr class="a">
<th>sort, aggregate, or access values</th>
<th>ES 2.x</th>
<th>ES 5.x</th>
<th>Example</th>
</tr>
<tr class="b">
<td>no</td>
<td>
<div>
<pre><tt>&quot;my_property&quot; : {
&quot;type&quot;: &quot;string&quot;,
&quot;index&quot;: &quot;analyzed&quot;
}
</tt></pre></div>
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot; : {
&quot;type&quot;: &quot;text&quot;
}
</tt></pre></div>
Additional defaults: &quot;index&quot;: &quot;true&quot;, &quot;fielddata&quot;: &quot;false&quot;
</td>
<td>
&quot;New York&quot; handled via in-mem search as &quot;New&quot; and &quot;York&quot; buckets. <b>No</b> aggregation or sort.
</td>
</tr>
<tr class="a">
<td>
yes
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot;: {
&quot;type&quot;: &quot;string&quot;,
&quot;index&quot;: &quot;analyzed&quot;
}
</tt></pre></div>
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot;: {
&quot;type&quot;: &quot;text&quot;,
&quot;fielddata&quot;: &quot;true&quot;
}
</tt></pre></div>
</td>
<td>
&quot;New York&quot; handled via in-mem search as &quot;New&quot; and &quot;York&quot; buckets. <b>Can</b> aggregate and sort.
</td>
</tr>
<tr class="b">
<td>
yes
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot;: {
&quot;type&quot;: &quot;string&quot;,
&quot;index&quot;: &quot;not_analyzed&quot;
}
</tt></pre></div>
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot; : {
&quot;type&quot;: &quot;keyword&quot;
}
</tt></pre></div>
</td>
<td>
&quot;New York&quot; searchable as single value. <b>Can</b> aggregate and sort. A search for &quot;New&quot; or &quot;York&quot; will not match against the whole value.
</td>
</tr>
<tr class="a">
<td>
yes
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot;: {
&quot;type&quot;: &quot;string&quot;,
&quot;index&quot;: &quot;analyzed&quot;
}
</tt></pre></div>
</td>
<td>
<div>
<pre><tt>&quot;my_property&quot;: {
&quot;type&quot;: &quot;text&quot;,
&quot;fields&quot;: {
&quot;keyword&quot;: {
&quot;type&quot;: &quot;keyword&quot;,
&quot;ignore_above&quot;: 256
}
}
}
</tt></pre></div>
</td>
<td>
&quot;New York&quot; searchable as single value or as text document, can aggregate and sort on the sub term &quot;keyword.&quot;
</td>
</tr>
</table>
<p>If you want to set default string behavior for all strings for a given index and type, you can do so with a mapping similar to the following (replace ${your_type_here} accordingly):</p>
<div>
<div>
<pre class="source"># curl -XPUT 'http://${ES_HOST}:${ES_PORT}/_template/default_string_template' -d '
{
&quot;template&quot;: &quot;*&quot;,
&quot;mappings&quot; : {
&quot;${your_type_here}&quot;: {
&quot;dynamic_templates&quot;: [
{
&quot;strings&quot;: {
&quot;match_mapping_type&quot;: &quot;string&quot;,
&quot;mapping&quot;: {
&quot;type&quot;: &quot;text&quot;
}
}
}
]
}
}
}
'
</pre></div></div>
<p>By specifying the &#x201c;template&#x201d; property with value &#x201c;*&#x201d; the template will apply to all indexes that have documents indexed of the specified type (${your_type_here}). This results in the following template.</p>
<div>
<div>
<pre class="source"># curl -XGET 'http://${ES_HOST}:${ES_PORT}/_template/default_string_template?pretty'
{
&quot;default_string_template&quot; : {
&quot;order&quot; : 0,
&quot;template&quot; : &quot;*&quot;,
&quot;settings&quot; : { },
&quot;mappings&quot; : {
&quot;${your_type_here}&quot; : {
&quot;dynamic_templates&quot; : [
{
&quot;strings&quot; : {
&quot;match_mapping_type&quot; : &quot;string&quot;,
&quot;mapping&quot; : {
&quot;type&quot; : &quot;text&quot;
}
}
}
]
}
},
&quot;aliases&quot; : { }
}
}
</pre></div></div>
<p>Notes on other settings for types in ES</p>
<ul>
<li>doc_values
<ul>
<li>on-disk data structure</li>
<li>provides access for sorting, aggregation, and field values</li>
<li>stores same values as _source, but in column-oriented fashion better for sorting and aggregating</li>
<li>not supported on text fields</li>
<li>enabled by default</li>
</ul>
</li>
<li>fielddata
<ul>
<li>in-memory data structure</li>
<li>provides access for sorting, aggregation, and field values</li>
<li>primarily for text fields</li>
<li>disabled by default because the heap space required can be large</li>
</ul>
</li>
</ul>
<div class="section">
<div class="section">
<div class="section">
<h5><a name="Type_Mapping_References"></a>Type Mapping References</h5>
<ul>
<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html</a></li>
<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_mapping_changes.html</a></li>
<li><a class="externalLink" href="https://www.elastic.co/blog/strings-are-dead-long-live-strings">https://www.elastic.co/blog/strings-are-dead-long-live-strings</a></li>
</ul></div></div></div>
<div class="section">
<h3><a name="Metron_Properties"></a>Metron Properties</h3>
<p>Metron depends on some internal fields being defined in sensor templates. A field is defined in Elasticsearch by adding an entry to the <tt>properties</tt> section of the template:</p>
<div>
<div>
<pre class="source">&quot;properties&quot;: {
&quot;metron_field&quot;: {
&quot;type&quot;: &quot;keyword&quot;
}
}
</pre></div></div>
<p>The following is a list of properties that need to be defined along with their type:</p>
<ul>
<li>source:type - keyword</li>
<li>alert_status - keyword</li>
<li>metron_alert - nested</li>
</ul></div></div>
<div class="section">
<h2><a name="Using_Metron_with_Elasticsearch_5.6"></a>Using Metron with Elasticsearch 5.6</h2>
<p>Although infrequent, sometimes an internal field is added in Metron and existing templates must be updated. The following steps outlines how to do this, using <tt>metron_alert</tt> as an example.</p>
<p>With the addition of the meta alert feature, there is a requirement that all sensors templates have a nested <tt>metron_alert</tt> field defined. This field is a dummy field. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields">Ignoring Unmapped Fields</a> for more information</p>
<p>Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service&#x2019;s logs.</p>
<p>Exception seen:</p>
<div>
<div>
<pre class="source">QueryParsingException[[nested] failed to find nested object under path [metron_alert]];
</pre></div></div>
<p>There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.</p>
<p>Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it&#x2019;s not necessary to do each index individually. The example here appends <tt>index*</tt> to get all indexes for the provided sensor.</p>
<div>
<div>
<pre class="source">export ELASTICSEARCH=&quot;node1&quot;
export SENSOR=&quot;bro&quot;
curl -XGET &quot;http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true&quot; -o &quot;${SENSOR}.template&quot;
sed -i '' '2d;$d' ./${SENSOR}.template
sed -i '' '/&quot;properties&quot; : {/ a\
&quot;metron_alert&quot;: { &quot;type&quot;: &quot;nested&quot;},' ${SENSOR}.template
</pre></div></div>
<p>To manually verify this, you can optionally pretty print it again with:</p>
<div>
<div>
<pre class="source">python -m json.tool bro.template
</pre></div></div>
<p>We&#x2019;ll want to put the template back into Elasticsearch:</p>
<div>
<div>
<pre class="source">curl -XPUT &quot;http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index&quot; -d @${SENSOR}.template
</pre></div></div>
<p>To update existing indexes, update Elasticsearch mappings with the new field for each sensor.</p>
<div>
<div>
<pre class="source">curl -XPUT &quot;http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc&quot; -d '
{
&quot;properties&quot; : {
&quot;metron_alert&quot; : {
&quot;type&quot; : &quot;nested&quot;
}
}
}
'
rm ${SENSOR}.template
</pre></div></div>
</div>
<div class="section">
<h2><a name="Installing_Elasticsearch_Templates"></a>Installing Elasticsearch Templates</h2>
<p>The stock set of Elasticsearch templates for bro, snort, yaf, error index and meta index are installed automatically during the first time install and startup of Metron Indexing service.</p>
<p>It is possible that Elasticsearch service is not available when the Metron Indexing Service startup, in that case the Elasticsearch template will not be installed.</p>
<p>For such a scenario, an Admin can have the template installed in two ways:</p>
<p><i>Method 1</i> - Manually from the Ambari UI by following the flow: Ambari UI -&gt; Services -&gt; Metron -&gt; Service Actions -&gt; Elasticsearch Template Install</p>
<p><i>Method 2</i> - Stop the Metron Indexing service, and start it again from Ambari UI. Note that the Metron Indexing service tracks if it has successfully installed the Elasticsearch templates, and will attempt to do so each time it is Started until successful.</p>
<blockquote>
<p>Note: If you have made any customization to your index templates, then installing Elasticsearch templates afresh will lead to overwriting your existing changes. Please exercise caution.</p>
</blockquote></div>
</div>
</div>
</div>
<hr/>
<footer>
<div class="container-fluid">
<div class="row-fluid">
© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo,
and the Apache Metron project logo are trademarks of The Apache Software Foundation.
</div>
</div>
</footer>
</body>
</html>