| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – </title> |
| <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../../../css/site.css" /> |
| <link rel="stylesheet" href="../../../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active "></li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> |
| <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> |
| <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> |
| <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> |
| <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> |
| <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> |
| <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-down"></span>Ambari</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../../metron-deployment/packaging/ambari/elasticsearch-mpack/index.html" title="Elasticsearch-mpack"><span class="none"></span>Elasticsearch-mpack</a></li> |
| <li class="active"><a href="#"><span class="none"></span>Mpack</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> |
| <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> |
| <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> |
| <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> |
| <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>This provides a Management Pack (MPack) extension for <a class="externalLink" href="https://ambari.apache.org/">Apache Ambari</a> that simplifies the provisioning, management and monitoring of Metron on clusters of any size.</p> |
| <p>This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos.</p> |
| <div class="section"> |
| <div class="section"> |
| <h3><a name="Prerequisites"></a>Prerequisites</h3> |
| <ul> |
| |
| <li> |
| |
| <p>Ambari 2.4.2+</p> |
| </li> |
| <li> |
| |
| <p>Installable Metron packages (either RPMs or DEBs) located in a repository on each host at <tt>/localrepo</tt>.</p> |
| </li> |
| <li> |
| |
| <p>A <a class="externalLink" href="https://nodejs.org/en/download/package-manager/">Node.js</a> repository installed on the host running the Management and Alarm UI.</p> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Quick_Start"></a>Quick Start</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Build the Metron MPack. Execute the following command from the project’s root directory.</p> |
| |
| <div> |
| <div> |
| <pre class="source">mvn clean package -Pmpack -DskipTests |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>This results in the Mpack being produced at the following location.</p> |
| |
| <div> |
| <div> |
| <pre class="source">metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Copy the tarball to the host where Ambari Server is installed.</p> |
| </li> |
| <li> |
| |
| <p>Ensure that Ambari Server is stopped.</p> |
| </li> |
| <li> |
| |
| <p>Install the MPack.</p> |
| |
| <div> |
| <div> |
| <pre class="source">ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at <tt>/localrepo</tt>.</p> |
| <p>On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands.</p> |
| |
| <div> |
| <div> |
| <pre class="source">yum install createrepo |
| mkdir /localrepo |
| cd /localrepo |
| createrepo |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Metron will now be available as an installable service within Ambari.</p> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Installation_Notes"></a>Installation Notes</h3> |
| <p>The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari’s user interface using “Add Services” or during an initial cluster install.</p> |
| <div class="section"> |
| <h4><a name="Kerberization"></a>Kerberization</h4> |
| <p>The MPack allows Metron to be automatically kerberized in two different ways.</p> |
| <ul> |
| |
| <li>Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari.</li> |
| <li>Alternatively, Metron can be installed on top of an already kerberized cluster.</li> |
| </ul> |
| <p>Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at <a href="../../Kerberos-manual-setup.html">Kerberos-manual-setup.md</a>.</p> |
| <div class="section"> |
| <h5><a name="Metron_Client"></a>Metron Client</h5> |
| <p>A “Metron Client” must be installed on each supervisor node in a kerberized cluster. This client ensures that the Metron keytab and <tt>client_jaas.conf</tt> get distributed to each node in order to allow reading and writing from Kafka.</p> |
| <ul> |
| |
| <li>When Metron is already installed on the cluster, installation of the “Metron Client” should be done before Kerberizing.</li> |
| <li>When adding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.</li> |
| </ul></div> |
| <div class="section"> |
| <h5><a name="Restarts"></a>Restarts</h5> |
| <p>Storm (and the Metron topologies) must be restarted after Metron is installed on an already Kerberized cluster. The restart triggers several Storm configurations to get updated and Metron will be unable to write to Kafka without a restart.</p> |
| <p>Kerberizing a cluster with a pre-existing Metron, automatically restarts all services during Kerberization. No additional manual restart is needed in this case.</p></div></div> |
| <div class="section"> |
| <h4><a name="Kibana_Dashboards"></a>Kibana Dashboards</h4> |
| <p>The dashboards installed by the Kibana custom action are managed by two JSON files:</p> |
| <ul> |
| |
| <li>metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/kibana.template</li> |
| <li>metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboard-bulkload.json</li> |
| </ul> |
| <p>The first file, <tt>kibana.template</tt>, is an Elasticsearch template that specifies the proper mapping types for the Kibana index. This configuration is necessary due to a bug in the default dynamic mappings provided by Elasticsearch for long types versus integer that are incompatible with Kibana [1]. The second file, <tt>dashboard-bulkload.json</tt>, contains all of the dashboard metadata necessary to create the Metron dashboard. It is an Elasticsearch bulk-insert formatted file [2] that contains a series of documents necessary for setting up the dashboard in Elasticsearch. The main features installed are index patterns, searches, and a variety of visualizations that are used in the Metron dashboard.</p> |
| <p>Deploying the existing dashboard is easy. Once the MPack is installed, run the Metron service’s action “Load Template” to install dashboards. This will no longer overwrite the .kibana in Elasticsearch. The bulk load is configured to fail inserts for existing documents. If you want to <i>completely</i> reload the dashboard, you would need to delete the .kibana index and reload again from Ambari.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li><a class="externalLink" href="https://github.com/elastic/kibana/issues/9888#issuecomment-298096954">https://github.com/elastic/kibana/issues/9888#issuecomment-298096954</a></li> |
| <li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html</a></li> |
| </ol> |
| <div class="section"> |
| <h5><a name="Modifying_Pre-Installed_Dashboards"></a>Modifying Pre-Installed Dashboards</h5> |
| <p>You can modify dashboards in Kibana and bring those changes into the core MPack distribution by performing the following steps:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Export the .kibana index from ES</li> |
| <li>Convert the data into the ES bulk load format</li> |
| <li>Replace the dashboard-bulkload.json file in the Metron MPack.</li> |
| </ol> |
| <p>You can export the .kibana index using a tool like <a class="externalLink" href="https://github.com/taskrabbit/elasticsearch-dump">https://github.com/taskrabbit/elasticsearch-dump</a>. The important feature is to have one document per line. Here’s an exmaple export using elasticsearch-dump</p> |
| |
| <div> |
| <div> |
| <pre class="source">elasticdump \ |
| --input=http://node1:9200/.kibana \ |
| --output=~/dashboard-data.json \ |
| --type=data |
| </pre></div></div> |
| |
| <p>Once you’ve exported the data, you can now format it as a bulk load ES file by running the import/export tool located in metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java. This tool can be run from full-dev as follows</p> |
| |
| <div> |
| <div> |
| <pre class="source">java -cp $METRON_HOME/lib/metron-elasticsearch-0.4.2-uber.jar org.apache.metron.elasticsearch.bulk.ElasticsearchImportExport \ |
| ~/dashboard-data.json \ |
| ~/dashboard-bulkload.json |
| </pre></div></div> |
| |
| <p>Locate the “create” command for setting the default index by searching for “5.6.14”. Change “create” to “index” so that it modifies the existing value. It should look similar to line 1 below.</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ "index" : { "_id": "5.6.14", "_type": "config" } } |
| {"defaultIndex":"AV-S2e81hKs1cXXnFMqN"} |
| </pre></div></div> |
| |
| <p>Now copy this file to the Kibana MPack, overwriting the existing bulk load file. That should be everything needed to backup the dashboard.</p> |
| <p><b>Note</b>: the dashboard Python Pickle binary file is deprecated and no longer used for backing up and restoring Kibana dashboards. The tooling is still provided as of this version but is expected to be removed in the future. A section describing the deprecated backup process remains below.</p></div> |
| <div class="section"> |
| <h5><a name="Deprecated_Dashboard_Install.2FBackup_Instructions"></a>Deprecated Dashboard Install/Backup Instructions</h5> |
| <p>The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.</p> |
| <p>To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.</p> |
| <p><b>Script Options</b></p> |
| |
| <div> |
| <div> |
| <pre class="source">[elasticsearch_host] ES host |
| [elasticsearch_port] ES port number |
| [input_output_filename] Filename used for reading or writing out pickle file |
| [-s] Flag to indicate that the .kibana index should be saved locally. Not including this flag will overwrite the .kibana |
| index completely with the contents of 'input_output_filename'. Careful with this. |
| </pre></div></div> |
| |
| <p><b>Saving a Backup</b></p> |
| |
| <div> |
| <div> |
| <pre class="source">python packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboardindex.py \ |
| $ES_HOST 9200 \ |
| ~/dashboard.p -s |
| </pre></div></div> |
| |
| <p><b>Restoring From a Backup</b></p> |
| |
| <div> |
| <div> |
| <pre class="source">python packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboardindex.py \ |
| $ES_HOST 9200 \ |
| ~/dashboard.p |
| </pre></div></div> |
| |
| <p><b>Note</b>: This method of writing the Kibana dashboard to Elasticsearch will overwrite the entire .kibana index. Be sure to first backup the index first using either the new JSON method described above, or writing out the dashboard.p pickle file using the old method (passing -s option to dashboardindex.py) described here.</p></div></div> |
| <div class="section"> |
| <h4><a name="Zeppelin_Import"></a>Zeppelin Import</h4> |
| <p>A custom action is available in Ambari to import Zeppelin dashboards. See the <a href="../../../../metron-platform/metron-indexing/index.html">metron-indexing documentation</a> for more information.</p></div> |
| <div class="section"> |
| <h4><a name="Offline_Installation"></a>Offline Installation</h4> |
| <p>Retrieval of the GeoIP and ASN databases (both from MaxMind) is the only point during installation that reaches out to the internet. For an offline installation, the URL for the databases can be manually set to a local path on the file system such as <tt>file:///home/root/geoip/GeoLite2-City.tar.gz</tt>.</p> |
| <p>The properties for configuration are <tt>geoip_url</tt> and <tt>asn_url</tt> in the <tt>Enrichment</tt> section.</p> |
| <p>The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in <tt>/localrepo</tt>.</p></div> |
| <div class="section"> |
| <h4><a name="Limitations"></a>Limitations</h4> |
| <p>There are a few limitations that should be addressed to improve the Metron MPack installation.</p> |
| <ul> |
| |
| <li> |
| |
| <p>There is no external hosting for Metron packages (either RPMs or DEBs). These have to be built locally and installed on each host in a repository located at <tt>/localrepo</tt>.</p> |
| </li> |
| <li> |
| |
| <p>Several configuration parameters used when installing Metron could retrieved from Ambari rather than requiring user input.</p> |
| </li> |
| <li> |
| |
| <p>The MPack does not support upgrades.</p> |
| </li> |
| </ul></div></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
