| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsing/metron-parsers-common/3rdPartyParser.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Custom Metron Parsers</title> |
| <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../../css/site.css" /> |
| <link rel="stylesheet" href="../../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Custom Metron Parsers</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> |
| <li><a href="../../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> |
| <li><a href="../../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> |
| <li><a href="../../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> |
| <li><a href="../../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> |
| <li><a href="../../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li> |
| <li><a href="../../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> |
| <li><a href="../../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> |
| <li><a href="../../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-down"></span>Parsing</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/index.html" title="Parsers-common"><span class="icon-chevron-down"></span>Parsers-common</a> |
| <ul class="nav nav-list"> |
| <li class="active"><a href="#"><span class="none"></span>3rdPartyParser</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/parser-testing.html" title="parser-testing"><span class="none"></span>parser-testing</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsing-storm/index.html" title="Parsing-storm"><span class="none"></span>Parsing-storm</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> |
| <li><a href="../../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> |
| <li><a href="../../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Custom Metron Parsers</h1> |
| <p><a name="Custom_Metron_Parsers"></a></p> |
| <p>We have many stock parsers for normal operations. Some of these are networking and cybersecurity focused (e.g. the ASA Parser), some of these are general purpose (e.g. the CSVParser), but inevitably users will want to extend the system to process their own data formats. To enable this, this is a walkthrough of how to create and use a custom parser within Metron.</p> |
| <p><a name="Writing_A_Custom_Parser"></a></p> |
| <h1>Writing A Custom Parser</h1> |
| <p>Before we can use a parser, we will need to create a custom parser. The parser is the workhorse of Metron ingest. It provides the mapping between the raw data coming in via the Kafka value and a <tt>JSONObject</tt>, the internal data structure provided.</p> |
| <div class="section"> |
| <h2><a name="Implementation"></a>Implementation</h2> |
| <p>In order to do create a custom parser, we need to do one of the following:</p> |
| <ul> |
| |
| <li>Write a class which conforms to the <tt>org.apache.metron.parsers.interfaces.MessageParser<JSONObject></tt> and <tt>java.util.Serializable</tt> interfaces |
| <ul> |
| |
| <li>Implement <tt>init()</tt>, <tt>validate(JSONObject message)</tt>, and <tt>List<JSONObject> parse(byte[] rawMessage)</tt></li> |
| </ul> |
| </li> |
| <li>Write a class which extends <tt>org.apache.metron.parsers.BasicParser</tt> |
| <ul> |
| |
| <li>Provides convenience implementations to <tt>validate</tt> which ensures <tt>timestamp</tt> and <tt>original_string</tt> fields exist.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="Example"></a>Example</h2> |
| <p>In order to illustrate how this might be done, let’s create a very simple parser that takes a comma separated pair and creates a couple of fields:</p> |
| <ul> |
| |
| <li><tt>original_string</tt> – the raw data</li> |
| <li><tt>timestamp</tt> – the current time</li> |
| <li><tt>first</tt> – the first field of the comma separated pair</li> |
| <li><tt>last</tt> – the last field of the comma separated pair</li> |
| </ul> |
| <p>For this demonstration, let’s create a maven project to compile our project. We’ll call it <tt>extra_parsers</tt>, so in your workspace, let’s set up the maven project:</p> |
| <ul> |
| |
| <li> |
| |
| <p>Create the maven infrastructure for <tt>extra_parsers</tt> via</p> |
| |
| <div> |
| <div> |
| <pre class="source">mkdir -p extra_parsers/src/{main,test}/java |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Create a pom file indicating how we should build our parsers by editing <tt>extra_parsers/pom.xml</tt> with the following content:</p> |
| |
| <div> |
| <div> |
| <pre class="source"><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> |
| <modelVersion>4.0.0</modelVersion> |
| <groupId>com.3rdparty</groupId> |
| <artifactId>extra-parsers</artifactId> |
| <packaging>jar</packaging> |
| <version>1.0-SNAPSHOT</version> |
| <name>extra-parsers</name> |
| <url>http://thirdpartysoftware.org</url> |
| <properties> |
| <!-- The java version to conform to. Metron works all the way to 1.8 --> |
| <java_version>1.8</java_version> |
| <!-- The version of Metron that we'll be targetting. --> |
| <metron_version>0.4.1</metron_version> |
| <!-- To complete the simulation, we'll depend on a common dependency --> |
| <guava_version>19.0</guava_version> |
| <!-- We will shade our dependencies to create a single jar at the end --> |
| <shade_version>2.4.3</shade_version> |
| </properties> |
| <dependencies> |
| <!-- |
| We want to depend on Metron, but ensure that the scope is "provided" |
| as we do not want to include it in our bundle. |
| --> |
| <dependency> |
| <groupId>org.apache.metron</groupId> |
| <artifactId>metron-parsers-common</artifactId> |
| <version>${metron_version}</version> |
| <scope>provided</scope> |
| </dependency> |
| <dependency> |
| <groupId>com.google.guava</groupId> |
| <artifactId>guava</artifactId> |
| <version>${guava_version}</version> |
| </dependency> |
| <dependency> |
| <groupId>junit</groupId> |
| <artifactId>junit</artifactId> |
| <version>3.8.1</version> |
| <scope>test</scope> |
| </dependency> |
| </dependencies> |
| <build> |
| <plugins> |
| <!-- We will set up the shade plugin to create a single jar at the |
| end of the build lifecycle. We will exclude some things and |
| relocate others to simulate a real situation. |
| |
| One thing to note is that it's a good practice to shade and |
| relocate common libraries that may be dependencies in Metron. |
| Your jar will be merged with the parsers jar, so the metron |
| version will be included for all overlapping classes. |
| So, shade and relocate to ensure that YOUR version of the library is used. |
| --> |
| |
| <plugin> |
| <groupId>org.apache.maven.plugins</groupId> |
| <artifactId>maven-shade-plugin</artifactId> |
| <version>${shade_version}</version> |
| <configuration> |
| <createDependencyReducedPom>true</createDependencyReducedPom> |
| <artifactSet> |
| <excludes> |
| <!-- Exclude slf4j for no reason other than to illustrate how to exclude dependencies. |
| The metron team has nothing against slf4j. :-) |
| --> |
| <exclude>*slf4j*</exclude> |
| </excludes> |
| </artifactSet> |
| </configuration> |
| <executions> |
| <execution> |
| <phase>package</phase> |
| <goals> |
| <goal>shade</goal> |
| </goals> |
| <configuration> |
| <shadedArtifactAttached>true</shadedArtifactAttached> |
| <shadedClassifierName>uber</shadedClassifierName> |
| <filters> |
| <filter> |
| <!-- Sometimes these get added and confuse the uber jar out of shade --> |
| <artifact>*:*</artifact> |
| <excludes> |
| <exclude>META-INF/*.SF</exclude> |
| <exclude>META-INF/*.DSA</exclude> |
| <exclude>META-INF/*.RSA</exclude> |
| </excludes> |
| </filter> |
| </filters> |
| <relocations> |
| <!-- Relocate guava as it's used in Metron and I really want 0.19 --> |
| <relocation> |
| <pattern>com.google</pattern> |
| <shadedPattern>com.thirdparty.guava</shadedPattern> |
| </relocation> |
| </relocations> |
| <artifactSet> |
| <excludes> |
| <!-- We can also exclude by artifactId and groupId --> |
| <exclude>storm:storm-core:*</exclude> |
| <exclude>storm:storm-lib:*</exclude> |
| <exclude>org.slf4j.impl*</exclude> |
| <exclude>org.slf4j:slf4j-log4j*</exclude> |
| </excludes> |
| </artifactSet> |
| </configuration> |
| </execution> |
| </executions> |
| </plugin> |
| <!-- |
| We want to make sure we compile using java 1.8. |
| --> |
| <plugin> |
| <groupId>org.apache.maven.plugins</groupId> |
| <artifactId>maven-compiler-plugin</artifactId> |
| <version>3.5.1</version> |
| <configuration> |
| <forceJavacCompilerUse>true</forceJavacCompilerUse> |
| <source>${java_version}</source> |
| <compilerArgument>-Xlint:unchecked</compilerArgument> |
| <target>${java_version}</target> |
| <showWarnings>true</showWarnings> |
| </configuration> |
| </plugin> |
| </plugins> |
| </build> |
| </project> |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Now let’s create our parser <tt>com.thirdparty.SimpleParser</tt> by creating the file <tt>extra-parsers/src/main/java/com/thirdparty/SimpleParser.java</tt> with the following content:</p> |
| |
| <div> |
| <div> |
| <pre class="source">package com.thirdparty; |
| |
| import com.google.common.base.Splitter; |
| import com.google.common.collect.ImmutableList; |
| import com.google.common.collect.Iterables; |
| import org.apache.metron.parsers.BasicParser; |
| import org.json.simple.JSONObject; |
| |
| import java.util.List; |
| import java.util.Map; |
| |
| public class SimpleParser extends BasicParser { |
| @Override |
| public void init() { |
| |
| } |
| |
| @Override |
| public List<JSONObject> parse(byte[] bytes) { |
| String input = new String(bytes); |
| Iterable<String> it = Splitter.on(",").split(input); |
| JSONObject ret = new JSONObject(); |
| ret.put("original_string", input); |
| ret.put("timestamp", System.currentTimeMillis()); |
| ret.put("first", Iterables.getFirst(it, "missing")); |
| ret.put("last", Iterables.getLast(it, "missing")); |
| return ImmutableList.of(ret); |
| } |
| |
| @Override |
| public void configure(Map<String, Object> map) { |
| |
| } |
| } |
| </pre></div></div> |
| </li> |
| <li>Compile the parser via <tt>mvn clean package</tt> in <tt>extra_parsers</tt></li> |
| <li> |
| |
| <p>This will create a jar containing your parser and its dependencies (sans Metron dependencies) in <tt>extra-parsers/target/extra-parsers-1.0-SNAPSHOT-uber.jar</tt></p> |
| </li> |
| </ul> |
| <p><a name="Deploying_Your_Custom_Parser"></a></p> |
| <h1>Deploying Your Custom Parser</h1> |
| <p>In order to deploy your newly built custom parser, you would place the jar file above in the <tt>$METRON_HOME/parser_contrib</tt> directory on the Metron host (i.e. any host you would start parsers from or, alternatively, where the Metron REST is hosted).</p></div> |
| <div class="section"> |
| <h2><a name="Example"></a>Example</h2> |
| <p>Let’s work through deploying the example above.</p> |
| <div class="section"> |
| <h3><a name="Preliminaries"></a>Preliminaries</h3> |
| <p>We assume that the following environment variables are set:</p> |
| <ul> |
| |
| <li><tt>METRON_HOME</tt> - the home directory for metron</li> |
| <li><tt>ZOOKEEPER</tt> - The zookeeper quorum (comma separated with port specified: e.g. <tt>node1:2181</tt> for full-dev)</li> |
| <li><tt>BROKERLIST</tt> - The Kafka broker list (comma separated with port specified: e.g. <tt>node1:6667</tt> for full-dev)</li> |
| <li><tt>ES_HOST</tt> - The elasticsearch master (and port) e.g. <tt>node1:9200</tt> for full-dev.</li> |
| </ul> |
| <p>Also, this does not assume that you are using a kerberized cluster. If you are, then the parser start command will adjust slightly to include the security protocol.</p></div> |
| <div class="section"> |
| <h3><a name="Copy_the_jar_file_up"></a>Copy the jar file up</h3> |
| <p>Copy the jar file located in <tt>extra-parsers/target/extra-parsers-1.0-SNAPSHOT-uber.jar</tt> to <tt>$METRON_HOME/parser_contrib</tt> and ensure the permissions are such that the <tt>metron</tt> user can read and execute.</p></div> |
| <div class="section"> |
| <h3><a name="Restart_the_REST_service_in_Ambari"></a>Restart the REST service in Ambari</h3> |
| <p>In order for new parsers to be picked up, the REST service must be restarted. You can do that from within Ambari by restarting the <tt>Metron REST</tt> service.</p></div> |
| <div class="section"> |
| <h3><a name="Create_a_Kafka_Topic"></a>Create a Kafka Topic</h3> |
| <p>Create a kafka topic, let’s call it <tt>test</tt>.</p> |
| |
| <div> |
| <div> |
| <pre class="source">KAFKA_HOME=/usr/hdp/current/kafka-broker |
| $KAFKA_HOME/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic test --partitions 1 --replication-factor 1 |
| </pre></div></div> |
| |
| <p>Note, in a real deployment, that topic would be named something more descriptive and would have replication factor and partitions set to something less trivial.</p></div> |
| <div class="section"> |
| <h3><a name="Configure_Test_Parser"></a>Configure Test Parser</h3> |
| <p>Create the a file called <tt>$METRON_HOME/config/zookeeper/parsers/test.json</tt> with the following content:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "parserClassName":"com.thirdparty.SimpleParser", |
| "sensorTopic":"test" |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Push_the_Zookeeper_Configs"></a>Push the Zookeeper Configs</h3> |
| <p>Now push the config to Zookeeper with the following command.</p> |
| |
| <div> |
| <div> |
| <pre class="source">$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper/ -z $ZOOKEEPER |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Start_Parser"></a>Start Parser</h3> |
| <p>Now we can start the parser and send some data through:</p> |
| <ul> |
| |
| <li> |
| |
| <p>Start the parser</p> |
| |
| <div> |
| <div> |
| <pre class="source">$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s test |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Send example data through:</p> |
| |
| <div> |
| <div> |
| <pre class="source">echo "apache,metron" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic test |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>Validate data was written in ES:</p> |
| |
| <div> |
| <div> |
| <pre class="source">curl -XPOST "http://$ES_HOST/test*/_search?pretty" -d ' |
| { |
| "_source" : [ "original_string", "timestamp", "first", "last"] |
| } |
| ' |
| </pre></div></div> |
| </li> |
| <li> |
| |
| <p>This should yield something like:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "took" : 23, |
| "timed_out" : false, |
| "_shards" : { |
| "total" : 1, |
| "successful" : 1, |
| "failed" : 0 |
| }, |
| "hits" : { |
| "total" : 1, |
| "max_score" : 1.0, |
| "hits" : [ { |
| "_index" : "test_index_2017.10.04.17", |
| "_type" : "test_doc", |
| "_id" : "3ae4dd4d-8c09-4f2a-93c0-26ec5508baaa", |
| "_score" : 1.0, |
| "_source" : { |
| "original_string" : "apache,metron", |
| "last" : "metron", |
| "first" : "apache", |
| "timestamp" : 1507138373223 |
| } |
| } ] |
| } |
| } |
| </pre></div></div> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Via_the_Management_UI"></a>Via the Management UI</h3> |
| <p>As long as the REST service is restarted after new parsers are added to <tt>$METRON_HOME/parser_contrib</tt>, they are available in the UI to creating and deploying parsers.</p></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
