| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsers/parser-testing.md at 2018-12-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20181214" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Parser Contribution and Testing</title> |
| <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../css/site.css" /> |
| <link rel="stylesheet" href="../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Parser Contribution and Testing</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-12-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.0</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> |
| <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> |
| <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> |
| <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> |
| <li><a href="../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> |
| <li><a href="../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> |
| <li><a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> |
| <li><a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> |
| <li><a href="../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> |
| <li><a href="../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> |
| <li><a href="../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> |
| <li><a href="../../metron-platform/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-down"></span>Parsers</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li> |
| <li><a href="../../metron-platform/metron-parsers/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li> |
| <li><a href="../../metron-platform/metron-parsers/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li> |
| <li class="active"><a href="#"><span class="none"></span>parser-testing</a></li> |
| <li><a href="../../metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li> |
| </ul> |
| </li> |
| <li><a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> |
| <li><a href="../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> |
| <li><a href="../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> |
| </ul> |
| </li> |
| <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Parser Contribution and Testing</h1> |
| <p><a name="Parser_Contribution_and_Testing"></a></p> |
| <p>So you want to contribute a parser to Apache Metron. First off, on behalf of the community, thank you very much! Now that you have implemented a parser by writing a java class which implements <tt>org.apache.metron.parsers.interfaces.MessageParser</tt> what are the testing expectations for a new parser?</p> |
| <p>It is expected that a new parser have two tests:</p> |
| <ul> |
| |
| <li>A JUnit test directly testing your parser class.</li> |
| <li>An Integration test validating that your parser class can parse messages inside the <tt>ParserBolt</tt>.</li> |
| </ul> |
| <div class="section"> |
| <h2><a name="The_JUnit_Test"></a>The JUnit Test</h2> |
| <p>The JUnit Test should be focused on testing your Parser directly. You should feel free to use mocks or stubs or whatever else you need to completely test that unit of functionality.</p></div> |
| <div class="section"> |
| <h2><a name="The_Integration_Test"></a>The Integration Test</h2> |
| <p>Integration tests are more structured. The intent is that the parser that you have implemented can be driven successfully from <tt>org.apache.metron.parsers.bolt.ParserBolt</tt>.</p> |
| <p>The procedure for creating a new test is as follows:</p> |
| <ul> |
| |
| <li>Create an integration test that extends <tt>org.apache.metron.parsers.integration.ParserIntegrationTest</tt> |
| <ul> |
| |
| <li>Override <tt>getSensorType()</tt> to return the sensor type to be used in the test (referred to as <tt>${sensor_type}</tt> at times)</li> |
| <li>Override <tt>getValidations()</tt> to indicate how you want the output of the parser to be validated (more on validations later)</li> |
| <li>Optionally <tt>readSensorConfig(String sensorType)</tt> to read the sensor config |
| <ul> |
| |
| <li>By default, we will pull this from <tt>metron-parsers/src/main/config/zookeeper/parsers/${sensor_type}</tt>. Override if you want to provide your own</li> |
| </ul> |
| </li> |
| <li>Optionally <tt>readGlobalConfig()</tt> to return the global config |
| <ul> |
| |
| <li>By default, we will pull this from <tt>metron-integration-test/src/main/config/zookeeper/global.json)</tt>. Override if you want to provide your own</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Place sample input data in <tt>metron-integration-test/src/main/sample/data/${sensor_type}/raw</tt> |
| <ul> |
| |
| <li>It should be one line per input record.</li> |
| </ul> |
| </li> |
| <li>Place expected output based on sample data in <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> |
| <ul> |
| |
| <li>Line <tt>k</tt> in the expected data should match with line <tt>k</tt></li> |
| </ul> |
| </li> |
| </ul> |
| <p>The way these tests function is by creating a <tt>ParserBolt</tt> instance with your specified global configuration and sensor configuration. It will then send your specified sample input data in line-by-line. It will then perform some basic sanity validation:</p> |
| <ul> |
| |
| <li>Ensure no errors were logged</li> |
| <li>Execute your specified validation methods</li> |
| </ul> |
| <div class="section"> |
| <h3><a name="Validations"></a>Validations</h3> |
| <p>Validations are functions which indicate how one should validate the parsed messages. The basic one which is sufficient for most cases is <tt>org.apache.metron.parsers.integration.validation.SampleDataValidation</tt>. This will read the expected results from <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> and validate that the actual parsed message conforms (excluding timestamp).</p> |
| <p>If you have special validations required, you may implement your own and return an instance of that in the <tt>getValidations()</tt> method of your Integration Test.</p></div> |
| <div class="section"> |
| <h3><a name="Sample_Integration_Test"></a>Sample Integration Test</h3> |
| <p>A sample integration test for the <tt>snort</tt> parser is as follows:</p> |
| |
| <div> |
| <div> |
| <pre class="source">public class SnortIntegrationTest extends ParserIntegrationTest { |
| @Override |
| String getSensorType() { |
| return "snort"; |
| } |
| |
| @Override |
| List<ParserValidation> getValidations() { |
| return new ArrayList<ParserValidation>() {{ |
| add(new SampleDataValidation()); |
| }}; |
| } |
| } |
| </pre></div></div></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
