| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-platform/metron-parsing/metron-parsers-common/parser-testing.md at 2019-05-14 |
| | Rendered using Apache Maven Fluido Skin 1.7 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20190514" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Metron – Parser Contribution and Testing</title> |
| <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> |
| <link rel="stylesheet" href="../../../css/site.css" /> |
| <link rel="stylesheet" href="../../../css/print.css" media="print" /> |
| <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> |
| <script type="text/javascript"> |
| $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); |
| </script> |
| </head> |
| <body class="topBarDisabled"> |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"><a href="http://metron.apache.org/" id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> |
| <div class="pull-right"></div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> |
| <li class=""><a href="http://metron.apache.org/" class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> |
| <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> |
| <li class="active ">Parser Contribution and Testing</li> |
| <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2019-05-14</li> |
| <li id="projectVersion" class="pull-right">Version: 0.7.1</li> |
| </ul> |
| </div> |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span2"> |
| <div class="well sidebar-nav"> |
| <ul class="nav nav-list"> |
| <li class="nav-header">User Documentation</li> |
| <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> |
| <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> |
| <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> |
| <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> |
| <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> |
| <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> |
| <li><a href="../../../metron-interface/index.html" title="Interface"><span class="icon-chevron-right"></span>Interface</a></li> |
| <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-down"></span>Platform</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide"><span class="none"></span>Performance-tuning-guide</a></li> |
| <li><a href="../../../metron-platform/metron-common/index.html" title="Common"><span class="none"></span>Common</a></li> |
| <li><a href="../../../metron-platform/metron-data-management/index.html" title="Data-management"><span class="none"></span>Data-management</a></li> |
| <li><a href="../../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li> |
| <li><a href="../../../metron-platform/metron-enrichment/index.html" title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li> |
| <li><a href="../../../metron-platform/metron-hbase-server/index.html" title="Hbase-server"><span class="none"></span>Hbase-server</a></li> |
| <li><a href="../../../metron-platform/metron-indexing/index.html" title="Indexing"><span class="none"></span>Indexing</a></li> |
| <li><a href="../../../metron-platform/metron-job/index.html" title="Job"><span class="none"></span>Job</a></li> |
| <li><a href="../../../metron-platform/metron-management/index.html" title="Management"><span class="none"></span>Management</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/index.html" title="Parsing"><span class="icon-chevron-down"></span>Parsing</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers/index.html" title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/index.html" title="Parsers-common"><span class="icon-chevron-down"></span>Parsers-common</a> |
| <ul class="nav nav-list"> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/3rdPartyParser.html" title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/ParserChaining.html" title="ParserChaining"><span class="none"></span>ParserChaining</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/message-parser-implementation-notes.html" title="message-parser-implementation-notes"><span class="none"></span>message-parser-implementation-notes</a></li> |
| <li class="active"><a href="#"><span class="none"></span>parser-testing</a></li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/paloalto/index.html" title="Paloalto"><span class="none"></span>Paloalto</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-platform/metron-parsing/metron-parsing-storm/index.html" title="Parsing-storm"><span class="none"></span>Parsing-storm</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li> |
| <li><a href="../../../metron-platform/metron-solr/index.html" title="Solr"><span class="none"></span>Solr</a></li> |
| <li><a href="../../../metron-platform/metron-writer/index.html" title="Writer"><span class="none"></span>Writer</a></li> |
| </ul> |
| </li> |
| <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> |
| <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> |
| <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> |
| <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> |
| <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> |
| </ul> |
| </li> |
| </ul> |
| <hr /> |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> |
| </div> |
| </div> |
| </div> |
| <div id="bodyColumn" class="span10" > |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1>Parser Contribution and Testing</h1> |
| <p><a name="Parser_Contribution_and_Testing"></a></p> |
| <p>So you want to contribute a parser to Apache Metron. First off, on behalf of the community, thank you very much! Now that you have implemented a parser by writing a java class which implements <tt>org.apache.metron.parsers.interfaces.MessageParser</tt> what are the testing expectations for a new parser?</p> |
| <p>It is expected that a new parser have two tests:</p> |
| <ul> |
| |
| <li>A JUnit test directly testing your parser class.</li> |
| <li>An Integration test validating that your parser class can parse messages inside the <tt>ParserBolt</tt>.</li> |
| </ul> |
| <div class="section"> |
| <h2><a name="The_JUnit_Test"></a>The JUnit Test</h2> |
| <p>The JUnit Test should be focused on testing your Parser directly. You should feel free to use mocks or stubs or whatever else you need to completely test that unit of functionality.</p></div> |
| <div class="section"> |
| <h2><a name="The_Integration_Test"></a>The Integration Test</h2> |
| <p>Integration tests are more structured. The intent is that the parser that you have implemented can be driven successfully from the appropriate driver, e.g. <tt>org.apache.metron.parsers.bolt.ParserBolt</tt> for Storm.</p> |
| <p>To add a new test, just add it to the list of sensorTypes in <tt>org.apache.metron.parsers.integration.ParserIntegrationTest</tt>.</p> |
| <p>To setup the tests for a new platform, extend <tt>ParserIntegrationTest</tt>, e.g. as in <tt>org.apache.metron.parsers.integration.StormParserIntegrationTests</tt>. This should be a parameterized test, so that each sensorType gets its own test. Use <tt>StormParserIntegrationTests</tt> as a template for the new platform’s class. The test method should just setup the appropriate <tt>ParserDriver</tt> implementation, and simply call back into the parent to run the test.</p> |
| <p>Customized versions of the tests can be added by extending <tt>ParserIntegrationTest</tt> and performing additional setup or validations as needed.</p> |
| <p>The way these tests function is by running the <tt>ParserDriver</tt> instance with your specified global configuration and sensor configuration. It will then send your specified sample input data in line-by-line. It will then perform some basic sanity validation:</p> |
| <ul> |
| |
| <li>Ensure no errors were logged</li> |
| <li>Execute your specified validation methods</li> |
| </ul> |
| <div class="section"> |
| <h3><a name="Validations"></a>Validations</h3> |
| <p>Validations are functions which indicate how one should validate the parsed messages. The basic one which is sufficient for most cases is <tt>org.apache.metron.parsers.integration.validation.SampleDataValidation</tt>. This will read the expected results from <tt>metron-integration-test/src/main/sample/data/${sensor_type}/parsed</tt> and validate that the actual parsed message conforms (excluding timestamp).</p> |
| <p>If you have special validations required, you may implement your own and return an instance of that in the <tt>getValidations()</tt> method of your Integration Test.</p></div></div> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row-fluid"> |
| © 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, |
| and the Apache Metron project logo are trademarks of The Apache Software Foundation. |
| </div> |
| </div> |
| </footer> |
| </body> |
| </html> |
