metron-platform/metron-parsers/parser-testing.md

# Parser Contribution and Testing

So you want to contribute a parser to Apache Metron.  First off, on behalf
of the community, thank you very much!  Now that you have implemented a parser
by writing a java class which implements `org.apache.metron.parsers.interfaces.MessageParser`
what are the testing expectations for a new parser?

It is expected that a new parser have two tests:
* A JUnit test directly testing your parser class.
* An Integration test validating that your parser class can parse messages 
inside the `ParserBolt`.

## The JUnit Test

The JUnit Test should be focused on testing your Parser directly.  You
should feel free to use mocks or stubs or whatever else you need to completely
test that unit of functionality.

## The Integration Test

Integration tests are more structured.  The intent is that the parser that
you have implemented can be driven successfully from `org.apache.metron.parsers.bolt.ParserBolt`.

The procedure for creating a new test is as follows:
* Create an integration test that extends `org.apache.metron.parsers.integration.ParserIntegrationTest`
  * Override `getSensorType()` to return the sensor type to be used in the test (referred to as `${sensor_type}` at times)
  * Override `getValidations()` to indicate how you want the output of the parser to be validated (more on validations later)
  * Optionally `readSensorConfig(String sensorType)` to read the sensor config
    * By default, we will pull this from `metron-parsers/src/main/config/zookeeper/parsers/${sensor_type}`.  Override if you want to provide your own
  * Optionally `readGlobalConfig()` to return the global config
    * By default, we will pull this from `metron-integration-test/src/main/config/zookeeper/global.json)`.  Override if you want to provide your own
* Place sample input data in `metron-integration-test/src/main/sample/data/${sensor_type}/raw`
  * It should be one line per input record.
* Place expected output based on sample data in `metron-integration-test/src/main/sample/data/${sensor_type}/parsed`
  * Line `k` in the expected data should match with line `k`

The way these tests function is by creating a `ParserBolt` instance with your specified global configuration and
sensor configuration.  It will then send your specified sample input data in line-by-line.  It will then
perform some basic sanity validation:
* Ensure no errors were logged
* Execute your specified validation methods

### Validations

Validations are functions which indicate how one should validate the parsed messages.  The basic one which is sufficient
for most cases is `org.apache.metron.parsers.integration.validation.SampleDataValidation`.  This will read the expected results
from `metron-integration-test/src/main/sample/data/${sensor_type}/parsed` and validate that the actual parsed message
conforms (excluding timestamp).

If you have special validations required, you may implement your own and return an instance of that in the `getValidations()`
method of your Integration Test.

### Sample Integration Test

A sample integration test for the `snort` parser is as follows:
```
public class SnortIntegrationTest extends ParserIntegrationTest {
  @Override
  String getSensorType() {
    return "snort";
  }

  @Override
  List<ParserValidation> getValidations() {
    return new ArrayList<ParserValidation>() {{
      add(new SampleDataValidation());
    }};
  }
}
```