#Metron-Indexing
##Module Description
This module provides the indexing capability to Metron components. The primary indexing engine for now is Elastic Search, but Solr may be supported at some point in the future as well. There are three types of messages that are commonly indexed in Metron topologies: messages, alerts, and errors. Messages are telemetry messages parsed by the parser bolt. Alerts are alerts generated by the alerts bolt. Errors are an optional feature where each Metron bolt in addition to outputting errors in the log file will also index them for immediate analysis.
###Index bolt
The signature of the index bolt is as follows:
TelemetryIndexingBolt indexing_bolt = new TelemetryIndexingBolt()
.withIndexIP(config.getString("es.ip"))
.withIndexPort(config.getInt("es.port"))
.withClusterName(config.getString("es.clustername"))
.withIndexName(
config.getString("bolt.error.indexing.indexname"))
.withDocumentName(
config.getString("bolt.error.indexing.documentname"))
.withBulk(config.getInt("bolt.error.indexing.bulk"))
.withIndexAdapter(adapter)
.withMetricConfiguration(config);
###IndexAdapters
*org.apache.metron.indexing.adapters.ESBaseBulkAdapter - bulk ingest messages into Elastic Search *org.apache.metron.indexing.adapters.ESBaseBulkRotatingAdapter - does everything adapter above does, but is able to rotate the index names based on size *org.apache.metron.indexing.adapters.ESTimedBulkRotatingAdapter - does everything adapter above does, but is able to rotate the index names based on size and time *org.apache.metron.indexing.adapters.SolrAdapter - currently under development
/etc/ directory contains all environment-related configs
##Sample Input and Generator Spout
The sample input for topologies provided in this release was checked in here:
https://github.com/apache/incubator-metron-streaming/tree/master/Metron-Topologies/src/main/resources/SampleInput
We provide a generator spout that is able to drive these topologies. In production we run with the kafka spout, but for documentation on that please reference the Storm project documentation
The generator spout comes with the following signature:
GenericInternalTestSpout testSpout = new GenericInternalTestSpout()
.withFilename(test_file_path).withRepeating(
config.getBoolean("spout.test.parallelism.repeat"));
###Additional Storm Bolts In addition to custom bolts developed for Metron we utilize standard bolts and spouts included with the Storm release. We will not provide documentation for these spouts and bolts since they are provided as part of Storm. These spouts bolts are: