Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.
For the latest information, please visit our website at http://metron.apache.org/
Metron can be divided into 4 areas:
A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates. Because security telemetry is constantly being generated, it requires a method for ingesting the data at high speeds and pushing it to various processing units for advanced computation and analytics.
Real time processing and application of enrichments such as threat intelligence, geolocation, and DNS information to telemetry being collected. The immediate application of this information to incoming telemetry provides the context and situational awareness, as well as the who and where information critical for investigation
Efficient information storage based on how the information will be used:
An interface that gives a security investigator a centralized view of data and alerts passed through the system. Metron’s interface presents alert summaries with threat intelligence and enrichment data specific to that alert on one single page. Furthermore, advanced search capabilities and full packet extraction tools are presented to the analyst for investigation without the need to pivot into additional tools.
Big data is a natural fit for powerful security analytics. The Metron framework integrates a number of elements from the Hadoop ecosystem to provide a scalable platform for security analytics, incorporating such functionality as full-packet capture, stream processing, batch processing, real-time search, and telemetry aggregation. With Metron, our goal is to tie big data into security analytics and drive towards an extensible centralized platform to effectively enable rapid detection and rapid response for advanced security threats.
To obtain a release of Metron, please visit http://metron.apache.org/documentation/#releases
This repository is a collection of submodules for convenience which is regularly updated to point to the latest versions. Github provides multiple ways to obtain Metron's code:
Option 3 is more likely to have the latest code.
To start exploring the capabilities of Apache Metron follow these instructions to launch Metron in a single-node VM on your own hardware.
Build the full project and run tests:
$ mvn clean install
Build without tests:
$ mvn clean install -DskipTests
Build with the HDP profile:
$ mvn clean install -PHDP-22.214.171.124
You can swap “install” for “package” in the commands above if you don't want to deploy the artifacts to your local .m2 repo.
To build and run reporting with code coverage:
$ mvn clean install $ mvn site site:stage-deploy site:deploy
Code coverage can be skipped by skipping tests:
$ mvn clean install -DskipTests site site:stage-deploy site:deploy
The staged site is deployed to /tmp/metron/site/index.html, and can be viewed by opening the file in a browser.
A Docker container with all the required software, with the proper versions, is available to be used as well. see ansible-docker
Metron is at its core a Kappa architecture with Apache Storm as the processing component and Apache Kafka as the unified data bus.
Some high level links to the relevant subparts of the architecture, for more information:
Some useful utilities that cross all of these parts of the architecture:
In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors.
Please see a description of the steps necessary to make this change in metron-elasticsearch Using Metron with Elasticsearch 5.6