This module provides a RESTful API for interacting with Metron.
mvn clean package
config rest_application.yml bin metron-rest lib metron-rest-$METRON_VERSION.jar
$METRON_HOME/bin/metron-rest
script to /etc/init.d/metron-rest
Deploy the RPM at /metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm
Install the RPM with:
rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
The REST application depends on several configuration parameters:
No optional parameter has a default.
Environment Variable | Description |
---|---|
METRON_JDBC_DRIVER | JDBC driver class |
METRON_JDBC_URL | JDBC url |
METRON_JDBC_USERNAME | JDBC username |
METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle |
ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) |
BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) |
HDFS_URL | HDFS url or fs.defaultFS Hadoop setting (ex. hdfs://node1:8020) |
Environment Variable | Description | Required | Default |
---|---|---|---|
METRON_USER | Run the application as this user | Optional | metron |
METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
METRON_REST_PORT | REST application port | Optional | 8082 |
METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
SECURITY_ENABLED | Enables Kerberos support | Optional | false |
Environment Variable | Description | Required |
---|---|---|
METRON_JVMFLAGS | JVM flags added to the start command | Optional |
METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see below) | Optional |
METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional |
METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional |
METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional |
These are set in the /etc/sysconfig/metron
file.
The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
For example, edit these variables in /etc/sysconfig/metron
before starting the application to configure H2:
METRON_JDBC_DRIVER="org.h2.Driver" METRON_JDBC_URL="jdbc:h2:file:~/metrondb" METRON_JDBC_USERNAME="root" METRON_JDBC_PASSWORD='root" METRON_JDBC_PLATFORM="h2"
The REST application should be configured with a production-grade database outside of development.
For example, the following configures the application for MySQL:
Install MySQL if not already available (this example uses version 5.7, installation instructions can be found here)
Create a metron user and REST database and permission the user for that database:
CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd'; CREATE DATABASE IF NOT EXISTS metronrest; GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
cd $METRON_HOME/lib wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz tar xf mysql-connector-java-5.1.41.tar.gz
/etc/sysconfig/metron
to configure the REST application for MySQL:METRON_JDBC_DRIVER="com.mysql.jdbc.Driver" METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest" METRON_JDBC_USERNAME="metron" METRON_JDBC_PLATFORM="mysql" METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
After configuration is complete, the REST application can be managed as a service:
service metron-rest start
If a production database is configured, the JDBC password should be passed in as the first argument on startup:
service metron-rest start Myp@ssw0rd
The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
The metron-rest module uses Spring Security for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
use metronrest; insert into users (username, password, enabled) values ('your_username','your_password',1); insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the /etc/sysconfig/metron
file:
SECURITY_ENABLED=true METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf" METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM" METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
The REST application comes with a few Spring Profiles to aid in testing and development.
Profile | Description |
---|---|
test | sets variables to in-memory services, only used for integration testing |
dev | adds a test user to the database with credentials user/password |
vagrant | sets configuration variables to match the Metron vagrant environment |
docker | sets configuration variables to match the Metron docker environment |
Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in /etc/sysconfig/metron
to configure the REST application for the Vagrant environment and add a test user:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.
POST /api/v1/alert/escalate
GET /api/v1/global/config
DELETE /api/v1/global/config
POST /api/v1/global/config
GET /api/v1/grok/get/statement
GET /api/v1/grok/list
POST /api/v1/grok/validate
POST /api/v1/hdfs
GET /api/v1/hdfs
DELETE /api/v1/hdfs
GET /api/v1/hdfs/list
GET /api/v1/kafka/topic
POST /api/v1/kafka/topic
GET /api/v1/kafka/topic/{name}
DELETE /api/v1/kafka/topic/{name}
GET /api/v1/kafka/topic/{name}/sample
GET /api/v1/search/findOne
bro
document with UUID of 000-000-0000
{ "guid" : "000-000-0000", "sensorType" : "bro" }
GET /api/v1/search/search
GET /api/v1/search/column/metadata
GET /api/v1/search/column/metadata/common
GET /api/v1/sensor/enrichment/config
GET /api/v1/sensor/enrichment/config/list/available/enrichments
GET /api/v1/sensor/enrichment/config/list/available/threat/triage/aggregators
DELETE /api/v1/sensor/enrichment/config/{name}
POST /api/v1/sensor/enrichment/config/{name}
GET /api/v1/sensor/enrichment/config/{name}
GET /api/v1/sensor/indexing/config
DELETE /api/v1/sensor/indexing/config/{name}
POST /api/v1/sensor/indexing/config/{name}
GET /api/v1/sensor/indexing/config/{name}
POST /api/v1/sensor/parser/config
GET /api/v1/sensor/parser/config
GET /api/v1/sensor/parser/config/list/available
POST /api/v1/sensor/parser/config/parseMessage
GET /api/v1/sensor/parser/config/reload/available
DELETE /api/v1/sensor/parser/config/{name}
GET /api/v1/sensor/parser/config/{name}
POST /api/v1/stellar/apply/transformations
GET /api/v1/stellar/list
GET /api/v1/stellar/list/functions
GET /api/v1/stellar/list/simple/functions
POST /api/v1/stellar/validate/rules
GET /api/v1/storm
GET /api/v1/storm/client/status
GET /api/v1/storm/enrichment
GET /api/v1/storm/enrichment/activate
GET /api/v1/storm/enrichment/deactivate
GET /api/v1/storm/enrichment/start
GET /api/v1/storm/enrichment/stop
GET /api/v1/storm/indexing
GET /api/v1/storm/indexing/activate
GET /api/v1/storm/indexing/deactivate
GET /api/v1/storm/indexing/start
GET /api/v1/storm/indexing/stop
GET /api/v1/storm/parser/activate/{name}
GET /api/v1/storm/parser/deactivate/{name}
GET /api/v1/storm/parser/start/{name}
GET /api/v1/storm/parser/stop/{name}
GET /api/v1/storm/{name}
GET /api/v1/storm/supervisors
PATCH /api/v1/update/patch
project
with value metron
to the bro
message with UUID of 000-000-0000
:{ "guid" : "000-000-0000", "sensorType" : "bro", "patch" : [ { "op": "add" , "path": "/project" , "value": "metron" } ] }
PUT /api/v1/update/replace
bro
message with guid of 000-000-0000
{ "guid" : "000-000-0000", "sensorType" : "bro", "replacement" : { "source:type": "bro", "guid" : "bro_index_2017.01.01.01:1", "ip_src_addr":"192.168.1.2", "ip_src_port": 8009, "timestamp":200, "rejected":false } }
GET /api/v1/user
Profiles are includes for both the metron-docker and Quick Dev environments.
Start the metron-docker environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
mvn clean package mvn spring-boot:run -Drun.profiles=docker,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
Start the Quick Dev environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
mvn clean package mvn spring-boot:run -Drun.profiles=vagrant,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
To run the application locally on the Quick Dev host (node1), follow the Installation instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in /etc/sysconfig/metron
:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
and start the application:
service metron-rest start
In a cluster with Kerberos enabled, update the security settings in /etc/sysconfig/metron
. Security is disabled by default in the vagrant
Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev" METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf" METRON_SPRING_OPTIONS="--kerberos.enabled=true"
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
This project depends on the Java Transaction API. See https://java.net/projects/jta-spec/ for more details.