Application manager aims to manage topology status on EAGLE UI. Users can easily start/start topologies remotely or locally without any shell commands. At the same, it should be capable to sync the latest status of topologies on the execution platform (e.g., storm cluster).
This tutorial will go through all parts of application manager and then give an example to use it.
Application manager consists of a daemon scheduler and an execution module. The scheduler periodically loads user operations(start/stop) from database, and the execution module executes these operations. For more details, please refer to here
The configuration file eagle-scheduler.conf
defines scheduler parameters, execution platform settings and parts of default topology configuration.
Scheduler properties
appCommandLoaderEnabled: enable application manager. TODO: change it to true.
appCommandLoaderIntervalSecs: defines the interval of the scheduler loads commands. Default value is 1 second.
appHealthCheckIntervalSecs: define the interval of health check, which tries to sync the topology execution status to Eagle.
Execution platform properties
envContextConfig.env: application execution platform. Default value is storm.
envContextConfig.url: execution platform http url. Default is “http://sandbox.hortonworks.com:8744”.
envContextConfig.nimbusHost: storm nimbus host. Default is “sandbox.hortonworks.com”.
envContextConfig.nimbusThriftPort: default is 6627.
envContextConfig.jarFile: storm fat jar path. TODO: change “/dir-to-jar/eagle-topology-0.3.0-incubating-assembly.jar” to your own jar path.
Topology default properties
Some default topology properties are defined here.
Note: these configurations can be overridden in the topology configurations, which is shown below. The only difference is to add a prefix app.
. For example, ‘app.envContextConfig.jarFile’ is to override ‘envContextConfig.jarFile’ in eagle-schedule.conf
Editing eagle-scheduler.conf, and start Eagle service
# enable application manager appCommandLoaderEnabled=true # provide jar path envContextConfig.jarFile=/path-to-jar
For more configurations, please back to Configuration.
After the configuration is ready, start Eagle service bin/eagle-service.sh start
.
Go to admin page
Go to management page, and create a topology description. There are three required fields
Back to monitoring page, and choose the site/application to deploy the topology
Go to site page, and edit site->application and add some new configurations. Blow are some example configurations for [site=sandbox, applicatoin=hbaseSecurityLog] These configurations have a higher priority than those in eagle-scheduler.conf
classification.hbase.zookeeper.property.clientPort=2181 classification.hbase.zookeeper.quorum=sandbox.hortonworks.com # platform related configurations app.envContextConfig.env=storm app.envContextConfig.mode=cluster # data source related configurations app.dataSourceConfig.topic=sandbox_hbase_security_log app.dataSourceConfig.zkConnection=sandbox.hortonworks.com:2181 app.dataSourceConfig.zkConnectionTimeoutMS=15000 app.dataSourceConfig.brokerZkPath=/brokers app.dataSourceConfig.fetchSize=1048586 app.dataSourceConfig.transactionZKServers=sandbox.hortonworks.com app.dataSourceConfig.transactionZKPort=2181 app.dataSourceConfig.transactionZKRoot=/consumers app.dataSourceConfig.consumerGroupId=eagle.hbasesecurity.consumer app.dataSourceConfig.transactionStateUpdateMS=2000 app.dataSourceConfig.deserializerClass=org.apache.eagle.security.hbase.parse.HbaseAuditLogKafkaDeserializer # service related configurations app.eagleProps.site=sandbox app.eagleProps.application=hbaseSecurityLog app.eagleProps.dataJoinPollIntervalSec=30 app.eagleProps.mailHost=some.mail.server app.eagleProps.mailSmtpPort=25 app.eagleProps.mailDebug=true app.eagleProps.eagleService.host=localhost app.eagleProps.eagleService.port=9099 app.eagleProps.eagleService.username=admin app.eagleProps.eagleService.password=secret
Go to monitoring page, and start topologies
stop topologies on monitoring page