Application manager aims to manage topology status on EAGLE UI. Users can easily start/start topologies remotely or locally without any shell commands. At the same, it should be capable to sync the latest status of topologies on the execution platform (e.g., storm cluster).

This tutorial will go through all parts of application manager and then give an example to use it.

Design

Application manager consists of a daemon scheduler and an execution module. The scheduler periodically loads user operations(start/stop) from database, and the execution module executes these operations. For more details, please refer to here

Configurations

The configuration file eagle-scheduler.conf defines scheduler parameters, execution platform settings and parts of default topology configuration.

• Scheduler properties

  appCommandLoaderEnabled: enable application manager. TODO: change it to true.
  appCommandLoaderIntervalSecs: defines the interval of the scheduler loads commands. Default value is 1 second.
  appHealthCheckIntervalSecs: define the interval of health check, which tries to sync the topology execution status to Eagle.
  
  

• Execution platform properties

  envContextConfig.env: application execution platform. Default value is storm.
  envContextConfig.url: execution platform http url. Default is “http://sandbox.hortonworks.com:8744”.
  envContextConfig.nimbusHost: storm nimbus host. Default is “sandbox.hortonworks.com”.
  envContextConfig.nimbusThriftPort: default is 6627.
  envContextConfig.jarFile: storm fat jar path. TODO: change “/dir-to-jar/eagle-topology-0.3.0-incubating-assembly.jar” to your own jar path.
  
  

• Topology default properties

  Some default topology properties are defined here.

Note: these configurations can be overridden in the topology configurations, which is shown below. The only difference is to add a prefix app.. For example, ‘app.envContextConfig.jarFile’ is to override ‘envContextConfig.jarFile’ in eagle-schedule.conf

Manual

1. Editing eagle-scheduler.conf, and start Eagle service

    # enable application manager         
    appCommandLoaderEnabled=true
    # provide jar path
    envContextConfig.jarFile=/path-to-jar
   

   For more configurations, please back to Configuration.
   After the configuration is ready, start Eagle service bin/eagle-service.sh start.

2. Go to admin page

3. Go to management page, and create a topology description. There are three required fields

   • name: topology name
   • type: topology type [CLASS, DYNAMIC]
   • execution entry: either the class which implements interface TopologyExecutable or eagle DSL based topology definition

4. Back to monitoring page, and choose the site/application to deploy the topology

5. Go to site page, and edit site->application and add some new configurations. Blow are some example configurations for [site=sandbox, applicatoin=hbaseSecurityLog] These configurations have a higher priority than those in eagle-scheduler.conf

       classification.hbase.zookeeper.property.clientPort=2181
       classification.hbase.zookeeper.quorum=sandbox.hortonworks.com
       # platform related configurations
       app.envContextConfig.env=storm
       app.envContextConfig.mode=cluster
       # data source related configurations
       app.dataSourceConfig.topic=sandbox_hbase_security_log
       app.dataSourceConfig.zkConnection=sandbox.hortonworks.com:2181
       app.dataSourceConfig.zkConnectionTimeoutMS=15000
       app.dataSourceConfig.brokerZkPath=/brokers
       app.dataSourceConfig.fetchSize=1048586
       app.dataSourceConfig.transactionZKServers=sandbox.hortonworks.com
       app.dataSourceConfig.transactionZKPort=2181
       app.dataSourceConfig.transactionZKRoot=/consumers
       app.dataSourceConfig.consumerGroupId=eagle.hbasesecurity.consumer
       app.dataSourceConfig.transactionStateUpdateMS=2000
       app.dataSourceConfig.deserializerClass=org.apache.eagle.security.hbase.parse.HbaseAuditLogKafkaDeserializer
       # service related configurations
       app.eagleProps.site=sandbox
       app.eagleProps.application=hbaseSecurityLog
       app.eagleProps.dataJoinPollIntervalSec=30
       app.eagleProps.mailHost=some.mail.server
       app.eagleProps.mailSmtpPort=25
       app.eagleProps.mailDebug=true
       app.eagleProps.eagleService.host=localhost
       app.eagleProps.eagleService.port=9099
       app.eagleProps.eagleService.username=admin
       app.eagleProps.eagleService.password=secret
   

   

6. Go to monitoring page, and start topologies

7. stop topologies on monitoring page