layout: doc title: “Topology Management” permalink: /docs/tutorial/topologymanagement.html

Since Apache Eagle 0.4.0-incubating. Apache Eagle will be called Eagle in the following.

Application manager aims to manage applications on EAGLE UI. Users can easily start/start topologies remotely or locally without any shell commands. At the same, it should be capable to sync the latest status of topologies on the execution platform (e.g., Storm[^STORM] cluster).

This tutorial will go through all parts of application manager and then give an example to use it.

Design

Application manager consists of a daemon scheduler and an execution module. The scheduler periodically loads user operations(start/stop) from database, and the execution module executes these operations. For more details, please refer to here.

Configurations

The configuration file eagle-scheduler.conf defines scheduler parameters, execution platform settings and parts of default topology configuration.

  • Scheduler properties

    Property NameDefaultDescription
    appCommandLoaderEnabledfalsetopology management is enabled or not
    appCommandLoaderIntervalSecs1defines the interval of the scheduler loads commands
    appHealthCheckIntervalSecs5define the interval of topology health checking, which syncs the topology execution status from storm cluster to Eagle

  • Execution platform properties

    Property NameDefaultDescription
    envContextConfig.envstormexecution environment, only storm is supported
    envContextConfig.urlhttp://sandbox.hortonworks.com:8744storm ui
    envContextConfig.nimbusHostsandbox.hortonworks.comstorm nimbus host
    envContextConfig.nimbusThriftPort6627storm nimbus thrift port
    envContextConfig.jarFileTODOstorm fat jar path

  • Topology default properties

    Some default topology properties are defined here.

Playbook

  1. Editing eagle-scheduler.conf, and start Eagle service

     # enable application manager       
 appCommandLoaderEnabled = true

 # provide jar path
 envContextConfig.jarFile =

 # storm nimbus
 envContextConfig.url = "http://sandbox.hortonworks.com:8744"
 envContextConfig.nimbusHost = "sandbox.hortonworks.com"

    For more configurations, please back to Application Configuration.
    After the configuration is ready, start Eagle service bin/eagle-service.sh start.

  2. Go to admin page admin-page topology-monitor

  3. Go to management page, and create a topology description. There are three required fields

    • name: topology name
    • type: topology type [CLASS, DYNAMIC]
    • execution entry: either the class which implements interface TopologyExecutable or eagle DSL based topology definition topology-description

  4. Back to monitoring page, and choose the site/application to deploy the topology topology-execution

  5. Go to site page, and add topology configurations.

    NOTICE topology configurations defined here are REQUIRED an extra prefix .app

    Blow are some example configurations for [site=sandbox, applicatoin=hbaseSecurityLog].

     classification.hbase.zookeeper.property.clientPort=2181
 classification.hbase.zookeeper.quorum=sandbox.hortonworks.com

 app.envContextConfig.env=storm
 app.envContextConfig.mode=cluster

 app.dataSourceConfig.topic=sandbox_hbase_security_log
 app.dataSourceConfig.zkConnection=sandbox.hortonworks.com:2181
 app.dataSourceConfig.zkConnectionTimeoutMS=15000
 app.dataSourceConfig.brokerZkPath=/brokers
 app.dataSourceConfig.fetchSize=1048586
 app.dataSourceConfig.transactionZKServers=sandbox.hortonworks.com
 app.dataSourceConfig.transactionZKPort=2181
 app.dataSourceConfig.transactionZKRoot=/consumers
 app.dataSourceConfig.consumerGroupId=eagle.hbasesecurity.consumer
 app.dataSourceConfig.transactionStateUpdateMS=2000
 app.dataSourceConfig.deserializerClass=org.apache.eagle.security.hbase.parse.HbaseAuditLogKafkaDeserializer

 app.eagleProps.site=sandbox
 app.eagleProps.application=hbaseSecurityLog
 app.eagleProps.dataJoinPollIntervalSec=30
 app.eagleProps.mailHost=some.mail.server
 app.eagleProps.mailSmtpPort=25
 app.eagleProps.mailDebug=true
 app.eagleProps.eagleService.host=localhost
 app.eagleProps.eagleService.port=9099
 app.eagleProps.eagleService.username=admin
 app.eagleProps.eagleService.password=secret

    topology-configuration-1 topology-configuration-2

  6. Go to monitoring page, and start topologies start-topology-1 start-topology-2

  7. stop topologies on monitoring page stop-topology-1 stop-topology-2 stop-topology-3

Footnotes

[^STORM]:All mentions of “storm” on this page represent Apache Storm.