Eagle Declarative Streaming DSL

DSL Format

{
	config {
	  config.key = configValue
	}

	schema {
	  metricStreamSchema {
	    metric: string
	    value: double
	    timestamp: long
	  }
	}

	dataflow {
	  kafkaSource.source1 {
	    schema = "metricStreamSchema"
	  }
	  kafkaSource.source2 {
	    schema = {
	      metric: string
	      value: double
	      timestamp: long
	    }
	  }
	}
}

Usage

val pipeline = Pipeline.parseResource("pipeline.conf")
val stream = Pipeline.compile(pipeline)
stream.submit[storm]

Features

• [x] Compile DSL Configure to Pipeline model
• [x] Compile Pipeline model to Stream Execution Graph
• [x] Submit Stream Execution Graph to actual running environment say storm
• [x] Support Alert and Persistence for metric monitoring
• [ ] Extensible stream module management and automatically scan and register module
• [x] Pipeline runner CLI tool and shell script
• [ ] Decouple pipeline compiler and scheduler into individual modules
• [ ] Stream Pipeline Scheduler
• [ ] Graph editor to define streaming graph in UI
• [?] JSON/Config & Scala Case Class Mapping (https://github.com/scala/pickling)
• [?] Raw message structure oriented programing is a little ugly, we should define a generic message/event consist of [payload:stream/timestamp/serializer/deserializer,data:message]
• [ ] Provide stream schema inline and send to metadata when submitting
• [ ] UI should support specify executorId when defining new stream
• [ ] Lack of a entity named StreamEntity for the workflow of defining topology&policy end-to-end
• [!] Fix configuration conflict, should pass through Config instead of ConfigFactory.load() manually
• [ ] Override application configuration with pipeline configuration
• [ ] Refactor schema registration structure and automatically submit stream schema when submitting pipeline
• [ ] Submit alertStream, alertExecutorId mapping to AlertExecutorService when submitting pipeline
• [x] Supports inputs field to define connector