tree: 0f7e47bda4173bfa79ba17e2af552236febd3647 [path history] [tgz]
  1. src/
  2. pom.xml
  3. README.md
eagle-core/eagle-data-process/eagle-stream-pipeline/README.md

Eagle Declarative Streaming DSL

DSL Format

{
	config {
	  config.key = configValue
	}

	schema {
	  metricStreamSchema {
	    metric: string
	    value: double
	    timestamp: long
	  }
	}

	dataflow {
	  kafkaSource.source1 {
	    schema = "metricStreamSchema"
	  }
	  kafkaSource.source2 {
	    schema = {
	      metric: string
	      value: double
	      timestamp: long
	    }
	  }
	}
}

Usage

val pipeline = Pipeline.parseResource("pipeline.conf")
val stream = Pipeline.compile(pipeline)
stream.submit[storm]

Features

  • [x] Compile DSL Configure to Pipeline model
  • [x] Compile Pipeline model to Stream Execution Graph
  • [x] Submit Stream Execution Graph to actual running environment say storm
  • [x] Support Alert and Persistence for metric monitoring
  • [ ] Extensible stream module management and automatically scan and register module
  • [x] Pipeline runner CLI tool and shell script
  • [ ] Decouple pipeline compiler and scheduler into individual modules
  • [ ] Stream Pipeline Scheduler
  • [ ] Graph editor to define streaming graph in UI
  • [?] JSON/Config & Scala Case Class Mapping (https://github.com/scala/pickling)
  • [?] Raw message structure oriented programing is a little ugly, we should define a generic message/event consist of [payload:stream/timestamp/serializer/deserializer,data:message]
  • [ ] Provide stream schema inline and send to metadata when submitting
  • [ ] UI should support specify executorId when defining new stream
  • [ ] Lack of a entity named StreamEntity for the workflow of defining topology&policy end-to-end
  • [!] Fix configuration conflict, should pass through Config instead of ConfigFactory.load() manually
  • [ ] Override application configuration with pipeline configuration
  • [ ] Refactor schema registration structure and automatically submit stream schema when submitting pipeline
  • [ ] Submit alertStream, alertExecutorId mapping to AlertExecutorService when submitting pipeline
  • [x] Supports inputs field to define connector