merge master into akka-streams branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the commit message is formatted like:
   `[GEARPUMP-<Jira issue #>] Meaningful description of pull request`
 - [x] Make sure tests pass via `sbt clean test`.
 - [x] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.

Author: manuzhang <>
Author: huafengw <>
Author: Kam Kasravi <>
Author: darionyaphet <>
Author: Roshanson <>
Author: Willem Jiang <>
Author: Christoph Seifert <>
Author: <>

Closes #136 from manuzhang/akka-streams.
364 files changed
tree: 65d1d88853dcd6b102f66e2f904d8409ada86777
  1. .github/
  2. bin/
  3. conf/
  4. core/
  5. dev-tools/
  6. docs/
  7. examples/
  8. experiments/
  9. external/
  10. gearpump-hadoop/
  11. integrationtest/
  12. licenses/
  13. notes/
  14. project/
  15. services/
  16. shaded/
  17. streaming/
  18. unmanagedlibs/
  19. yarnconf/
  20. .coveragerc
  21. .gitignore
  22. .rat-excludes
  23. .sbtopts
  24. .travis.yml
  26. codecov.yml
  29. jvm.sbt
  30. KEYS
  32. LICENSE.bin
  33. NOTICE
  34. pubring.asc
  37. scalastyle-config.xml
  38. secring.asc.enc
  39. version.sbt

Apache Gearpump (incubating) Maven Central GitHub license Build Status

Gearpump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks.

The name Gearpump is a reference to the engineering term “gear pump”, which is a super simple pump that consists of only two gears, but is very powerful at streaming water.

We model streaming within the Akka actor hierarchy.

Per initial benchmarks we are able to process near 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.

For steps to reproduce the performance test, please check Performance benchmark.

Useful Resources

How to Build

1). Clone the Gearpump repository

  git clone
  cd gearpump

2). Build package

  ## Please use scala 2.11 or 2.10
  ## The target package path: output/target/gearpump-${version}.zip
  sbt clean +assembly +packArchiveZip

After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder.

To build scala document, use

   ## Will generate the scala doc under target/scala_2.xx/unidoc/
   sbt unidoc

NOTE: The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands. For windows:

set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port

For Linux:

export HTTP_PROXY=http://host:port
export HTTPS_PROXY= http://host:port

How to run Gearpump integration test

Gearpump has an integration test system which is based on Docker. Please check the instructions.

How to do style check before submitting a pull request?

Before submitting a PR, you should always run style check first:

  ## Run style check for compile, test, and integration test.
  sbt scalastyle test:scalastyle it:scalastyle

How to generate the license report to generate a list of all dependencies

  sbt dumpLicenseReport

How to generate dependencies by declared license

  sbt dependencyLicenseInfo

Contributors (time order)



Gearpump itself is licensed under the Apache License (2.0). For library it used, please see LICENSE.


The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.

The cgroup code work is based on JStorm. Thanks JStorm contributors.

Thanks to Jetbrains for providing a IntelliJ IDEA Free Open Source License.