[STREAMCOMP-2885] broadcast GCC completition (#3330)

* amend gitignore

* decouple GCC post-save logic from stateful controller

* tmaster broadcast checkpoint completion

the message will be first sent to all stmgrs, and then each stmgr will forward
it to every heron instance it connects to.

* heron instance handles global checkpoint saved msg

* expose necessary API for the 2PC support

unit tests for this will be in the following commit

* test: refactor mock physical plan to a builder

* test preSave and postSave hook

* [STREAMCOMP-2916] block execute on postSave

* add stateful tests for spouts

* test preRestore

* rename interface to ITwoPhaseStatefulComponent

* block spout from reading data tuples as well

* fix log format

* do not check interface

this is because the boolean will only be set to true for tasks that implement ITwoPhaseStatefulComponent. No need to do the check again.
31 files changed
tree: 9cad270ef52b1bbffb2739120cec93ca31b43a9b
  1. .bazelrc
  2. .gitattributes
  3. .gitignore
  4. .gitmodules
  5. .htaccess
  6. .rat-excludes
  7. .travis.yml
  8. ADOPTERS.md
  9. BUILD
  10. DISCLAIMER
  11. LICENSE
  12. NOTICE
  13. README.md
  14. WORKSPACE
  15. bazel_configure.py
  16. config/
  17. contrib/
  18. deploy/
  19. docker/
  20. eco-heron-examples/
  21. eco-storm-examples/
  22. eco/
  23. examples/
  24. heron/
  25. heronpy/
  26. index.html
  27. integration_test/
  28. licenses/
  29. release/
  30. scripts/
  31. storm-compatibility-examples/
  32. storm-compatibility/
  33. third_party/
  34. tools/
  35. vagrant/
  36. website/
  37. website2/
README.md

Build Status

logo

Heron is realtime analytics platform developed by Twitter. It has a wide array of architectural improvements over it's predecessor.

Heron in Apache Incubation

Heron Website

Documentation

https://apache.github.io/incubator-heron

Heron Requirements:

  • Java JDK 1.8
  • Python 2.7
  • Bazel latest

Contact

Mailing lists

NameScope
user@heron.incubator.apache.orgUser-related discussionsSubscribeUnsubscribeArchives
dev@heron.incubator.apache.orgDevelopment-related discussionsSubscribeUnsubscribeArchives

Slack

Self-Register to our Heron Slack Workspace

Meetup Group

Bay Area Heron Meetup, We meet on Third Monday of Every Month in Palo Alto.

Twitter

Heron on Twitter: @heronstreaming

For more information:

Update

We recently merged updates to run Heron natively using Mesos in AWS, Mesos/Aurora in AWS, and locally on a laptop.

We also added beta testing for Apache YARN support using Apache REEF, in addition to the Simple Linux Utility for Resource Management (SLURM).

We are working to add support for Mesosphere DC/OS and Kubernetes. We will continue to post updates as we progress.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0