[REEF-1448] Support for upstream heartbeats
Adding job cancellation functionality into IMRU driver.
And exposing ICancellationDetector interface in API which enables client to provide custom cancellation detection from upstream.
Implementation: JobLifecycleManager is registered as IjobCancelled event source.
the manager checks IsJobCancelled method of configured CancellationonDetector on each timer event and
if cancellation signal is detected JobLifecycleManager fires JobCancelled event.
IMRU driver is registered as IJobCancelled observable and if event is fired, shuts down all evaluators and moves to Fail state.
Similar to failure handling, when job is cancelled, IMRU driver fires and exception instead of retrying the job.
(there is no other way to communicate driver status)
In addition, JobLifecycleManager is making a request to RM to try to kill current job - if succeeded, this will mark job status as KILLED instead of FAILED.

Client can provide implementation of IJobCancelledDetector in job definition.
This configuration is optional, if no implementation is configured in the definition, the driver assumes there is no cancellation signal.

JIRA:
  [REEF-1448] (https://issues.apache.org/jira/browse/REEF-1448)

This closes #1161
24 files changed
tree: e41fb5b8111e363ec95b3ea1cfbfe200d46f12bd
  1. bin/
  2. dev/
  3. lang/
  4. website/
  5. .gitattributes
  6. .gitignore
  7. .travis.yml
  8. appveyor.yml
  9. Doxyfile
  10. HEADER
  11. LICENSE
  12. NOTICE
  13. pom.xml
  14. README.md
README.md

Apache REEF™

Apache REEF™ (Retainable Evaluator Execution Framework) is a library for developing portable applications for cluster resource managers such as Apache Hadoop YARN or Apache Mesos. For example, Microsoft Azure Stream Analytics is built on REEF and Hadoop.

Online Documentation

Detailed information on REEF can be found in the following places:

The developer mailing list is the best way to reach REEF‘s developers when the above aren’t sufficient.

Build Status

ComponentOSStatus
REEF JavaUbuntuBuild Status
REEF.NETWindowsBuild status

Building REEF

Java.NET
Build & run unit testsjava\BUILD.mdcs\BUILD.md

Releases

downloads NuGet package

License

Apache 2.0