commit | 1fd4f2655406b7ce0f9e9b8c0d954d1941e60880 | [log] [tgz] |
---|---|---|
author | andrey-me <andreym@microsoft.com> | Fri Oct 14 15:57:12 2016 -0700 |
committer | Julia Wang <jwang98052@yahoo.com> | Mon Nov 21 12:21:54 2016 -0800 |
tree | e41fb5b8111e363ec95b3ea1cfbfe200d46f12bd | |
parent | 4e4e568faa3c3d625433b17bb4d15e0cd7897d7a [diff] |
[REEF-1448] Support for upstream heartbeats Adding job cancellation functionality into IMRU driver. And exposing ICancellationDetector interface in API which enables client to provide custom cancellation detection from upstream. Implementation: JobLifecycleManager is registered as IjobCancelled event source. the manager checks IsJobCancelled method of configured CancellationonDetector on each timer event and if cancellation signal is detected JobLifecycleManager fires JobCancelled event. IMRU driver is registered as IJobCancelled observable and if event is fired, shuts down all evaluators and moves to Fail state. Similar to failure handling, when job is cancelled, IMRU driver fires and exception instead of retrying the job. (there is no other way to communicate driver status) In addition, JobLifecycleManager is making a request to RM to try to kill current job - if succeeded, this will mark job status as KILLED instead of FAILED. Client can provide implementation of IJobCancelledDetector in job definition. This configuration is optional, if no implementation is configured in the definition, the driver assumes there is no cancellation signal. JIRA: [REEF-1448] (https://issues.apache.org/jira/browse/REEF-1448) This closes #1161
Apache REEF™ (Retainable Evaluator Execution Framework) is a library for developing portable applications for cluster resource managers such as Apache Hadoop YARN or Apache Mesos. For example, Microsoft Azure Stream Analytics is built on REEF and Hadoop.
Detailed information on REEF can be found in the following places:
The developer mailing list is the best way to reach REEF‘s developers when the above aren’t sufficient.
Component | OS | Status |
---|---|---|
REEF Java | Ubuntu | |
REEF.NET | Windows |
Java | .NET | |
---|---|---|
Build & run unit tests | java\BUILD.md | cs\BUILD.md |