With [JobManager High Availability]({{ site.baseurl}}/ops/ha/jobmanager_high_availability.html), you can recover from JobManager failures and thereby eliminate the SPOF. However, when the new leader takes over the job, it first needs to make a full restart.
With the support of JobManager failover, a job can continue running without any interruption during the recovery of JobManager failures. This is especially useful for batch jobs.
The general idea to support JobManager failover is that for each modification of ExecutionGraph, the JobManager will write an operation log. The operation log will be persisted to external storage, so that when a new leader takes over the job, it first replays the operation log to rebuild the ExecutionGraph, and then it reconciles with the TaskManagers to make sure the real status of tasks are the same with that in the ExecutionGraph.If all the task status in ExecutionGraph are consistent with the status reported by TaskManagers, the JobManager will continue running the job without any interruption. Or else, it will fail the tasks whose status are conflict with report of TaskManagers and restart them.
To enable JobManager failover you have to set the jobmanager.failover.operation-log-store to filesystem and config the high-availability.storageDir to the path where you want to store the operation log. The path can be a local path or HDFS path according to the running mode of your cluster.
{% top %}