| |
| |
| :mod:`airflow.contrib.executors.mesos_executor` |
| =============================================== |
| |
| .. py:module:: airflow.contrib.executors.mesos_executor |
| |
| |
| |
| |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| |
| |
| |
| |
| |
| .. data:: DEFAULT_FRAMEWORK_NAME |
| :annotation: = Airflow |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. data:: FRAMEWORK_CONNID_PREFIX |
| :annotation: = mesos_framework_ |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. function:: get_framework_name() |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. py:class:: AirflowMesosScheduler(task_queue, result_queue, task_cpu=1, task_mem=256) |
| |
| Bases::class:`mesos.interface.Scheduler` |
| |
| |
| |
| Airflow Mesos scheduler implements mesos scheduler interface |
| to schedule airflow tasks on mesos. |
| Basically, it schedules a command like |
| 'airflow run <dag_id> <task_instance_id> <start_date> --local -p=<pickle>' |
| to run on a mesos slave. |
| |
| |
| |
| |
| |
| |
| |
| |
| .. method:: registered(self, driver, frameworkId, masterInfo) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: reregistered(self, driver, masterInfo) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: disconnected(self, driver) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: offerRescinded(self, driver, offerId) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: frameworkMessage(self, driver, executorId, slaveId, message) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: executorLost(self, driver, executorId, slaveId, status) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: slaveLost(self, driver, slaveId) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: error(self, driver, message) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: resourceOffers(self, driver, offers) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: statusUpdate(self, driver, update) |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| .. py:class:: MesosExecutor |
| |
| Bases::class:`airflow.executors.base_executor.BaseExecutor` |
| |
| |
| |
| MesosExecutor allows distributing the execution of task |
| instances to multiple mesos workers. |
| |
| Apache Mesos is a distributed systems kernel which abstracts |
| CPU, memory, storage, and other compute resources away from |
| machines (physical or virtual), enabling fault-tolerant and |
| elastic distributed systems to easily be built and run effectively. |
| See http://mesos.apache.org/ |
| |
| |
| |
| |
| |
| |
| .. method:: start(self) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: execute_async(self, key, command, queue=None, executor_config=None) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: sync(self) |
| |
| |
| |
| |
| |
| |
| |
| .. method:: end(self) |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
