| :py:mod:`airflow.executors.celery_kubernetes_executor` |
| ====================================================== |
| |
| .. py:module:: airflow.executors.celery_kubernetes_executor |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.executors.celery_kubernetes_executor.CeleryKubernetesExecutor |
| |
| |
| |
| |
| .. py:class:: CeleryKubernetesExecutor(celery_executor, kubernetes_executor) |
| |
| Bases: :py:obj:`airflow.utils.log.logging_mixin.LoggingMixin` |
| |
| CeleryKubernetesExecutor consists of CeleryExecutor and KubernetesExecutor. |
| It chooses an executor to use based on the queue defined on the task. |
| When the queue is the value of ``kubernetes_queue`` in section ``[celery_kubernetes_executor]`` |
| of the configuration (default value: `kubernetes`), KubernetesExecutor is selected to run the task, |
| otherwise, CeleryExecutor is used. |
| |
| .. py:attribute:: supports_ad_hoc_ti_run |
| :annotation: :bool = True |
| |
| |
| |
| .. py:attribute:: callback_sink |
| :annotation: :Optional[airflow.callbacks.base_callback_sink.BaseCallbackSink] |
| |
| |
| |
| .. py:attribute:: KUBERNETES_QUEUE |
| |
| |
| |
| |
| .. py:method:: queued_tasks(self) |
| :property: |
| |
| Return queued tasks from celery and kubernetes executor |
| |
| |
| .. py:method:: running(self) |
| :property: |
| |
| Return running tasks from celery and kubernetes executor |
| |
| |
| .. py:method:: job_id(self) |
| :property: |
| |
| This is a class attribute in BaseExecutor but since this is not really an executor, but a wrapper |
| of executors we implement as property so we can have custom setter. |
| |
| |
| .. py:method:: start(self) |
| |
| Start celery and kubernetes executor |
| |
| |
| .. py:method:: slots_available(self) |
| :property: |
| |
| Number of new tasks this executor instance can accept |
| |
| |
| .. py:method:: queue_command(self, task_instance, command, priority = 1, queue = None) |
| |
| Queues command via celery or kubernetes executor |
| |
| |
| .. py:method:: queue_task_instance(self, task_instance, mark_success = False, pickle_id = None, ignore_all_deps = False, ignore_depends_on_past = False, ignore_task_deps = False, ignore_ti_state = False, pool = None, cfg_path = None) |
| |
| Queues task instance via celery or kubernetes executor |
| |
| |
| .. py:method:: has_task(self, task_instance) |
| |
| Checks if a task is either queued or running in either celery or kubernetes executor. |
| |
| :param task_instance: TaskInstance |
| :return: True if the task is known to this executor |
| |
| |
| .. py:method:: heartbeat(self) |
| |
| Heartbeat sent to trigger new jobs in celery and kubernetes executor |
| |
| |
| .. py:method:: get_event_buffer(self, dag_ids = None) |
| |
| Returns and flush the event buffer from celery and kubernetes executor |
| |
| :param dag_ids: dag_ids to return events for, if None returns all |
| :return: a dict of events |
| |
| |
| .. py:method:: try_adopt_task_instances(self, tis) |
| |
| Try to adopt running task instances that have been abandoned by a SchedulerJob dying. |
| |
| Anything that is not adopted will be cleared by the scheduler (and then become eligible for |
| re-scheduling) |
| |
| :return: any TaskInstances that were unable to be adopted |
| :rtype: list[airflow.models.TaskInstance] |
| |
| |
| .. py:method:: end(self) |
| |
| End celery and kubernetes executor |
| |
| |
| .. py:method:: terminate(self) |
| |
| Terminate celery and kubernetes executor |
| |
| |
| .. py:method:: debug_dump(self) |
| |
| Called in response to SIGUSR2 by the scheduler |
| |
| |
| .. py:method:: send_callback(self, request) |
| |
| Sends callback for execution. |
| |
| :param request: Callback request to be executed. |
| |
| |
| |