docs-archive/1.10.13/_sources/_api/airflow/executors/celery_executor/index.rst.txt

:mod:`airflow.executors.celery_executor`
========================================

.. py:module:: airflow.executors.celery_executor

.. autoapi-nested-parse::

   CeleryExecutor

   .. seealso::
       For more information on how the CeleryExecutor works, take a look at the guide:
       :ref:`executor:CeleryExecutor`



Module Contents
---------------

.. data:: log
   

   

.. data:: CELERY_FETCH_ERR_MSG_HEADER
   :annotation: = Error fetching Celery task state

   

.. data:: CELERY_SEND_ERR_MSG_HEADER
   :annotation: = Error sending Celery task

   

.. data:: OPERATION_TIMEOUT
   

   To start the celery worker, run the command:
   airflow worker


.. data:: celery_configuration
   

   

.. data:: app
   

   

.. function:: execute_command(command_to_exec)
   Executes command.


.. py:class:: ExceptionWithTraceback(exception, exception_traceback)

   Bases: :class:`object`

   Wrapper class used to propagate exceptions to parent processes from subprocesses.

   :param exception: The exception to wrap
   :type exception: Exception
   :param exception_traceback: The stacktrace to wrap
   :type exception_traceback: str


.. function:: fetch_celery_task_state(celery_task)
   Fetch and return the state of the given celery task. The scope of this function is
   global so that it can be called by subprocesses in the pool.

   :param celery_task: a tuple of the Celery task key and the async Celery object used
       to fetch the task's state
   :type celery_task: tuple(str, celery.result.AsyncResult)
   :return: a tuple of the Celery task key and the Celery state of the task
   :rtype: tuple[str, str]


.. function:: send_task_to_executor(task_tuple)

.. py:class:: CeleryExecutor

   Bases: :class:`airflow.executors.base_executor.BaseExecutor`

   CeleryExecutor is recommended for production use of Airflow. It allows
   distributing the execution of task instances to multiple worker nodes.

   Celery is a simple, flexible and reliable distributed system to process
   vast amounts of messages, while providing operations with the tools
   required to maintain such a system.

   
   .. method:: start(self)



   
   .. method:: _num_tasks_per_send_process(self, to_send_count)

      How many Celery tasks should each worker process send.

      :return: Number of tasks that should be sent per process
      :rtype: int



   
   .. method:: _num_tasks_per_fetch_process(self)

      How many Celery tasks should be sent to each worker process.

      :return: Number of tasks that should be used per process
      :rtype: int



   
   .. method:: trigger_tasks(self, open_slots)

      Overwrite trigger_tasks function from BaseExecutor

      :param open_slots: Number of open slots
      :return:



   
   .. method:: sync(self)



   
   .. method:: end(self, synchronous=False)