docs-archive/1.10.5/_sources/_api/airflow/executors/local_executor/index.rst.txt

:mod:`airflow.executors.local_executor`
=======================================

.. py:module:: airflow.executors.local_executor

.. autoapi-nested-parse::

   LocalExecutor runs tasks by spawning processes in a controlled fashion in different
   modes. Given that BaseExecutor has the option to receive a `parallelism` parameter to
   limit the number of process spawned, when this parameter is `0` the number of processes
   that LocalExecutor can spawn is unlimited.

   The following strategies are implemented:
   1. Unlimited Parallelism (self.parallelism == 0): In this strategy, LocalExecutor will
   spawn a process every time `execute_async` is called, that is, every task submitted to the
   LocalExecutor will be executed in its own process. Once the task is executed and the
   result stored in the `result_queue`, the process terminates. There is no need for a
   `task_queue` in this approach, since as soon as a task is received a new process will be
   allocated to the task. Processes used in this strategy are of class LocalWorker.

   2. Limited Parallelism (self.parallelism > 0): In this strategy, the LocalExecutor spawns
   the number of processes equal to the value of `self.parallelism` at `start` time,
   using a `task_queue` to coordinate the ingestion of tasks and the work distribution among
   the workers, which will take a task as soon as they are ready. During the lifecycle of
   the LocalExecutor, the worker processes are running waiting for tasks, once the
   LocalExecutor receives the call to shutdown the executor a poison token is sent to the
   workers to terminate them. Processes used in this strategy are of class QueuedLocalWorker.

   Arguably, `SequentialExecutor` could be thought as a LocalExecutor with limited
   parallelism of just 1 worker, i.e. `self.parallelism = 1`.
   This option could lead to the unification of the executor implementations, running
   locally, into just one `LocalExecutor` with multiple modes.



Module Contents
---------------

.. py:class:: LocalWorker(result_queue)

   Bases: :class:`multiprocessing.Process`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`

   LocalWorker Process implementation to run airflow commands. Executes the given
   command and puts the result into a result queue when done, terminating execution.

   
   .. method:: execute_work(self, key, command)

      Executes command received and stores result state in queue.
      :param key: the key to identify the TI
      :type key: tuple(dag_id, task_id, execution_date)
      :param command: the command to execute
      :type command: str



   
   .. method:: run(self)




.. py:class:: QueuedLocalWorker(task_queue, result_queue)

   Bases: :class:`airflow.executors.local_executor.LocalWorker`

   LocalWorker implementation that is waiting for tasks from a queue and will
   continue executing commands as they become available in the queue. It will terminate
   execution once the poison token is found.

   
   .. method:: run(self)




.. py:class:: LocalExecutor

   Bases: :class:`airflow.executors.base_executor.BaseExecutor`

   LocalExecutor executes tasks locally in parallel. It uses the
   multiprocessing Python library and queues to parallelize the execution
   of tasks.

   .. py:class:: _UnlimitedParallelism(executor)

      Bases: :class:`object`

      Implements LocalExecutor with unlimited parallelism, starting one process
      per each command to execute.

      
      .. method:: start(self)



      
      .. method:: execute_async(self, key, command)

         :param key: the key to identify the TI
         :type key: tuple(dag_id, task_id, execution_date)
         :param command: the command to execute
         :type command: str



      
      .. method:: sync(self)



      
      .. method:: end(self)




   .. py:class:: _LimitedParallelism(executor)

      Bases: :class:`object`

      Implements LocalExecutor with limited parallelism using a task queue to
      coordinate work distribution.

      
      .. method:: start(self)



      
      .. method:: execute_async(self, key, command)

         :param key: the key to identify the TI
         :type key: tuple(dag_id, task_id, execution_date)
         :param command: the command to execute
         :type command: str



      
      .. method:: sync(self)



      
      .. method:: end(self)




   
   .. method:: start(self)



   
   .. method:: execute_async(self, key, command, queue=None, executor_config=None)



   
   .. method:: sync(self)



   
   .. method:: end(self)