:py:mod:`airflow.providers.apache.beam.operators.beam`
======================================================

.. py:module:: airflow.providers.apache.beam.operators.beam

.. autoapi-nested-parse::

   This module contains Apache Beam operators.


Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   airflow.providers.apache.beam.operators.beam.BeamDataflowMixin
   airflow.providers.apache.beam.operators.beam.BeamBasePipelineOperator
   airflow.providers.apache.beam.operators.beam.BeamRunPythonPipelineOperator
   airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator
   airflow.providers.apache.beam.operators.beam.BeamRunGoPipelineOperator


.. py:class:: BeamDataflowMixin

   Helper class to store common, Dataflow specific logic for both
   :class:`~airflow.providers.apache.beam.operators.beam.BeamRunPythonPipelineOperator`,
   :class:`~airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator` and
   :class:`~airflow.providers.apache.beam.operators.beam.BeamRunGoPipelineOperator`.

   .. py:attribute:: dataflow_hook
      :annotation: :Optional[airflow.providers.google.cloud.hooks.dataflow.DataflowHook]

      
   .. py:attribute:: dataflow_config
      :annotation: :airflow.providers.google.cloud.operators.dataflow.DataflowConfiguration

      
   .. py:attribute:: gcp_conn_id
      :annotation: :str

      
   .. py:attribute:: delegate_to
      :annotation: :Optional[str]

      
.. py:class:: BeamBasePipelineOperator(*, runner = 'DirectRunner', default_pipeline_options = None, pipeline_options = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, dataflow_config = None, **kwargs)

   Bases: :py:obj:`airflow.models.BaseOperator`, :py:obj:`BeamDataflowMixin`, :py:obj:`abc.ABC`

   Abstract base class for Beam Pipeline Operators.

   :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
       Other possible options: DataflowRunner, SparkRunner, FlinkRunner, PortableRunner.
       See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
       See: https://beam.apache.org/documentation/runners/capability-matrix/

   :param default_pipeline_options: Map of default pipeline options.
   :param pipeline_options: Map of pipeline options.The key must be a dictionary.
       The value can contain different types:

       * If the value is None, the single option - ``--key`` (without value) will be added.
       * If the value is False, this option will be skipped
       * If the value is True, the single option - ``--key`` (without value) will be added.
       * If the value is list, the many options will be added for each key.
         If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key=B`` options
         will be left
       * Other value types will be replaced with the Python textual representation.

       When defining labels (labels option), you can also provide a dictionary.
   :param gcp_conn_id: Optional.
       The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
   :param delegate_to:  Optional.
       The account to impersonate using domain-wide delegation of authority,
       if any. For this to work, the service account making the request must have
       domain-wide delegation enabled.
   :param dataflow_config: Dataflow configuration, used when runner type is set to DataflowRunner,
       (optional) defaults to None.


.. py:class:: BeamRunPythonPipelineOperator(*, py_file, runner = 'DirectRunner', default_pipeline_options = None, pipeline_options = None, py_interpreter = 'python3', py_options = None, py_requirements = None, py_system_site_packages = False, gcp_conn_id = 'google_cloud_default', delegate_to = None, dataflow_config = None, **kwargs)

   Bases: :py:obj:`BeamBasePipelineOperator`

   Launching Apache Beam pipelines written in Python. Note that both
   ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
   execution parameter, and ``default_pipeline_options`` is expected to save
   high-level options, for instances, project and zone information, which
   apply to all beam operators in the DAG.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:BeamRunPythonPipelineOperator`

   .. seealso::
       For more detail on Apache Beam have a look at the reference:
       https://beam.apache.org/documentation/

   :param py_file: Reference to the python Apache Beam pipeline file.py, e.g.,
       /some/local/file/path/to/your/python/pipeline/file. (templated)
   :param py_options: Additional python options, e.g., ["-m", "-v"].
   :param py_interpreter: Python version of the beam pipeline.
       If None, this defaults to the python3.
       To track python versions supported by beam and related
       issues check: https://issues.apache.org/jira/browse/BEAM-1251
   :param py_requirements: Additional python package(s) to install.
       If a value is passed to this parameter, a new virtual environment has been created with
       additional packages installed.

       You could also install the apache_beam package if it is not installed on your system or you want
       to use a different version.
   :param py_system_site_packages: Whether to include system_site_packages in your virtualenv.
       See virtualenv documentation for more information.

       This option is only relevant if the ``py_requirements`` parameter is not None.

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['py_file', 'runner', 'pipeline_options', 'default_pipeline_options', 'dataflow_config']

      
   .. py:attribute:: template_fields_renderers
      

   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      Execute the Apache Beam Pipeline.


   .. py:method:: on_kill(self)

      Override this method to cleanup subprocesses when a task instance
      gets killed. Any use of the threading, subprocess or multiprocessing
      module within an operator needs to be cleaned up or it will leave
      ghost processes behind.


.. py:class:: BeamRunJavaPipelineOperator(*, jar, runner = 'DirectRunner', job_class = None, default_pipeline_options = None, pipeline_options = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, dataflow_config = None, **kwargs)

   Bases: :py:obj:`BeamBasePipelineOperator`

   Launching Apache Beam pipelines written in Java.

   Note that both
   ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
   execution parameter, and ``default_pipeline_options`` is expected to save
   high-level pipeline_options, for instances, project and zone information, which
   apply to all Apache Beam operators in the DAG.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:BeamRunJavaPipelineOperator`

   .. seealso::
       For more detail on Apache Beam have a look at the reference:
       https://beam.apache.org/documentation/

   You need to pass the path to your jar file as a file reference with the ``jar``
   parameter, the jar needs to be a self executing jar (see documentation here:
   https://beam.apache.org/documentation/runners/dataflow/#self-executing-jar).
   Use ``pipeline_options`` to pass on pipeline_options to your job.

   :param jar: The reference to a self executing Apache Beam jar (templated).
   :param job_class: The name of the Apache Beam pipeline class to be executed, it
       is often not the main class configured in the pipeline jar file.

   .. py:attribute:: template_fields
      :annotation: :Sequence[str] = ['jar', 'runner', 'job_class', 'pipeline_options', 'default_pipeline_options', 'dataflow_config']

      
   .. py:attribute:: template_fields_renderers
      

   .. py:attribute:: ui_color
      :annotation: = #0273d4

      
   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      Execute the Apache Beam Pipeline.


   .. py:method:: on_kill(self)

      Override this method to cleanup subprocesses when a task instance
      gets killed. Any use of the threading, subprocess or multiprocessing
      module within an operator needs to be cleaned up or it will leave
      ghost processes behind.


.. py:class:: BeamRunGoPipelineOperator(*, go_file, runner = 'DirectRunner', default_pipeline_options = None, pipeline_options = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, dataflow_config = None, **kwargs)

   Bases: :py:obj:`BeamBasePipelineOperator`

   Launching Apache Beam pipelines written in Go. Note that both
   ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
   execution parameter, and ``default_pipeline_options`` is expected to save
   high-level options, for instances, project and zone information, which
   apply to all beam operators in the DAG.

   .. seealso::
       For more information on how to use this operator, take a look at the guide:
       :ref:`howto/operator:BeamRunGoPipelineOperator`

   .. seealso::
       For more detail on Apache Beam have a look at the reference:
       https://beam.apache.org/documentation/

   :param go_file: Reference to the Go Apache Beam pipeline e.g.,
       /some/local/file/path/to/your/go/pipeline/file.go

   .. py:attribute:: template_fields
      :annotation: = ['go_file', 'runner', 'pipeline_options', 'default_pipeline_options', 'dataflow_config']

      
   .. py:attribute:: template_fields_renderers
      

   .. py:attribute:: operator_extra_links
      

   .. py:method:: execute(self, context)

      Execute the Apache Beam Pipeline.


   .. py:method:: on_kill(self)

      Override this method to cleanup subprocesses when a task instance
      gets killed. Any use of the threading, subprocess or multiprocessing
      module within an operator needs to be cleaned up or it will leave
      ghost processes behind.