| :py:mod:`airflow.models.mappedoperator` |
| ======================================= |
| |
| .. py:module:: airflow.models.mappedoperator |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.mappedoperator.OperatorPartial |
| airflow.models.mappedoperator.MappedOperator |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.mappedoperator.get_mappable_types |
| airflow.models.mappedoperator.validate_mapping_kwargs |
| airflow.models.mappedoperator.prevent_duplicates |
| airflow.models.mappedoperator.ensure_xcomarg_return_value |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.mappedoperator.Mappable |
| airflow.models.mappedoperator.ValidationSource |
| airflow.models.mappedoperator.MAPPABLE_LITERAL_TYPES |
| |
| |
| .. py:data:: Mappable |
| |
| |
| |
| |
| .. py:data:: ValidationSource |
| |
| |
| |
| |
| .. py:data:: MAPPABLE_LITERAL_TYPES |
| |
| |
| |
| |
| .. py:function:: get_mappable_types() |
| |
| |
| .. py:function:: validate_mapping_kwargs(op, func, value) |
| |
| |
| .. py:function:: prevent_duplicates(kwargs1, kwargs2, *, fail_reason) |
| |
| |
| .. py:function:: ensure_xcomarg_return_value(arg) |
| |
| |
| .. py:class:: OperatorPartial |
| |
| An "intermediate state" returned by ``BaseOperator.partial()``. |
| |
| This only exists at DAG-parsing time; the only intended usage is for the |
| user to call ``.expand()`` on it at some point (usually in a method chain) to |
| create a ``MappedOperator`` to add into the DAG. |
| |
| .. py:attribute:: operator_class |
| :annotation: :Type[airflow.models.baseoperator.BaseOperator] |
| |
| |
| |
| .. py:attribute:: kwargs |
| :annotation: :Dict[str, Any] |
| |
| |
| |
| .. py:method:: __attrs_post_init__(self) |
| |
| |
| .. py:method:: __repr__(self) |
| |
| Return repr(self). |
| |
| |
| .. py:method:: __del__(self) |
| |
| |
| .. py:method:: expand(self, **mapped_kwargs) |
| |
| |
| |
| .. py:class:: MappedOperator(context=None) |
| |
| Bases: :py:obj:`airflow.models.abstractoperator.AbstractOperator` |
| |
| Object representing a mapped operator in a DAG. |
| |
| .. py:attribute:: operator_class |
| :annotation: :Union[Type[airflow.models.baseoperator.BaseOperator], Dict[str, Any]] |
| |
| |
| |
| .. py:attribute:: mapped_kwargs |
| :annotation: :Dict[str, Mappable] |
| |
| |
| |
| .. py:attribute:: partial_kwargs |
| :annotation: :Dict[str, Any] |
| |
| |
| |
| .. py:attribute:: task_id |
| :annotation: :str |
| |
| |
| |
| .. py:attribute:: params |
| :annotation: :Optional[dict] |
| |
| |
| |
| .. py:attribute:: deps |
| :annotation: :FrozenSet[airflow.ti_deps.deps.base_ti_dep.BaseTIDep] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| :annotation: :Collection[airflow.models.baseoperator.BaseOperatorLink] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] |
| |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Collection[str] |
| |
| |
| |
| .. py:attribute:: template_fields_renderers |
| :annotation: :Dict[str, str] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: :str |
| |
| |
| |
| .. py:attribute:: ui_fgcolor |
| :annotation: :str |
| |
| |
| |
| .. py:attribute:: dag |
| :annotation: :Optional[airflow.models.dag.DAG] |
| |
| |
| |
| .. py:attribute:: task_group |
| :annotation: :Optional[airflow.utils.task_group.TaskGroup] |
| |
| |
| |
| .. py:attribute:: start_date |
| :annotation: :Optional[pendulum.DateTime] |
| |
| |
| |
| .. py:attribute:: end_date |
| :annotation: :Optional[pendulum.DateTime] |
| |
| |
| |
| .. py:attribute:: upstream_task_ids |
| :annotation: :Set[str] |
| |
| |
| |
| .. py:attribute:: downstream_task_ids |
| :annotation: :Set[str] |
| |
| |
| |
| .. py:attribute:: is_mapped |
| :annotation: :ClassVar[bool] = True |
| |
| |
| |
| .. py:attribute:: subdag |
| :annotation: :None |
| |
| |
| |
| .. py:attribute:: HIDE_ATTRS_FROM_UI |
| :annotation: :ClassVar[FrozenSet[str]] |
| |
| |
| |
| .. py:method:: __hash__(self) |
| |
| Return hash(self). |
| |
| |
| .. py:method:: __repr__(self) |
| |
| Return repr(self). |
| |
| |
| .. py:method:: __attrs_post_init__(self) |
| |
| |
| .. py:method:: get_serialized_fields(cls) |
| :classmethod: |
| |
| |
| .. py:method:: deps_for(operator_class) |
| :staticmethod: |
| |
| |
| .. py:method:: task_type(self) |
| :property: |
| |
| Implementing Operator. |
| |
| |
| .. py:method:: inherits_from_empty_operator(self) |
| :property: |
| |
| Implementing Operator. |
| |
| |
| .. py:method:: roots(self) |
| :property: |
| |
| Implementing DAGNode. |
| |
| |
| .. py:method:: leaves(self) |
| :property: |
| |
| Implementing DAGNode. |
| |
| |
| .. py:method:: owner(self) |
| :property: |
| |
| |
| .. py:method:: email(self) |
| :property: |
| |
| |
| .. py:method:: trigger_rule(self) |
| :property: |
| |
| |
| .. py:method:: depends_on_past(self) |
| :property: |
| |
| |
| .. py:method:: wait_for_downstream(self) |
| :property: |
| |
| |
| .. py:method:: retries(self) |
| :property: |
| |
| |
| .. py:method:: queue(self) |
| :property: |
| |
| |
| .. py:method:: pool(self) |
| :property: |
| |
| |
| .. py:method:: pool_slots(self) |
| :property: |
| |
| |
| .. py:method:: execution_timeout(self) |
| :property: |
| |
| |
| .. py:method:: retry_delay(self) |
| :property: |
| |
| |
| .. py:method:: retry_exponential_backoff(self) |
| :property: |
| |
| |
| .. py:method:: priority_weight(self) |
| :property: |
| |
| |
| .. py:method:: weight_rule(self) |
| :property: |
| |
| |
| .. py:method:: sla(self) |
| :property: |
| |
| |
| .. py:method:: max_active_tis_per_dag(self) |
| :property: |
| |
| |
| .. py:method:: resources(self) |
| :property: |
| |
| |
| .. py:method:: on_execute_callback(self) |
| :property: |
| |
| |
| .. py:method:: on_failure_callback(self) |
| :property: |
| |
| |
| .. py:method:: on_retry_callback(self) |
| :property: |
| |
| |
| .. py:method:: on_success_callback(self) |
| :property: |
| |
| |
| .. py:method:: run_as_user(self) |
| :property: |
| |
| |
| .. py:method:: executor_config(self) |
| :property: |
| |
| |
| .. py:method:: inlets(self) |
| :property: |
| |
| |
| .. py:method:: outlets(self) |
| :property: |
| |
| |
| .. py:method:: get_dag(self) |
| |
| Implementing Operator. |
| |
| |
| .. py:method:: serialize_for_task_group(self) |
| |
| Implementing DAGNode. |
| |
| |
| .. py:method:: unmap(self, unmap_kwargs = None) |
| |
| Get the "normal" Operator after applying the current mapping. |
| |
| If ``operator_class`` is not a class (i.e. this DAG has been deserialized) then this will return a |
| SerializedBaseOperator that aims to "look like" the real operator. |
| |
| :param unmap_kwargs: Override the args to pass to the Operator constructor. Only used when |
| ``operator_class`` is still an actual class. |
| |
| :meta private: |
| |
| |
| .. py:method:: expand_mapped_task(self, run_id, *, session) |
| |
| Create the mapped task instances for mapped task. |
| |
| :return: The newly created mapped TaskInstances (if any) in ascending order by map index, and the |
| maximum map_index. |
| |
| |
| .. py:method:: prepare_for_execution(self) |
| |
| |
| .. py:method:: render_template_fields(self, context, jinja_env = None) |
| |
| Template all attributes listed in template_fields. |
| |
| Different from the BaseOperator implementation, this renders the |
| template fields on the *unmapped* BaseOperator. |
| |
| :param context: Dict with values to apply on content |
| :param jinja_env: Jinja environment |
| :return: The unmapped, populated BaseOperator |
| |
| |
| .. py:method:: iter_mapped_dependencies(self) |
| |
| Upstream dependencies that provide XComs used by this task for task mapping. |
| |
| |
| .. py:method:: parse_time_mapped_ti_count(self) |
| |
| Number of mapped TaskInstances that can be created at DagRun create time. |
| |
| :return: None if non-literal mapped arg encountered, or else total number of mapped TIs this task |
| should have |
| |
| |
| .. py:method:: run_time_mapped_ti_count(self, run_id, *, session) |
| |
| Number of mapped TaskInstances that can be created at run time, or None if upstream tasks are not |
| complete yet. |
| |
| :return: None if upstream tasks are not complete yet, or else total number of mapped TIs this task |
| should have |
| |
| |
| |