| :py:mod:`airflow.providers.amazon.aws.operators.ecs` |
| ==================================================== |
| |
| .. py:module:: airflow.providers.amazon.aws.operators.ecs |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.operators.ecs.EcsProtocol |
| airflow.providers.amazon.aws.operators.ecs.EcsTaskLogFetcher |
| airflow.providers.amazon.aws.operators.ecs.EcsOperator |
| airflow.providers.amazon.aws.operators.ecs.ECSOperator |
| airflow.providers.amazon.aws.operators.ecs.ECSTaskLogFetcher |
| airflow.providers.amazon.aws.operators.ecs.ECSProtocol |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.operators.ecs.should_retry |
| airflow.providers.amazon.aws.operators.ecs.should_retry_eni |
| |
| |
| |
| .. py:function:: should_retry(exception) |
| |
| Check if exception is related to ECS resource quota (CPU, MEM). |
| |
| |
| .. py:function:: should_retry_eni(exception) |
| |
| Check if exception is related to ENI (Elastic Network Interfaces). |
| |
| |
| .. py:class:: EcsProtocol |
| |
| Bases: :py:obj:`airflow.typing_compat.Protocol` |
| |
| A structured Protocol for ``boto3.client('ecs')``. This is used for type hints on |
| :py:meth:`.EcsOperator.client`. |
| |
| .. seealso:: |
| |
| - https://mypy.readthedocs.io/en/latest/protocols.html |
| - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html |
| |
| .. py:method:: run_task(self, **kwargs) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task |
| |
| |
| .. py:method:: get_waiter(self, x) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.get_waiter |
| |
| |
| .. py:method:: describe_tasks(self, cluster, tasks) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.describe_tasks |
| |
| |
| .. py:method:: stop_task(self, cluster, task, reason) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.stop_task |
| |
| |
| .. py:method:: describe_task_definition(self, taskDefinition) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.describe_task_definition |
| |
| |
| .. py:method:: list_tasks(self, cluster, launchType, desiredStatus, family) |
| |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.list_tasks |
| |
| |
| |
| .. py:class:: EcsTaskLogFetcher(*, aws_conn_id = 'aws_default', region_name = None, log_group, log_stream_name, fetch_interval, logger) |
| |
| Bases: :py:obj:`threading.Thread` |
| |
| Fetches Cloudwatch log events with specific interval as a thread |
| and sends the log events to the info channel of the provided logger. |
| |
| .. py:method:: run(self) |
| |
| Method representing the thread's activity. |
| |
| You may override this method in a subclass. The standard run() method |
| invokes the callable object passed to the object's constructor as the |
| target argument, if any, with sequential and keyword arguments taken |
| from the args and kwargs arguments, respectively. |
| |
| |
| |
| .. py:method:: get_last_log_messages(self, number_messages) |
| |
| |
| .. py:method:: get_last_log_message(self) |
| |
| |
| .. py:method:: is_stopped(self) |
| |
| |
| .. py:method:: stop(self) |
| |
| |
| |
| .. py:class:: EcsOperator(*, task_definition, cluster, overrides, aws_conn_id = None, region_name = None, launch_type = 'EC2', capacity_provider_strategy = None, group = None, placement_constraints = None, placement_strategy = None, platform_version = None, network_configuration = None, tags = None, awslogs_group = None, awslogs_region = None, awslogs_stream_prefix = None, awslogs_fetch_interval = timedelta(seconds=30), propagate_tags = None, quota_retry = None, reattach = False, number_logs_exception = 10, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Execute a task on AWS ECS (Elastic Container Service) |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EcsOperator` |
| |
| :param task_definition: the task definition name on Elastic Container Service |
| :param cluster: the cluster name on Elastic Container Service |
| :param overrides: the same parameter that boto3 will receive (templated): |
| https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task |
| :param aws_conn_id: connection id of AWS credentials / region name. If None, |
| credential boto3 strategy will be used |
| (http://boto3.readthedocs.io/en/latest/guide/configuration.html). |
| :param region_name: region name to use in AWS Hook. |
| Override the region_name in connection (if provided) |
| :param launch_type: the launch type on which to run your task ('EC2', 'EXTERNAL', or 'FARGATE') |
| :param capacity_provider_strategy: the capacity provider strategy to use for the task. |
| When capacity_provider_strategy is specified, the launch_type parameter is omitted. |
| If no capacity_provider_strategy or launch_type is specified, |
| the default capacity provider strategy for the cluster is used. |
| :param group: the name of the task group associated with the task |
| :param placement_constraints: an array of placement constraint objects to use for |
| the task |
| :param placement_strategy: an array of placement strategy objects to use for |
| the task |
| :param platform_version: the platform version on which your task is running |
| :param network_configuration: the network configuration for the task |
| :param tags: a dictionary of tags in the form of {'tagKey': 'tagValue'}. |
| :param awslogs_group: the CloudWatch group where your ECS container logs are stored. |
| Only required if you want logs to be shown in the Airflow UI after your job has |
| finished. |
| :param awslogs_region: the region in which your CloudWatch logs are stored. |
| If None, this is the same as the `region_name` parameter. If that is also None, |
| this is the default AWS region based on your connection settings. |
| :param awslogs_stream_prefix: the stream prefix that is used for the CloudWatch logs. |
| This is usually based on some custom name combined with the name of the container. |
| Only required if you want logs to be shown in the Airflow UI after your job has |
| finished. |
| :param awslogs_fetch_interval: the interval that the ECS task log fetcher should wait |
| in between each Cloudwatch logs fetches. |
| :param quota_retry: Config if and how to retry the launch of a new ECS task, to handle |
| transient errors. |
| :param reattach: If set to True, will check if the task previously launched by the task_instance |
| is already running. If so, the operator will attach to it instead of starting a new task. |
| This is to avoid relaunching a new task when the connection drops between Airflow and ECS while |
| the task is running (when the Airflow worker is restarted for example). |
| :param number_logs_exception: Number of lines from the last Cloudwatch logs to return in the |
| AirflowException if an ECS task is stopped (to receive Airflow alerts with the logs of what |
| failed in the code running in ECS). |
| |
| .. py:attribute:: ui_color |
| :annotation: = #f0ede4 |
| |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['overrides'] |
| |
| |
| |
| .. py:attribute:: template_fields_renderers |
| |
| |
| |
| |
| .. py:attribute:: REATTACH_XCOM_KEY |
| :annotation: = ecs_task_arn |
| |
| |
| |
| .. py:attribute:: REATTACH_XCOM_TASK_ID_TEMPLATE |
| :annotation: = {task_id}_task_arn |
| |
| |
| |
| .. py:method:: execute(self, context, session=None) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: get_hook(self) |
| |
| Create and return an AwsHook. |
| |
| |
| .. py:method:: on_kill(self) |
| |
| Override this method to cleanup subprocesses when a task instance |
| gets killed. Any use of the threading, subprocess or multiprocessing |
| module within an operator needs to be cleaned up or it will leave |
| ghost processes behind. |
| |
| |
| |
| .. py:class:: ECSOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EcsOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.ecs.EcsOperator`. |
| |
| |
| .. py:class:: ECSTaskLogFetcher(*args, **kwargs) |
| |
| Bases: :py:obj:`EcsTaskLogFetcher` |
| |
| This class is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.ecs.EcsTaskLogFetcher`. |
| |
| |
| .. py:class:: ECSProtocol |
| |
| Bases: :py:obj:`EcsProtocol` |
| |
| This class is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.ecs.EcsProtocol`. |
| |
| |
