blob: 64eb31277a6280b4951c111ff0ea816ef8c52000 [file] [log] [blame]
:mod:`airflow.contrib.operators.awsbatch_operator`
==================================================
.. py:module:: airflow.contrib.operators.awsbatch_operator
Module Contents
---------------
.. py:class:: AWSBatchOperator(job_name, job_definition, job_queue, overrides, array_properties=None, max_retries=4200, aws_conn_id=None, region_name=None, **kwargs)
Bases: :class:`airflow.models.BaseOperator`
Execute a job on AWS Batch Service
.. warning: the queue parameter was renamed to job_queue to segregate the
internal CeleryExecutor queue from the AWS Batch internal queue.
:param job_name: the name for the job that will run on AWS Batch (templated)
:type job_name: str
:param job_definition: the job definition name on AWS Batch
:type job_definition: str
:param job_queue: the queue name on AWS Batch
:type job_queue: str
:param overrides: the same parameter that boto3 will receive on
containerOverrides (templated):
http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job
:type overrides: dict
:param array_properties: the same parameter that boto3 will receive on
arrayProperties:
http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job
:type array_properties: dict
:param max_retries: exponential backoff retries while waiter is not
merged, 4200 = 48 hours
:type max_retries: int
:param aws_conn_id: connection id of AWS credentials / region name. If None,
credential boto3 strategy will be used
(http://boto3.readthedocs.io/en/latest/guide/configuration.html).
:type aws_conn_id: str
:param region_name: region name to use in AWS Hook.
Override the region_name in connection (if provided)
:type region_name: str
.. attribute:: ui_color
:annotation: = #c3dae0
.. attribute:: client
.. attribute:: arn
.. attribute:: template_fields
:annotation: = ['job_name', 'overrides']
.. method:: execute(self, context)
.. method:: _wait_for_task_ended(self)
Try to use a waiter from the below pull request
* https://github.com/boto/botocore/pull/1307
If the waiter is not available apply a exponential backoff
* docs.aws.amazon.com/general/latest/gr/api-retries.html
.. method:: _check_success_task(self)
.. method:: get_hook(self)
.. method:: on_kill(self)