| :py:mod:`airflow.providers.amazon.aws.operators.datasync` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.amazon.aws.operators.datasync |
| |
| .. autoapi-nested-parse:: |
| |
| Create, get, update, execute and delete an AWS DataSync Task. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.operators.datasync.DataSyncOperator |
| airflow.providers.amazon.aws.operators.datasync.AWSDataSyncOperator |
| |
| |
| |
| |
| .. py:class:: DataSyncOperator(*, aws_conn_id = 'aws_default', wait_interval_seconds = 30, max_iterations = 60, task_arn = None, source_location_uri = None, destination_location_uri = None, allow_random_task_choice = False, allow_random_location_choice = False, create_task_kwargs = None, create_source_location_kwargs = None, create_destination_location_kwargs = None, update_task_kwargs = None, task_execution_kwargs = None, delete_task_after_execution = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Find, Create, Update, Execute and Delete AWS DataSync Tasks. |
| |
| If ``do_xcom_push`` is True, then the DataSync TaskArn and TaskExecutionArn |
| which were executed will be pushed to an XCom. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:DataSyncOperator` |
| |
| .. note:: There may be 0, 1, or many existing DataSync Tasks defined in your AWS |
| environment. The default behavior is to create a new Task if there are 0, or |
| execute the Task if there was 1 Task, or fail if there were many Tasks. |
| |
| :param aws_conn_id: AWS connection to use. |
| :param wait_interval_seconds: Time to wait between two |
| consecutive calls to check TaskExecution status. |
| :param max_iterations: Maximum number of |
| consecutive calls to check TaskExecution status. |
| :param task_arn: AWS DataSync TaskArn to use. If None, then this operator will |
| attempt to either search for an existing Task or attempt to create a new Task. |
| :param source_location_uri: Source location URI to search for. All DataSync |
| Tasks with a LocationArn with this URI will be considered. |
| Example: ``smb://server/subdir`` |
| :param destination_location_uri: Destination location URI to search for. |
| All DataSync Tasks with a LocationArn with this URI will be considered. |
| Example: ``s3://airflow_bucket/stuff`` |
| :param allow_random_task_choice: If multiple Tasks match, one must be chosen to |
| execute. If allow_random_task_choice is True then a random one is chosen. |
| :param allow_random_location_choice: If multiple Locations match, one must be chosen |
| when creating a task. If allow_random_location_choice is True then a random one is chosen. |
| :param create_task_kwargs: If no suitable TaskArn is identified, |
| it will be created if ``create_task_kwargs`` is defined. |
| ``create_task_kwargs`` is then used internally like this: |
| ``boto3.create_task(**create_task_kwargs)`` |
| Example: ``{'Name': 'xyz', 'Options': ..., 'Excludes': ..., 'Tags': ...}`` |
| :param create_source_location_kwargs: If no suitable LocationArn is found, |
| a Location will be created if ``create_source_location_kwargs`` is defined. |
| ``create_source_location_kwargs`` is then used internally like this: |
| ``boto3.create_location_xyz(**create_source_location_kwargs)`` |
| The xyz is determined from the prefix of source_location_uri, eg ``smb:/...`` or ``s3:/...`` |
| Example: ``{'Subdirectory': ..., 'ServerHostname': ..., ...}`` |
| :param create_destination_location_kwargs: If no suitable LocationArn is found, |
| a Location will be created if ``create_destination_location_kwargs`` is defined. |
| ``create_destination_location_kwargs`` is used internally like this: |
| ``boto3.create_location_xyz(**create_destination_location_kwargs)`` |
| The xyz is determined from the prefix of destination_location_uri, eg ``smb:/...` or ``s3:/...`` |
| Example: ``{'S3BucketArn': ..., 'S3Config': {'BucketAccessRoleArn': ...}, ...}`` |
| :param update_task_kwargs: If a suitable TaskArn is found or created, |
| it will be updated if ``update_task_kwargs`` is defined. |
| ``update_task_kwargs`` is used internally like this: |
| ``boto3.update_task(TaskArn=task_arn, **update_task_kwargs)`` |
| Example: ``{'Name': 'xyz', 'Options': ..., 'Excludes': ...}`` |
| :param task_execution_kwargs: Additional kwargs passed directly when starting the |
| Task execution, used internally like this: |
| ``boto3.start_task_execution(TaskArn=task_arn, **task_execution_kwargs)`` |
| :param delete_task_after_execution: If True then the TaskArn which was executed |
| will be deleted from AWS DataSync on successful completion. |
| :raises AirflowException: If ``task_arn`` was not specified, or if |
| either ``source_location_uri`` or ``destination_location_uri`` were |
| not specified. |
| :raises AirflowException: If source or destination Location were not found |
| and could not be created. |
| :raises AirflowException: If ``choose_task`` or ``choose_location`` fails. |
| :raises AirflowException: If Task creation, update, execution or delete fails. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['task_arn', 'source_location_uri', 'destination_location_uri', 'create_task_kwargs',... |
| |
| |
| |
| .. py:attribute:: template_fields_renderers |
| |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #44b5e2 |
| |
| |
| |
| .. py:method:: get_hook(self) |
| |
| Create and return DataSyncHook. |
| |
| :return DataSyncHook: An DataSyncHook instance. |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: choose_task(self, task_arn_list) |
| |
| Select 1 DataSync TaskArn from a list |
| |
| |
| .. py:method:: choose_location(self, location_arn_list) |
| |
| Select 1 DataSync LocationArn from a list |
| |
| |
| .. py:method:: on_kill(self) |
| |
| Override this method to cleanup subprocesses when a task instance |
| gets killed. Any use of the threading, subprocess or multiprocessing |
| module within an operator needs to be cleaned up or it will leave |
| ghost processes behind. |
| |
| |
| |
| .. py:class:: AWSDataSyncOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`DataSyncOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.datasync.DataSyncOperator`. |
| |
| |
