| :py:mod:`airflow.providers.microsoft.azure.operators.batch` |
| =========================================================== |
| |
| .. py:module:: airflow.providers.microsoft.azure.operators.batch |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.microsoft.azure.operators.batch.AzureBatchOperator |
| |
| |
| |
| |
| .. py:class:: AzureBatchOperator(*, batch_pool_id, batch_pool_vm_size, batch_job_id, batch_task_command_line, batch_task_id, vm_publisher = None, vm_offer = None, sku_starts_with = None, vm_sku = None, vm_version = None, vm_node_agent_sku_id = None, os_family = None, os_version = None, batch_pool_display_name = None, batch_job_display_name = None, batch_job_manager_task = None, batch_job_preparation_task = None, batch_job_release_task = None, batch_task_display_name = None, batch_task_container_settings = None, batch_start_task = None, batch_max_retries = 3, batch_task_resource_files = None, batch_task_output_files = None, batch_task_user_identity = None, target_low_priority_nodes = None, target_dedicated_nodes = None, enable_auto_scale = False, auto_scale_formula = None, azure_batch_conn_id='azure_batch_default', use_latest_verified_vm_image_and_sku = False, timeout = 25, should_delete_job = False, should_delete_pool = False, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Executes a job on Azure Batch Service |
| |
| :param batch_pool_id: A string that uniquely identifies the Pool within the Account. |
| :param batch_pool_vm_size: The size of virtual machines in the Pool |
| :param batch_job_id: A string that uniquely identifies the Job within the Account. |
| :param batch_task_command_line: The command line of the Task |
| :param batch_task_id: A string that uniquely identifies the task within the Job. |
| :param batch_pool_display_name: The display name for the Pool. |
| The display name need not be unique |
| :param batch_job_display_name: The display name for the Job. |
| The display name need not be unique |
| :param batch_job_manager_task: Details of a Job Manager Task to be launched when the Job is started. |
| :param batch_job_preparation_task: The Job Preparation Task. If set, the Batch service will |
| run the Job Preparation Task on a Node before starting any Tasks of that |
| Job on that Compute Node. Required if batch_job_release_task is set. |
| :param batch_job_release_task: The Job Release Task. Use to undo changes to Compute Nodes |
| made by the Job Preparation Task |
| :param batch_task_display_name: The display name for the task. |
| The display name need not be unique |
| :param batch_task_container_settings: The settings for the container under which the Task runs |
| :param batch_start_task: A Task specified to run on each Compute Node as it joins the Pool. |
| The Task runs when the Compute Node is added to the Pool or |
| when the Compute Node is restarted. |
| :param batch_max_retries: The number of times to retry this batch operation before it's |
| considered a failed operation. Default is 3 |
| :param batch_task_resource_files: A list of files that the Batch service will |
| download to the Compute Node before running the command line. |
| :param batch_task_output_files: A list of files that the Batch service will upload |
| from the Compute Node after running the command line. |
| :param batch_task_user_identity: The user identity under which the Task runs. |
| If omitted, the Task runs as a non-administrative user unique to the Task. |
| :param target_low_priority_nodes: The desired number of low-priority Compute Nodes in the Pool. |
| This property must not be specified if enable_auto_scale is set to true. |
| :param target_dedicated_nodes: The desired number of dedicated Compute Nodes in the Pool. |
| This property must not be specified if enable_auto_scale is set to true. |
| :param enable_auto_scale: Whether the Pool size should automatically adjust over time. Default is false |
| :param auto_scale_formula: A formula for the desired number of Compute Nodes in the Pool. |
| This property must not be specified if enableAutoScale is set to false. |
| It is required if enableAutoScale is set to true. |
| :param azure_batch_conn_id: The :ref:`Azure Batch connection id<howto/connection:azure_batch>` |
| :param use_latest_verified_vm_image_and_sku: Whether to use the latest verified virtual |
| machine image and sku in the batch account. Default is false. |
| :param vm_publisher: The publisher of the Azure Virtual Machines Marketplace Image. |
| For example, Canonical or MicrosoftWindowsServer. Required if |
| use_latest_image_and_sku is set to True |
| :param vm_offer: The offer type of the Azure Virtual Machines Marketplace Image. |
| For example, UbuntuServer or WindowsServer. Required if |
| use_latest_image_and_sku is set to True |
| :param sku_starts_with: The starting string of the Virtual Machine SKU. Required if |
| use_latest_image_and_sku is set to True |
| :param vm_sku: The name of the virtual machine sku to use |
| :param vm_version: The version of the virtual machine |
| :param vm_version: Optional[str] |
| :param vm_node_agent_sku_id: The node agent sku id of the virtual machine |
| :param os_family: The Azure Guest OS family to be installed on the virtual machines in the Pool. |
| :param os_version: The OS family version |
| :param timeout: The amount of time to wait for the job to complete in minutes. Default is 25 |
| :param should_delete_job: Whether to delete job after execution. Default is False |
| :param should_delete_pool: Whether to delete pool after execution of jobs. Default is False |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['batch_pool_id', 'batch_pool_vm_size', 'batch_job_id', 'batch_task_id', 'batch_task_command_line'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #f0f0e4 |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: on_kill(self) |
| |
| Override this method to cleanup subprocesses when a task instance |
| gets killed. Any use of the threading, subprocess or multiprocessing |
| module within an operator needs to be cleaned up or it will leave |
| ghost processes behind. |
| |
| |
| .. py:method:: get_hook(self) |
| |
| Create and return an AzureBatchHook. |
| |
| |
| .. py:method:: clean_up(self, pool_id = None, job_id = None) |
| |
| Delete the given pool and job in the batch account |
| |
| :param pool_id: The id of the pool to delete |
| :param job_id: The id of the job to delete |
| |
| |
| |
| |
