| :py:mod:`airflow.providers.amazon.aws.operators.eks` |
| ==================================================== |
| |
| .. py:module:: airflow.providers.amazon.aws.operators.eks |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Amazon EKS operators. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.operators.eks.EksCreateClusterOperator |
| airflow.providers.amazon.aws.operators.eks.EksCreateNodegroupOperator |
| airflow.providers.amazon.aws.operators.eks.EksCreateFargateProfileOperator |
| airflow.providers.amazon.aws.operators.eks.EksDeleteClusterOperator |
| airflow.providers.amazon.aws.operators.eks.EksDeleteNodegroupOperator |
| airflow.providers.amazon.aws.operators.eks.EksDeleteFargateProfileOperator |
| airflow.providers.amazon.aws.operators.eks.EksPodOperator |
| airflow.providers.amazon.aws.operators.eks.EKSCreateClusterOperator |
| airflow.providers.amazon.aws.operators.eks.EKSCreateNodegroupOperator |
| airflow.providers.amazon.aws.operators.eks.EKSCreateFargateProfileOperator |
| airflow.providers.amazon.aws.operators.eks.EKSDeleteClusterOperator |
| airflow.providers.amazon.aws.operators.eks.EKSDeleteNodegroupOperator |
| airflow.providers.amazon.aws.operators.eks.EKSDeleteFargateProfileOperator |
| airflow.providers.amazon.aws.operators.eks.EKSPodOperator |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.amazon.aws.operators.eks.CHECK_INTERVAL_SECONDS |
| airflow.providers.amazon.aws.operators.eks.TIMEOUT_SECONDS |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_COMPUTE_TYPE |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_CONN_ID |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_FARGATE_PROFILE_NAME |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_NAMESPACE_NAME |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_NODEGROUP_NAME |
| airflow.providers.amazon.aws.operators.eks.DEFAULT_POD_NAME |
| airflow.providers.amazon.aws.operators.eks.ABORT_MSG |
| airflow.providers.amazon.aws.operators.eks.CAN_NOT_DELETE_MSG |
| airflow.providers.amazon.aws.operators.eks.MISSING_ARN_MSG |
| airflow.providers.amazon.aws.operators.eks.SUCCESS_MSG |
| airflow.providers.amazon.aws.operators.eks.SUPPORTED_COMPUTE_VALUES |
| airflow.providers.amazon.aws.operators.eks.NODEGROUP_FULL_NAME |
| airflow.providers.amazon.aws.operators.eks.FARGATE_FULL_NAME |
| |
| |
| .. py:data:: CHECK_INTERVAL_SECONDS |
| :annotation: = 15 |
| |
| |
| |
| .. py:data:: TIMEOUT_SECONDS |
| |
| |
| |
| |
| .. py:data:: DEFAULT_COMPUTE_TYPE |
| :annotation: = nodegroup |
| |
| |
| |
| .. py:data:: DEFAULT_CONN_ID |
| :annotation: = aws_default |
| |
| |
| |
| .. py:data:: DEFAULT_FARGATE_PROFILE_NAME |
| :annotation: = profile |
| |
| |
| |
| .. py:data:: DEFAULT_NAMESPACE_NAME |
| :annotation: = default |
| |
| |
| |
| .. py:data:: DEFAULT_NODEGROUP_NAME |
| :annotation: = nodegroup |
| |
| |
| |
| .. py:data:: DEFAULT_POD_NAME |
| :annotation: = pod |
| |
| |
| |
| .. py:data:: ABORT_MSG |
| :annotation: = {compute} are still active after the allocated time limit. Aborting. |
| |
| |
| |
| .. py:data:: CAN_NOT_DELETE_MSG |
| :annotation: = A cluster can not be deleted with attached {compute}. Deleting {count} {compute}. |
| |
| |
| |
| .. py:data:: MISSING_ARN_MSG |
| :annotation: = Creating an {compute} requires {requirement} to be passed in. |
| |
| |
| |
| .. py:data:: SUCCESS_MSG |
| :annotation: = No {compute} remain, deleting cluster. |
| |
| |
| |
| .. py:data:: SUPPORTED_COMPUTE_VALUES |
| |
| |
| |
| |
| .. py:data:: NODEGROUP_FULL_NAME |
| :annotation: = Amazon EKS managed node groups |
| |
| |
| |
| .. py:data:: FARGATE_FULL_NAME |
| :annotation: = AWS Fargate profiles |
| |
| |
| |
| .. py:class:: EksCreateClusterOperator(cluster_name, cluster_role_arn, resources_vpc_config, compute = DEFAULT_COMPUTE_TYPE, create_cluster_kwargs = None, nodegroup_name = DEFAULT_NODEGROUP_NAME, nodegroup_role_arn = None, create_nodegroup_kwargs = None, fargate_profile_name = DEFAULT_FARGATE_PROFILE_NAME, fargate_pod_execution_role_arn = None, fargate_selectors = None, create_fargate_profile_kwargs = None, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates an Amazon EKS Cluster control plane. |
| |
| Optionally, can also create the supporting compute architecture: |
| |
| - If argument 'compute' is provided with a value of 'nodegroup', will also |
| attempt to create an Amazon EKS Managed Nodegroup for the cluster. |
| See :class:`~airflow.providers.amazon.aws.operators.EksCreateNodegroupOperator` |
| documentation for requirements. |
| |
| - If argument 'compute' is provided with a value of 'fargate', will also attempt to create an AWS |
| Fargate profile for the cluster. |
| See :class:`~airflow.providers.amazon.aws.operators.EksCreateFargateProfileOperator` |
| documentation for requirements. |
| |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksCreateClusterOperator` |
| |
| :param cluster_name: The unique name to give to your Amazon EKS Cluster. (templated) |
| :param cluster_role_arn: The Amazon Resource Name (ARN) of the IAM role that provides permissions for the |
| Kubernetes control plane to make calls to AWS API operations on your behalf. (templated) |
| :param resources_vpc_config: The VPC configuration used by the cluster control plane. (templated) |
| :param compute: The type of compute architecture to generate along with the cluster. (templated) |
| Defaults to 'nodegroup' to generate an EKS Managed Nodegroup. |
| :param create_cluster_kwargs: Optional parameters to pass to the CreateCluster API (templated) |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| If compute is assigned the value of 'nodegroup': |
| |
| :param nodegroup_name: *REQUIRED* The unique name to give your Amazon EKS managed node group. (templated) |
| :param nodegroup_role_arn: *REQUIRED* The Amazon Resource Name (ARN) of the IAM role to associate with |
| the Amazon EKS managed node group. (templated) |
| :param create_nodegroup_kwargs: Optional parameters to pass to the CreateNodegroup API (templated) |
| |
| |
| If compute is assigned the value of 'fargate': |
| |
| :param fargate_profile_name: *REQUIRED* The unique name to give your AWS Fargate profile. (templated) |
| :param fargate_pod_execution_role_arn: *REQUIRED* The Amazon Resource Name (ARN) of the pod execution |
| role to use for pods that match the selectors in the AWS Fargate profile. (templated) |
| :param fargate_selectors: The selectors to match for pods to use this AWS Fargate profile. (templated) |
| :param create_fargate_profile_kwargs: Optional parameters to pass to the CreateFargateProfile API |
| (templated) |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'cluster_role_arn', 'resources_vpc_config', 'create_cluster_kwargs', 'compute',... |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EksCreateNodegroupOperator(cluster_name, nodegroup_subnets, nodegroup_role_arn, nodegroup_name = DEFAULT_NODEGROUP_NAME, create_nodegroup_kwargs = None, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates an Amazon EKS managed node group for an existing Amazon EKS Cluster. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksCreateNodegroupOperator` |
| |
| :param cluster_name: The name of the Amazon EKS Cluster to create the managed nodegroup in. (templated) |
| :param nodegroup_name: The unique name to give your managed nodegroup. (templated) |
| :param nodegroup_subnets: |
| The subnets to use for the Auto Scaling group that is created for the managed nodegroup. (templated) |
| :param nodegroup_role_arn: |
| The Amazon Resource Name (ARN) of the IAM role to associate with the managed nodegroup. (templated) |
| :param create_nodegroup_kwargs: Optional parameters to pass to the Create Nodegroup API (templated) |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'nodegroup_subnets', 'nodegroup_role_arn', 'nodegroup_name',... |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EksCreateFargateProfileOperator(cluster_name, pod_execution_role_arn, selectors, fargate_profile_name = DEFAULT_FARGATE_PROFILE_NAME, create_fargate_profile_kwargs = None, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates an AWS Fargate profile for an Amazon EKS cluster. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksCreateFargateProfileOperator` |
| |
| :param cluster_name: The name of the Amazon EKS cluster to apply the AWS Fargate profile to. (templated) |
| :param pod_execution_role_arn: The Amazon Resource Name (ARN) of the pod execution role to |
| use for pods that match the selectors in the AWS Fargate profile. (templated) |
| :param selectors: The selectors to match for pods to use this AWS Fargate profile. (templated) |
| :param fargate_profile_name: The unique name to give your AWS Fargate profile. (templated) |
| :param create_fargate_profile_kwargs: Optional parameters to pass to the CreateFargate Profile API |
| (templated) |
| |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'pod_execution_role_arn', 'selectors', 'fargate_profile_name',... |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EksDeleteClusterOperator(cluster_name, force_delete_compute = False, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes the Amazon EKS Cluster control plane and all nodegroups attached to it. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksDeleteClusterOperator` |
| |
| :param cluster_name: The name of the Amazon EKS Cluster to delete. (templated) |
| :param force_delete_compute: If True, will delete any attached resources. (templated) |
| Defaults to False. |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'force_delete_compute', 'aws_conn_id', 'region'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: delete_any_nodegroups(self, eks_hook) |
| |
| Deletes all Amazon EKS managed node groups for a provided Amazon EKS Cluster. |
| |
| Amazon EKS managed node groups can be deleted in parallel, so we can send all |
| of the delete commands in bulk and move on once the count of nodegroups is zero. |
| |
| |
| .. py:method:: delete_any_fargate_profiles(self, eks_hook) |
| |
| Deletes all EKS Fargate profiles for a provided Amazon EKS Cluster. |
| |
| EKS Fargate profiles must be deleted one at a time, so we must wait |
| for one to be deleted before sending the next delete command. |
| |
| |
| |
| .. py:class:: EksDeleteNodegroupOperator(cluster_name, nodegroup_name, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes an Amazon EKS managed node group from an Amazon EKS Cluster. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksDeleteNodegroupOperator` |
| |
| :param cluster_name: The name of the Amazon EKS Cluster associated with your nodegroup. (templated) |
| :param nodegroup_name: The name of the nodegroup to delete. (templated) |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'nodegroup_name', 'aws_conn_id', 'region'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EksDeleteFargateProfileOperator(cluster_name, fargate_profile_name, aws_conn_id = DEFAULT_CONN_ID, region = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes an AWS Fargate profile from an Amazon EKS Cluster. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksDeleteFargateProfileOperator` |
| |
| :param cluster_name: The name of the Amazon EKS cluster associated with your Fargate profile. (templated) |
| :param fargate_profile_name: The name of the AWS Fargate profile to delete. (templated) |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['cluster_name', 'fargate_profile_name', 'aws_conn_id', 'region'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EksPodOperator(cluster_name, in_cluster = False, namespace = DEFAULT_NAMESPACE_NAME, pod_context = None, pod_name = None, pod_username = None, aws_conn_id = DEFAULT_CONN_ID, region = None, is_delete_operator_pod = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator` |
| |
| Executes a task in a Kubernetes pod on the specified Amazon EKS Cluster. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:EksPodOperator` |
| |
| :param cluster_name: The name of the Amazon EKS Cluster to execute the task on. (templated) |
| :param cluster_role_arn: The Amazon Resource Name (ARN) of the IAM role that provides permissions |
| for the Kubernetes control plane to make calls to AWS API operations on your behalf. (templated) |
| :param in_cluster: If True, look for config inside the cluster; if False look for a local file path. |
| :param namespace: The namespace in which to execute the pod. (templated) |
| :param pod_name: The unique name to give the pod. (templated) |
| :param aws_profile: The named profile containing the credentials for the AWS CLI tool to use. |
| :param aws_profile: str |
| :param region: Which AWS region the connection should use. (templated) |
| If this is None or empty then the default boto3 behaviour is used. |
| :param aws_conn_id: The Airflow connection used for AWS credentials. (templated) |
| If this is None or empty then the default boto3 behaviour is used. If |
| running Airflow in a distributed manner and aws_conn_id is None or |
| empty, then the default boto3 configuration would be used (and must be |
| maintained on each worker node). |
| :param is_delete_operator_pod: What to do when the pod reaches its final |
| state, or the execution is interrupted. If True, delete the |
| pod; if False, leave the pod. Current default is False, but this will be |
| changed in the next major release of this provider. |
| |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: EKSCreateClusterOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksCreateClusterOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksCreateClusterOperator`. |
| |
| |
| .. py:class:: EKSCreateNodegroupOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksCreateNodegroupOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksCreateNodegroupOperator`. |
| |
| |
| .. py:class:: EKSCreateFargateProfileOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksCreateFargateProfileOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksCreateFargateProfileOperator`. |
| |
| |
| .. py:class:: EKSDeleteClusterOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksDeleteClusterOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksDeleteClusterOperator`. |
| |
| |
| .. py:class:: EKSDeleteNodegroupOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksDeleteNodegroupOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksDeleteNodegroupOperator`. |
| |
| |
| .. py:class:: EKSDeleteFargateProfileOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksDeleteFargateProfileOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksDeleteFargateProfileOperator`. |
| |
| |
| .. py:class:: EKSPodOperator(*args, **kwargs) |
| |
| Bases: :py:obj:`EksPodOperator` |
| |
| This operator is deprecated. |
| Please use :class:`airflow.providers.amazon.aws.operators.eks.EksPodOperator`. |
| |
| |
