tree: c4fd239ad44e098d4f4653e76dadd47d95191217 [path history] [tgz]
  1. __init__.py
  2. appflow.json
  3. athena.json
  4. base_waiter.py
  5. batch.json
  6. databrew.json
  7. dynamodb.json
  8. ecs.json
  9. eks.json
  10. emr-containers.json
  11. emr-serverless.json
  12. emr.json
  13. glue.json
  14. neptune.json
  15. README.md
  16. redshift.json
  17. sagemaker.json
  18. stepfunctions.json
airflow/providers/amazon/aws/waiters/README.md

This module is for custom Boto3 waiter configuration files. Since documentation on creating custom waiters is pretty sparse out in the wild, this document can act as a rough quickstart guide. It is not meant to cover all edge cases.

To add a new custom waiter

Create or modify the service waiter config file

Find or create a file for the service it is related to, for example waiters/eks.json

In the service waiter config file

Build or add to the waiter model config json in that file. For examples of what these should look like, have a look through some official waiter models. Some examples:

Below is an example of a working waiter model config that will make an EKS waiter which will wait for all Nodegroups in a cluster to be deleted. An explanation follows the code snippet. Note the backticks to escape the integers in the “argument” values.

{
    "version": 2,
    "waiters": {
        "all_nodegroups_deleted": {
            "operation": "ListNodegroups",
            "delay": 30,
            "maxAttempts": 60,
            "acceptors": [
                {
                    "matcher": "path",
                    "argument": "length(nodegroups[]) == `0`",
                    "expected": true,
                    "state": "success"
                },
                {
                    "matcher": "path",
                    "expected": true,
                    "argument": "length(nodegroups[]) > `0`",
                    "state": "retry"
                }
            ]
        }
    }
}

In the model config above we create a new waiter called all_nodegroups_deleted which calls the ListNodegroups API endpoint. The parameters for the endpoint call must be passed into the waiter.wait() call, the same as when using an official waiter. The waiter then performs “argument” (in this case len(result) == 0) on the result. If the argument returns the value in “expected” (in this case True) then the waiter's state is set to success, the waiter can close down, and the operator which called it can continue. If len(result) > 0 is True then the state is set to retry. The waiter will “delay” 30 seconds before trying again. If the state does not go to success before the maxAttempts number of tries, the waiter raises a WaiterException. Both retry and maxAttempts can be overridden by the user when calling waiter.wait() like any other waiter.

That's It!

The AwsBaseHook handles the rest. Using the above waiter will look like this: EksHook().get_waiter("all_nodegroups_deleted").wait(clusterName="my_cluster") and for testing purposes, a list_custom_waiters() helper method is proved which can be used the same way: EksHook().list_custom_waiters()

In your Operators (How to use these)

Once configured correctly, the custom waiter will be nearly indistinguishable from an official waiter. Below is an example of an official waiter followed by a custom one.

EksHook().conn.get_waiter("nodegroup_deleted").wait(clusterName=cluster_name, nodegroupName=nodegroup_name)
EksHook().get_waiter("all_nodegroups_deleted").wait(clusterName=cluster_name)

Note that since the get_waiter is in the hook instead of on the client side, a custom waiter is just hook.get_waiter and not hook.conn.get_waiter. Other than that, they should be identical.

Note the custom waiter doesn't work with resource_type, only client_type is supported.