emr executor

The emr executor allows you to run tasks on AWS EMR cluster.

  - executor: my_emr_cluster
    type: emr
    cluser_name: liminal-cluster

attributes

cluster_name: the name of the cluster uses by the executor

OR

cluster_id: the unique identifier of the cluster used by the executor.

aws_conn_id: a reference to the emr connection. default: aws_default

cluster_states: the cluster state filters to apply when searching for an existing cluster. default: ['RUNNING', 'WAITING']

properties: any attribute of aws-resource-emr-step-properties (Optional)

supported task types

• spark
• sql