Updates to enable launch of EC2 instances via user-defined templates (#274)
* Updates to enable launch of EC2 instances that use
EBS-backed (non-ephemeral) data storage, etc, via
user-defined EC2 request templates
* - Removed redundant MinCount/MaxCount updates from
init_request method in Ec2Cluster base class
- Updated muchos.props.example to improve documentation
for the new cluster_template variable
* PR feedback addressed:
- Added conf/templates/user/.gitignore, removed
conf/templates/.gitignore
- Renamed .devices file to 'devices' and updated README
Other updates:
- Renamed 'cluster_template' dict field to 'cluster_template_d'
for consistency with other dictionaries in the class
- Removed 'cluster_template_id' field, added that info to
the dictionary instead
- Fixed typo(s) in README.md
* README.md fix
diff --git a/conf/muchos.props.example b/conf/muchos.props.example
index 9932c40..599c812 100644
--- a/conf/muchos.props.example
+++ b/conf/muchos.props.example
@@ -62,6 +62,9 @@
# Type of AWS instance launched for any node running 'worker' service
# Leave default below to use same instance type set by 'default_instance_type' property
worker_instance_type = %(default_instance_type)s
+# Enable template mode by selecting a template from conf/templates, in order to leverage your own
+# custom EC2 launch requests (optional). See conf/templates/README.md for more information
+#cluster_template = example
# VPC to launch instances in (optional)
#vpc_id = vpc-xxxxx
# VPC Subnet to launch instances in (optional)
diff --git a/conf/templates/README.md b/conf/templates/README.md
new file mode 100644
index 0000000..235d5be
--- /dev/null
+++ b/conf/templates/README.md
@@ -0,0 +1,188 @@
+## Muchos EC2 Cluster Templates
+
+Cluster templates are intended to provide greater flexibility, if needed,
+with respect to instance type selection and launch configuration for
+your EC2 hosts. For example, cluster templates may be ideal for use
+cases that require distinct, per-host launch configurations, and
+for use cases that require hosts to have persistent, EBS-backed data
+volumes (rather than ephemeral volumes, the muchos default for EC2
+clusters)
+
+If you are already familiar with muchos and with the basics of EC2
+launch requests, then creating your own launch templates will be simple
+and straightforward.
+
+Please follow the guidance provided here to ensure compatibility between
+your custom templates and muchos automation
+
+## Configuration
+
+### Select a cluster template in *muchos.props*
+```
+[ec2]
+...
+cluster_template = example
+...
+```
+The configured value must match the name of a subdirectory under
+`conf/templates`
+
+```
+~$ ls -1a fluo-muchos/conf/templates/example
+.
+..
+devices
+metrics.json
+namenode.json
+resourcemanager.json
+zookeeper.json
+worker.json
+```
+The subdirectory will contain one or more user-defined EC2 launch
+templates (*\*.json*) for your various host types, and it will
+include a *devices* file specifying the desired mount points for all
+data volumes (excluding root volumes, as they are mounted
+automatically)
+
+### Defining EC2 launch templates and device mounts for your hosts
+
+#### Launch Templates: *{service-name}.json* files
+
+Each JSON file represents a standard EC2 launch request, and each file
+name must match one of the predefined muchos service names, as
+defined in the **nodes** section of *muchos.props*. E.g.,
+```
+...
+[nodes]
+leader1 = namenode,resourcemanager,accumulomaster
+leader2 = metrics,zookeeper
+worker1 = worker
+worker2 = worker
+worker3 = worker
+worker4 = worker
+```
+In template mode, the first service listed for a given host denotes the
+template to be selected for its launch.
+
+Based on the example given above:
+* **leader1** selects **namenode.json**
+* **leader2** selects **metrics.json**
+* **worker1** selects **worker.json**
+* and so on...
+
+For example, *namenode.json* might be defined as follows...
+```
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1", # Here, /dev/sda1 denotes the root volume
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf", # Here, /dev/sdf (a.k.a "/dev/xvdf") denotes
+ "Ebs": { # our single EBS-backed data volume
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.4xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
+```
+*Property Placeholders*
+
+The `${property name}` placeholders demonstrated above are optional and
+are intended to simplify template creation and reduce maintenance burden.
+They allow any matching properties from the **ec2** section of *muchos.props*
+to be interpolated automatically.
+
+If needed, you may also define your own custom properties and have them be injected
+automatically by simply adding them to the **ec2** section of
+*muchos.props* and to your templates
+
+#### Device Mounts: *devices* file
+
+The *devices* file contains the user-defined mapping of storage
+devices and mount points for all data (i.e., non-root) volumes in your
+cluster.
+
+Two (and only two) device mappings should exist within *devices*:
+* One map to represent your **worker** device mounts, and
+* One map to represent the device mounts on all other hosts, i.e., the
+ **default** map
+
+For example, the *devices* file below specifies 4 mount points for all
+`worker` instance types, and specifies 1 mount point for all
+other hosts via the `default` map.
+```
+{
+ "default": {
+ "mounts": [
+ "/data0" # For non-workers, mount the /data0 directory
+ ], # on /dev/xvdf (a.k.a "/dev/sdf")
+ "devices": [
+ "/dev/xvdf"
+ ]
+ },
+ "worker": {
+ "mounts": [
+ "/data0",
+ "/data1",
+ "/data2",
+ "/data3" # For workers, mount the /data0 directory on
+ ], # /dev/xvdf (a.k.a "/dev/sdf"), mount /data1 on
+ "devices": [ # /dev/xvdg (a.k.a "/dev/sdg"), and so on...
+ "/dev/xvdf",
+ "/dev/xvdg",
+ "/dev/xvdh",
+ "/dev/xvdi"
+ ]
+ }
+}
+```
+Naturally, you should take care to ensure that your **BlockDeviceMappings**
+also align to *default* vs *worker* node type semantics. As you
+explore the example files, you should observe the implicit link betweeen
+a data volume denoted by *BlockDeviceMappings\[N].DeviceName* and its
+respective device map entry in the *devices* file.
+
+* **Note**: While *DeviceName* mount profiles should not vary among
+ your default (non-worker) nodes, other attributes within *BlockDeviceMappings*,
+ such as *DeleteOnTermination*, *VolumeSize*, *etc*, may vary as needed
+
+* **Note**: Be aware that the device names used by AWS EC2 launch requests
+ may differ from the actual names assigned by the EC2 block device driver, which
+ can be confusing. Please read
+ [this](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html)
+ and [also this](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html)
+ for additional information and guidance, if needed
+
+* **Note**: Root EBS volumes may be configured as desired in your JSON
+ launch templates, but only non-root, "data" storage devices should be
+ specified in *devices*, as root devices are mounted automatically
+
+
+## Beyond the Launch Phase: *Setup*, *Terminate*, *Etc*
+
+Aside from the configuration differences described above, which impact
+the `launch` phase, all other muchos operations behave the same in
+template mode as in default mode
+
diff --git a/conf/templates/example/accumulomaster.json b/conf/templates/example/accumulomaster.json
new file mode 100644
index 0000000..263c192
--- /dev/null
+++ b/conf/templates/example/accumulomaster.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.2xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/client.json b/conf/templates/example/client.json
new file mode 100644
index 0000000..c5d4de1
--- /dev/null
+++ b/conf/templates/example/client.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.2xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/devices b/conf/templates/example/devices
new file mode 100644
index 0000000..f338f9f
--- /dev/null
+++ b/conf/templates/example/devices
@@ -0,0 +1,24 @@
+{
+ "default": {
+ "mounts": [
+ "/data0"
+ ],
+ "devices": [
+ "/dev/xvdf"
+ ]
+ },
+ "worker": {
+ "mounts": [
+ "/data0",
+ "/data1",
+ "/data2",
+ "/data3"
+ ],
+ "devices": [
+ "/dev/xvdf",
+ "/dev/xvdg",
+ "/dev/xvdh",
+ "/dev/xvdi"
+ ]
+ }
+}
\ No newline at end of file
diff --git a/conf/templates/example/metrics.json b/conf/templates/example/metrics.json
new file mode 100644
index 0000000..c5d4de1
--- /dev/null
+++ b/conf/templates/example/metrics.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.2xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/namenode.json b/conf/templates/example/namenode.json
new file mode 100644
index 0000000..b281782
--- /dev/null
+++ b/conf/templates/example/namenode.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.4xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/resourcemanager.json b/conf/templates/example/resourcemanager.json
new file mode 100644
index 0000000..a406feb
--- /dev/null
+++ b/conf/templates/example/resourcemanager.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.2xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/worker.json b/conf/templates/example/worker.json
new file mode 100644
index 0000000..ae693ee
--- /dev/null
+++ b/conf/templates/example/worker.json
@@ -0,0 +1,58 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 250,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdg",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 250,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdh",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 250,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdi",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 250,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "c4.4xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/example/zookeeper.json b/conf/templates/example/zookeeper.json
new file mode 100644
index 0000000..16ae67e
--- /dev/null
+++ b/conf/templates/example/zookeeper.json
@@ -0,0 +1,34 @@
+{
+ "KeyName": "${key_name}",
+ "BlockDeviceMappings": [
+ {
+ "DeviceName": "/dev/sda1",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 40,
+ "VolumeType": "gp2"
+ }
+ },
+ {
+ "DeviceName": "/dev/sdf",
+ "Ebs": {
+ "DeleteOnTermination": true,
+ "VolumeSize": 500,
+ "VolumeType": "gp2"
+ }
+ }
+ ],
+ "ImageId": "${aws_ami}",
+ "InstanceType": "m4.xlarge",
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": ${associate_public_ip},
+ "Groups": [
+ "${security_group_id}"
+ ]
+ }
+ ],
+ "EbsOptimized": true,
+ "InstanceInitiatedShutdownBehavior": "${shutdown_behavior}"
+}
diff --git a/conf/templates/user/.gitignore b/conf/templates/user/.gitignore
new file mode 100644
index 0000000..f59ec20
--- /dev/null
+++ b/conf/templates/user/.gitignore
@@ -0,0 +1 @@
+*
\ No newline at end of file
diff --git a/lib/main.py b/lib/main.py
index 383c242..62df08c 100644
--- a/lib/main.py
+++ b/lib/main.py
@@ -49,7 +49,9 @@
hosts_path = join(hosts_dir, opts.cluster)
- config = DeployConfig(deploy_path, config_path, hosts_path, checksums_path, opts.cluster)
+ templates_path = join(deploy_path, "conf/templates/")
+
+ config = DeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, opts.cluster)
config.verify_config(action)
if action == 'config':
@@ -64,8 +66,11 @@
cluster = ExistingCluster(config)
cluster.perform(action)
elif cluster_type == 'ec2':
- from muchos.ec2 import Ec2Cluster
- cluster = Ec2Cluster(config)
+ from muchos.ec2 import Ec2Cluster, Ec2ClusterTemplate
+ if config.has_option('ec2', 'cluster_template'):
+ cluster = Ec2ClusterTemplate(config)
+ else:
+ cluster = Ec2Cluster(config)
cluster.perform(action)
else:
exit('Unknown cluster_type: ' + cluster_type)
diff --git a/lib/muchos/config.py b/lib/muchos/config.py
index b0a3fe5..3b07c27 100644
--- a/lib/muchos/config.py
+++ b/lib/muchos/config.py
@@ -19,6 +19,8 @@
from sys import exit
from .util import get_ephemeral_devices, get_arch
import os
+import json
+import glob
SERVICES = ['zookeeper', 'namenode', 'resourcemanager', 'accumulomaster', 'mesosmaster', 'worker', 'fluo', 'fluo_yarn', 'metrics', 'spark', 'client', 'swarmmanager']
@@ -27,7 +29,7 @@
class DeployConfig(ConfigParser):
- def __init__(self, deploy_path, config_path, hosts_path, checksums_path, cluster_name):
+ def __init__(self, deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
ConfigParser.__init__(self)
self.optionxform = str
self.deploy_path = deploy_path
@@ -43,6 +45,8 @@
self.checksums_path = checksums_path
self.checksums_d = None
self.init_nodes()
+ self.cluster_template_d = None
+ self.init_template(templates_path)
def verify_config(self, action):
proxy = self.get('general', 'proxy_hostname')
@@ -86,6 +90,9 @@
return max((len(self.default_ephemeral_devices()), len(self.worker_ephemeral_devices())))
def node_type_map(self):
+ if self.cluster_template_d:
+ return self.cluster_template_d['devices']
+
node_types = {}
if self.get_cluster_type() == 'ec2':
node_list = [('default', self.default_ephemeral_devices()), ('worker', self.worker_ephemeral_devices())]
@@ -189,9 +196,10 @@
return self.checksums_d[key]
def verify_instance_type(self, instance_type):
- if get_arch(instance_type) == 'pvm':
- exit("ERROR - Configuration contains instance type '{0}' that uses pvm architecture."
- "Only hvm architecture is supported!".format(instance_type))
+ if not self.cluster_template_d:
+ if get_arch(instance_type) == 'pvm':
+ exit("ERROR - Configuration contains instance type '{0}' that uses pvm architecture."
+ "Only hvm architecture is supported!".format(instance_type))
def instance_tags(self):
retd = {}
@@ -272,8 +280,6 @@
else:
exit('ERROR - Bad line %s in hosts %s' % (line, self.hosts_path))
-
-
def get_hosts(self):
if self.hosts is None:
self.parse_hosts()
@@ -332,6 +338,77 @@
return
exit("Property '{0}' was not found".format(key))
+ def init_template(self, templates_path):
+ if not self.has_option('ec2', 'cluster_template'):
+ return
+
+ template_id = self.get('ec2', 'cluster_template')
+ for root, dirs, files in os.walk(templates_path):
+ for dir_name in dirs:
+ if dir_name == template_id:
+ self.cluster_template_d = {'id': template_id}
+ template_dir = os.path.join(root, dir_name)
+ self.load_template_ec2_requests(template_dir)
+ self.load_template_device_map(template_dir)
+ break
+ break
+
+ self.validate_template()
+
+ def load_template_ec2_requests(self, template_dir):
+ for json_path in glob.glob(os.path.join(template_dir, '*.json')):
+ service = os.path.basename(json_path).rsplit('.', 1)[0]
+ if service not in SERVICES:
+ exit("ERROR - Template '{0}' has unrecognized option '{1}'. Must be one of {2}".format(
+ self.cluster_template_d['id'], service, str(SERVICES)))
+ with open(json_path, 'r') as json_file:
+ # load as string, so we can use string.Template to inject config values
+ self.cluster_template_d[service] = json_file.read()
+
+ def load_template_device_map(self, template_dir):
+ device_map_path = os.path.join(template_dir, 'devices')
+ if not os.path.isfile(device_map_path):
+ exit("ERROR - template '{0}' is missing 'devices' config".format(self.cluster_template_d['id']))
+ with open(device_map_path, 'r') as json_file:
+ self.cluster_template_d['devices'] = json.load(json_file)
+
+ def validate_template(self):
+ if not self.cluster_template_d:
+ exit("ERROR - Template '{0}' is not defined!".format(self.cluster_template_d['id']))
+
+ if 'worker' not in self.cluster_template_d:
+ exit("ERROR - '{0}' template config is invalid. No 'worker' launch request is defined".format(
+ self.cluster_template_d['id']))
+
+ if 'worker' not in self.cluster_template_d['devices']:
+ exit("ERROR - '{0}' template is invalid. The devices file must have a 'worker' device map".format(
+ self.cluster_template_d['id']))
+
+ if 'default' not in self.cluster_template_d['devices']:
+ exit("ERROR - '{0}' template is invalid. The devices file must have a 'default' device map".format(
+ self.cluster_template_d['id']))
+
+ # Validate the selected launch template for each host
+
+ worker_count = 0
+ for hostname in self.node_d:
+ # first service listed denotes the selected template
+ selected_ec2_request = self.node_d[hostname][0]
+ if 'worker' == selected_ec2_request:
+ worker_count = worker_count + 1
+ else:
+ if 'worker' in self.node_d[hostname]:
+ exit("ERROR - '{0}' node config is invalid. The 'worker' service should be listed first".format(
+ hostname))
+ if selected_ec2_request not in self.cluster_template_d:
+ if len(self.node_d[hostname]) > 1:
+ print('Hint: In template mode, the first service listed for a host denotes its EC2 template')
+ exit("ERROR - '{0}' node config is invalid. No EC2 template defined for the '{1}' service".format(
+ hostname, selected_ec2_request))
+
+ if worker_count == 0:
+ exit("ERROR - No worker instances are defined for template '{0}'".format(self.cluster_template_d['id']))
+
HOST_VAR_DEFAULTS = {
'accumulo_home': '"{{ install_dir }}/accumulo-{{ accumulo_version }}"',
diff --git a/lib/muchos/ec2.py b/lib/muchos/ec2.py
index 57a354f..2973d87 100644
--- a/lib/muchos/ec2.py
+++ b/lib/muchos/ec2.py
@@ -24,6 +24,8 @@
import time
import boto3
from .existing import ExistingCluster
+import json
+from string import Template
class Ec2Cluster(ExistingCluster):
@@ -33,40 +35,17 @@
def launch_node(self, hostname, services, sg_id):
- associate_public_ip = True
- if self.config.has_option('ec2', 'associate_public_ip'):
- associate_public_ip = self.config.get('ec2', 'associate_public_ip').strip().lower() == 'true'
+ request = self.init_request(hostname, services, sg_id)
- request = {'MinCount': 1, 'MaxCount': 1,
- 'NetworkInterfaces': [{'DeviceIndex': 0, 'AssociatePublicIpAddress': associate_public_ip,
- 'Groups': [sg_id]}]}
-
- if self.config.has_option('ec2', 'subnet_id'):
- request['NetworkInterfaces'][0]['SubnetId'] = self.config.get('ec2', 'subnet_id')
-
- if 'worker' in services:
- instance_type = self.config.get('ec2', 'worker_instance_type')
- else:
- instance_type = self.config.get('ec2', 'default_instance_type')
- request['InstanceType'] = instance_type
- request['InstanceInitiatedShutdownBehavior'] = self.config.get('ec2', 'shutdown_behavior')
-
- if not self.config.has_option('ec2', 'aws_ami'):
- exit('aws_ami property must be set!')
- image_id = self.config.get('ec2', 'aws_ami')
- if not image_id:
- exit('aws_ami property was not properly')
-
- request['ImageId'] = image_id
- request['BlockDeviceMappings'] = get_block_device_map(instance_type)
-
- if self.config.has_option('ec2', 'key_name'):
- request['KeyName'] = self.config.get('ec2', 'key_name')
+ request['MinCount'] = 1
+ request['MaxCount'] = 1
tags = [{'Key': 'Name', 'Value': self.config.cluster_name + '-' + hostname},
{'Key': 'Muchos', 'Value': self.config.cluster_name}]
+
for key, val in self.config.instance_tags().items():
tags.append({'Key': key, 'Value': val})
+
request['TagSpecifications'] = [{'ResourceType': 'instance', 'Tags': tags}]
if self.config.has_option('ec2', 'user_data_path'):
@@ -85,7 +64,7 @@
if response is None or len(response['Instances']) != 1:
exit('ERROR - Failed to start {0} node'.format(hostname))
- print('Launching {0} node using {1}'.format(hostname, image_id))
+ print('Launching {0} node using {1}'.format(hostname, request['ImageId']))
return response['Instances'][0]
def create_security_group(self):
@@ -140,6 +119,38 @@
time.sleep(10)
print("Deleted security group")
+ def init_request(self, hostname, services, sg_id):
+ associate_public_ip = True
+ if self.config.has_option('ec2', 'associate_public_ip'):
+ associate_public_ip = self.config.get('ec2', 'associate_public_ip').strip().lower() == 'true'
+
+ request = {'NetworkInterfaces': [{'DeviceIndex': 0, 'AssociatePublicIpAddress': associate_public_ip,
+ 'Groups': [sg_id]}]}
+
+ if self.config.has_option('ec2', 'subnet_id'):
+ request['NetworkInterfaces'][0]['SubnetId'] = self.config.get('ec2', 'subnet_id')
+
+ if 'worker' in services:
+ instance_type = self.config.get('ec2', 'worker_instance_type')
+ else:
+ instance_type = self.config.get('ec2', 'default_instance_type')
+ request['InstanceType'] = instance_type
+ request['InstanceInitiatedShutdownBehavior'] = self.config.get('ec2', 'shutdown_behavior')
+
+ if not self.config.has_option('ec2', 'aws_ami'):
+ exit('aws_ami property must be set!')
+ image_id = self.config.get('ec2', 'aws_ami')
+ if not image_id:
+ exit('aws_ami property was not properly')
+
+ request['ImageId'] = image_id
+ request['BlockDeviceMappings'] = get_block_device_map(instance_type)
+
+ if self.config.has_option('ec2', 'key_name'):
+ request['KeyName'] = self.config.get('ec2', 'key_name')
+
+ return request
+
def launch(self):
if self.active_nodes():
exit('ERROR - There are already instances running for {0} cluster'.format(self.config.cluster_name))
@@ -227,3 +238,21 @@
print("Removed hosts file at ", self.config.hosts_path)
else:
print("Aborted termination")
+
+
+class Ec2ClusterTemplate(Ec2Cluster):
+
+ def __init__(self, config):
+ Ec2Cluster.__init__(self, config)
+
+ def launch(self):
+ print("Using cluster template '{0}' to launch nodes".format(self.config.cluster_template_d['id']))
+ super().launch()
+
+ def init_request(self, hostname, services, sg_id):
+ # the first service in the list denotes the node's target template
+ print("Template '{0}' selected for {1}".format(services[0], hostname))
+ # interpolate any values from the ec2 config section and create request
+ ec2_d = dict(self.config.items('ec2'))
+ ec2_d['security_group_id'] = sg_id
+ return json.loads(Template(self.config.cluster_template_d[services[0]]).substitute(ec2_d))
diff --git a/lib/tests/test_config.py b/lib/tests/test_config.py
index 99ac626..3d78275 100644
--- a/lib/tests/test_config.py
+++ b/lib/tests/test_config.py
@@ -20,7 +20,7 @@
def test_ec2_cluster():
c = DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', 'mycluster')
+ '../conf/checksums', '../conf/templates', 'mycluster')
assert c.checksum_ver('accumulo', '1.9.0') == 'f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe'
assert c.checksum('accumulo') == 'df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47'
assert c.get('ec2', 'default_instance_type') == 'm5d.large'
@@ -80,7 +80,7 @@
def test_existing_cluster():
c = DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', 'mycluster')
+ '../conf/checksums', '../conf/templates', 'mycluster')
c.cluster_type = 'existing'
assert c.get_cluster_type() == 'existing'
assert c.node_type_map() == {}
@@ -95,9 +95,27 @@
def test_case_sensitive():
c = DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', 'mycluster')
+ '../conf/checksums', '../conf/templates', 'mycluster')
assert c.has_option('ec2', 'default_instance_type') == True
assert c.has_option('ec2', 'Default_instance_type') == False
c.set('nodes', 'CamelCaseWorker', 'worker,fluo')
c.init_nodes()
assert c.get_node('CamelCaseWorker') == ['worker', 'fluo']
+
+
+def test_ec2_cluster_template():
+ c = DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
+ '../conf/checksums', '../conf/templates', 'mycluster')
+
+ c.set('ec2', 'cluster_template', 'example')
+ c.init_template('../conf/templates')
+ # init_template already calls validate_template, so just ensure that
+ # we've loaded all the expected dictionary items from the example
+ assert 'accumulomaster' in c.cluster_template_d
+ assert 'client' in c.cluster_template_d
+ assert 'metrics' in c.cluster_template_d
+ assert 'namenode' in c.cluster_template_d
+ assert 'resourcemanager' in c.cluster_template_d
+ assert 'worker' in c.cluster_template_d
+ assert 'zookeeper' in c.cluster_template_d
+ assert 'devices' in c.cluster_template_d
