Support Accumulo installs on Microsoft Azure
* Add new cluster type 'azure' which leverages VM Scale Sets
* Increase some CentOS defaults to improve cluster stability
* Add checksums for specific Spark and Hadoop versions, as well as for
Accumulo 2.0.0
diff --git a/README.md b/README.md
index 8a9f9f3..c8f83c2 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,7 @@
**Muchos automates setting up [Apache Accumulo][accumulo] or [Apache Fluo][fluo] (and their dependencies) on a cluster**
-Muchos makes it easy to launch a cluster in Amazon's EC2 and deploy Accumulo or Fluo to it. Muchos
+Muchos makes it easy to launch a cluster in Amazon's EC2 or Microsoft Azure and deploy Accumulo or Fluo to it. Muchos
enables developers to experiment with Accumulo or Fluo in a realistic, distributed environment.
Muchos installs all software using tarball distributions which makes its easy to experiment
with the latest versions of Accumulo, Hadoop, Zookeeper, etc without waiting for downstream packaging.
@@ -17,35 +17,77 @@
* [Ansible] scripts that install and configure Fluo and its dependencies on a cluster.
* Python scripts that push the Ansible scripts from a local development machine to a cluster and
- run them. These Python scripts can also optionally launch a cluster in EC2 using [boto].
+ run them. These Python scripts can also optionally launch a cluster in EC2 using [boto] or in Azure using Azure CLI.
Checkout [Uno] for setting up Accumulo or Fluo on a single machine.
## Requirements
-Muchos requires the following:
+### Common
-* Python 3
+Muchos requires the following common components for installation and setup:
+
+* Python 3 with a virtual environment setup
+Create a Python 3 environment and switch to it. (We tested using Python 3.6.8,
+but this should work in later versions as well. If you encounter problems,
+please file an issue.)
+```bash
+cd ~
+python3.6 -m venv env
+source env/bin/activate
+```
+* `ssh-agent` installed and running and ssh-agent forwarding. Note that this may
+also require the creation of SSH public-private [key pair](https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
+```bash
+eval $(ssh-agent -s)
+ssh-add ~/.ssh/id_rsa
+```
+* Git (current version)
+
+### EC2
+
+Muchos requires the following for EC2 installations:
+
* [awscli] & [boto3] libraries - Install using `pip3 install awscli boto3 --upgrade --user`
-* `ssh-agent` installed and running
* An AWS account with your SSH public key uploaded. When you configure [muchos.props], set `key.name`
to name of your key pair in AWS.
* `~/.aws` [configured][aws-config] on your machine. Can be created manually or using [aws configure][awscli-config].
+### Azure
+
+Muchos requires the following for Azure installations:
+
+* [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) must be installed,
+ configured and authenticated to an Azure subscription. Please note - you should install
+ [Azure CLI 2.0.69](https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.0.69-1.el7.x86_64.rpm) on CentOS.
+ Higher versions of Azure CLI are unsupported for Muchos on CentOS at this time until
+ [this issue](https://github.com/Azure/azure-cli/issues/10128) in the Azure CLI 2.0.70 is fixed.
+ Example command to install Azure CLI 2.0.69 on CentOS is below:
+```bash
+wget https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.0.69-1.el7.x86_64.rpm
+sudo yum install azure-cli-2.0.69-1.el7.x86_64.rpm
+```
+* An Azure account with permissions to either use an existing or create new Resource Groups, Virtual Networks and Subnets
+* A machine which can connect to securely deploy the cluster in Azure.
+* Install [Ansible for Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/ansible-install-configure) within
+ the Python virtual environment by using `pip install ansible[azure]`
+
## Quickstart
-The following commands will install Muchos, launch an EC2 cluster, and setup/run Accumulo:
+The following commands will install Muchos, launch a cluster, and setup/run Accumulo:
```bash
git clone https://github.com/apache/fluo-muchos.git
+
cd fluo-muchos/
cp conf/muchos.props.example conf/muchos.props
vim conf/muchos.props # Edit to configure Muchos cluster
-./bin/muchos launch -c mycluster # Launches Muchos cluster in EC2
+./bin/muchos launch -c mycluster # Launches Muchos cluster in EC2 or Azure
./bin/muchos setup # Set up cluster and start Accumulo
```
-The `setup` command can be run repeatedly to fix any failures and will not repeat successful operations.
+The `launch` command will create a cluster with the name specified in the command (e.g. 'mycluster'). The `setup`
+command can be run repeatedly to fix any failures and will not repeat successful operations.
After your cluster is launched, SSH to it using the following command:
@@ -92,11 +134,47 @@
./bin/muchos status
+## Launching an Azure cluster
+
+Before launching a cluster, you will need to complete the requirements for Azure above, clone the Muchos repo, and
+create [muchos.props] by making a copy of existing [muchos.props.example]. If you want to give others access to your
+cluster, add their public keys to a file named `keys` in your `conf/` directory. During the setup of your cluster,
+this file will be appended on each node to the `~/.ssh/authorized_keys` file for the user set by the
+`cluster.username` property. You will also need to ensure you have authenticated to Azure and set the target
+subscription using the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli?view=azure-cli-latest).
+
+Muchos by default uses a CentOS 7 image that is hosted in the Azure marketplace. The Azure Linux Agent is already
+pre-installed on the Azure Marketplace images and is typically available from the distribution's package repository.
+Azure requires that the publishers of the endorsed Linux distributions regularly update their images in the Azure
+Marketplace with the latest patches and security fixes, at a quarterly or faster cadence. Updated images in the Azure
+Marketplace are available automatically to customers as new versions of an image SKU.
+
+Edit the values in the sections within [muchos.props] as below
+Under the `general` section, edit following values as per your configuration
+* `cluster_type = azure`
+* `cluster_user` should be set to the name of the administrative user
+* `proxy_hostname` (optional) is the name of the machine which has access to the cluster VNET
+
+Under the `azure` section, edit following values as per your configuration
+* `resource_group` and provide the same created in your Azure subscription for the cluster deployment
+* `vnet` and provide the same created in your Azure subscription for the cluster deployment
+* `subnet` and provide the same created in your Azure subscription for the cluster deployment
+* `numnodes` can be changed as per the cluster size
+* `vm_sku` can be specified from the available skus in the [selected Azure region](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/cli-ps-findimage)
+
+Within Azure the `nodes` section is auto populated with the hostnames and their default roles.
+
+After following the steps above, run the following command to launch an Azure VMSS cluster called `mycluster`
+(where 'mycluster' is the name assigned to your cluster):
+```bash
+.bin/muchos launch -c `mycluster` # Launches Muchos cluster in Azure
+```
+
## Set up the cluster
-The `./bin/muchos setup` command will set up your cluster and start Hadoop, Zookeeper, & Accumulo. It
-will download release tarballs of Fluo, Accumulo, Hadoop, etc. The versions of these tarballs are
-specified in [muchos.props] and can be changed if desired.
+Once your cluster is built in EC2 or Azure, the `./bin/muchos setup` command will set up your cluster and
+start Hadoop, Zookeeper & Accumulo. It will download release tarballs of Fluo, Accumulo, Hadoop, etc. The
+versions of these tarballs are specified in [muchos.props] and can be changed if desired.
Optionally, Muchos can setup the cluster using an Accumulo or Fluo tarball that is placed in the
`conf/upload` directory of Muchos. This option is only necessary if you want to use an unreleased
@@ -199,16 +277,19 @@
useful in user playbooks. It is recommended that any user-defined Ansible playbooks should be
managed in their own git repository (see [mikewalch/muchos-custom][mc] for an example).
-## Terminating your EC2 cluster
+## Terminating your cluster
-If you launched your cluster on EC2, run the following command terminate your cluster. WARNING - All
+If you launched your cluster, run the following command to terminate your cluster. WARNING - All
data on your cluster will be lost:
./bin/muchos terminate
-## Automatic shutdown of EC2 clusters
+Note: The terminate command is currently unsupported for Azure based clusters. Instead, you should delete
+underlying Azure VMSS resources when you need to terminate the cluster.
-With the default configuration, EC2 clusters will not shutdown automatically after a delay and the default
+## Automatic shutdown of clusters
+
+With the default configuration, clusters will not shutdown automatically after a delay and the default
shutdown behavior will be stopping the node. If you would like your cluster to terminate after 8 hours,
set the following configuration in [muchos.props]:
@@ -233,8 +314,10 @@
Muchos is powered by the following projects:
* [boto] - Python library used by `muchos launch` to start a cluster in AWS EC2.
- * [Ansible] - Cluster management tool that is used by `muchos setup` to install, configure, and
+ * [ansible] - Cluster management tool that is used by `muchos setup` to install, configure, and
start Fluo, Accumulo, Hadoop, etc on an existing EC2 or bare metal cluster.
+ * [azure-cli] - The Azure CLI is a command-line tool for managing Azure resources.
+ * [ansible-azure] - Ansible includes a suite of modules for interacting with Azure Resource Manager.
## Muchos Testing
@@ -250,6 +333,8 @@
[aws-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
[awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
[awscli-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration
+[azure-cli]: https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.0.69-1.el7.x86_64.rpm
+[ansible-azure]: https://docs.ansible.com/ansible/latest/scenario_guides/guide_azure.html
[fluo-app]: https://github.com/apache/fluo/blob/master/docs/applications.md
[WebIndex]: https://github.com/apache/fluo-examples/tree/master/webindex
[Phrasecount]: https://github.com/apache/fluo-examples/tree/master/phrasecount
diff --git a/ansible/accumulo.yml b/ansible/accumulo.yml
index dca90eb..125ee56 100644
--- a/ansible/accumulo.yml
+++ b/ansible/accumulo.yml
@@ -19,7 +19,7 @@
tasks:
- import_tasks: roles/accumulo/tasks/download.yml
when: download_software
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
roles:
- accumulo
- hosts: accumulomaster
diff --git a/ansible/azure.yml b/ansible/azure.yml
new file mode 100644
index 0000000..c587558
--- /dev/null
+++ b/ansible/azure.yml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+---
+- hosts: localhost
+ roles:
+ - azure
diff --git a/ansible/common.yml b/ansible/common.yml
index ef26613..744ec5e 100644
--- a/ansible/common.yml
+++ b/ansible/common.yml
@@ -15,16 +15,18 @@
# limitations under the License.
#
+- hosts: proxy
+ become: yes
+ roles:
+ - proxy
+ tasks:
+ - import_tasks: roles/proxy/tasks/main.yml
+ - import_tasks: roles/proxy/tasks/download.yml
+ when: download_software
- hosts: nodes
become: yes
tasks:
- import_tasks: roles/common/tasks/hosts.yml
-- hosts: proxy
- roles:
- - proxy
- tasks:
- - import_tasks: roles/proxy/tasks/download.yml
- when: download_software
- hosts: all
become: yes
roles:
@@ -32,6 +34,8 @@
tasks:
- import_tasks: roles/common/tasks/ssh.yml
- import_tasks: roles/common/tasks/os.yml
+ - import_tasks: roles/common/tasks/azure.yml
+ when: cluster_type == 'azure'
- import_tasks: roles/common/tasks/ec2.yml
when: cluster_type == 'ec2'
- import_tasks: roles/common/tasks/existing.yml
diff --git a/ansible/conf/ansible.cfg b/ansible/conf/ansible.cfg
index 2ba1c3e..e99d4ed 100644
--- a/ansible/conf/ansible.cfg
+++ b/ansible/conf/ansible.cfg
@@ -17,3 +17,5 @@
host_key_checking = False
forks = 50
gathering = smart
+callback_whitelist = profile_tasks
+timeout=30
diff --git a/ansible/docker.yml b/ansible/docker.yml
index 83870be..0e893ad 100644
--- a/ansible/docker.yml
+++ b/ansible/docker.yml
@@ -15,7 +15,7 @@
# limitations under the License.
#
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
become: yes
roles:
- docker
diff --git a/ansible/hadoop.yml b/ansible/hadoop.yml
index ec88699..30fd7f7 100644
--- a/ansible/hadoop.yml
+++ b/ansible/hadoop.yml
@@ -15,7 +15,7 @@
# limitations under the License.
#
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
roles:
- hadoop
- hosts: namenode
diff --git a/ansible/mesos.yml b/ansible/mesos.yml
index fd8d8d0..60d6527 100644
--- a/ansible/mesos.yml
+++ b/ansible/mesos.yml
@@ -15,7 +15,7 @@
# limitations under the License.
#
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
become: yes
roles:
- mesos
diff --git a/ansible/roles/azure/tasks/create_vmss.yml b/ansible/roles/azure/tasks/create_vmss.yml
new file mode 100644
index 0000000..1feb360
--- /dev/null
+++ b/ansible/roles/azure/tasks/create_vmss.yml
@@ -0,0 +1,243 @@
+---
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# These Ansible tasks only run on the client machine where Muchos runs
+# At a high level, the various sections in this file do the following:
+# 1. Create (if not already existing): an Azure resource group, virtual network / subnet
+# 2. Optionally (if the user specified) create a VM and related resources to use as a proxy host
+# 3. Create the Azure VMSS to support the nodes for use with Muchos
+# 4. Automatically populate the hosts file and associated [nodes] section in muchos.props
+
+# SECTION 1: Create Azure RG, network and subnet
+- name: Create a resource group
+ azure_rm_resourcegroup:
+ name: "{{ resource_group }}"
+ location: "{{ location }}"
+ tags:
+ deployment_type: muchos
+ application: accumulo
+
+- name: Create a virtual network
+ azure_rm_virtualnetwork:
+ resource_group: "{{ resource_group }}"
+ name: "{{ vnet }}"
+ address_prefixes_cidr:
+ - "{{ vnet_cidr }}"
+ tags:
+ deployment_type: muchos
+ application: accumulo
+
+- name: Create a subnet
+ azure_rm_subnet:
+ resource_group: "{{ resource_group }}"
+ virtual_network_name: "{{ vnet }}"
+ name: "{{ subnet }}"
+ address_prefix_cidr: "{{ subnet_cidr }}"
+
+# SECTION 2: Optionally create a VM with a public IP which can act as a proxy host
+- name: Create public IP address
+ azure_rm_publicipaddress:
+ resource_group: "{{ resource_group }}"
+ allocation_method: Static
+ name: "{{ azure_proxy_host }}-ip"
+ register: azure_proxy_public_ip
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Create Network Security Group that allows SSH
+ azure_rm_securitygroup:
+ resource_group: "{{ resource_group }}"
+ name: "{{ azure_proxy_host }}-nsg"
+ rules:
+ - name: SSH
+ protocol: Tcp
+ destination_port_range: 22
+ access: Allow
+ priority: 1001
+ direction: Inbound
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Create NIC
+ azure_rm_networkinterface:
+ resource_group: "{{ resource_group }}"
+ name: "{{ azure_proxy_host }}-nic"
+ virtual_network: "{{ vnet }}"
+ subnet: "{{ subnet }}"
+ public_ip_name: "{{ azure_proxy_host }}-ip"
+ security_group: "{{ azure_proxy_host }}-nsg"
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Create azure proxy virtual machine
+ azure_rm_virtualmachine:
+ resource_group: "{{ resource_group }}"
+ name: "{{ azure_proxy_host }}"
+ network_interface_names:
+ - "{{ azure_proxy_host }}-nic"
+ vm_size: Standard_D8s_v3
+ admin_username: "{{ admin_username }}"
+ ssh_password_enabled: false
+ ssh_public_keys:
+ - path: /home/{{admin_username}}/.ssh/authorized_keys
+ key_data: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+ os_disk_caching: ReadWrite
+ image:
+ offer: CentOS
+ publisher: OpenLogic
+ sku: 7.5
+ version: latest
+ data_disks:
+ - lun: 0
+ disk_size_gb: 64
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+# SECTION 3: Create the Azure VMSS for the nodes used by Muchos
+- name: Create luns dictionary
+ set_fact:
+ luns_dict: "{{ luns_dict | default ([]) + [{ 'lun': item, 'disk_size_gb': disk_size_gb , 'caching': None } ] }}"
+ with_sequence: start=0 end={{numdisks-1}}
+
+- name: Set single placement group to correct value
+ set_fact:
+ single_placement_group: False
+ when: numnodes > 100
+
+- name: Create Scale Set
+ azure_rm_virtualmachinescaleset:
+ resource_group: "{{ resource_group }}"
+ location: "{{ location }}"
+ name: "{{ vmss_name }}"
+ vm_size: "{{ vm_sku }}"
+ admin_username: "{{ admin_username }}"
+ ssh_password_enabled: false
+ ssh_public_keys:
+ - path: /home/{{admin_username}}/.ssh/authorized_keys
+ key_data: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+ capacity: "{{ numnodes }}"
+ virtual_network_name: "{{ vnet }}"
+ subnet_name: "{{ subnet }}"
+ upgrade_policy: Manual
+ tier: Standard
+ managed_disk_type: "{{ managed_disk_type }}"
+ os_disk_caching: ReadWrite
+ enable_accelerated_networking: yes
+ single_placement_group: "{{ single_placement_group | default(omit) }}"
+ image:
+ offer: CentOS
+ publisher: OpenLogic
+ sku: 7.5
+ version: latest
+ data_disks: "{{ luns_dict }}"
+ tags: create_vmss
+
+# SECTION 4: Automatically populate entries in the hosts file and in the muchos.props file, based on the VMSS node details
+- name: get instance ids
+ shell: "az vmss list-instances --resource-group {{ resource_group }} --name {{ vmss_name }} --query '[].[{name: name,vmresourceid: id}]' -o json | sed -e 's/{{vmss_name}}_/{{vmss_name}}-/g'"
+ register: instance_list
+- set_fact:
+ instances: "{{ instance_list.stdout|from_json }}"
+
+- name: get private ip addresses
+ command: "az vmss nic list --resource-group {{ resource_group }} --vmss-name {{ vmss_name }} --query '[].[{ip: ipConfigurations[0].privateIpAddress, vmresourceid: virtualMachine.id}]' -o json"
+ register: ip_list
+- set_fact:
+ addresses: "{{ ip_list.stdout|from_json }}"
+- set_fact:
+ instances_dict: "{{ instances | json_query('[].{key: vmresourceid, value: name}') }}"
+ addresses_dict: "{{ addresses | json_query('[].{key: vmresourceid, value: ip}') }}"
+
+- name: Ensures hosts sub-dir exists
+ file:
+ path: "{{ deploy_path }}/conf/hosts/"
+ state: directory
+ recurse: yes
+
+- name: Ensure empty hosts file is created
+ copy:
+ content: ""
+ dest: "{{ deploy_path }}/conf/hosts/{{ vmss_name }}"
+ force: yes
+ mode: 0644
+
+- name: Write azure proxy to hosts file
+ lineinfile:
+ path: "{{ deploy_path }}/conf/hosts/{{ vmss_name }}"
+ line: "{{ azure_proxy_host }} {{ azure_proxy_public_ip.state.ip_address }}"
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Write VMSS vms to hosts file
+ lineinfile:
+ path: "{{ deploy_path }}/conf/hosts/{{ vmss_name }}"
+ line: "{{ item1[0] }} {{ item2[0] }}"
+ with_items: "{{ instances_dict | json_query('[].key')}}"
+ vars:
+ myquery: "[?key=='{{ item }}'].value"
+ item1: "{{ instances_dict | json_query(myquery) }}"
+ item2: "{{ addresses_dict | json_query(myquery) }}"
+
+- name: Clear section
+ ini_file:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ section: "nodes"
+ state: absent
+
+- name: Recreate section
+ ini_file:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ section: "nodes"
+ option: "#host0"
+ value: "service"
+ state: present
+
+- name: add azure proxy host
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ line: "{{ azure_proxy_host }} = client"
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Assign Accumulo master, HDFS components to the first node of the cluster
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ line: "{{ item }} = namenode,resourcemanager,accumulomaster,zookeeper"
+ with_items: "{{ instances_dict | json_query('[0].value') }}"
+
+- name: Assign metrics to the second node of the cluster
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ line: "{{ item }} = metrics"
+ with_items: "{{ instances_dict | json_query('[1].value') }}"
+
+- name: Add worker nodes to muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ line: "{{ item }} = worker"
+ with_items: "{{ instances_dict | json_query('[2:].value') }}"
+
+- name: Change proxy hostname to azure proxy host in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^proxy_hostname\s*=\s*'
+ line: "proxy_hostname = {{ azure_proxy_host }}"
+ when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+
+- name: Change proxy hostname to first node in vmss in muchos.props
+ lineinfile:
+ path: "{{ deploy_path }}/conf/muchos.props"
+ regexp: '^proxy_hostname\s*=\s*'
+ line: "proxy_hostname = {{ item }}"
+ with_items: "{{ instances_dict | json_query('[0].value') }}"
+ when: not (azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None)
diff --git a/ansible/roles/azure/tasks/main.yml b/ansible/roles/azure/tasks/main.yml
new file mode 100644
index 0000000..6ec80d7
--- /dev/null
+++ b/ansible/roles/azure/tasks/main.yml
@@ -0,0 +1,21 @@
+---
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# tasks file for azure
+- import_tasks: create_vmss.yml
diff --git a/ansible/roles/common/tasks/azure.yml b/ansible/roles/common/tasks/azure.yml
new file mode 100644
index 0000000..bcbe967
--- /dev/null
+++ b/ansible/roles/common/tasks/azure.yml
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+- name: Find luns
+ find:
+ paths: "/dev/disk/azure/scsi1"
+ patterns: "lun*"
+ file_type: link
+ register: files_matched
+- name: Create xfs filesytems
+ filesystem:
+ fstype: xfs
+ dev: "{{ item.path }}"
+ with_items: "{{ files_matched.files }}"
+- name: Get UUID
+ command: "blkid {{ item.path }} -s UUID -o value"
+ with_items: "{{ files_matched.files }}"
+ register: disk_uuids
+- name: Create mount points
+ file:
+ path: '{{ mount_root }}{{ item.0 + 1 }}'
+ state: directory
+ with_indexed_items: "{{ files_matched.files }}"
+- name: Mount filesystems
+ mount:
+ path: '{{ mount_root }}{{ item.0 + 1 }}'
+ src: "UUID={{ item.1.stdout }}"
+ fstype: xfs
+ state: mounted
+ with_indexed_items: "{{ disk_uuids.results }}"
+- name: Set mount point ownership
+ file:
+ path: '{{ mount_root }}{{ item.0 + 1 }}'
+ state: directory
+ owner: "{{ cluster_user }}"
+ group: "{{ cluster_group }}"
+ with_indexed_items: "{{ files_matched.files }}"
+- name: Create directry to mount Azure File share
+ file:
+ path: "{{ azure_fileshare_mount }}"
+ state: directory
+ owner: '{{ cluster_user }}'
+ group: '{{ cluster_group }}'
+ when: azure_fileshare_mount is defined and azure_fileshare_mount != '' and azure_fileshare_mount != None
+- name: Mount Azure File share
+ become: yes
+ mount:
+ fstype: cifs
+ src: "{{ azure_fileshare }}"
+ path: "{{ azure_fileshare_mount }}"
+ opts: vers=3.0,username={{ azure_fileshare_username }},password={{ azure_fileshare_password }},dir_mode=0777,file_mode=0777,serverino
+ state: mounted
+ when: azure_fileshare_mount is defined and azure_fileshare_mount != '' and azure_fileshare_mount != None
diff --git a/ansible/roles/common/tasks/hosts.yml b/ansible/roles/common/tasks/hosts.yml
index 1e3d68b..abb2d3a 100644
--- a/ansible/roles/common/tasks/hosts.yml
+++ b/ansible/roles/common/tasks/hosts.yml
@@ -18,4 +18,4 @@
- name: "ensure hostname is correct"
hostname: name={{ inventory_hostname }}
- name: "ensure /etc/hosts is correct"
- template: src=roles/common/templates/etc_hosts dest=/etc/hosts owner=root group=root mode=0644
+ copy: src=/etc/etc_hosts dest=/etc/hosts owner=root group=root mode=0644
diff --git a/ansible/roles/common/tasks/os.yml b/ansible/roles/common/tasks/os.yml
index 8e6f02b..83df0df 100644
--- a/ansible/roles/common/tasks/os.yml
+++ b/ansible/roles/common/tasks/os.yml
@@ -17,6 +17,24 @@
- name: "set swappiness to zero"
sysctl: name=vm.swappiness value=0
+
+# The kernel memory settings below have been added based on performance testing done
+# with Microsoft Azure based clusters. For more details please refer to
+# https://access.redhat.com/solutions/90883 and https://bugzilla.kernel.org/show_bug.cgi?id=107111
+- name: "set vm.min_free_kbytes to 1G"
+ sysctl: name=vm.min_free_kbytes value=1000000
+ when: cluster_type == 'azure' or cluster_type == 'ec2'
+- name: "set vm.zone_reclaim_mode to 1"
+ sysctl: name=vm.zone_reclaim_mode value=1
+ when: cluster_type == 'azure' or cluster_type == 'ec2'
+
+# The TCP/IP setting below has been added based on performance testing done
+# with Microsoft Azure based clusters. For more details refer to
+# https://access.redhat.com/solutions/30453
+- name: "set net.core.somaxconn=2048"
+ sysctl: name=net.core.somaxconn value=2048
+ when: cluster_type == 'azure' or cluster_type == 'ec2'
+
- name: "remove old limits file"
file: path=/etc/security/limits.d/20-nproc.conf state=absent
- name: "copy new limits.conf"
diff --git a/ansible/roles/proxy/tasks/main.yml b/ansible/roles/proxy/tasks/main.yml
index 21a7fa5..9a4fe07 100644
--- a/ansible/roles/proxy/tasks/main.yml
+++ b/ansible/roles/proxy/tasks/main.yml
@@ -15,8 +15,10 @@
# limitations under the License.
#
-- name: "ensure cluster user exists and generate ssh key"
+- name: "ensure cluster user exists and generate ssh key"
user: name={{ cluster_user }} generate_ssh_key=yes ssh_key_bits=4096 state=present
become: yes
- name: "create tarball directory on proxy"
file: path={{ tarballs_dir }} state=directory
+- name: "copy /etc/hosts to proxy"
+ template: src=roles/proxy/templates/etc_hosts dest=/etc/etc_hosts owner=root group=root mode=0644
diff --git a/ansible/roles/proxy/templates/etc_hosts b/ansible/roles/proxy/templates/etc_hosts
new file mode 100644
index 0000000..d8c79db
--- /dev/null
+++ b/ansible/roles/proxy/templates/etc_hosts
@@ -0,0 +1,6 @@
+127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
+::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
+
+{% for host in groups['nodes'] %}
+{{ hostvars[host]['ansible_ssh_host'] }} {{ host }}
+{% endfor %}
diff --git a/ansible/spark.yml b/ansible/spark.yml
index adf62b1..40d01db 100644
--- a/ansible/spark.yml
+++ b/ansible/spark.yml
@@ -19,7 +19,7 @@
tasks:
- import_tasks: roles/spark/tasks/download.yml
when: download_software
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
roles:
- spark
- hosts: spark
diff --git a/ansible/zookeeper.yml b/ansible/zookeeper.yml
index 3e625ed..650e524 100644
--- a/ansible/zookeeper.yml
+++ b/ansible/zookeeper.yml
@@ -15,7 +15,7 @@
# limitations under the License.
#
-- hosts: all
+- hosts: all:!{{ azure_proxy_host }}
roles:
- zookeeper
- hosts: zookeepers
diff --git a/conf/checksums b/conf/checksums
index 9b130e3..2b2d38e 100644
--- a/conf/checksums
+++ b/conf/checksums
@@ -8,15 +8,20 @@
fluo:1.2.0:037f89cd2bfdaf76a1368256c52de46d6b9a85c9c1bfc776ec4447d02c813fb2
fluo_yarn:1.0.0:c6220d35cf23127272f3b5638c44586504dc17a46f5beecdfee5027b5ff874b0
hadoop:3.2.0:226b6cbdf769467250054b3abdf26df9f05fde44bbb82fe5d12d6993ea848f64
+hadoop:3.1.2:1C02CCC60A09C63A48DC4234FFD3AED1B75E5A1F2B49D60927EDA114B93DD31A
hadoop:3.1.1:f837fe260587f71629aad1f4fb6719274e948111dc96ffc5a8e26f27deac5602
hadoop:3.0.2:0d507aa71007b2685e292343c11c2cb90a92ea7625446b57d1fb47c5721e2f82
+hadoop:3.0.3:db96e2c0d0d5352d8984892dfac4e27c0e682d98a497b7e04ee97c3e2019277a
hadoop:2.9.2:3d2023c46b1156c1b102461ad08cbc17c8cc53004eae95dab40a1f659839f28a
hadoop:2.9.0:8d48666f29f9ade6ed2762b7a9edab177bad2c57396f43d0ffd6a269d54f6fe1
+hadoop:2.8.5:F9C726DF693CE2DAA4107886F603270D66E7257F77A92C9886502D6CD4A884A4
hadoop:2.8.4:6b545972fdd73173887cdbc3e1cbd3cc72068271924edea82a0e7e653199b115
hadoop:2.8.3:e8bf9a53337b1dca3b152b0a5b5e277dc734e76520543e525c301a050bb27eae
hadoop:2.7.6:f2327ea93f4bc5a5d7150dee8e0ede196d3a77ff8526a7dd05a48a09aae25669
hadoop:2.7.5:0bfc4d9b04be919be2fdf36f67fa3b4526cdbd406c512a7a1f5f1b715661f831
hadoop:2.6.5:001ad18d4b6d0fe542b15ddadba2d092bc97df1c4d2d797381c8d12887691898
+spark:2.4.3:4db62e110c8080f28ac8f1d701b7477cede23e60b51231ab63687b2cc6150faa
+spark:2.3.3:a5c4a54db5163df25110534f36e13d6ae65758d45d1bc8bfcd7e27304e716a23
spark:2.3.2:3387107155d62f04ccf6bcaf2e00a69a0de5ae5df875348d93147743c206f0a8
spark:2.2.2:023b2fea378b3dd0fee2d5d1de6bfaf2d8349aefe7be97a9cbcf03bbacc428d7
zookeeper:3.4.14:b14f7a0fece8bd34c7fffa46039e563ac5367607c612517aa7bd37306afbd1cd
diff --git a/conf/muchos.props.example b/conf/muchos.props.example
index 599c812..2615af4 100644
--- a/conf/muchos.props.example
+++ b/conf/muchos.props.example
@@ -14,7 +14,7 @@
# limitations under the License.
[general]
-# Cluster type (ec2 or existing)
+# Cluster type (Azure, ec2, or existing)
cluster_type = ec2
# Cluster user name (install command will SSH to cluster using this user)
# Leave default below if launching cluster in AWS
@@ -108,6 +108,29 @@
# Below are different performance profiles that can be selected. Each profile
# has the same properties with different values.
+[azure]
+resource_group = accumulo-rg
+vnet = vnet1
+vnet_cidr = "10.0.0.0/8"
+subnet = subnet1
+subnet_cidr = "10.1.0.0/16"
+numnodes = 8
+vm_sku = Standard_D8s_v3
+managed_disk_type = Standard_LRS
+numdisks = 3
+disk_size_gb = 128
+mount_root = /var/data
+metrics_drive_root = var-data
+# Optional proxy VM. If not set, the first node of the cluster will be selected as the proxy.
+azure_proxy_host =
+location = westus2
+# Optional Azure fileshare to mount on all nodes.
+# Path and credentials must be updated to enable this.
+#azure_fileshare_mount = /mnt/azure-fileshare
+#azure_fileshare = //fileshare-to-mount.file.core.windows.net/path
+#azure_fileshare_username = fs_username
+#azure_fileshare_password = fs_password
+
[perf-small]
# Amount of JVM heap for each tserver
accumulo_tserv_mem=2G
@@ -156,18 +179,42 @@
fluo_worker_instances_multiplier=2
yarn_nm_mem_mb=16384
+[azd16s]
+accumulo_tserv_mem=4G
+accumulo_dcache_size=2G
+accumulo_icache_size=1G
+accumulo_imap_size=512M
+fluo_worker_mem_mb=4096
+twill_reserve_mem_mb=512
+fluo_worker_threads=64
+fluo_worker_instances_multiplier=2
+yarn_nm_mem_mb=16384
+
+[azd8s]
+accumulo_tserv_mem=4G
+accumulo_dcache_size=2G
+accumulo_icache_size=1G
+accumulo_imap_size=512M
+fluo_worker_mem_mb=4096
+twill_reserve_mem_mb=512
+fluo_worker_threads=64
+fluo_worker_instances_multiplier=2
+yarn_nm_mem_mb=16384
+
[ansible-vars]
# This section is used to override Ansible variables. Any variable set below will be placed in the hosts file created by Muchos.
# Expected format: variable = value
[nodes]
-# Describes nodes in cluster in the following format:
+# If cluster_type=existing, the list of nodes below needs to manually populated with a list of cluster nodes in the following format:
# <Hostname> = <Service1>[,<Service2>,<Service3>]
# Where:
# Hostname = Must be unique. Will be used for hostname in EC2 or should match hostname on your own cluster
# Service = Service to run on node (possible values: zookeeper, namenode, resourcemanager, accumulomaster, client, swarmmanager,
# mesosmaster, worker, fluo, metrics, spark). The following services are required: namenode, resourcemanager,
# accumulomaster, zookeeper & worker
+# If cluster_type=azure, the list of nodes below is auto-generated by the launch action e.g. "muchos launch --cluster accumuloclstr"
+# For the 'azure' cluster type, it is perfectly normal if the auto-generated list of node names is not sequential
leader1 = namenode,resourcemanager,accumulomaster,zookeeper
leader2 = metrics
worker1 = worker,swarmmanager
diff --git a/lib/main.py b/lib/main.py
index 62df08c..e2149c6 100644
--- a/lib/main.py
+++ b/lib/main.py
@@ -72,6 +72,10 @@
else:
cluster = Ec2Cluster(config)
cluster.perform(action)
+ elif cluster_type == 'azure':
+ from muchos.azure import VmssCluster
+ cluster = VmssCluster(config)
+ cluster.perform(action)
else:
exit('Unknown cluster_type: ' + cluster_type)
diff --git a/lib/muchos/azure.py b/lib/muchos/azure.py
new file mode 100644
index 0000000..0b3d3ea
--- /dev/null
+++ b/lib/muchos/azure.py
@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import json
+import subprocess
+from os.path import join
+from .existing import ExistingCluster
+from azure.common.client_factory import get_client_from_cli_profile
+from azure.mgmt.compute import ComputeManagementClient
+
+
+# For Muchos deployments on Microsoft Azure, we use Virtual Machine Scale Sets
+# VMSS is the most efficient way to provision large numbers of VMs in Azure
+class VmssCluster(ExistingCluster):
+
+ def __init__(self, config):
+ ExistingCluster.__init__(self, config)
+
+ def launch(self):
+ config = self.config
+ azure_config = dict(config.items("azure"))
+ azure_config["admin_username"] = config.get("general", "cluster_user")
+ azure_config["vmss_name"] = config.cluster_name
+ azure_config["deploy_path"] = config.deploy_path
+ azure_config = {k: int(v) if v.isdigit() else v
+ for k, v in azure_config.items()}
+ subprocess.call(["ansible-playbook",
+ join(config.deploy_path, "ansible/azure.yml"),
+ "--extra-vars", json.dumps(azure_config)])
+
+ def status(self):
+ config = self.config
+ azure_vars_dict = dict(config.items("azure"))
+ compute_client = get_client_from_cli_profile(ComputeManagementClient)
+ vmss_status = compute_client.virtual_machine_scale_sets.get(
+ azure_vars_dict["resource_group"],
+ self.config.cluster_name)
+ print('name:', vmss_status.name,
+ '\nprovisioning_state:', vmss_status.provisioning_state)
diff --git a/lib/muchos/config.py b/lib/muchos/config.py
index 1b0c196..b0f004b 100644
--- a/lib/muchos/config.py
+++ b/lib/muchos/config.py
@@ -50,11 +50,13 @@
def verify_config(self, action):
proxy = self.get('general', 'proxy_hostname')
- if not proxy:
- exit("ERROR - proxy.hostname must be set in muchos.props")
+ cluster_type = self.get('general', 'cluster_type')
+ if cluster_type not in ['azure']:
+ if not proxy:
+ exit("ERROR - proxy.hostname must be set in muchos.props")
- if proxy not in self.node_d:
- exit("ERROR - The proxy (set by property proxy_hostname={0}) cannot be found in 'nodes' section of "
+ if proxy not in self.node_d:
+ exit("ERROR - The proxy (set by property proxy_hostname={0}) cannot be found in 'nodes' section of "
"muchos.props".format(proxy))
if action in ['launch', 'setup']:
@@ -120,6 +122,8 @@
return '/media/' + self.ephemeral_root
elif self.get_cluster_type() == 'existing':
return self.get('existing', 'mount_root')
+ elif self.get_cluster_type() == 'azure':
+ return self.get('azure', 'mount_root')
def fstype(self):
retval = None
@@ -137,17 +141,27 @@
return 'no'
return retval
- def worker_data_dirs(self):
+ def data_dirs_common(self, nodeType):
+ data_dirs = []
+
if self.get_cluster_type() == 'ec2':
- return self.node_type_map()['worker']['mounts']
+ data_dirs = self.node_type_map()[nodeType]['mounts']
elif self.get_cluster_type() == 'existing':
- return self.get('existing', 'data_dirs').split(",")
+ data_dirs = self.get('existing', 'data_dirs').split(",")
+ elif self.get_cluster_type() == 'azure':
+ num_disks = int(self.get("azure","numdisks"))
+ range_var = num_disks + 1
+ for diskNum in range(1, range_var):
+ data_dirs.append(self.get("azure","mount_root") + str(diskNum))
+
+
+ return data_dirs
+
+ def worker_data_dirs(self):
+ return self.data_dirs_common("worker")
def default_data_dirs(self):
- if self.get_cluster_type() == 'ec2':
- return self.node_type_map()['default']['mounts']
- elif self.get_cluster_type() == 'existing':
- return self.get('existing', 'data_dirs').split(",")
+ return self.data_dirs_common("default")
def metrics_drive_ids(self):
if self.get_cluster_type() == 'ec2':
@@ -157,6 +171,12 @@
return drive_ids
elif self.get_cluster_type() == 'existing':
return self.get("existing", "metrics_drive_ids").split(",")
+ elif self.get_cluster_type() == 'azure':
+ drive_ids = []
+ range_var = int(self.get("azure","numdisks")) + 1
+ for i in range(1, range_var):
+ drive_ids.append(self.get("azure","metrics_drive_root") + str(i))
+ return drive_ids
def shutdown_delay_minutes(self):
retval = '0'
@@ -292,7 +312,7 @@
return self.get_hosts()[hostname][1]
def get_cluster_type(self):
- if self.cluster_type not in ('ec2', 'existing'):
+ if self.cluster_type not in ('azure', 'ec2', 'existing'):
exit('ERROR - Unknown cluster type' + self.cluster_type)
return self.cluster_type
@@ -327,6 +347,9 @@
for (name, val) in self.items('ec2'):
print(name, '=', val)
+ for (name, val) in self.items('azure'):
+ print(name, '=', val)
+
def print_property(self, key):
if key == 'proxy.public.ip':
print(self.proxy_public_ip())
@@ -474,3 +497,10 @@
'yarn_nm_mem_mb': None,
'zookeeper_sha256': None
}
+
+AZURE_VAR_DEFAULTS = {
+ 'azure_fileshare_mount': None,
+ 'azure_fileshare': None,
+ 'azure_fileshare_username': None,
+ 'azure_fileshare_password': None,
+}
diff --git a/lib/muchos/existing.py b/lib/muchos/existing.py
index 6b9247a..5253d6d 100644
--- a/lib/muchos/existing.py
+++ b/lib/muchos/existing.py
@@ -23,7 +23,7 @@
from sys import exit
from os import listdir
-from .config import HOST_VAR_DEFAULTS, PLAY_VAR_DEFAULTS
+from .config import HOST_VAR_DEFAULTS, PLAY_VAR_DEFAULTS, AZURE_VAR_DEFAULTS
class ExistingCluster:
@@ -41,13 +41,17 @@
host_vars = HOST_VAR_DEFAULTS
play_vars = PLAY_VAR_DEFAULTS
- for section in ("general", "ansible-vars", config.get('performance', 'profile')):
+ azure_vars = AZURE_VAR_DEFAULTS
+
+ for section in ("general", "ansible-vars", config.get('performance', 'profile'), "azure"):
for (name, value) in config.items(section):
if name not in ('proxy_hostname', 'proxy_socks_port'):
if name in host_vars:
host_vars[name] = value
if name in play_vars:
play_vars[name] = value
+ if name in azure_vars:
+ azure_vars[name] = value
play_vars['accumulo_sha256'] = config.checksum('accumulo')
play_vars['fluo_sha256'] = config.checksum('fluo')
@@ -66,10 +70,13 @@
with open(join(config.deploy_path, "ansible/site.yml"), 'w') as site_file:
print("- import_playbook: common.yml", file=site_file)
+
+ print("- import_playbook: zookeeper.yml", file=site_file)
+ print("- import_playbook: hadoop.yml", file=site_file)
+
if config.has_service("spark"):
print("- import_playbook: spark.yml", file=site_file)
- print("- import_playbook: hadoop.yml", file=site_file)
- print("- import_playbook: zookeeper.yml", file=site_file)
+
if config.has_service("metrics"):
print("- import_playbook: metrics.yml", file=site_file)
print("- import_playbook: accumulo.yml", file=site_file)
@@ -127,6 +134,8 @@
print("\n[all:vars]", file=hosts_file)
for (name, value) in sorted(host_vars.items()):
print("{0} = {1}".format(name, value), file=hosts_file)
+ for (name, value) in sorted(azure_vars.items()):
+ print("{0} = {1}".format(name, value), file=hosts_file)
with open(join(config.deploy_path, "ansible/group_vars/all"), 'w') as play_vars_file:
for (name, value) in sorted(play_vars.items()):
@@ -210,8 +219,10 @@
def execute_playbook(self, playbook):
print("Executing '{0}' playbook".format(playbook))
- self.exec_on_proxy_verified("time -p ansible-playbook {base}/ansible/{playbook}"
- .format(base=self.config.user_home(), playbook=playbook), opts='-t')
+ azure_proxy_host = self.config.get("azure","azure_proxy_host")
+ var_azure_proxy_host = "_" if (azure_proxy_host==None or azure_proxy_host.strip()=='') else azure_proxy_host
+ self.exec_on_proxy_verified("time -p ansible-playbook {base}/ansible/{playbook} --extra-vars \"azure_proxy_host={var_azure_proxy_host}\""
+ .format(base=self.config.user_home(), playbook=playbook, var_azure_proxy_host=var_azure_proxy_host), opts='-t')
def send_to_proxy(self, path, target, skip_if_exists=True):
print("Copying to proxy: ", path)
diff --git a/lib/muchos/util.py b/lib/muchos/util.py
index 202544b..0a665d0 100644
--- a/lib/muchos/util.py
+++ b/lib/muchos/util.py
@@ -142,8 +142,8 @@
parser = OptionParser(
usage="muchos [options] <action>\n\n"
+ "where <action> can be:\n"
- + " launch Launch cluster in EC2\n"
- + " status Check status of EC2 cluster\n"
+ + " launch Launch cluster in Azure or EC2\n"
+ + " status Check status of Azure or EC2 cluster\n"
+ " setup Set up cluster\n"
+ " sync Sync ansible directory on cluster proxy node\n"
+ " config Print configuration for that cluster. Requires '-p'. Use '-p all' for all config.\n"
diff --git a/lib/tests/test_config.py b/lib/tests/test_config.py
index 3d78275..f6501ff 100644
--- a/lib/tests/test_config.py
+++ b/lib/tests/test_config.py
@@ -46,6 +46,7 @@
assert c.instance_tags() == {}
assert len(c.nodes()) == 6
assert c.get_node('leader1') == ['namenode', 'resourcemanager', 'accumulomaster', 'zookeeper']
+ assert c.get_node('leader2') == ['metrics']
assert c.get_node('worker1') == ['worker', 'swarmmanager']
assert c.get_node('worker2') == ['worker']
assert c.get_node('worker3') == ['worker']
@@ -74,7 +75,77 @@
('10.0.0.4', 'worker3'), ('10.0.0.5', 'worker4')]
assert c.get_host_services() == [('leader1', 'namenode resourcemanager accumulomaster zookeeper'),
('leader2', 'metrics'),
- ('worker1', 'worker swarmmanager'), ('worker2', 'worker'), ('worker3', 'worker'),
+ ('worker1', 'worker swarmmanager'),
+ ('worker2', 'worker'),
+ ('worker3', 'worker'),
+ ('worker4', 'worker')]
+
+
+def test_azure_cluster():
+ c = DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
+ '../conf/checksums', '../conf/templates', 'mycluster')
+
+ # since we are sharing a single muchos.props.example file, we need
+ # to stub the cluster type to be azure (as the file itself has a default of ec2)
+
+ c.cluster_type = 'azure'
+
+ assert c.checksum_ver('accumulo', '1.9.0') == 'f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe'
+ assert c.checksum('accumulo') == 'df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47'
+ assert c.get('azure', 'vm_sku') == 'Standard_D8s_v3'
+ assert c.get('azure', 'managed_disk_type') == 'Standard_LRS'
+ assert c.user_home() == '/home/centos'
+ assert c.mount_root() == '/var/data'
+ assert c.force_format() == 'no'
+ assert c.worker_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
+ assert c.default_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
+ assert c.metrics_drive_ids() == ['var-data1', 'var-data2', 'var-data3']
+ assert c.shutdown_delay_minutes() == '0'
+ assert c.mounts(2) == ['/var/data0', '/var/data1']
+ assert c.node_type('worker1') == 'worker'
+ assert c.node_type('leader1') == 'default'
+ assert c.has_option('azure', 'resource_group')
+ assert c.has_option('azure', 'vnet')
+ assert c.has_option('azure', 'vnet_cidr')
+ assert c.has_option('azure', 'subnet')
+ assert c.has_option('azure', 'subnet_cidr')
+ assert c.has_option('azure', 'numnodes')
+ assert c.has_option('azure', 'location')
+ assert c.instance_tags() == {}
+ assert len(c.nodes()) == 6
+ assert c.get_node('leader1') == ['namenode', 'resourcemanager', 'accumulomaster', 'zookeeper']
+ assert c.get_node('leader2') == ['metrics']
+ assert c.get_node('worker1') == ['worker', 'swarmmanager']
+ assert c.get_node('worker2') == ['worker']
+ assert c.get_node('worker3') == ['worker']
+ assert c.has_service('accumulomaster')
+ assert not c.has_service('fluo')
+ assert c.get_service_hostnames('worker') == ['worker1', 'worker2', 'worker3', 'worker4']
+ assert c.get_service_hostnames('zookeeper') == ['leader1']
+ assert c.get_hosts() == {'leader2': ('10.0.0.1', None), 'leader1': ('10.0.0.0', '23.0.0.0'),
+ 'worker1': ('10.0.0.2', None), 'worker3': ('10.0.0.4', None),
+ 'worker2': ('10.0.0.3', None), 'worker4': ('10.0.0.5', None)}
+ assert c.get_public_ip('leader1') == '23.0.0.0'
+ assert c.get_private_ip('leader1') == '10.0.0.0'
+ assert c.cluster_name == 'mycluster'
+ assert c.get_cluster_type() == 'azure'
+ assert c.version("accumulo").startswith('2.')
+ assert c.version("fluo").startswith('1.')
+ assert c.version("hadoop").startswith('3.')
+ assert c.version("zookeeper").startswith('3.')
+ assert c.get_service_private_ips("worker") == ['10.0.0.2', '10.0.0.3', '10.0.0.4', '10.0.0.5']
+ assert c.get('general', 'proxy_hostname') == "leader1"
+ assert c.proxy_public_ip() == "23.0.0.0"
+ assert c.proxy_private_ip() == "10.0.0.0"
+ assert c.get('general', 'cluster_user') == "centos"
+ assert c.get('general', 'cluster_group') == "centos"
+ assert c.get_non_proxy() == [('10.0.0.1', 'leader2'), ('10.0.0.2', 'worker1'), ('10.0.0.3', 'worker2'),
+ ('10.0.0.4', 'worker3'), ('10.0.0.5', 'worker4')]
+ assert c.get_host_services() == [('leader1', 'namenode resourcemanager accumulomaster zookeeper'),
+ ('leader2', 'metrics'),
+ ('worker1', 'worker swarmmanager'),
+ ('worker2', 'worker'),
+ ('worker3', 'worker'),
('worker4', 'worker')]
