Cleanup Python and Ansible Code (#340)
* Format Python code using Black (https://github.com/psf/black)
* Normalize Python string literals to use double quotes
* Use specific Python imports where possible
* Address number of issues flagged by ansible-lint
* Refactor wipe.yml to avoid rm -rf commands
* Run flake8 and ansible-lint as part of Travis CI
* Add a script to detect files without EOF newline
* Call the EOF newline checker from Travis CI build script
* Add a CONTRIBUTING.md file with instructions for developers
* Move info about nose tests into the CONTRIBUTING.md file
* Link to the CONTRIBUTING.md file from README.md
* Fix minor typos in Ansible task names
Co-Authored-By: Christopher Tubbs <ctubbsii@apache.org>
diff --git a/.gitignore b/.gitignore
index 3c468ed..2ab6a2f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
*.pyc
/.idea/
/pom.xml
+.vscode/
diff --git a/.travis.yml b/.travis.yml
index 441ad01..b6ed779 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,4 +15,8 @@
language: python
python:
- "3.5"
-script: nosetests -w lib/
+install:
+ - pip install flake8
+ - pip install ansible-lint
+script:
+ - ./scripts/cibuild
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..8a8b06b
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,49 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Contributors Guide
+
+If you believe that you have found a bug, please search for an existing [issue](https://github.com/apache/fluo-muchos/issues) to see if it has already been reported. For simple changes, its ok to just submit a pull request without an issue.
+
+## Muchos Testing
+
+Muchos has unit tests. To run them, first install nose using pip:
+```
+ pip install nose
+```
+The following command runs the unit tests:
+```
+ nosetests -w lib/
+```
+
+## Before you submit a PR
+
+If you are modifying any of the Python code in this project, please use [Black](https://github.com/psf/black) to enforce that Python code found under the [lib](https://github.com/apache/fluo-muchos/tree/master/lib) folder is formatted correctly. Before submitting a PR, please ensure that you have used Black to format the code with max line length set to 79 as below (it is to be run from the repo root):
+```
+black lib --line-length 79
+```
+
+The [CI](https://github.com/apache/fluo-muchos/tree/master/.travis.yml) for this project runs tools to detect common coding issues with Python and Ansible files. Rather than wait for the CI to flag any issues with your work, please run the [cibuild](https://github.com/apache/fluo-muchos/tree/master/scripts/cibuild.sh) script on your dev machine, which in turn runs the following tools:
+- [flake8](https://github.com/pycqa/flake8) to validate that the Python code in the project conforms to known good practices.
+- [Ansible-lint](https://github.com/ansible/ansible-lint/) prior to submitting a PR. This will ensure that you align with known good practices. Please also review the guidance on [false positives](https://docs.ansible.com/ansible-lint/rules/rules.html#false-positives-skipping-rules) from Ansible-lint.
+
+Please ensure that you address issues flagged by the above CI build script before creating a PR.
+
+## Review
+
+- We welcome reviews from anyone. Any committer can approve and merge the changes.
+- Reviewers will likely have questions and comments. They may use terms such as those in [RFC2119](https://tools.ietf.org/html/rfc2119).
diff --git a/README.md b/README.md
index 0b27230..98630b9 100644
--- a/README.md
+++ b/README.md
@@ -297,7 +297,7 @@
## High-Availability (optional)
Additionally, Muchos can be configured to provide High-Availability for HDFS & Accumulo components. By default,
-this feature is off, however it can be turned on by editing the following settings in [muchos.props]
+this feature is off, however it can be turned on by editing the following settings in [muchos.props]
under the `general` section as shown below:
```ini
@@ -305,7 +305,7 @@
nameservice_id = muchoshacluster # Logical name for the cluster, no special characters
```
-Before enabling HA, it is strongly recommended you read the Apache doc for [HDFS HA] & [Accumulo HA]
+Before enabling HA, it is strongly recommended you read the Apache doc for [HDFS HA] & [Accumulo HA]
Also in the `[nodes]` section of [muchos.props] ensure the `journalnode` and `zkfc` service are configured to run.
@@ -340,6 +340,10 @@
10.10.10.10
```
+## Contributions
+
+We welcome contributions to the project. [These notes](./CONTRIBUTING.md) should be helpful.
+
## Powered by
Muchos is powered by the following projects:
@@ -350,16 +354,6 @@
* [azure-cli] - The Azure CLI is a command-line tool for managing Azure resources.
* [ansible-azure] - Ansible includes a suite of modules for interacting with Azure Resource Manager.
-## Muchos Testing
-
-Muchos has unit tests. To run them, first install nose using pip:
-
- pip install nose
-
-The following command runs the unit tests:
-
- nosetests -w lib/
-
[centos7]: https://aws.amazon.com/marketplace/pp/B00O7WM7QW
[aws-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
[awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
diff --git a/ansible/accumulo.yml b/ansible/accumulo.yml
index c35406f..affbd0a 100644
--- a/ansible/accumulo.yml
+++ b/ansible/accumulo.yml
@@ -43,14 +43,14 @@
command: "{{ accumulo_home }}/bin/start-here.sh"
register: start_result
changed_when: "'Starting' in start_result.stdout"
- when: accumulo_major_version == '1' and use_systemd == False
+ when: accumulo_major_version == '1' and not use_systemd
- hosts: workers
tasks:
- name: "start accumulo 2.0 tablet servers"
command: "nohup {{ accumulo_home }}/bin/accumulo-service tserver start"
register: start_result
changed_when: "'Starting' in start_result.stdout"
- when: accumulo_major_version == '2' and use_systemd == False
+ when: accumulo_major_version == '2' and not use_systemd
- hosts: accumulomaster
tasks:
- name: "start accumulo 2.0 master, monitor, gc & tracer"
@@ -62,7 +62,7 @@
- monitor
- gc
- tracer
- when: accumulo_major_version == '2' and use_systemd == False
+ when: accumulo_major_version == '2' and not use_systemd
- hosts: accumulomaster
tasks:
- name: "install and start all the accumulo services using systemd"
@@ -71,12 +71,12 @@
- import_tasks: roles/accumulo/tasks/start-gc.yml
- import_tasks: roles/accumulo/tasks/start-monitor.yml
- import_tasks: roles/accumulo/tasks/start-tracer.yml
- when: use_systemd == True
+ when: use_systemd
become: yes
-
+
- hosts: workers
gather_facts: false
tasks:
- import_tasks: roles/accumulo/tasks/start-tserver.yml
- when: use_systemd == True
+ when: use_systemd
become: yes
diff --git a/ansible/roles/azure/tasks/terminate_cluster.yml b/ansible/azure_terminate.yml
similarity index 96%
rename from ansible/roles/azure/tasks/terminate_cluster.yml
rename to ansible/azure_terminate.yml
index 3676cae..6c82c01 100644
--- a/ansible/roles/azure/tasks/terminate_cluster.yml
+++ b/ansible/azure_terminate.yml
@@ -16,11 +16,10 @@
#
---
-
- hosts: localhost
tasks:
- block:
- - import_tasks: log_analytics_ws_common.yml
+ - import_tasks: roles/azure/tasks/log_analytics_ws_common.yml
- name: Delete workbook
azure_rm_resource:
@@ -56,7 +55,7 @@
resource_group: "{{ resource_group }}"
provider: ManagedIdentity
resource_type: userAssignedIdentities
- resource_name: "{{ user_assigned_identity if user_assigned_identity !='' else vmss_name + '-ua-msi' }}"
+ resource_name: "{{ user_assigned_identity if user_assigned_identity else vmss_name + '-ua-msi' }}"
api_version: '2018-11-30'
state: absent
when: use_adlsg2
diff --git a/ansible/roles/azure/tasks/wipe_adlsg2.yml b/ansible/azure_wipe.yml
similarity index 92%
rename from ansible/roles/azure/tasks/wipe_adlsg2.yml
rename to ansible/azure_wipe.yml
index 47c32d7..f5fca2a 100644
--- a/ansible/roles/azure/tasks/wipe_adlsg2.yml
+++ b/ansible/azure_wipe.yml
@@ -36,7 +36,7 @@
retries: 20
delay: 30
register: result
- until: result is succeeded and (result.changed == False or (result.changed == True and result.container|length > 0))
+ until: result is succeeded and (not result.changed or (result.changed and result.container|length > 0))
loop:
"{{ instance_volumes_preferred.split(',') }}"
when: cluster_type == 'azure' and use_adlsg2
diff --git a/ansible/cancel_shutdown.yml b/ansible/cancel_shutdown.yml
index 5be7184..88350b0 100644
--- a/ansible/cancel_shutdown.yml
+++ b/ansible/cancel_shutdown.yml
@@ -19,4 +19,4 @@
become: yes
tasks:
- name: "cancelling node shutdown"
- shell: shutdown -c
+ command: shutdown -c
diff --git a/ansible/docker.yml b/ansible/docker.yml
index 43eddf7..b8322bf 100644
--- a/ansible/docker.yml
+++ b/ansible/docker.yml
@@ -24,7 +24,9 @@
tasks:
- name: get swarm status
shell: >
- docker info | egrep '^Swarm: ' | cut -d ' ' -f 2
+ set -o pipefail && docker info | egrep '^Swarm: ' | cut -d ' ' -f 2
+ args:
+ executable: bash
register: swarm_status
changed_when: "'active' not in swarm_status.stdout_lines"
- name: initialize swarm
@@ -32,7 +34,7 @@
docker swarm init --advertise-addr={{ ansible_default_ipv4.address }}:2377
when: "'active' not in swarm_status.stdout_lines"
- name: get swarm token
- shell: docker swarm join-token -q worker
+ command: docker swarm join-token -q worker
register: swarm_token
changed_when: "'active' not in swarm_status.stdout_lines"
- hosts: workers:!swarmmanager
@@ -43,7 +45,9 @@
tasks:
- name: get swarm status
shell: >
- docker info | egrep '^Swarm: ' | cut -d ' ' -f 2
+ set -o pipefail && docker info | egrep '^Swarm: ' | cut -d ' ' -f 2
+ args:
+ executable: bash
register: swarm_status
changed_when: "'active' not in swarm_status.stdout_lines"
- name: join worker to swarm
diff --git a/ansible/hadoop.yml b/ansible/hadoop.yml
index c4832ed..37cdafa 100644
--- a/ansible/hadoop.yml
+++ b/ansible/hadoop.yml
@@ -21,25 +21,25 @@
- hosts: journalnode
tasks:
- import_tasks: roles/hadoop/tasks/start-journal.yml
- when: hdfs_ha == True
+ when: hdfs_ha
- hosts: namenode[0]
tasks:
- import_tasks: roles/hadoop/tasks/format-nn.yml
- hosts: namenode[0]
tasks:
- import_tasks: roles/hadoop/tasks/format-zk.yml
- when: hdfs_ha == True
+ when: hdfs_ha
- hosts: namenode
tasks:
- import_tasks: roles/hadoop/tasks/start-zkfc.yml
- when: hdfs_ha == True
+ when: hdfs_ha
- hosts: namenode[0]
tasks:
- import_tasks: roles/hadoop/tasks/start-nn1.yml
- hosts: namenode:!namenode[0]
tasks:
- import_tasks: roles/hadoop/tasks/start-nn2.yml
- when: hdfs_ha == True
+ when: hdfs_ha
- hosts: namenode
tasks:
- import_tasks: roles/hadoop/tasks/start-hdfs.yml
diff --git a/ansible/roles/accumulo/files/accumulo-cluster-systemd b/ansible/roles/accumulo/files/accumulo-cluster-systemd
index e40a4eb..2d311c1 100755
--- a/ansible/roles/accumulo/files/accumulo-cluster-systemd
+++ b/ansible/roles/accumulo/files/accumulo-cluster-systemd
@@ -379,4 +379,4 @@
esac
}
-main "$@"
\ No newline at end of file
+main "$@"
diff --git a/ansible/roles/accumulo/tasks/download.yml b/ansible/roles/accumulo/tasks/download.yml
index e46ca1a..322cafc 100644
--- a/ansible/roles/accumulo/tasks/download.yml
+++ b/ansible/roles/accumulo/tasks/download.yml
@@ -26,4 +26,4 @@
retries: 3
with_items:
- { urlp: "{{ apache_mirror.stdout }}/accumulo/{{ accumulo_version }}", fn: "{{ accumulo_tarball }}", sum: "{{ accumulo_checksum }}" }
- when: accumulo.stat.exists == False
+ when: not accumulo.stat.exists
diff --git a/ansible/roles/accumulo/tasks/main.yml b/ansible/roles/accumulo/tasks/main.yml
index d5a3a3a..bb6dd78 100644
--- a/ansible/roles/accumulo/tasks/main.yml
+++ b/ansible/roles/accumulo/tasks/main.yml
@@ -56,20 +56,20 @@
- name: "configure tservers using managed templates"
template: src=tservers dest={{ accumulo_home }}/conf/{{ accumulo_tservers_fn[accumulo_major_version] }}
- name: "build accumulo native libraries"
- shell: "{{ accumulo_build_native_cmd[accumulo_major_version] }}"
+ command: "{{ accumulo_build_native_cmd[accumulo_major_version] }}"
args:
creates: "{{ accumulo_home }}/lib/native/libaccumulo.so"
- name: "Create accumulo log dir"
file: path={{ worker_data_dirs[0] }}/logs/accumulo state=directory
- name: "Copy the modified accumulo-cluster script that supports systemd to bin"
copy: src=roles/accumulo/files/accumulo-cluster-systemd dest={{ accumulo_home }}/bin/accumulo-cluster-systemd mode=0755
- when: use_systemd == True
+ when: use_systemd
- name: "Remove the exisiting accumulo-service and accumulo-cluster scripts"
file: path={{ accumulo_home }}/bin/{{ item }} state=absent
with_items:
- accumulo-service
- accumulo-cluster
- when: use_systemd == True
+ when: use_systemd
- name: "Create a symlink for accumulo-cluster-systemd"
file: src={{ accumulo_home }}/bin/accumulo-cluster-systemd dest={{ accumulo_home }}/bin/accumulo-cluster mode=0755 state=link
- when: use_systemd == True
+ when: use_systemd
diff --git a/ansible/roles/accumulo/tasks/start-gc.yml b/ansible/roles/accumulo/tasks/start-gc.yml
index ee52c9d..ca94a22 100644
--- a/ansible/roles/accumulo/tasks/start-gc.yml
+++ b/ansible/roles/accumulo/tasks/start-gc.yml
@@ -17,13 +17,13 @@
##
- name: install accumulo gc systemd unit file
template:
- src: roles/accumulo/templates/gc.j2
+ src: roles/accumulo/templates/gc.j2
dest: "/etc/systemd/system/accumulo-gc.service"
mode: 0644
- name: start accumulo-gc using systemd
- systemd:
- state: started
+ systemd:
+ state: started
name: "accumulo-gc"
daemon_reload: yes
enabled: yes
diff --git a/ansible/roles/accumulo/tasks/start-master.yml b/ansible/roles/accumulo/tasks/start-master.yml
index 20b2fdd..da8e3f7 100644
--- a/ansible/roles/accumulo/tasks/start-master.yml
+++ b/ansible/roles/accumulo/tasks/start-master.yml
@@ -17,13 +17,13 @@
##
- name: install accumulo master systemd unit file
template:
- src: roles/accumulo/templates/master.j2
+ src: roles/accumulo/templates/master.j2
dest: "/etc/systemd/system/accumulo-master.service"
mode: 0644
- name: start accumulo-master using systemd
- systemd:
- state: started
+ systemd:
+ state: started
name: "accumulo-master"
daemon_reload: yes
enabled: yes
diff --git a/ansible/roles/accumulo/tasks/start-monitor.yml b/ansible/roles/accumulo/tasks/start-monitor.yml
index 59f300b..13d054c 100644
--- a/ansible/roles/accumulo/tasks/start-monitor.yml
+++ b/ansible/roles/accumulo/tasks/start-monitor.yml
@@ -17,13 +17,13 @@
##
- name: install accumulo monitor systemd unit file
template:
- src: roles/accumulo/templates/monitor.j2
+ src: roles/accumulo/templates/monitor.j2
dest: "/etc/systemd/system/accumulo-monitor.service"
mode: 0644
- name: start accumulo-monitor using systemd
- systemd:
- state: started
+ systemd:
+ state: started
name: "accumulo-monitor"
daemon_reload: yes
enabled: yes
diff --git a/ansible/roles/accumulo/tasks/start-tracer.yml b/ansible/roles/accumulo/tasks/start-tracer.yml
index a38888f..2f7cc36 100644
--- a/ansible/roles/accumulo/tasks/start-tracer.yml
+++ b/ansible/roles/accumulo/tasks/start-tracer.yml
@@ -17,13 +17,13 @@
##
- name: install accumulo tracer systemd unit file
template:
- src: roles/accumulo/templates/tracer.j2
+ src: roles/accumulo/templates/tracer.j2
dest: "/etc/systemd/system/accumulo-tracer.service"
mode: 0644
- name: start accumulo-tracer using systemd
- systemd:
- state: started
+ systemd:
+ state: started
name: "accumulo-tracer"
daemon_reload: yes
enabled: yes
diff --git a/ansible/roles/accumulo/tasks/start-tserver.yml b/ansible/roles/accumulo/tasks/start-tserver.yml
index 508de3e..4ae20b8 100644
--- a/ansible/roles/accumulo/tasks/start-tserver.yml
+++ b/ansible/roles/accumulo/tasks/start-tserver.yml
@@ -17,13 +17,13 @@
##
- name: install accumulo tserver systemd unit file
template:
- src: roles/accumulo/templates/tserver.j2
+ src: roles/accumulo/templates/tserver.j2
dest: "/etc/systemd/system/accumulo-tserver@.service"
mode: 0644
- name: start accumulo-tserver(s) using systemd
- systemd:
- state: started
+ systemd:
+ state: started
name: "accumulo-tserver@{{ item }}.service"
daemon_reload: yes
enabled: yes
diff --git a/ansible/roles/accumulo/templates/gc.j2 b/ansible/roles/accumulo/templates/gc.j2
index defbc52..ffca3a4 100644
--- a/ansible/roles/accumulo/templates/gc.j2
+++ b/ansible/roles/accumulo/templates/gc.j2
@@ -1,6 +1,6 @@
[Unit]
Description=GC Service for Accumulo
-Requires=network.target
+Requires=network.target
After=network.target
[Service]
diff --git a/ansible/roles/accumulo/templates/master.j2 b/ansible/roles/accumulo/templates/master.j2
index 8065d59..b2ef35f 100644
--- a/ansible/roles/accumulo/templates/master.j2
+++ b/ansible/roles/accumulo/templates/master.j2
@@ -1,6 +1,6 @@
[Unit]
Description=Master Service for Accumulo
-Requires=network.target
+Requires=network.target
After=network.target
[Service]
diff --git a/ansible/roles/accumulo/templates/monitor.j2 b/ansible/roles/accumulo/templates/monitor.j2
index f6f496f..782da9d 100644
--- a/ansible/roles/accumulo/templates/monitor.j2
+++ b/ansible/roles/accumulo/templates/monitor.j2
@@ -1,6 +1,6 @@
[Unit]
Description=Monitor Service for Accumulo
-Requires=network.target
+Requires=network.target
After=network.target
[Service]
diff --git a/ansible/roles/accumulo/templates/tserver.j2 b/ansible/roles/accumulo/templates/tserver.j2
index a026483..4e22af5 100644
--- a/ansible/roles/accumulo/templates/tserver.j2
+++ b/ansible/roles/accumulo/templates/tserver.j2
@@ -1,6 +1,6 @@
[Unit]
Description=TServer Service for Accumulo
-Requires=network.target
+Requires=network.target
After=network.target
[Service]
diff --git a/ansible/roles/azure/tasks/create_adlsgen2.yml b/ansible/roles/azure/tasks/create_adlsgen2.yml
index 042903e..8a0e660 100644
--- a/ansible/roles/azure/tasks/create_adlsgen2.yml
+++ b/ansible/roles/azure/tasks/create_adlsgen2.yml
@@ -21,12 +21,14 @@
# 1. Create an Azure ADLS Gen2 storage account.
# 2. Create User Assigned Identity.
# 3. Assign roles to storage accounts.
-# 4. Create filesysystem/container in storage accounts.
+# 4. Create filesystem / container in storage accounts.
# 5. Update tenant_id, client_id and instance_volumes_preferred in muchos.props.
# 6. Assign User Assigned Identity to VMSS.
- name: Generate MD5 checksum based on resource_group name, vmss_name and cluster name
- shell: echo -n {{ resource_group + vmss_name + location }}|md5sum|tr -cd "[:alnum:]"|cut -c 1-16|tr '[:upper:]' '[:lower:]'
+ shell: set -o pipefail && echo -n {{ resource_group + vmss_name + location }}|md5sum|tr -cd "[:alnum:]"|cut -c 1-16|tr '[:upper:]' '[:lower:]'
+ args:
+ executable: bash
register: StorageAccountMD5
- name: Generate random names for storage account names
@@ -38,15 +40,15 @@
InstanceVolumesAuto: []
InstanceVolumesManual: []
-- name: Validate instance_volumes_input
+- name: Validate instance_volumes_input
fail: msg="Variable instance_volumes_input incorrectly specified, Both Manual and Auto cannot be specified at same time"
when: instance_volumes_input.split('|')[0].split(',') != [''] and instance_volumes_input.split('|')[1].split(',') != ['']
-- name: Assign manual or autogenerated volumes
+- name: Assign manual or auto-generated volumes
set_fact:
- InstanceVolumesTemp: "{{ instance_volumes_input.split('|')[0].split(',')|list if instance_volumes_input.split('|')[0].split(',') != [''] else instance_volumes_input.split('|')[1].split(',')|list }}"
+ InstanceVolumesTemp: "{{ instance_volumes_input.split('|')[0].split(',')|list if instance_volumes_input.split('|')[0].split(',') != [''] else instance_volumes_input.split('|')[1].split(',')|list }}"
-- name: Retrieve sequence end number to get the number of storage accounts
+- name: Retrieve sequence end number to get the number of storage accounts
set_fact:
InstanceVolumesEndSequence: "{{ '1' if instance_volumes_input.split('|')[0].split(',') == [''] else InstanceVolumesTemp[0]|int }}"
@@ -56,17 +58,17 @@
with_sequence: start=1 end={{ InstanceVolumesEndSequence|int }}
when: InstanceVolumesTemp[0]|int != 0
-- name: Retrieve ABFSS values when specified manually
+- name: Retrieve ABFSS values when specified manually
set_fact:
InstanceVolumesManual: "{{ InstanceVolumesManual + [ item ] }}"
loop:
"{{ InstanceVolumesTemp }}"
when: item.split('://')[0] == 'abfss' and instance_volumes_input.split('|')[0].split(',') == ['']
-# This is final list of instance volumes
-- name: Assign variables for autogeneration or manual for storage account creation
+# This is final list of instance volumes
+- name: Assign variables for auto-generation or manual for storage account creation
set_fact:
- InstanceVolumes: "{{ InstanceVolumesManual if instance_volumes_input.split('|')[0].split(',') == [''] else InstanceVolumesAuto }}"
+ InstanceVolumes: "{{ InstanceVolumesManual if instance_volumes_input.split('|')[0].split(',') == [''] else InstanceVolumesAuto }}"
- name: Update instance_volumes_preferred in muchos.props
lineinfile:
@@ -74,8 +76,8 @@
regexp: '^instance_volumes_preferred\s*=\s*|^[#]instance_volumes_preferred\s*=\s*'
line: "instance_volumes_preferred = {{ InstanceVolumes|join(',') }}"
-# Not registering variable because storage values are not visible immediately
-- name: Create ADLS Gen2 storage acount using REST API
+# Not registering variable because storage values are not visible immediately
+- name: Create ADLS Gen2 storage account using REST API
azure_rm_resource:
resource_group: "{{ resource_group }}"
provider: Storage
@@ -96,25 +98,25 @@
# Creating User Assigned identity with vmss_name suffixed by ua-msi if not specified in muchos.props
# Not registering variable because user identity values are not visible immediately
-- name: Create User Assigned Identity
+- name: Create User Assigned Identity
azure_rm_resource:
resource_group: "{{ resource_group }}"
provider: ManagedIdentity
resource_type: userAssignedIdentities
- resource_name: "{{ user_assigned_identity if user_assigned_identity !='' else vmss_name + '-ua-msi' }}"
+ resource_name: "{{ user_assigned_identity if user_assigned_identity else vmss_name + '-ua-msi' }}"
api_version: '2018-11-30'
idempotency: yes
state: present
body:
- location: "{{ location }}"
+ location: "{{ location }}"
-# Retrieving facts about User Assigned Identity
+# Retrieving facts about User Assigned Identity
- name: Get facts for User Assigned Identity
azure_rm_resource_info:
resource_group: "{{ resource_group }}"
provider: ManagedIdentity
resource_type: userAssignedIdentities
- resource_name: "{{ user_assigned_identity if user_assigned_identity !='' else vmss_name + '-ua-msi' }}"
+ resource_name: "{{ user_assigned_identity if user_assigned_identity else vmss_name + '-ua-msi' }}"
api_version: '2018-11-30'
register: UserAssignedIdentityInfo
retries: 20
@@ -130,7 +132,7 @@
# This will be used to assign the MSI for VMSS
- name: Format User Assigned Identity for API
set_fact:
- UserAssignedIdentityArr: "{{ UserAssignedIdentityInfo.response|default({})|map(attribute='id')|map('regex_replace','^(.*)$','{\"\\1\":{}}')|list}}"
+ UserAssignedIdentityArr: "{{ UserAssignedIdentityInfo.response|default({})|map(attribute='id')|map('regex_replace','^(.*)$','{\"\\1\":{}}')|list }}" # noqa 206
# Retrieve facts about role assignment
- name: Get role definition id for "Storage Blob Data Owner"
@@ -142,7 +144,7 @@
api_version: '2015-07-01'
register: RoleDefinitionInfo
-# Retrieve storage acount informationn.
+# Retrieve storage account information
- name: Check if the storage accounts are visible
azure_rm_storageaccount_info:
resource_group: "{{ resource_group }}"
@@ -154,12 +156,12 @@
loop:
"{{ InstanceVolumes }}"
-# Retrieve storage accounts id creeated -- Used for account assignments
+# Retrieve storage accounts id created; these are used for account assignments
- name: Get the id of storage accounts created
set_fact:
- StorageAccountsId: "{{StorageAccountsInfo.results | sum(attribute='storageaccounts', start=[]) | map(attribute='id') | list | unique }}"
-
-# Adding this module since role aassignment fails if it already exists.
+ StorageAccountsId: "{{ StorageAccountsInfo.results | sum(attribute='storageaccounts', start=[]) | map(attribute='id') | list | unique }}"
+
+# Adding this module since role assignment fails if it already exists
- name: Get facts about role assignment
azure_rm_roleassignment_info:
scope: "{{ item }}"
@@ -177,7 +179,7 @@
StorageAccountRoles: "{{ item|map(attribute='scope')|list|unique }}"
no_log: True
loop:
- "{{RoleAssignmentResults.results|map(attribute='roleassignments')|list }}"
+ "{{ RoleAssignmentResults.results|map(attribute='roleassignments')|list }}"
# This retry logic is needed due to race condition between storage account create complete and role assignment
- name: Create a role assignment
@@ -186,7 +188,7 @@
assignee_object_id: "{{ UserAssignedIdentityInfo.response|map(attribute='properties')|map(attribute='principalId')|list|join('') }}"
role_definition_id: "{{ RoleDefinitionInfo.response|map(attribute='id')|list|join('') }}"
state: present
- retries: 30
+ retries: 30
delay: 15
register: roleassignresult
until: roleassignresult is succeeded
@@ -195,25 +197,25 @@
when: item not in StorageAccountRoles
# This retry logic is needed due to race condition between storage account creation and creating filesystem
-- name: Create container/Filesystem on ADLS Gen2
+- name: Create container/Filesystem on ADLS Gen2
azure_rm_storageblob:
- resource_group: "{{ resource_group }}"
+ resource_group: "{{ resource_group }}"
storage_account_name: "{{ item.split('@')[1].split('.')[0] }}"
container: "{{ item.split('@')[0].split('://')[1] }}"
retries: 20
delay: 30
- register: createfsresult
- until: createfsresult is succeeded and (createfsresult.changed == False or (createfsresult.changed == True and createfsresult.container|length > 0))
+ register: createfsresult
+ until: createfsresult is succeeded and ((not createfsresult.changed) or (createfsresult.changed and createfsresult.container|length > 0))
loop:
"{{ InstanceVolumes }}"
-# Retrieve tenantId for core-site.xml
+# Retrieve tenantId for core-site.xml
- name: Update tenantId in muchos.props
lineinfile:
path: "{{ deploy_path }}/conf/muchos.props"
regexp: '^azure_tenant_id\s*=\s*|^[#]azure_tenant_id\s*=\s*'
line: "azure_tenant_id = {{ UserAssignedIdentityInfo.response|map(attribute='properties')|map(attribute='tenantId')|list|join('') }}"
-
+
# Retrieve clientId for core-site.xml
- name: Update clientid in muchos.props
lineinfile:
diff --git a/ansible/roles/azure/tasks/create_log_analytics_ws.yml b/ansible/roles/azure/tasks/create_log_analytics_ws.yml
index 126da4d..b7dbbc3 100644
--- a/ansible/roles/azure/tasks/create_log_analytics_ws.yml
+++ b/ansible/roles/azure/tasks/create_log_analytics_ws.yml
@@ -21,11 +21,11 @@
# 1. Create Log Analytics workspace
# 2. Create dashboard and workbook using json templates
# 3. Update az_logs_id and az_logs_key in muchos.props
-#
+#
#
- import_tasks: log_analytics_ws_common.yml
-
+
- name: Deploy log analytics workspace and performance counters
azure_rm_deployment:
resource_group_name: "{{ resource_group }}"
@@ -39,7 +39,7 @@
- name: Gather information about log analytics workspace
azure_rm_loganalyticsworkspace_info:
resource_group: "{{ resource_group }}"
- name: "{{ log_workspace_name }}"
+ name: "{{ log_workspace_name }}"
show_shared_keys: True
retries: 20
delay: 15
diff --git a/ansible/roles/azure/tasks/create_vmss.yml b/ansible/roles/azure/tasks/create_vmss.yml
index 00eb9cd..3094237 100644
--- a/ansible/roles/azure/tasks/create_vmss.yml
+++ b/ansible/roles/azure/tasks/create_vmss.yml
@@ -57,7 +57,7 @@
allocation_method: Static
name: "{{ azure_proxy_host }}-ip"
register: azure_proxy_public_ip
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Create Network Security Group that allows SSH
azure_rm_securitygroup:
@@ -70,7 +70,7 @@
access: Allow
priority: 1001
direction: Inbound
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Create NIC
azure_rm_networkinterface:
@@ -83,7 +83,7 @@
public_ip_address_name: "{{ azure_proxy_host }}-ip"
primary: True
security_group: "{{ azure_proxy_host }}-nsg"
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Create azure proxy virtual machine
azure_rm_virtualmachine:
@@ -95,7 +95,7 @@
admin_username: "{{ admin_username }}"
ssh_password_enabled: false
ssh_public_keys:
- - path: /home/{{admin_username}}/.ssh/authorized_keys
+ - path: /home/{{ admin_username }}/.ssh/authorized_keys
key_data: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
os_disk_caching: ReadWrite
managed_disk_type: Standard_LRS
@@ -108,7 +108,7 @@
- lun: 0
disk_size_gb: 64
managed_disk_type: Standard_LRS
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
# SECTION 3: Create the Azure VMSS for the nodes used by Muchos
- name: Create luns dictionary
@@ -130,7 +130,7 @@
admin_username: "{{ admin_username }}"
ssh_password_enabled: false
ssh_public_keys:
- - path: /home/{{admin_username}}/.ssh/authorized_keys
+ - path: /home/{{ admin_username }}/.ssh/authorized_keys
key_data: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
capacity: "{{ numnodes }}"
virtual_network_name: "{{ vnet }}"
@@ -151,17 +151,22 @@
# SECTION 4: Automatically populate entries in the hosts file and in the muchos.props file, based on the VMSS node details
- name: get instance ids
- shell: "az vmss list-instances --resource-group {{ resource_group }} --name {{ vmss_name }} --query '[].[{name: name,vmresourceid: id}]' -o json | sed -e 's/{{vmss_name}}_/{{vmss_name}}-/g'"
+ shell: "set -o pipefail && az vmss list-instances --resource-group {{ resource_group }} --name {{ vmss_name }} --query '[].[{name: name,vmresourceid: id}]' -o json | sed -e 's/{{ vmss_name }}_/{{ vmss_name }}-/g'"
+ args:
+ executable: bash
register: instance_list
-- set_fact:
+- name: set instance_list
+ set_fact:
instances: "{{ instance_list.stdout|from_json }}"
- name: get private ip addresses
command: "az vmss nic list --resource-group {{ resource_group }} --vmss-name {{ vmss_name }} --query '[].[{ip: ipConfigurations[0].privateIpAddress, vmresourceid: virtualMachine.id}]' -o json"
register: ip_list
-- set_fact:
+- name: obtain ip address list
+ set_fact:
addresses: "{{ ip_list.stdout|from_json }}"
-- set_fact:
+- name: create instance name and ip address dictionaries
+ set_fact:
instances_dict: "{{ instances | json_query('[].{key: vmresourceid, value: name}') }}"
addresses_dict: "{{ addresses | json_query('[].{key: vmresourceid, value: ip}') }}"
@@ -182,13 +187,13 @@
lineinfile:
path: "{{ deploy_path }}/conf/hosts/{{ vmss_name }}"
line: "{{ azure_proxy_host }} {{ azure_proxy_public_ip.state.ip_address }}"
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Write VMSS vms to hosts file
lineinfile:
path: "{{ deploy_path }}/conf/hosts/{{ vmss_name }}"
line: "{{ item1[0] }} {{ item2[0] }}"
- with_items: "{{ instances_dict | json_query('[].key')}}"
+ with_items: "{{ instances_dict | json_query('[].key') }}"
vars:
myquery: "[?key=='{{ item }}'].value"
item1: "{{ instances_dict | json_query(myquery) }}"
@@ -212,7 +217,7 @@
lineinfile:
path: "{{ deploy_path }}/conf/muchos.props"
line: "{{ azure_proxy_host }} = client"
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Assign Accumulo master, HDFS components to the first node of the cluster
lineinfile:
@@ -268,7 +273,7 @@
path: "{{ deploy_path }}/conf/muchos.props"
regexp: '^proxy_hostname\s*=\s*'
line: "proxy_hostname = {{ azure_proxy_host }}"
- when: azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None
+ when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
- name: Change proxy hostname to first node in vmss in muchos.props
lineinfile:
@@ -276,4 +281,4 @@
regexp: '^proxy_hostname\s*=\s*'
line: "proxy_hostname = {{ item }}"
with_items: "{{ instances_dict | json_query('[0].value') }}"
- when: not (azure_proxy_host is defined and azure_proxy_host != '' and azure_proxy_host != None)
+ when: not (azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None)
diff --git a/ansible/roles/azure/tasks/log_analytics_ws_common.yml b/ansible/roles/azure/tasks/log_analytics_ws_common.yml
index 4d9e8fd..86fa102 100644
--- a/ansible/roles/azure/tasks/log_analytics_ws_common.yml
+++ b/ansible/roles/azure/tasks/log_analytics_ws_common.yml
@@ -20,7 +20,9 @@
#
- name: Generate name for Workspace, Dashboard and Workbook
- shell: echo -n {{resource_group + vmss_name + location }}|md5sum|tr -cd "[:alnum:]"|cut -c 1-48
+ shell: set -o pipefail && echo -n {{ resource_group + vmss_name + location }} | md5sum | tr -cd "[:alnum:]" | cut -c 1-48
+ args:
+ executable: bash
register: monitor_name
- name: Set name for log analytics workspace
diff --git a/ansible/roles/azure/tasks/main.yml b/ansible/roles/azure/tasks/main.yml
index 51cea53..bea2bb8 100644
--- a/ansible/roles/azure/tasks/main.yml
+++ b/ansible/roles/azure/tasks/main.yml
@@ -29,4 +29,4 @@
- import_tasks: create_adlsgen2.yml
when: use_adlsg2
- import_tasks: create_log_analytics_ws.yml
- when: az_oms_integration_needed and (az_logs_id is not defined or az_logs_id == '' or az_logs_id == None)
+ when: az_oms_integration_needed and (az_logs_id is not defined or (not az_logs_id) or az_logs_id == None)
diff --git a/ansible/roles/common/tasks/azure.yml b/ansible/roles/common/tasks/azure.yml
index 2329f39..e20bdd0 100644
--- a/ansible/roles/common/tasks/azure.yml
+++ b/ansible/roles/common/tasks/azure.yml
@@ -62,7 +62,7 @@
state: directory
owner: '{{ cluster_user }}'
group: '{{ cluster_group }}'
- when: azure_fileshare_mount is defined and azure_fileshare_mount != '' and azure_fileshare_mount != None
+ when: azure_fileshare_mount is defined and azure_fileshare_mount and azure_fileshare_mount != None
- name: Mount Azure File share
become: yes
mount:
@@ -71,4 +71,4 @@
path: "{{ azure_fileshare_mount }}"
opts: vers=3.0,username={{ azure_fileshare_username }},password={{ azure_fileshare_password }},dir_mode=0777,file_mode=0777,serverino
state: mounted
- when: azure_fileshare_mount is defined and azure_fileshare_mount != '' and azure_fileshare_mount != None
+ when: azure_fileshare_mount is defined and azure_fileshare_mount and azure_fileshare_mount != None
diff --git a/ansible/roles/common/tasks/azure_oms.yml b/ansible/roles/common/tasks/azure_oms.yml
index 756cb81..a502aec 100644
--- a/ansible/roles/common/tasks/azure_oms.yml
+++ b/ansible/roles/common/tasks/azure_oms.yml
@@ -18,7 +18,7 @@
# The tasks below have been added to enable integration with Azure Monitor Logs
- name: "Configure required plugins in collectd"
- template: src=../templates/etc/collectd.d/azure.conf.j2 dest=/etc/collectd.d/azure.conf
+ template: src=roles/common/templates/etc/collectd.d/azure.conf.j2 dest=/etc/collectd.d/azure.conf
- name: "Ensure OMS Agent is installed"
script: "{{ tarballs_dir }}/omsagent.x64.sh --upgrade"
@@ -26,7 +26,7 @@
creates: /opt/microsoft/omsagent/bin/omsadmin.sh
- name: "Configure OMS Agent for workspace"
- shell: /opt/microsoft/omsagent/bin/omsadmin.sh -w {{ az_logs_id }} -s {{ az_logs_key }}
+ command: /opt/microsoft/omsagent/bin/omsadmin.sh -w {{ az_logs_id }} -s {{ az_logs_key }}
args:
creates: /etc/opt/microsoft/omsagent/{{ az_logs_id }}/conf/omsadmin.conf
register: omsagent_configured
@@ -40,5 +40,5 @@
copy: remote_src=true src=/etc/opt/microsoft/omsagent/sysconf/omsagent.d/collectd.conf dest=/etc/opt/microsoft/omsagent/{{ az_logs_id }}/conf/omsagent.d/collectd.conf
- name: "Restart service omsagent, in all cases"
- shell: /opt/microsoft/omsagent/bin/service_control restart {{ az_logs_id }}
+ command: /opt/microsoft/omsagent/bin/service_control restart {{ az_logs_id }}
when: omsagent_configured is succeeded
diff --git a/ansible/roles/common/tasks/azure_oms_selinux.yml b/ansible/roles/common/tasks/azure_oms_selinux.yml
index 1cea400..85ae915 100644
--- a/ansible/roles/common/tasks/azure_oms_selinux.yml
+++ b/ansible/roles/common/tasks/azure_oms_selinux.yml
@@ -16,88 +16,76 @@
- name: Creates directory for SE Policy files
- file: path=/home/{{ cluster_user}}/SEPolicyFiles state=directory
-
+ file: path=/home/{{ cluster_user }}/SEPolicyFiles state=directory
- name: Copy Policy file OMS-collectd.te
- copy: src=roles/common/files/OMS-collectd.te dest=/home/{{ cluster_user}}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
+ copy: src=roles/common/files/OMS-collectd.te dest=/home/{{ cluster_user }}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
- name: Copy Policy file OMS-logrotate.te
- copy: src=roles/common/files/OMS-logrotate.te dest=/home/{{ cluster_user}}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
+ copy: src=roles/common/files/OMS-logrotate.te dest=/home/{{ cluster_user }}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
- name: Copy Policy file OMS-reader3.te
- copy: src=roles/common/files/OMS-reader3.te dest=/home/{{ cluster_user}}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
+ copy: src=roles/common/files/OMS-reader3.te dest=/home/{{ cluster_user }}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
- name: Copy Policy file OMS-writer3.te
- copy: src=roles/common/files/OMS-writer3.te dest=/home/{{ cluster_user}}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
-
-
+ copy: src=roles/common/files/OMS-writer3.te dest=/home/{{ cluster_user }}/SEPolicyFiles owner={{ cluster_user }} group={{ cluster_group }}
- name: Run checkmodule for OMS-collectd.te
command: checkmodule -M -m -o my-collectd.mod OMS-collectd.te
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run checkmodule for OMS-reader3.te
command: checkmodule -M -m -o my-reader3.mod OMS-reader3.te
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run checkmodule for OMS-writer3.te
command: checkmodule -M -m -o my-writer3.mod OMS-writer3.te
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run checkmodule for OMS-logrotate.te
command: checkmodule -M -m -o my-logrotate.mod OMS-logrotate.te
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
-
-
-
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule_package for OMS-collectd.te
command: semodule_package -o OMS-collectd.pp -m my-collectd.mod
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule_package for OMS-reader3.te
command: semodule_package -o OMS-reader3.pp -m my-reader3.mod
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule_package for OMS-writer3.te
command: semodule_package -o OMS-writer3.pp -m my-writer3.mod
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule_package for OMS-logrotate.te
command: semodule_package -o OMS-logrotate.pp -m my-logrotate.mod
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
-
-
-
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule for OMS-collectd.pp
command: semodule -i OMS-collectd.pp
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule for OMS-reader3.pp
command: semodule -i OMS-reader3.pp
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule for OMS-writer3.pp
command: semodule -i OMS-writer3.pp
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
- name: Run semodule for OMS-logrotate.pp
command: semodule -i OMS-logrotate.pp
args:
- chdir: /home/{{ cluster_user}}/SEPolicyFiles
-
-
-
+ chdir: /home/{{ cluster_user }}/SEPolicyFiles
diff --git a/ansible/roles/common/tasks/main.yml b/ansible/roles/common/tasks/main.yml
index 49536cd..bfa6e2f 100644
--- a/ansible/roles/common/tasks/main.yml
+++ b/ansible/roles/common/tasks/main.yml
@@ -47,12 +47,12 @@
- name: "install maven"
unarchive: src={{ tarballs_dir }}/{{ maven_tarball }} dest={{ install_dir }} creates={{ maven_home }}
- name: "chown maven home"
- file: path={{ maven_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group}}
+ file: path={{ maven_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group }}
- name: "install hub"
unarchive: src={{ tarballs_dir }}/{{ hub_tarball }} dest={{ install_dir }} creates={{ hub_home }}
when: install_hub
- name: "chown hub home"
- file: path={{ hub_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group}}
+ file: path={{ hub_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group }}
when: install_hub
- name: "configure collectd"
template: src=etc/collectd.conf.j2 dest=/etc/collectd.conf
diff --git a/ansible/roles/common/tasks/ssh.yml b/ansible/roles/common/tasks/ssh.yml
index 77e8af8..0e94130 100644
--- a/ansible/roles/common/tasks/ssh.yml
+++ b/ansible/roles/common/tasks/ssh.yml
@@ -26,4 +26,4 @@
- name: "add conf/keys to ~/.ssh/authorized_keys"
authorized_key: user={{ cluster_user }} key="{{ lookup('file', 'conf/keys') }}"
- name: "set ssh config"
- copy: src=roles/common/files/ssh_config dest=/home/{{ cluster_user}}/.ssh/config owner={{ cluster_user }} group={{ cluster_group }} mode=0600
+ copy: src=roles/common/files/ssh_config dest=/home/{{ cluster_user }}/.ssh/config owner={{ cluster_user }} group={{ cluster_group }} mode=0600
diff --git a/ansible/roles/fluo/tasks/download.yml b/ansible/roles/fluo/tasks/download.yml
index 5dd2296..0baa72c 100644
--- a/ansible/roles/fluo/tasks/download.yml
+++ b/ansible/roles/fluo/tasks/download.yml
@@ -26,4 +26,4 @@
retries: 3
with_items:
- { urlp: "{{ apache_mirror.stdout }}/fluo/fluo/{{ fluo_version }}", fn: "{{ fluo_tarball }}", sum: "{{ fluo_checksum }}" }
- when: fluo.stat.exists == False
+ when: not fluo.stat.exists
diff --git a/ansible/roles/fluo/tasks/main.yml b/ansible/roles/fluo/tasks/main.yml
index 103b69e..7d88f62 100644
--- a/ansible/roles/fluo/tasks/main.yml
+++ b/ansible/roles/fluo/tasks/main.yml
@@ -38,4 +38,4 @@
- name: "fetch extra fluo dependencies"
command: "{{ fluo_home }}/lib/fetch.sh extra"
- name: "set correct owner & group"
- file: path={{ fluo_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group}}
+ file: path={{ fluo_home }} recurse=yes owner={{ cluster_user }} group={{ cluster_group }}
diff --git a/ansible/roles/fluo_yarn/tasks/download.yml b/ansible/roles/fluo_yarn/tasks/download.yml
index 5b995e3..90115d0 100644
--- a/ansible/roles/fluo_yarn/tasks/download.yml
+++ b/ansible/roles/fluo_yarn/tasks/download.yml
@@ -26,4 +26,4 @@
retries: 3
with_items:
- { urlp: "{{ apache_mirror.stdout }}/fluo/fluo-yarn/{{ fluo_yarn_version }}", fn: "{{ fluo_yarn_tarball }}", sum: "{{ fluo_yarn_checksum }}" }
- when: fluo_yarn.stat.exists == False
+ when: not fluo_yarn.stat.exists
diff --git a/ansible/roles/hadoop/tasks/start-jhs.yml b/ansible/roles/hadoop/tasks/start-jhs.yml
index f701f4e..70e15f7 100644
--- a/ansible/roles/hadoop/tasks/start-jhs.yml
+++ b/ansible/roles/hadoop/tasks/start-jhs.yml
@@ -16,7 +16,7 @@
#
- name: Check if JobHistoryServer is running
- shell: jps | grep "JobHistoryServer"
+ shell: jps | grep "JobHistoryServer"
ignore_errors: yes
changed_when: false
register: jhs_status
diff --git a/ansible/roles/hadoop/tasks/start-nn2.yml b/ansible/roles/hadoop/tasks/start-nn2.yml
index 98ba326..094210b 100644
--- a/ansible/roles/hadoop/tasks/start-nn2.yml
+++ b/ansible/roles/hadoop/tasks/start-nn2.yml
@@ -22,7 +22,7 @@
register: namenode_status
- name: "bootstrap standby"
command: "nohup {{ hadoop_home }}/bin/hdfs namenode -bootstrapStandby -force"
- when: namenode_status.rc == 1
+ when: namenode_status.rc == 1
- name: "start namenode"
command: "nohup {{ hadoop_home }}/sbin/hadoop-daemon.sh start namenode"
- when: namenode_status.rc == 1
+ when: namenode_status.rc == 1
diff --git a/ansible/roles/hadoop/templates/core-site.xml b/ansible/roles/hadoop/templates/core-site.xml
index 583d42e..54563d6 100644
--- a/ansible/roles/hadoop/templates/core-site.xml
+++ b/ansible/roles/hadoop/templates/core-site.xml
@@ -34,12 +34,12 @@
<name>dfs.domain.socket.path</name>
<value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>
-{% if hdfs_ha == True %}
+{% if hdfs_ha %}
<property>
<name>ha.zookeeper.quorum</name>
<value>{{ zookeeper_connect }}</value>
</property>
-{% endif %}
+{% endif %}
{% if cluster_type == 'azure' and use_adlsg2 %}
<property>
<name>fs.azure.account.auth.type</name>
diff --git a/ansible/roles/hadoop/templates/hdfs-site.xml b/ansible/roles/hadoop/templates/hdfs-site.xml
index 2accf01..ff9fe2b 100644
--- a/ansible/roles/hadoop/templates/hdfs-site.xml
+++ b/ansible/roles/hadoop/templates/hdfs-site.xml
@@ -37,7 +37,7 @@
{%- if not loop.last -%} , {%- endif -%}
{%- endfor %}</value>
</property>
-{% if hdfs_ha == False %}
+{% if not hdfs_ha %}
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>{{ groups['namenode'][0] }}:50090</value>
diff --git a/ansible/roles/influxdb/handlers/main.yml b/ansible/roles/influxdb/handlers/main.yml
index 156c040..492026b 100644
--- a/ansible/roles/influxdb/handlers/main.yml
+++ b/ansible/roles/influxdb/handlers/main.yml
@@ -18,7 +18,7 @@
- name: "restart influxdb"
service: name=influxdb state=restarted
- name: "setup influxdb"
- shell: /opt/influxdb/influx -import -path /etc/opt/influxdb/fluo_metrics_setup.txt
+ command: /opt/influxdb/influx -import -path /etc/opt/influxdb/fluo_metrics_setup.txt
register: cresult
until: cresult.rc == 0
retries: 5
diff --git a/ansible/roles/mesos/tasks/main.yml b/ansible/roles/mesos/tasks/main.yml
index aea70e7..75ea56a 100644
--- a/ansible/roles/mesos/tasks/main.yml
+++ b/ansible/roles/mesos/tasks/main.yml
@@ -18,10 +18,10 @@
- name: "add mesosphere repo"
yum: name={{ mesosphere_yum_repo }} state=present
- name: "install mesos, marathon & docker"
- yum: name={{item}} state=present
+ yum: name={{ item }} state=present
with_items:
- - mesos-{{mesos_version}}
- - marathon-{{marathon_version}}
+ - mesos-{{ mesos_version }}
+ - marathon-{{ marathon_version }}
- docker
- name: "docker is running"
service: name=docker state=started
diff --git a/ansible/roles/proxy/tasks/download.yml b/ansible/roles/proxy/tasks/download.yml
index 85ef2e9..446336e 100644
--- a/ansible/roles/proxy/tasks/download.yml
+++ b/ansible/roles/proxy/tasks/download.yml
@@ -27,7 +27,7 @@
- { urlp: "{{ apache_mirror.stdout }}/maven/maven-3/{{ maven_version }}/binaries", fn: "{{ maven_tarball }}", sum: "{{ maven_checksum }}" }
- { urlp: "https://github.com/github/hub/releases/download/v{{ hub_version }}", fn: "{{ hub_tarball }}", sum: "{{ hub_checksum }}" }
-# This is currently needed to run hadoop with Java 11 (see https://github.com/apache/fluo-muchos/issues/266)
+# This is currently needed to run hadoop with Java 11 (see https://github.com/apache/fluo-muchos/issues/266)
- name: "Download javax.activation-api for Hadoop 3 when Java 11 is used"
maven_artifact:
group_id: javax.activation
@@ -36,10 +36,12 @@
version: 1.2.0
dest: "{{ user_home }}/mvn_dep/"
mode: 0644
- when: hadoop_major_version == '3' and java_product_version == 11
+ when: hadoop_major_version == '3' and java_product_version == 11
- name: "download omsagent installer to proxy"
- shell: eval $(curl -s https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh | grep -e 'GITHUB_RELEASE_X64=' -e 'BUNDLE_X64=') && wget ${GITHUB_RELEASE_X64}${BUNDLE_X64} -O {{ tarballs_dir }}/omsagent.x64.sh
+ shell: set -o pipefail && eval $(curl -s https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh | grep -e 'GITHUB_RELEASE_X64=' -e 'BUNDLE_X64=') && wget ${GITHUB_RELEASE_X64}${BUNDLE_X64} -O {{ tarballs_dir }}/omsagent.x64.sh
+ args:
+ executable: bash
register: result
until: result is not failed
retries: 3
diff --git a/ansible/roles/proxy/tasks/get-asf-mirror.yml b/ansible/roles/proxy/tasks/get-asf-mirror.yml
index 9a2a220..c381965 100644
--- a/ansible/roles/proxy/tasks/get-asf-mirror.yml
+++ b/ansible/roles/proxy/tasks/get-asf-mirror.yml
@@ -1,6 +1,7 @@
- name: "determine best apache mirror to use"
- shell: curl -sk https://www.apache.org/dyn/closer.cgi?as_json | jq -r .preferred
+ shell: set -o pipefail && curl -sk https://www.apache.org/dyn/closer.cgi?as_json | jq -r .preferred
args:
+ executable: bash
warn: no
register: apache_mirror
retries: 10
diff --git a/ansible/roles/spark/tasks/download.yml b/ansible/roles/spark/tasks/download.yml
index 65afc5e..886cbf6 100644
--- a/ansible/roles/spark/tasks/download.yml
+++ b/ansible/roles/spark/tasks/download.yml
@@ -26,4 +26,4 @@
retries: 3
with_items:
- { urlp: "{{ apache_mirror.stdout }}/spark/spark-{{ spark_version }}", fn: "{{ spark_tarball }}", sum: "{{ spark_checksum }}" }
- when: spark.stat.exists == False
+ when: not spark.stat.exists
diff --git a/ansible/roles/spark/tasks/start-spark-history.yml b/ansible/roles/spark/tasks/start-spark-history.yml
index ad136c8..a709fda 100644
--- a/ansible/roles/spark/tasks/start-spark-history.yml
+++ b/ansible/roles/spark/tasks/start-spark-history.yml
@@ -16,7 +16,7 @@
#
- name: "ensure spark history directory exists in hdfs"
- command: "{{ hadoop_home}}/bin/hdfs dfs -mkdir -p /spark/history"
+ command: "{{ hadoop_home }}/bin/hdfs dfs -mkdir -p /spark/history"
register: mk_hist_dir
changed_when: mk_hist_dir.rc != 0
- name: "start spark history server"
diff --git a/ansible/roles/zookeeper/tasks/start-zookeeper.yml b/ansible/roles/zookeeper/tasks/start-zookeeper.yml
index cb2f9dc..22b2d3c 100644
--- a/ansible/roles/zookeeper/tasks/start-zookeeper.yml
+++ b/ansible/roles/zookeeper/tasks/start-zookeeper.yml
@@ -20,12 +20,12 @@
- name: "ensure myid file is set"
template: src=roles/zookeeper/templates/myid dest={{ default_data_dirs[0] }}/zookeeper/myid
- name: Check if Zookeeper is running
- shell: jps | grep "QuorumPeerMain"
+ shell: jps | grep "QuorumPeerMain"
ignore_errors: yes
changed_when: false
register: zookeeper_status
- name: "start zookeeper"
- shell: "{{ zookeeper_home }}/bin/zkServer.sh start"
+ command: "{{ zookeeper_home }}/bin/zkServer.sh start"
register: zk_start
changed_when: "'STARTED' in zk_start.stdout"
when: zookeeper_status.rc == 1
diff --git a/ansible/wipe-systemd.yml b/ansible/wipe-systemd.yml
index 8c4e6b0..0d8d9f9 100644
--- a/ansible/wipe-systemd.yml
+++ b/ansible/wipe-systemd.yml
@@ -61,5 +61,5 @@
- name: "reload config changes"
systemd: daemon_reload=yes
- - name: "reset failed on every node"
- shell: systemctl reset-failed
+ - name: "reset failed on every node" # noqa 303
+ command: systemctl reset-failed
diff --git a/ansible/wipe.yml b/ansible/wipe.yml
index 429adff..8cd5c36 100644
--- a/ansible/wipe.yml
+++ b/ansible/wipe.yml
@@ -25,11 +25,20 @@
- grafana-server
- name: "wipe influxdb data"
file: path={{ default_data_dirs[0] }}/influxdb state=absent
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
+
- name: "wipe grafana db"
file: path=/var/lib/grafana/grafana.db state=absent
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
- import_playbook: wipe-systemd.yml
- when: use_systemd == True
+ when: use_systemd
- import_playbook: kill.yml
@@ -46,37 +55,58 @@
- "{{ maven_home }}"
- "{{ hub_home }}"
- "{{ fluo_yarn_home }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
- hosts: hadoop
tasks:
- name: "wipe hadoop data"
- file: path={{item}}/hadoop state=absent
+ file: path={{ item }}/hadoop state=absent
with_items: "{{ worker_data_dirs }}"
- when: cluster_type != 'azure'
- - name: "remove hadoop logs"
- shell: rm -rf {{ hadoop_home }}/logs/*
- when: cluster_type != 'azure'
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
+ - name: "remove logs folder"
+ file: path={{ item }}/logs state=absent
+ with_items: "{{ worker_data_dirs }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
- hosts: zookeepers
tasks:
- name: "wipe zookeeper data"
- file: path={{ default_data_dirs[0] }}/zookeeper state=absent
- when: cluster_type != 'azure'
- - name: "remove zookeeper logs"
- file: path={{ zookeeper_home }}/zookeeper.out state=absent
- when: cluster_type != 'azure'
+ file: path={{ item }}/zookeeper state=absent
+ with_items: "{{ default_data_dirs }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
+ - name: "remove logs folder"
+ file: path={{ item }}/logs state=absent
+ with_items: "{{ default_data_dirs }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
- hosts: accumulo
tasks:
- - name: "remove accumulo logs"
- shell: rm -rf {{ accumulo_home }}/logs/*
- when: cluster_type != 'azure'
-
-# Using rm -rf instead of Ansible file module to remove multiple levels of subfolder
-- hosts: all
- tasks:
- - name: "wipe data and log directories for Azure"
- shell: rm -rf {{ item }}/*
- loop:
- "{{ worker_data_dirs }}"
- when: cluster_type == 'azure'
+ - name: "remove accumulo folder"
+ file: path={{ item }}/accumulo state=absent
+ with_items: "{{ worker_data_dirs }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
+ - name: "remove logs folder"
+ file: path={{ item }}/logs state=absent
+ with_items: "{{ worker_data_dirs }}"
+ register: delete_task
+ retries: 10
+ delay: 3
+ until: delete_task is success
diff --git a/docs/azure-ephemeral-disks.md b/docs/azure-ephemeral-disks.md
index 98e2135..b4e2cf5 100644
--- a/docs/azure-ephemeral-disks.md
+++ b/docs/azure-ephemeral-disks.md
@@ -8,7 +8,7 @@
However, if you'd like to use only the ephemeral / temporary disk storage for HDFS, you first need to understand that
using temp storage will result in lost data across VM deallocate - start cycles. If that behavior is acceptable
for your dev/test scenario, there are two options available to use ephemeral storage within Azure:
-* Use the temporary SSD disk which is available on most VM types. This tends to be smaller in size. Refer to the
+* Use the temporary SSD disk which is available on most VM types. This tends to be smaller in size. Refer to the
[Azure VM sizes](https://docs.microsoft.com/en-us/azure/virtual-machines/dv3-dsv3-series) page for details on temp storage sizes
* Use the [Lsv2 series VMs](https://docs.microsoft.com/en-us/azure/virtual-machines/lsv2-series) which offer larger amounts of NVME based temp storage
@@ -16,10 +16,10 @@
* `data_disk_count` needs to be set to 0
* `mount_root` within the `azure` section needs to be set to `/mnt/resource'
-If you'd like larger NVME temporary disks, another option is to use the storage-optimized Lsv2 VM type in Azure. To use the
+If you'd like larger NVME temporary disks, another option is to use the storage-optimized Lsv2 VM type in Azure. To use the
NVME disks available in these VMs, you must change the following within the `azure` section within muchos.props:
* `vm_sku` needs to be set to one of the sizes from [this page](https://docs.microsoft.com/en-us/azure/virtual-machines/lsv2-series), for example Standard_L8s_v2
* `data_disk_count` needs to be set to 0
* `mount_root` within the `azure` section should be set to `/var/data` (which is also the default)
* `azure_disk_device_path` should be set to `/dev`
-* `azure_disk_device_pattern` should be set to `nvme*n1`
\ No newline at end of file
+* `azure_disk_device_pattern` should be set to `nvme*n1`
diff --git a/lib/main.py b/lib/main.py
index e2149c6..cef6dc3 100644
--- a/lib/main.py
+++ b/lib/main.py
@@ -16,68 +16,80 @@
# limitations under the License.
#
-import os
-import sys
+from os import environ
+from os import path
from sys import exit
from muchos.config import DeployConfig
from muchos.util import parse_args
-from os.path import isfile, join
def main():
- deploy_path = os.environ.get('MUCHOS_HOME')
+ deploy_path = environ.get("MUCHOS_HOME")
if not deploy_path:
- exit('ERROR - MUCHOS_HOME env variable must be set!')
- if not os.path.isdir(deploy_path):
- exit('ERROR - Directory set by MUCHOS_HOME does not exist: ' + deploy_path)
+ exit("ERROR - MUCHOS_HOME env variable must be set!")
+ if not path.isdir(deploy_path):
+ exit(
+ "ERROR - Directory set by MUCHOS_HOME does not exist: "
+ + deploy_path
+ )
- config_path = join(deploy_path, "conf/muchos.props")
- if not isfile(config_path):
- exit('ERROR - A config file does not exist at ' + config_path)
- checksums_path = join(deploy_path, "conf/checksums")
- if not isfile(checksums_path):
- exit('ERROR - A checksums file does not exist at ' + checksums_path)
+ config_path = path.join(deploy_path, "conf/muchos.props")
+ if not path.isfile(config_path):
+ exit("ERROR - A config file does not exist at " + config_path)
+ checksums_path = path.join(deploy_path, "conf/checksums")
+ if not path.isfile(checksums_path):
+ exit("ERROR - A checksums file does not exist at " + checksums_path)
- hosts_dir = join(deploy_path, "conf/hosts/")
+ hosts_dir = path.join(deploy_path, "conf/hosts/")
# parse command line args
retval = parse_args(hosts_dir)
if not retval:
print("Invalid command line arguments. For help, use 'muchos -h'")
- sys.exit(1)
+ exit(1)
(opts, action, args) = retval
- hosts_path = join(hosts_dir, opts.cluster)
+ hosts_path = path.join(hosts_dir, opts.cluster)
- templates_path = join(deploy_path, "conf/templates/")
+ templates_path = path.join(deploy_path, "conf/templates/")
- config = DeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, opts.cluster)
+ config = DeployConfig(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ opts.cluster,
+ )
config.verify_config(action)
- if action == 'config':
- if opts.property == 'all':
+ if action == "config":
+ if opts.property == "all":
config.print_all()
else:
config.print_property(opts.property)
else:
- cluster_type = config.get('general', 'cluster_type')
- if cluster_type == 'existing':
+ cluster_type = config.get("general", "cluster_type")
+ if cluster_type == "existing":
from muchos.existing import ExistingCluster
+
cluster = ExistingCluster(config)
cluster.perform(action)
- elif cluster_type == 'ec2':
+ elif cluster_type == "ec2":
from muchos.ec2 import Ec2Cluster, Ec2ClusterTemplate
- if config.has_option('ec2', 'cluster_template'):
+
+ if config.has_option("ec2", "cluster_template"):
cluster = Ec2ClusterTemplate(config)
else:
cluster = Ec2Cluster(config)
cluster.perform(action)
- elif cluster_type == 'azure':
+ elif cluster_type == "azure":
from muchos.azure import VmssCluster
+
cluster = VmssCluster(config)
cluster.perform(action)
else:
- exit('Unknown cluster_type: ' + cluster_type)
+ exit("Unknown cluster_type: " + cluster_type)
main()
diff --git a/lib/muchos/__init__.py b/lib/muchos/__init__.py
index 555eccf..cce3aca 100644
--- a/lib/muchos/__init__.py
+++ b/lib/muchos/__init__.py
@@ -14,4 +14,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
diff --git a/lib/muchos/azure.py b/lib/muchos/azure.py
index d6ac740..b7ae53e 100644
--- a/lib/muchos/azure.py
+++ b/lib/muchos/azure.py
@@ -18,16 +18,15 @@
import json
import subprocess
-from os.path import join
-from .existing import ExistingCluster
+from os import path
from azure.common.client_factory import get_client_from_cli_profile
from azure.mgmt.compute import ComputeManagementClient
+from .existing import ExistingCluster
# For Muchos deployments on Microsoft Azure, we use Virtual Machine Scale Sets
# VMSS is the most efficient way to provision large numbers of VMs in Azure
class VmssCluster(ExistingCluster):
-
def __init__(self, config):
ExistingCluster.__init__(self, config)
@@ -38,38 +37,59 @@
azure_config["hdfs_ha"] = config.get("general", "hdfs_ha")
azure_config["vmss_name"] = config.cluster_name
azure_config["deploy_path"] = config.deploy_path
- azure_config = {k: VmssCluster._parse_config_value(v)
- for k, v in azure_config.items()}
- subprocess.call(["ansible-playbook",
- join(config.deploy_path, "ansible/azure.yml"),
- "--extra-vars", json.dumps(azure_config)])
+ azure_config = {
+ k: VmssCluster._parse_config_value(v)
+ for k, v in azure_config.items()
+ }
+ subprocess.call(
+ [
+ "ansible-playbook",
+ path.join(config.deploy_path, "ansible/azure.yml"),
+ "--extra-vars",
+ json.dumps(azure_config),
+ ]
+ )
def status(self):
config = self.config
azure_vars_dict = dict(config.items("azure"))
compute_client = get_client_from_cli_profile(ComputeManagementClient)
vmss_status = compute_client.virtual_machine_scale_sets.get(
- azure_vars_dict["resource_group"],
- self.config.cluster_name)
- print('name:', vmss_status.name,
- '\nprovisioning_state:', vmss_status.provisioning_state)
+ azure_vars_dict["resource_group"], self.config.cluster_name
+ )
+ print(
+ "name:",
+ vmss_status.name,
+ "\nprovisioning_state:",
+ vmss_status.provisioning_state,
+ )
def terminate(self):
config = self.config
azure_config = dict(config.items("azure"))
azure_config["vmss_name"] = config.cluster_name
azure_config["deploy_path"] = config.deploy_path
- azure_config = {k: VmssCluster._parse_config_value(v)
- for k, v in azure_config.items()}
- print("All of the Muchos resources provisioned in resource group '{0}'"
- " will be deleted!".format(azure_config['resource_group']))
+ azure_config = {
+ k: VmssCluster._parse_config_value(v)
+ for k, v in azure_config.items()
+ }
+ print(
+ "All of the Muchos resources provisioned in resource group '{0}'"
+ " will be deleted!".format(azure_config["resource_group"])
+ )
response = input("Do you want to continue? (y/n) ")
if response == "y":
- subprocess.call(["ansible-playbook",
- join(config.deploy_path,
- "ansible/roles/azure/tasks/terminate_cluster.yml"),
- "--extra-vars", json.dumps(azure_config)])
+ subprocess.call(
+ [
+ "ansible-playbook",
+ path.join(
+ config.deploy_path, "ansible/azure_terminate.yml",
+ ),
+ "--extra-vars",
+ json.dumps(azure_config),
+ ]
+ )
else:
print("Aborted termination")
@@ -81,19 +101,30 @@
azure_config["vmss_name"] = config.cluster_name
azure_config["cluster_type"] = config.get("general", "cluster_type")
azure_config["deploy_path"] = config.deploy_path
- azure_config = {k: VmssCluster._parse_config_value(v)
- for k, v in azure_config.items()}
- retcode = subprocess.call(["ansible-playbook",
- join(config.deploy_path, "ansible/roles/azure/tasks/wipe_adlsg2.yml"),
- "--extra-vars", json.dumps(azure_config)])
+ azure_config = {
+ k: VmssCluster._parse_config_value(v)
+ for k, v in azure_config.items()
+ }
+ retcode = subprocess.call(
+ [
+ "ansible-playbook",
+ path.join(config.deploy_path, "ansible/azure_wipe.yml",),
+ "--extra-vars",
+ json.dumps(azure_config),
+ ]
+ )
if retcode != 0:
- exit("ERROR - Command failed with return code of {0}".format(retcode))
+ exit(
+ "ERROR - Command failed with return code of {0}".format(
+ retcode
+ )
+ )
- def _parse_config_value(v):
+ def _parse_config_value(v): # noqa
if v.isdigit():
return int(v)
- if v.lower() in ('true', 'yes'):
+ if v.lower() in ("true", "yes"):
return True
- if v.lower() in ('false', 'no'):
+ if v.lower() in ("false", "no"):
return False
return v
diff --git a/lib/muchos/config/__init__.py b/lib/muchos/config/__init__.py
index 13294f1..99a88b5 100644
--- a/lib/muchos/config/__init__.py
+++ b/lib/muchos/config/__init__.py
@@ -15,23 +15,51 @@
# limitations under the License.
#
-from muchos.config.base import BaseConfig, SERVICES, OPTIONAL_SERVICES
-from muchos.config.existing import ExistingDeployConfig
-from muchos.config.ec2 import Ec2DeployConfig
-from muchos.config.azure import AzureDeployConfig
+from .existing import ExistingDeployConfig
+from .ec2 import Ec2DeployConfig
+from .azure import AzureDeployConfig
from configparser import ConfigParser
-def DeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
+
+def DeployConfig(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+):
c = ConfigParser()
c.read(config_path)
- cluster_type = c.get('general', 'cluster_type')
+ cluster_type = c.get("general", "cluster_type")
- if cluster_type == 'existing':
- return ExistingDeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
+ if cluster_type == "existing":
+ return ExistingDeployConfig(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
- if cluster_type == 'ec2':
- return Ec2DeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
+ if cluster_type == "ec2":
+ return Ec2DeployConfig(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
- if cluster_type == 'azure':
- return AzureDeployConfig(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
\ No newline at end of file
+ if cluster_type == "azure":
+ return AzureDeployConfig(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
diff --git a/lib/muchos/config/azure.py b/lib/muchos/config/azure.py
index b07fba6..2c3a802 100644
--- a/lib/muchos/config/azure.py
+++ b/lib/muchos/config/azure.py
@@ -15,32 +15,46 @@
# limitations under the License.
#
-from muchos.config import BaseConfig
-from muchos.config.decorators import *
-from muchos.config.validators import *
from sys import exit
-from muchos.util import get_ephemeral_devices, get_arch
-import os
-import json
-import glob
+from .base import BaseConfig
+from .decorators import ansible_host_var, is_valid, default
+from .validators import is_type, is_in
class AzureDeployConfig(BaseConfig):
- def __init__(self, deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
- super(AzureDeployConfig, self).__init__(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
+ def __init__(
+ self,
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ ):
+ super(AzureDeployConfig, self).__init__(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
def verify_config(self, action):
self._verify_config(action)
- proxy = self.get('general', 'proxy_hostname')
- cluster_type = self.get('general', 'cluster_type')
- if cluster_type not in ['azure']:
+ proxy = self.get("general", "proxy_hostname")
+ cluster_type = self.get("general", "cluster_type")
+ if cluster_type not in ["azure"]:
if not proxy:
exit("ERROR - proxy.hostname must be set in muchos.props")
if proxy not in self.node_d:
- exit("ERROR - The proxy (set by property proxy_hostname={0}) cannot be found in 'nodes' section of "
- "muchos.props".format(proxy))
+ exit(
+ "ERROR - The proxy (set by property proxy_hostname={0}) "
+ "cannot be found in 'nodes' section of "
+ "muchos.props".format(proxy)
+ )
def verify_launch(self):
pass
@@ -49,7 +63,7 @@
return {}
def mount_root(self):
- return self.get('azure', 'mount_root')
+ return self.get("azure", "mount_root")
def data_dirs_common(self, nodeType):
data_dirs = []
@@ -62,15 +76,18 @@
return data_dirs
# Check if using Lsv2 NVME temp storage for HDFS
- lsv2_vm_disk_map = { "Standard_L8s_v2": 1,
+ lsv2_vm_disk_map = {
+ "Standard_L8s_v2": 1,
"Standard_L16s_v2": 2,
"Standard_L32s_v2": 4,
"Standard_L48s_v2": 6,
"Standard_L64s_v2": 8,
- "Standard_L80s_v2": 10}
+ "Standard_L80s_v2": 10,
+ }
if num_disks == 0 and self.vm_sku() in lsv2_vm_disk_map.keys():
- # pretend that we have N data disks - in this case those are NVME temp disks
+ # pretend that we have N data disks
+ # in this case those are NVME temp disks
num_disks = lsv2_vm_disk_map[self.vm_sku()]
# Persistent data disks attached to VMs
@@ -84,87 +101,86 @@
drive_ids = []
range_var = self.data_disk_count() + 1
for i in range(1, range_var):
- drive_ids.append(self.get("azure", "metrics_drive_root") +
- str(i))
+ drive_ids.append(self.get("azure", "metrics_drive_root") + str(i))
return drive_ids
@ansible_host_var
def vm_sku(self):
- return self.get('azure', 'vm_sku')
+ return self.get("azure", "vm_sku")
@ansible_host_var
@is_valid(is_type(int))
def data_disk_count(self):
- return self.getint('azure', 'data_disk_count')
+ return self.getint("azure", "data_disk_count")
@ansible_host_var
- @default('/dev/disk/azure/scsi1')
+ @default("/dev/disk/azure/scsi1")
def azure_disk_device_path(self):
- return self.get('azure', 'azure_disk_device_path')
+ return self.get("azure", "azure_disk_device_path")
@ansible_host_var
- @default('lun*')
+ @default("lun*")
def azure_disk_device_pattern(self):
- return self.get('azure', 'azure_disk_device_pattern')
+ return self.get("azure", "azure_disk_device_pattern")
@ansible_host_var
@default(None)
def azure_fileshare_mount(self):
- return self.get('azure', 'azure_fileshare_mount')
+ return self.get("azure", "azure_fileshare_mount")
@ansible_host_var
@default(None)
def azure_fileshare(self):
- return self.get('azure', 'azure_fileshare')
-
+ return self.get("azure", "azure_fileshare")
+
@ansible_host_var
@default(None)
def azure_fileshare_username(self):
- return self.get('azure', 'azure_fileshare_username')
+ return self.get("azure", "azure_fileshare_username")
@ansible_host_var
@default(None)
def azure_fileshare_password(self):
- return self.get('azure', 'azure_fileshare_password')
+ return self.get("azure", "azure_fileshare_password")
- @ansible_host_var(name='az_oms_integration_needed')
+ @ansible_host_var(name="az_oms_integration_needed")
@default(False)
@is_valid(is_in([True, False]))
def omsIntegrationNeeded(self):
- return self.getboolean('azure', 'az_oms_integration_needed')
+ return self.getboolean("azure", "az_oms_integration_needed")
- @ansible_host_var(name='az_logs_id')
+ @ansible_host_var(name="az_logs_id")
@default(None)
def logs_id(self):
- return self.get('azure', 'az_logs_id')
+ return self.get("azure", "az_logs_id")
- @ansible_host_var(name='az_logs_key')
+ @ansible_host_var(name="az_logs_key")
@default(None)
def logs_key(self):
- return self.get('azure', 'az_logs_key')
+ return self.get("azure", "az_logs_key")
- @ansible_host_var(name='use_adlsg2')
+ @ansible_host_var(name="use_adlsg2")
@is_valid(is_in([True, False]))
@default(False)
def use_adlsg2(self):
- return self.getboolean('azure', 'use_adlsg2')
+ return self.getboolean("azure", "use_adlsg2")
- @ansible_host_var(name='azure_tenant_id')
+ @ansible_host_var(name="azure_tenant_id")
@default(None)
def azure_tenant_id(self):
- return self.get('azure', 'azure_tenant_id')
+ return self.get("azure", "azure_tenant_id")
- @ansible_host_var(name='azure_client_id')
+ @ansible_host_var(name="azure_client_id")
@default(None)
def azure_client_id(self):
- return self.get('azure', 'azure_client_id')
+ return self.get("azure", "azure_client_id")
- @ansible_host_var(name='principal_id')
+ @ansible_host_var(name="principal_id")
@default(None)
def principal_id(self):
- return self.get('azure', 'principal_id')
+ return self.get("azure", "principal_id")
- @ansible_host_var(name='instance_volumes_preferred')
+ @ansible_host_var(name="instance_volumes_preferred")
@default(None)
def instance_volumes_preferred(self):
- return self.get('azure', 'instance_volumes_preferred')
+ return self.get("azure", "instance_volumes_preferred")
diff --git a/lib/muchos/config/base.py b/lib/muchos/config/base.py
index 7ffd3aa..2956b57 100644
--- a/lib/muchos/config/base.py
+++ b/lib/muchos/config/base.py
@@ -16,118 +16,174 @@
#
from abc import ABCMeta, abstractmethod
-
from collections import ChainMap
from configparser import ConfigParser
-from sys import exit
-from muchos.config.decorators import *
-from muchos.config.validators import *
-from muchos.util import get_ephemeral_devices, get_arch
-import os
-import json
-import glob
from distutils.version import StrictVersion
+from os.path import isfile
+from sys import exit
+from .decorators import (
+ ansible_host_var,
+ ansible_play_var,
+ get_ansible_vars,
+ is_valid,
+)
+from .validators import is_in
-SERVICES = ['zookeeper', 'namenode', 'resourcemanager', 'accumulomaster', 'mesosmaster', 'worker', 'fluo', 'fluo_yarn', 'metrics', 'spark', 'client', 'swarmmanager', 'journalnode', 'zkfc']
+SERVICES = [
+ "zookeeper",
+ "namenode",
+ "resourcemanager",
+ "accumulomaster",
+ "mesosmaster",
+ "worker",
+ "fluo",
+ "fluo_yarn",
+ "metrics",
+ "spark",
+ "client",
+ "swarmmanager",
+ "journalnode",
+ "zkfc",
+]
-OPTIONAL_SERVICES = ['fluo', 'fluo_yarn', 'metrics', 'mesosmaster', 'spark', 'client', 'swarmmanager', 'journalnode', 'zkfc']
+OPTIONAL_SERVICES = [
+ "fluo",
+ "fluo_yarn",
+ "metrics",
+ "mesosmaster",
+ "spark",
+ "client",
+ "swarmmanager",
+ "journalnode",
+ "zkfc",
+]
_HOST_VAR_DEFAULTS = {
- 'accumulo_home': '"{{ install_dir }}/accumulo-{{ accumulo_version }}"',
- 'accumulo_instance': None,
- 'accumulo_major_version': '"{{ accumulo_version.split(\'.\')[0] }}"',
- 'accumulo_password': None,
- 'accumulo_tarball': 'accumulo-{{ accumulo_version }}-bin.tar.gz',
- 'accumulo_version': None,
- 'cluster_type': None,
- 'cluster_group': None,
- 'cluster_user': None,
- 'default_data_dirs': None,
- 'download_software': None,
- 'fluo_home': '"{{ install_dir }}/fluo-{{ fluo_version }}"',
- 'fluo_tarball': 'fluo-{{ fluo_version }}-bin.tar.gz',
- 'fluo_version': None,
- 'fluo_yarn_home': '"{{ install_dir }}/fluo-yarn-{{ fluo_yarn_version }}"',
- 'fluo_yarn_tarball': 'fluo-yarn-{{ fluo_yarn_version }}-bin.tar.gz',
- 'fluo_yarn_version': None,
- 'hadoop_home': '"{{ install_dir }}/hadoop-{{ hadoop_version }}"',
- 'hadoop_tarball': 'hadoop-{{ hadoop_version }}.tar.gz',
- 'hadoop_version': None,
- 'hadoop_major_version': '"{{ hadoop_version.split(\'.\')[0] }}"',
- 'hdfs_root': "{% if hdfs_ha %}hdfs://{{ nameservice_id }}{% else %}hdfs://{{ groups[\'namenode\'][0] }}:8020{% endif %}",
- 'hdfs_ha': None,
- 'nameservice_id': None,
- 'num_tservers': 1,
- 'install_dir': None,
- 'install_hub': None,
- 'java_home': '"/usr/lib/jvm/java"',
- 'java_package': '"java-1.8.0-openjdk-devel"',
- 'journal_quorum': "{% for host in groups['journalnode'] %}{{ host }}:8485{% if not loop.last %};{% endif %}{% endfor %}",
- 'maven_home': '"{{ install_dir }}/apache-maven-{{ maven_version }}"',
- 'maven_tarball': 'apache-maven-{{ maven_version }}-bin.tar.gz',
- 'maven_version': '3.6.3',
- 'spark_home': '"{{ install_dir }}/spark-{{ spark_version }}-bin-without-hadoop"',
- 'spark_tarball': 'spark-{{ spark_version }}-bin-without-hadoop.tgz',
- 'spark_version': None,
- 'tarballs_dir': '"{{ user_home }}/tarballs"',
- 'user_home': None,
- 'worker_data_dirs': None,
- 'zookeeper_connect': "{% for host in groups['zookeepers'] %}{{ host }}:2181{% if not loop.last %},{% endif %}{% endfor %}",
- 'zookeeper_client_port': '"2181"',
- 'zookeeper_basename': "{% if zookeeper_version is version('3.5', '>=') %}apache-zookeeper-{{ zookeeper_version }}-bin{% else %}zookeeper-{{ zookeeper_version }}{% endif %}",
- 'zookeeper_home': "{{ install_dir }}/{{ zookeeper_basename }}",
- 'zookeeper_tarball': "{{ zookeeper_basename }}.tar.gz",
- 'zookeeper_version': None
+ "accumulo_home": "'{{ install_dir }}/accumulo-{{ accumulo_version }}'",
+ "accumulo_instance": None,
+ "accumulo_major_version": "{{ accumulo_version.split('.')[0] }}",
+ "accumulo_password": None,
+ "accumulo_tarball": "accumulo-{{ accumulo_version }}-bin.tar.gz",
+ "accumulo_version": None,
+ "cluster_type": None,
+ "cluster_group": None,
+ "cluster_user": None,
+ "default_data_dirs": None,
+ "download_software": None,
+ "fluo_home": "'{{ install_dir }}/fluo-{{ fluo_version }}'",
+ "fluo_tarball": "fluo-{{ fluo_version }}-bin.tar.gz",
+ "fluo_version": None,
+ "fluo_yarn_home": "'{{ install_dir }}/fluo-yarn-{{ fluo_yarn_version }}'",
+ "fluo_yarn_tarball": "fluo-yarn-{{ fluo_yarn_version }}-bin.tar.gz",
+ "fluo_yarn_version": None,
+ "hadoop_home": "'{{ install_dir }}/hadoop-{{ hadoop_version }}'",
+ "hadoop_tarball": "hadoop-{{ hadoop_version }}.tar.gz",
+ "hadoop_version": None,
+ "hadoop_major_version": "{{ hadoop_version.split('.')[0] }}",
+ "hdfs_root": (
+ "{% if hdfs_ha %}hdfs://{{ nameservice_id }}{% else %}"
+ "hdfs://{{ groups['namenode'][0] }}:8020{% endif %}"
+ ),
+ "hdfs_ha": None,
+ "nameservice_id": None,
+ "num_tservers": 1,
+ "install_dir": None,
+ "install_hub": None,
+ "java_home": "/usr/lib/jvm/java",
+ "java_package": "java-1.8.0-openjdk-devel",
+ "journal_quorum": (
+ "{% for host in groups['journalnode'] %}{{ host }}"
+ ":8485{% if not loop.last %};{% endif %}{% endfor %}"
+ ),
+ "maven_home": "{{ install_dir }}/apache-maven-{{ maven_version }}",
+ "maven_tarball": "apache-maven-{{ maven_version }}-bin.tar.gz",
+ "maven_version": "3.6.3",
+ "spark_home": (
+ "'{{ install_dir }}/spark-{{ spark_version }}" "-bin-without-hadoop'"
+ ),
+ "spark_tarball": "spark-{{ spark_version }}-bin-without-hadoop.tgz",
+ "spark_version": None,
+ "tarballs_dir": "'{{ user_home }}/tarballs'",
+ "user_home": None,
+ "worker_data_dirs": None,
+ "zookeeper_connect": (
+ "{% for host in groups['zookeepers'] %}{{ host }}"
+ ":2181{% if not loop.last %},{% endif %}{% endfor %}"
+ ),
+ "zookeeper_client_port": "2181",
+ "zookeeper_basename": (
+ "{% if zookeeper_version is version('3.5', '>=') %}"
+ "apache-zookeeper-{{ zookeeper_version }}-bin{% else %}"
+ "zookeeper-{{ zookeeper_version }}{% endif %}"
+ ),
+ "zookeeper_home": "{{ install_dir }}/{{ zookeeper_basename }}",
+ "zookeeper_tarball": "{{ zookeeper_basename }}.tar.gz",
+ "zookeeper_version": None,
}
_PLAY_VAR_DEFAULTS = {
- 'accumulo_dcache_size': None,
- 'accumulo_icache_size': None,
- 'accumulo_imap_size': None,
- 'accumulo_checksum': None,
- 'accumulo_tserv_mem': None,
- 'fluo_checksum': None,
- 'fluo_worker_instances_multiplier': None,
- 'fluo_worker_mem_mb': None,
- 'fluo_worker_threads': None,
- 'fluo_yarn_checksum': None,
- 'hadoop_checksum': None,
- 'hub_version': '2.2.3',
- 'hub_home': '"{{ install_dir }}/hub-linux-amd64-{{ hub_version }}"',
- 'hub_tarball': 'hub-linux-amd64-{{ hub_version }}.tgz',
- 'hub_checksum': 'sha256:54c35a459a4241b7ae4c28bcfea0ceef849dd2f8a9dd2b82ba2ba964a743e6bc',
- 'maven_checksum': 'sha512:c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0',
- 'metrics_drive_ids': None,
- 'mount_root': None,
- 'node_type_map': None,
- 'spark_checksum': None,
- 'shutdown_delay_minutes': None,
- 'twill_reserve_mem_mb': None,
- 'yarn_nm_mem_mb': None,
- 'zookeeper_checksum': None,
- 'use_systemd': False
+ "accumulo_dcache_size": None,
+ "accumulo_icache_size": None,
+ "accumulo_imap_size": None,
+ "accumulo_checksum": None,
+ "accumulo_tserv_mem": None,
+ "fluo_checksum": None,
+ "fluo_worker_instances_multiplier": None,
+ "fluo_worker_mem_mb": None,
+ "fluo_worker_threads": None,
+ "fluo_yarn_checksum": None,
+ "hadoop_checksum": None,
+ "hub_version": "2.2.3",
+ "hub_home": '"{{ install_dir }}/hub-linux-amd64-{{ hub_version }}"',
+ "hub_tarball": "hub-linux-amd64-{{ hub_version }}.tgz",
+ "hub_checksum": (
+ "sha256:"
+ "54c35a459a4241b7ae4c28bcfea0ceef849dd2f8a9dd2b82ba2ba964a743e6bc"
+ ),
+ "maven_checksum": (
+ "sha512:"
+ "c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0" # noqa
+ ),
+ "metrics_drive_ids": None,
+ "mount_root": None,
+ "node_type_map": None,
+ "spark_checksum": None,
+ "shutdown_delay_minutes": None,
+ "twill_reserve_mem_mb": None,
+ "yarn_nm_mem_mb": None,
+ "zookeeper_checksum": None,
+ "use_systemd": False,
}
_EXTRA_VAR_DEFAULTS = {}
HASHLEN_ALGO_MAP = {
- 32: "md5",
- 40: "sha1",
- 56: "sha224",
- 64: "sha256",
- 96: "sha384",
- 128: "sha512"
+ 32: "md5",
+ 40: "sha1",
+ 56: "sha224",
+ 64: "sha256",
+ 96: "sha384",
+ 128: "sha512",
}
+
class BaseConfig(ConfigParser, metaclass=ABCMeta):
- def __init__(self, deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
+ def __init__(
+ self,
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ ):
super(BaseConfig, self).__init__()
self.optionxform = str
self.deploy_path = deploy_path
self.read(config_path)
self.hosts_path = hosts_path
self.cluster_name = cluster_name
- self.cluster_type = self.get('general', 'cluster_type')
+ self.cluster_type = self.get("general", "cluster_type")
self.node_d = None
self.hosts = None
self.checksums_path = checksums_path
@@ -136,37 +192,57 @@
def ansible_host_vars(self):
return dict(
- ChainMap(self._ansible_vars_from_decorators('host'),
- getattr(self, 'HOST_VAR_DEFAULTS', {}),
- _HOST_VAR_DEFAULTS))
+ ChainMap(
+ self._ansible_vars_from_decorators("host"),
+ getattr(self, "HOST_VAR_DEFAULTS", {}),
+ _HOST_VAR_DEFAULTS,
+ )
+ )
def ansible_play_vars(self):
software_checksums = {
- '{}_checksum'.format(k): self.checksum(k) for
- k in ['accumulo', 'fluo', 'fluo_yarn', 'hadoop', 'spark', 'zookeeper']
+ "{}_checksum".format(k): self.checksum(k)
+ for k in [
+ "accumulo",
+ "fluo",
+ "fluo_yarn",
+ "hadoop",
+ "spark",
+ "zookeeper",
+ ]
}
return dict(
- ChainMap(self._ansible_vars_from_decorators('play'),
- software_checksums,
- getattr(self, 'PLAY_VAR_DEFAULTS', {}),
- _PLAY_VAR_DEFAULTS))
+ ChainMap(
+ self._ansible_vars_from_decorators("play"),
+ software_checksums,
+ getattr(self, "PLAY_VAR_DEFAULTS", {}),
+ _PLAY_VAR_DEFAULTS,
+ )
+ )
def ansible_extra_vars(self):
return dict(
- ChainMap(self._ansible_vars_from_decorators('extra'),
- getattr(self, 'EXTRA_VAR_DEFAULTS', {}),
- _EXTRA_VAR_DEFAULTS))
+ ChainMap(
+ self._ansible_vars_from_decorators("extra"),
+ getattr(self, "EXTRA_VAR_DEFAULTS", {}),
+ _EXTRA_VAR_DEFAULTS,
+ )
+ )
def _ansible_vars_from_decorators(self, var_type):
# only render play_vars for base and cluster specific config
f = ["{}deployconfig".format(self.get_cluster_type()), "baseconfig"]
return {
- v.var_name: getattr(self, v.property_name)() for v in
- # filter out any classes that are not baseconfig or
- # the cluster specific config
- filter(lambda t: t.class_name.lower() in f,
- # get all ansible vars of var_type
- get_ansible_vars(var_type, type(self)))}
+ v.var_name: getattr(self, v.property_name)()
+ for v in
+ # filter out any classes that are not baseconfig or
+ # the cluster specific config
+ filter(
+ lambda t: t.class_name.lower() in f,
+ # get all ansible vars of var_type
+ get_ansible_vars(var_type, type(self)),
+ )
+ }
@abstractmethod
def verify_config(self, action):
@@ -174,20 +250,39 @@
# helper for verify_config to call for common checks
def _verify_config(self, action):
- if action in ['launch', 'setup']:
+ if action in ["launch", "setup"]:
for service in SERVICES:
if service not in OPTIONAL_SERVICES:
if not self.has_service(service):
- exit("ERROR - Missing '{0}' service from [nodes] section of muchos.props".format(service))
+ exit(
+ "ERROR - Missing '{0}' service from [nodes] "
+ "section of muchos.props".format(service)
+ )
- # validate if we are using Java 11 and fail if we are using Accumulo 1.x
+ # fail if we are using Java 11 along with Accumulo 1.x
# See https://github.com/apache/accumulo/issues/958 for details
- if self.java_product_version() >= 11 and StrictVersion(self.version('accumulo').replace('-SNAPSHOT','')) <= StrictVersion("1.9.3"):
- exit("ERROR - Java 11 is not supported with Accumulo version '{0}'".format(self.version('accumulo')))
+ if self.java_product_version() >= 11 and StrictVersion(
+ self.version("accumulo").replace("-SNAPSHOT", "")
+ ) <= StrictVersion("1.9.3"):
+ exit(
+ "ERROR - Java 11 is not supported with Accumulo version "
+ "'{0}'".format(self.version("accumulo"))
+ )
- # validate and fail if we are using ZooKeeper 3.5.5 or greater and Accumulo 1.9.x or less
- if StrictVersion(self.version('zookeeper')) >= StrictVersion("3.5.5") and StrictVersion(self.version('accumulo').replace('-SNAPSHOT','')) < StrictVersion("1.10.0"):
- exit("ERROR - ZooKeeper version '{0}' is not supported with Accumulo version '{1}'".format(self.version('zookeeper'), self.version('accumulo')))
+ # fail if we are using ZooKeeper >= 3.5.5 with Accumulo <= 1.9.x
+ if StrictVersion(self.version("zookeeper")) >= StrictVersion(
+ "3.5.5"
+ ) and StrictVersion(
+ self.version("accumulo").replace("-SNAPSHOT", "")
+ ) < StrictVersion(
+ "1.10.0"
+ ):
+ exit(
+ "ERROR - ZooKeeper version '{0}' is not supported with "
+ "Accumulo version '{1}'".format(
+ self.version("zookeeper"), self.version("accumulo")
+ )
+ )
@abstractmethod
def verify_launch(self):
@@ -195,15 +290,23 @@
def _init_nodes(self):
self.node_d = {}
- for (hostname, value) in self.items('nodes'):
+ for (hostname, value) in self.items("nodes"):
if hostname in self.node_d:
- exit('Hostname {0} already exists twice in nodes'.format(hostname))
+ exit(
+ "Hostname {0} already exists twice in nodes".format(
+ hostname
+ )
+ )
service_list = []
- for service in value.split(','):
+ for service in value.split(","):
if service in SERVICES:
service_list.append(service)
else:
- exit('Unknown service "%s" declared for node %s' % (service, hostname))
+ exit(
+ "Unknown service '{}' declared for node {}".format(
+ service, hostname
+ )
+ )
self.node_d[hostname] = service_list
@abstractmethod
@@ -211,14 +314,14 @@
def node_type_map(self):
raise NotImplementedError()
- @is_valid(is_in(['default', 'worker']))
+ @is_valid(is_in(["default", "worker"]))
def node_type(self, hostname):
- if 'worker' in self.node_d[hostname]:
- return 'worker'
- return 'default'
+ if "worker" in self.node_d[hostname]:
+ return "worker"
+ return "default"
def user_home(self):
- return self.get('general', 'user_home')
+ return self.get("general", "user_home")
def mounts(self, num_ephemeral):
mounts = []
@@ -250,25 +353,34 @@
@ansible_play_var
def shutdown_delay_minutes(self):
- return '0'
+ return "0"
def version(self, software_id):
- return self.get('general', software_id + '_version')
+ return self.get("general", software_id + "_version")
@ansible_host_var
def java_product_version(self):
java_version_map = {
- "java-1.8.0-openjdk": 8,
- "java-11-openjdk": 11,
- "java-latest-openjdk": 13
- }
+ "java-1.8.0-openjdk": 8,
+ "java-11-openjdk": 11,
+ "java-latest-openjdk": 13,
+ }
- # Given that a user might chose to install a specific JDK version (where the version is suffixed to package name)
- # it is safer to check if the configured Java version starts with one of the above prefixes defined in the map.
+ # Given that a user might chose to install a specific JDK version
+ # (where the version is suffixed to package name) it is safer
+ # to check if the configured Java version starts with one of the above
+ # prefixes defined in the version map
configured_java_version = self.get("general", "java_package")
- filtered_java_versions = {k:v for (k,v) in java_version_map.items() if configured_java_version.startswith(k)}
+ filtered_java_versions = {
+ k: v
+ for (k, v) in java_version_map.items()
+ if configured_java_version.startswith(k)
+ }
if len(filtered_java_versions) != 1:
- exit('ERROR - unknown or ambiguous Java version \'{0}\' specified in properties'.format(configured_java_version))
+ exit(
+ "ERROR - unknown or ambiguous Java version '{0}' specified"
+ " in properties".format(configured_java_version)
+ )
return next(iter(filtered_java_versions.values()))
@@ -276,7 +388,7 @@
return self.checksum_ver(software, self.version(software))
def infer_hash_algo(self, hashstring):
- # assign the algorithm based on length. These are the default supported algorithms for Ansible
+ # infer the algorithm based on length
hashlen = len(hashstring)
if hashlen in HASHLEN_ALGO_MAP:
@@ -285,8 +397,12 @@
return None
def checksum_ver(self, software, version):
- if not os.path.isfile(self.checksums_path):
- exit('ERROR - A checksums file does not exist at %s' % self.hosts_path)
+ if not isfile(self.checksums_path):
+ exit(
+ "ERROR - A checksums file does not exist at {}".format(
+ self.hosts_path
+ )
+ )
if "SNAPSHOT" in version:
return ""
@@ -298,24 +414,47 @@
line = line.strip()
if line.startswith("#") or not line:
continue
- args = line.split(':')
+ args = line.split(":")
if len(args) == 3:
inferred_algo = self.infer_hash_algo(args[2])
if inferred_algo is not None:
- self.checksums_d["{0}:{1}".format(args[0], args[1])] = "{0}:{1}".format(self.infer_hash_algo(args[2]), args[2])
+ self.checksums_d[
+ "{0}:{1}".format(args[0], args[1])
+ ] = "{0}:{1}".format(
+ self.infer_hash_algo(args[2]), args[2]
+ )
else:
- exit('ERROR - Bad line %s in checksums %s' % (line, self.checksums_path))
+ exit(
+ "ERROR - Bad line {} in checksums {}".format(
+ line, self.checksums_path
+ )
+ )
elif len(args) == 4:
if args[2] not in HASHLEN_ALGO_MAP.values():
- exit('ERROR - Unsupported hash algorithm %s in checksums %s' % (line, self.checksums_path))
- self.checksums_d["{0}:{1}".format(args[0], args[1])] = "{0}:{1}".format(args[2], args[3])
+ exit(
+ "ERROR - Unsupported hash algorithm {} "
+ "in checksums {}".format(
+ line, self.checksums_path
+ )
+ )
+ self.checksums_d[
+ "{0}:{1}".format(args[0], args[1])
+ ] = "{0}:{1}".format(args[2], args[3])
else:
- exit('ERROR - Bad line %s in checksums %s' % (line, self.checksums_path))
+ exit(
+ "ERROR - Bad line {} in checksums {}".format(
+ line, self.checksums_path
+ )
+ )
key = "{0}:{1}".format(software, version)
if key not in self.checksums_d:
- exit('ERROR - Failed to find checksums for %s %s in %s' % (software, version, self.checksums_path))
+ exit(
+ "ERROR - Failed to find checksums for {} {} in {}".format(
+ software, version, self.checksums_path
+ )
+ )
return self.checksums_d[key]
def nodes(self):
@@ -334,7 +473,7 @@
def get_host_services(self):
retval = []
for (hostname, service_list) in list(self.node_d.items()):
- retval.append((hostname, ' '.join(service_list)))
+ retval.append((hostname, " ".join(service_list)))
retval.sort()
return retval
@@ -358,8 +497,10 @@
# test method, might want to make private or just move to test module
def get_non_proxy(self):
retval = []
- proxy_ip = self.get_private_ip(self.get('general', 'proxy_hostname'))
- for (hostname, (private_ip, public_ip)) in list(self.get_hosts().items()):
+ proxy_ip = self.get_private_ip(self.get("general", "proxy_hostname"))
+ for (hostname, (private_ip, public_ip)) in list(
+ self.get_hosts().items()
+ ):
if private_ip != proxy_ip:
retval.append((private_ip, hostname))
retval.sort()
@@ -367,14 +508,20 @@
def get_private_ip_hostnames(self):
retval = []
- for (hostname, (private_ip, public_ip)) in list(self.get_hosts().items()):
+ for (hostname, (private_ip, public_ip)) in list(
+ self.get_hosts().items()
+ ):
retval.append((private_ip, hostname))
retval.sort()
return retval
def parse_hosts(self):
- if not os.path.isfile(self.hosts_path):
- exit('ERROR - A hosts file does not exist at %s' % self.hosts_path)
+ if not isfile(self.hosts_path):
+ exit(
+ "ERROR - A hosts file does not exist at {}".format(
+ self.hosts_path
+ )
+ )
self.hosts = {}
with open(self.hosts_path) as f:
@@ -382,13 +529,17 @@
line = line.strip()
if line.startswith("#") or not line:
continue
- args = line.split(' ')
+ args = line.split(" ")
if len(args) == 2:
self.hosts[args[0]] = (args[1], None)
elif len(args) == 3:
self.hosts[args[0]] = (args[1], args[2])
else:
- exit('ERROR - Bad line %s in hosts %s' % (line, self.hosts_path))
+ exit(
+ "ERROR - Bad line {} in hosts {}".format(
+ line, self.hosts_path
+ )
+ )
def get_hosts(self):
if self.hosts is None:
@@ -402,17 +553,21 @@
return self.get_hosts()[hostname][1]
def get_cluster_type(self):
- if self.cluster_type not in ('azure', 'ec2', 'existing'):
- exit('ERROR - Unknown cluster type' + self.cluster_type)
+ if self.cluster_type not in ("azure", "ec2", "existing"):
+ exit("ERROR - Unknown cluster type" + self.cluster_type)
return self.cluster_type
def proxy_hostname(self):
- return self.get('general', 'proxy_hostname')
+ return self.get("general", "proxy_hostname")
def proxy_public_ip(self):
retval = self.get_public_ip(self.proxy_hostname())
if not retval:
- exit("ERROR - Proxy '{0}' does not have a public IP".format(self.proxy_hostname()))
+ exit(
+ "ERROR - Proxy '{0}' does not have a public IP".format(
+ self.proxy_hostname()
+ )
+ )
return retval
def get_proxy_ip(self):
@@ -426,22 +581,22 @@
return self.get_private_ip(self.proxy_hostname())
def get_performance_prop(self, prop):
- profile = self.get('performance', 'profile')
+ profile = self.get("performance", "profile")
return self.get(profile, prop)
def print_all(self):
- print('proxy_public_ip = ', self.proxy_public_ip())
- for (name, val) in self.items('general'):
- print(name, '=', val)
+ print("proxy_public_ip = ", self.proxy_public_ip())
+ for (name, val) in self.items("general"):
+ print(name, "=", val)
- for (name, val) in self.items('ec2'):
- print(name, '=', val)
+ for (name, val) in self.items("ec2"):
+ print(name, "=", val)
- for (name, val) in self.items('azure'):
- print(name, '=', val)
+ for (name, val) in self.items("azure"):
+ print(name, "=", val)
def print_property(self, key):
- if key == 'proxy.public.ip':
+ if key == "proxy.public.ip":
print(self.proxy_public_ip())
return
else:
@@ -450,11 +605,18 @@
print(self.get(section, key))
return
exit("Property '{0}' was not found".format(key))
-
+
def resolve_value(self, config_name, default=None):
# listed low to high priority
- section_priority = ['general', 'ansible-vars', self.get('performance', 'profile')]
- all_values = [self.get(section, config_name, fallback=None) for section in section_priority]
+ section_priority = [
+ "general",
+ "ansible-vars",
+ self.get("performance", "profile"),
+ ]
+ all_values = [
+ self.get(section, config_name, fallback=None)
+ for section in section_priority
+ ]
try:
*_, val = filter(None, all_values)
except ValueError:
diff --git a/lib/muchos/config/decorators.py b/lib/muchos/config/decorators.py
index 3fcd6fb..7ff0936 100644
--- a/lib/muchos/config/decorators.py
+++ b/lib/muchos/config/decorators.py
@@ -33,42 +33,55 @@
self.module_name = module_name
def __str__(self):
- return 'var_name={}, class_name={}, property_name={}, module_name={}'.format(
- self.var_name, self.class_name, self.property_name, self.module_name
+ return (
+ "var_name={}, class_name={}, " "property_name={}, module_name={}"
+ ).format(
+ self.var_name,
+ self.class_name,
+ self.property_name,
+ self.module_name,
)
+
# each entry of _ansible_vars will contain a list of _ansible_var instances
-_ansible_vars = dict(
- host=[],
- play=[],
- extra=[]
-)
+_ansible_vars = dict(host=[], play=[], extra=[])
+
def get_ansible_vars(var_type, class_in_scope):
# return variables for the complete class hierarchy
- return list(filter(lambda v:
- issubclass(class_in_scope, locate(v.module_name + "." + v.class_name)),
- _ansible_vars.get(var_type)))
+ return list(
+ filter(
+ lambda v: issubclass(
+ class_in_scope, locate(v.module_name + "." + v.class_name)
+ ),
+ _ansible_vars.get(var_type),
+ )
+ )
+
# ansible hosts inventory variables
def ansible_host_var(name=None):
- return ansible_var_decorator('host', name)
+ return ansible_var_decorator("host", name)
+
# ansible group/all variables
def ansible_play_var(name=None):
- return ansible_var_decorator('play', name)
+ return ansible_var_decorator("play", name)
+
# ansible extra variables
def ansible_extra_var(name=None):
- return ansible_var_decorator('extra', name)
+ return ansible_var_decorator("extra", name)
+
def ansible_var_decorator(var_type, name):
def _decorator(func):
ansible_var = _ansible_var(
var_name=name if isinstance(name, str) else func.__name__,
- class_name=func.__qualname__.split('.')[0],
+ class_name=func.__qualname__.split(".")[0],
property_name=func.__name__,
- module_name=func.__module__)
+ module_name=func.__module__,
+ )
_ansible_vars[var_type].append(ansible_var)
return func
@@ -76,46 +89,62 @@
return _decorator(name)
return _decorator
+
def default(val):
def _default(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
res = func(*args, **kwargs)
- except:
+ except: # noqa
return val
else:
if res is None or (isinstance(res, str) and len(res) == 0):
return val
return res
+
return wrapper
+
return _default
+
def required(func):
@wraps(func)
def wrapper(*args, **kwargs):
res = func(*args, **kwargs)
- if res in [None, 0, ''] or len(res) == 0:
+ if res in [None, 0, ""] or len(res) == 0:
raise ConfigMissingError(func.__name__)
return res
+
return wrapper
+
def is_valid(validators):
if not isinstance(validators, Iterable):
validators = [validators]
+
def _validate(func):
@wraps(func)
def wrapper(*args, **kwargs):
res = func(*args, **kwargs)
- failed_checks = list(filter(lambda f: f(res) is not True, validators))
+ failed_checks = list(
+ filter(lambda f: f(res) is not True, validators)
+ )
if len(failed_checks) > 0:
- raise Exception("{}={} checked validation {}".format(
- func.__name__, res,
- [str(v) for v in failed_checks]))
+ raise Exception(
+ "{}={} checked validation {}".format(
+ func.__name__, res, [str(v) for v in failed_checks]
+ )
+ )
return res
+
return wrapper
+
return _validate
+
class ConfigMissingError(Exception):
def __init__(self, name):
- super(ConfigMissingError, self).__init__("{} is missing from the configuration".format(name))
+ super(ConfigMissingError, self).__init__(
+ "{} is missing from the configuration".format(name)
+ )
diff --git a/lib/muchos/config/ec2.py b/lib/muchos/config/ec2.py
index 7eaa4ab..c2853f9 100644
--- a/lib/muchos/config/ec2.py
+++ b/lib/muchos/config/ec2.py
@@ -15,84 +15,119 @@
# limitations under the License.
#
-from muchos.config import SERVICES, OPTIONAL_SERVICES
-from muchos.config.base import BaseConfig
-from sys import exit
-from muchos.config.decorators import *
-from muchos.config.validators import *
-from muchos.util import get_ephemeral_devices, get_arch
-import os
-import json
import glob
+import json
+from sys import exit
+import os
+from .base import SERVICES
+from .base import BaseConfig
+from .decorators import (
+ ansible_play_var,
+ default,
+)
+from ..util import get_ephemeral_devices, get_arch
class Ec2DeployConfig(BaseConfig):
-
- def __init__(self, deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
- super(Ec2DeployConfig, self).__init__(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
- self.sg_name = cluster_name + '-group'
- self.ephemeral_root = 'ephemeral'
+ def __init__(
+ self,
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ ):
+ super(Ec2DeployConfig, self).__init__(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
+ self.sg_name = cluster_name + "-group"
+ self.ephemeral_root = "ephemeral"
self.cluster_template_d = None
- self.metrics_drive_root = 'media-' + self.ephemeral_root
+ self.metrics_drive_root = "media-" + self.ephemeral_root
self.init_template(templates_path)
def verify_config(self, action):
self._verify_config(action)
def verify_launch(self):
- self.verify_instance_type(self.get('ec2', 'default_instance_type'))
- self.verify_instance_type(self.get('ec2', 'worker_instance_type'))
+ self.verify_instance_type(self.get("ec2", "default_instance_type"))
+ self.verify_instance_type(self.get("ec2", "worker_instance_type"))
def init_nodes(self):
self.node_d = {}
- for (hostname, value) in self.items('nodes'):
+ for (hostname, value) in self.items("nodes"):
if hostname in self.node_d:
- exit('Hostname {0} already exists twice in nodes'.format(hostname))
+ exit(
+ "Hostname {0} already exists twice in nodes".format(
+ hostname
+ )
+ )
service_list = []
- for service in value.split(','):
+ for service in value.split(","):
if service in SERVICES:
service_list.append(service)
else:
- exit('Unknown service "%s" declared for node %s' % (service, hostname))
+ exit(
+ "Unknown service '{}' declared for node {}".format(
+ service, hostname
+ )
+ )
self.node_d[hostname] = service_list
def default_ephemeral_devices(self):
- return get_ephemeral_devices(self.get('ec2', 'default_instance_type'))
+ return get_ephemeral_devices(self.get("ec2", "default_instance_type"))
def worker_ephemeral_devices(self):
- return get_ephemeral_devices(self.get('ec2', 'worker_instance_type'))
+ return get_ephemeral_devices(self.get("ec2", "worker_instance_type"))
def max_ephemeral(self):
- return max((len(self.default_ephemeral_devices()), len(self.worker_ephemeral_devices())))
+ return max(
+ (
+ len(self.default_ephemeral_devices()),
+ len(self.worker_ephemeral_devices()),
+ )
+ )
def node_type_map(self):
if self.cluster_template_d:
- return self.cluster_template_d['devices']
+ return self.cluster_template_d["devices"]
node_types = {}
- node_list = [('default', self.default_ephemeral_devices()), ('worker', self.worker_ephemeral_devices())]
+ node_list = [
+ ("default", self.default_ephemeral_devices()),
+ ("worker", self.worker_ephemeral_devices()),
+ ]
for (ntype, devices) in node_list:
- node_types[ntype] = {'mounts': self.mounts(len(devices)), 'devices': devices}
+ node_types[ntype] = {
+ "mounts": self.mounts(len(devices)),
+ "devices": devices,
+ }
return node_types
def mount_root(self):
- return '/media/' + self.ephemeral_root
+ return "/media/" + self.ephemeral_root
@ansible_play_var
- @default('ext3')
+ @default("ext3")
def fstype(self):
- return self.get('ec2', 'fstype')
+ return self.get("ec2", "fstype")
@ansible_play_var
- @default('no')
+ @default("no")
def force_format(self):
- return self.get('ec2', 'force_format')
+ return self.get("ec2", "force_format")
def data_dirs_common(self, nodeType):
- return self.node_type_map()[nodeType]['mounts']
+ return self.node_type_map()[nodeType]["mounts"]
def metrics_drive_ids(self):
drive_ids = []
@@ -105,62 +140,90 @@
def verify_instance_type(self, instance_type):
if not self.cluster_template_d:
- if get_arch(instance_type) == 'pvm':
- exit("ERROR - Configuration contains instance type '{0}' that uses pvm architecture."
- "Only hvm architecture is supported!".format(instance_type))
+ if get_arch(instance_type) == "pvm":
+ exit(
+ "ERROR - Configuration contains instance type '{0}' "
+ "that uses pvm architecture."
+ "Only hvm architecture is supported!".format(instance_type)
+ )
def instance_tags(self):
retd = {}
- if self.has_option('ec2', 'instance_tags'):
- value = self.get('ec2', 'instance_tags')
+ if self.has_option("ec2", "instance_tags"):
+ value = self.get("ec2", "instance_tags")
if value:
- for kv in value.split(','):
- (key, val) = kv.split(':')
+ for kv in value.split(","):
+ (key, val) = kv.split(":")
retd[key] = val
return retd
def init_template(self, templates_path):
- if self.has_option('ec2', 'cluster_template'):
- template_id = self.get('ec2', 'cluster_template')
+ if self.has_option("ec2", "cluster_template"):
+ template_id = self.get("ec2", "cluster_template")
template_path = os.path.join(templates_path, template_id)
if os.path.exists(template_path):
- self.cluster_template_d = {'id': template_id}
+ self.cluster_template_d = {"id": template_id}
self.load_template_ec2_requests(template_path)
self.load_template_device_map(template_path)
self.validate_template()
def load_template_ec2_requests(self, template_dir):
- for json_path in glob.glob(os.path.join(template_dir, '*.json')):
- service = os.path.basename(json_path).rsplit('.', 1)[0]
+ for json_path in glob.glob(os.path.join(template_dir, "*.json")):
+ service = os.path.basename(json_path).rsplit(".", 1)[0]
if service not in SERVICES:
- exit("ERROR - Template '{0}' has unrecognized option '{1}'. Must be one of {2}".format(
- self.cluster_template_d['id'], service, str(SERVICES)))
- with open(json_path, 'r') as json_file:
- # load as string, so we can use string.Template to inject config values
+ exit(
+ "ERROR - Template '{0}' has unrecognized option '{1}'. "
+ "Must be one of {2}".format(
+ self.cluster_template_d["id"], service, str(SERVICES)
+ )
+ )
+ with open(json_path, "r") as json_file:
+ # load as string, so we can use string.Template
+ # to inject config values
self.cluster_template_d[service] = json_file.read()
def load_template_device_map(self, template_dir):
- device_map_path = os.path.join(template_dir, 'devices')
+ device_map_path = os.path.join(template_dir, "devices")
if not os.path.isfile(device_map_path):
- exit("ERROR - template '{0}' is missing 'devices' config".format(self.cluster_template_d['id']))
- with open(device_map_path, 'r') as json_file:
- self.cluster_template_d['devices'] = json.load(json_file)
+ exit(
+ "ERROR - template '{0}' is missing 'devices' config".format(
+ self.cluster_template_d["id"]
+ )
+ )
+ with open(device_map_path, "r") as json_file:
+ self.cluster_template_d["devices"] = json.load(json_file)
def validate_template(self):
if not self.cluster_template_d:
- exit("ERROR - Template '{0}' is not defined!".format(self.get('ec2', 'cluster_template')))
+ exit(
+ "ERROR - Template '{0}' is not defined!".format(
+ self.get("ec2", "cluster_template")
+ )
+ )
- if 'worker' not in self.cluster_template_d:
- exit("ERROR - '{0}' template config is invalid. No 'worker' launch request is defined".format(
- self.cluster_template_d['id']))
+ if "worker" not in self.cluster_template_d:
+ exit(
+ "ERROR - '{0}' template config is invalid. No 'worker' "
+ "launch request is defined".format(
+ self.cluster_template_d["id"]
+ )
+ )
- if 'worker' not in self.cluster_template_d['devices']:
- exit("ERROR - '{0}' template is invalid. The devices file must have a 'worker' device map".format(
- self.cluster_template_d['id']))
+ if "worker" not in self.cluster_template_d["devices"]:
+ exit(
+ "ERROR - '{0}' template is invalid. The devices file must "
+ "have a 'worker' device map".format(
+ self.cluster_template_d["id"]
+ )
+ )
- if 'default' not in self.cluster_template_d['devices']:
- exit("ERROR - '{0}' template is invalid. The devices file must have a 'default' device map".format(
- self.cluster_template_d['id']))
+ if "default" not in self.cluster_template_d["devices"]:
+ exit(
+ "ERROR - '{0}' template is invalid. The devices file must "
+ "have a 'default' device map".format(
+ self.cluster_template_d["id"]
+ )
+ )
# Validate the selected launch template for each host
@@ -168,17 +231,29 @@
for hostname in self.node_d:
# first service listed denotes the selected template
selected_ec2_request = self.node_d[hostname][0]
- if 'worker' == selected_ec2_request:
+ if "worker" == selected_ec2_request:
worker_count = worker_count + 1
else:
- if 'worker' in self.node_d[hostname]:
- exit("ERROR - '{0}' node config is invalid. The 'worker' service should be listed first".format(
- hostname))
+ if "worker" in self.node_d[hostname]:
+ exit(
+ "ERROR - '{0}' node config is invalid. The 'worker'"
+ " service should be listed first".format(hostname)
+ )
if selected_ec2_request not in self.cluster_template_d:
if len(self.node_d[hostname]) > 1:
- print('Hint: In template mode, the first service listed for a host denotes its EC2 template')
- exit("ERROR - '{0}' node config is invalid. No EC2 template defined for the '{1}' service".format(
- hostname, selected_ec2_request))
+ print(
+ "Hint: In template mode, the first service listed"
+ " for a host denotes its EC2 template"
+ )
+ exit(
+ "ERROR - '{0}' node config is invalid. No EC2 "
+ "template defined for the '{1}' service".format(
+ hostname, selected_ec2_request
+ )
+ )
if worker_count == 0:
- exit("ERROR - No worker instances are defined for template '{0}'".format(self.cluster_template_d['id']))
+ exit(
+ "ERROR - No worker instances are defined "
+ "for template '{0}'".format(self.cluster_template_d["id"])
+ )
diff --git a/lib/muchos/config/existing.py b/lib/muchos/config/existing.py
index e418b06..6fa1ad9 100644
--- a/lib/muchos/config/existing.py
+++ b/lib/muchos/config/existing.py
@@ -15,20 +15,27 @@
# limitations under the License.
#
-from muchos.config import BaseConfig
-from sys import exit
-from muchos.config.decorators import *
-from muchos.config.validators import *
-from muchos.util import get_ephemeral_devices, get_arch
-import os
-import json
-import glob
+from .base import BaseConfig
class ExistingDeployConfig(BaseConfig):
-
- def __init__(self, deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name):
- super(ExistingDeployConfig, self).__init__(deploy_path, config_path, hosts_path, checksums_path, templates_path, cluster_name)
+ def __init__(
+ self,
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ ):
+ super(ExistingDeployConfig, self).__init__(
+ deploy_path,
+ config_path,
+ hosts_path,
+ checksums_path,
+ templates_path,
+ cluster_name,
+ )
def verify_config(self, action):
self._verify_config(action)
@@ -41,10 +48,10 @@
return node_types
def mount_root(self):
- return self.get('existing', 'mount_root')
+ return self.get("existing", "mount_root")
def data_dirs_common(self, nodeType):
- return self.get('existing', 'data_dirs').split(",")
+ return self.get("existing", "data_dirs").split(",")
def metrics_drive_ids(self):
return self.get("existing", "metrics_drive_ids").split(",")
diff --git a/lib/muchos/config/validators.py b/lib/muchos/config/validators.py
index 1e8d49b..e5ac692 100644
--- a/lib/muchos/config/validators.py
+++ b/lib/muchos/config/validators.py
@@ -15,6 +15,7 @@
# limitations under the License.
#
+
class _validator(object):
def __init__(self, f, msg):
self.f = f
@@ -28,31 +29,26 @@
def greater_than(val):
- return _validator(
- lambda n: n > val,
- "must be greater than {}".format(val))
+ return _validator(lambda n: n > val, "must be greater than {}".format(val))
+
def less_than(val):
- return _validator(
- lambda n: n < val,
- "must be less than {}".format(val))
+ return _validator(lambda n: n < val, "must be less than {}".format(val))
+
def equals(val):
- return _validator(
- lambda n: n == val,
- "must equal {}".format(val))
+ return _validator(lambda n: n == val, "must equal {}".format(val))
+
def contains(val):
- return _validator(
- lambda n: val in n,
- "must contain {}".format(val))
+ return _validator(lambda n: val in n, "must contain {}".format(val))
+
def is_in(val):
- return _validator(
- lambda n: n in val,
- "must be in {}".format(val))
+ return _validator(lambda n: n in val, "must be in {}".format(val))
+
def is_type(t):
return _validator(
- lambda n: isinstance(n, t),
- "must be of type {}".format(t))
\ No newline at end of file
+ lambda n: isinstance(n, t), "must be of type {}".format(t)
+ )
diff --git a/lib/muchos/ec2.py b/lib/muchos/ec2.py
index a572b59..90cb572 100644
--- a/lib/muchos/ec2.py
+++ b/lib/muchos/ec2.py
@@ -20,7 +20,7 @@
from sys import exit
from botocore.exceptions import ClientError
from .util import AMI_HELP_MSG, get_block_device_map
-from os.path import isfile
+from os import path
import time
import boto3
from .existing import ExistingCluster
@@ -29,7 +29,6 @@
class Ec2Cluster(ExistingCluster):
-
def __init__(self, config):
ExistingCluster.__init__(self, config)
@@ -37,203 +36,290 @@
request = self.init_request(hostname, services, sg_id)
- request['MinCount'] = 1
- request['MaxCount'] = 1
+ request["MinCount"] = 1
+ request["MaxCount"] = 1
- tags = [{'Key': 'Name', 'Value': self.config.cluster_name + '-' + hostname},
- {'Key': 'Muchos', 'Value': self.config.cluster_name}]
+ tags = [
+ {
+ "Key": "Name",
+ "Value": self.config.cluster_name + "-" + hostname,
+ },
+ {"Key": "Muchos", "Value": self.config.cluster_name},
+ ]
for key, val in self.config.instance_tags().items():
- tags.append({'Key': key, 'Value': val})
+ tags.append({"Key": key, "Value": val})
- request['TagSpecifications'] = [{'ResourceType': 'instance', 'Tags': tags}]
+ request["TagSpecifications"] = [
+ {"ResourceType": "instance", "Tags": tags}
+ ]
- if self.config.has_option('ec2', 'user_data_path'):
- user_data_path = self.config.get('ec2', 'user_data_path')
- with open(user_data_path, 'r') as user_data_file:
+ if self.config.has_option("ec2", "user_data_path"):
+ user_data_path = self.config.get("ec2", "user_data_path")
+ with open(user_data_path, "r") as user_data_file:
user_data = user_data_file.read()
- request['UserData'] = user_data
+ request["UserData"] = user_data
- ec2 = boto3.client('ec2')
+ ec2 = boto3.client("ec2")
response = None
try:
response = ec2.run_instances(**request)
except ClientError as e:
- exit("ERROR - Failed to launch EC2 instance due to exception:\n\n{0}\n\n{1}".format(e, AMI_HELP_MSG))
+ exit(
+ "ERROR - Failed to launch EC2 instance due to exception:"
+ "\n\n{0}\n\n{1}".format(e, AMI_HELP_MSG)
+ )
- if response is None or len(response['Instances']) != 1:
- exit('ERROR - Failed to start {0} node'.format(hostname))
+ if response is None or len(response["Instances"]) != 1:
+ exit("ERROR - Failed to start {0} node".format(hostname))
- print('Launching {0} node using {1}'.format(hostname, request['ImageId']))
- return response['Instances'][0]
+ print(
+ "Launching {0} node using {1}".format(hostname, request["ImageId"])
+ )
+ return response["Instances"][0]
def create_security_group(self):
- ec2 = boto3.client('ec2')
+ ec2 = boto3.client("ec2")
sg = self.config.sg_name
create_group = True
group_id = None
try:
- response = ec2.describe_security_groups(Filters=[{'Name': 'group-name', 'Values': [sg]}])
- if len(response['SecurityGroups']) > 0:
- group_id = response['SecurityGroups'][0]['GroupId']
+ response = ec2.describe_security_groups(
+ Filters=[{"Name": "group-name", "Values": [sg]}]
+ )
+ if len(response["SecurityGroups"]) > 0:
+ group_id = response["SecurityGroups"][0]["GroupId"]
create_group = False
except ClientError:
pass
if create_group:
print("Creating security group " + sg)
- request = {'Description': "Security group created by Muchos", 'GroupName': sg}
- if self.config.has_option('ec2', 'vpc_id'):
- request['VpcId'] = self.config.get('ec2', 'vpc_id')
+ request = {
+ "Description": "Security group created by Muchos",
+ "GroupName": sg,
+ }
+ if self.config.has_option("ec2", "vpc_id"):
+ request["VpcId"] = self.config.get("ec2", "vpc_id")
response = ec2.create_security_group(**request)
- group_id = response['GroupId']
- ec2.authorize_security_group_ingress(GroupName=sg, SourceSecurityGroupName=sg)
- ec2.authorize_security_group_ingress(GroupName=sg, IpProtocol='tcp', FromPort=22, ToPort=22,
- CidrIp='0.0.0.0/0')
+ group_id = response["GroupId"]
+ ec2.authorize_security_group_ingress(
+ GroupName=sg, SourceSecurityGroupName=sg
+ )
+ ec2.authorize_security_group_ingress(
+ GroupName=sg,
+ IpProtocol="tcp",
+ FromPort=22,
+ ToPort=22,
+ CidrIp="0.0.0.0/0",
+ )
return group_id
def delete_security_group(self):
sg_id = None
- ec2 = boto3.client('ec2')
+ ec2 = boto3.client("ec2")
try:
- response = ec2.describe_security_groups(Filters=[{'Name': 'group-name', 'Values': [self.config.sg_name]}])
- if len(response['SecurityGroups']) > 0:
- sg_id = response['SecurityGroups'][0]['GroupId']
+ response = ec2.describe_security_groups(
+ Filters=[
+ {"Name": "group-name", "Values": [self.config.sg_name]}
+ ]
+ )
+ if len(response["SecurityGroups"]) > 0:
+ sg_id = response["SecurityGroups"][0]["GroupId"]
except ClientError:
pass
if not sg_id:
- print("Could not find security group '{0}'".format(self.config.sg_name))
+ print(
+ "Could not find security group '{0}'".format(
+ self.config.sg_name
+ )
+ )
return
- print("Attempting to delete security group '{0}' with id '{1}'...".format(self.config.sg_name, sg_id))
+ print(
+ "Attempting to delete security group '{0}' "
+ "with id '{1}'...".format(self.config.sg_name, sg_id)
+ )
sg_exists = True
while sg_exists:
try:
- request = {'GroupId': sg_id}
+ request = {"GroupId": sg_id}
ec2.delete_security_group(**request)
sg_exists = False
except ClientError as e:
- print("Failed to delete security group '{0}' due exception below:\n{1}\nRetrying in 10 sec..."
- .format(self.config.sg_name, e))
+ print(
+ "Failed to delete security group '{0}' due to "
+ "exception below:\n{1}\nRetrying in 10 sec...".format(
+ self.config.sg_name, e
+ )
+ )
time.sleep(10)
print("Deleted security group")
def init_request(self, hostname, services, sg_id):
associate_public_ip = True
- if self.config.has_option('ec2', 'associate_public_ip'):
- associate_public_ip = self.config.get('ec2', 'associate_public_ip').strip().lower() == 'true'
+ if self.config.has_option("ec2", "associate_public_ip"):
+ associate_public_ip = (
+ self.config.get("ec2", "associate_public_ip").strip().lower()
+ == "true"
+ )
- request = {'NetworkInterfaces': [{'DeviceIndex': 0, 'AssociatePublicIpAddress': associate_public_ip,
- 'Groups': [sg_id]}]}
+ request = {
+ "NetworkInterfaces": [
+ {
+ "DeviceIndex": 0,
+ "AssociatePublicIpAddress": associate_public_ip,
+ "Groups": [sg_id],
+ }
+ ]
+ }
- if self.config.has_option('ec2', 'subnet_id'):
- request['NetworkInterfaces'][0]['SubnetId'] = self.config.get('ec2', 'subnet_id')
+ if self.config.has_option("ec2", "subnet_id"):
+ request["NetworkInterfaces"][0]["SubnetId"] = self.config.get(
+ "ec2", "subnet_id"
+ )
- if 'worker' in services:
- instance_type = self.config.get('ec2', 'worker_instance_type')
+ if "worker" in services:
+ instance_type = self.config.get("ec2", "worker_instance_type")
else:
- instance_type = self.config.get('ec2', 'default_instance_type')
- request['InstanceType'] = instance_type
- request['InstanceInitiatedShutdownBehavior'] = self.config.get('ec2', 'shutdown_behavior')
+ instance_type = self.config.get("ec2", "default_instance_type")
+ request["InstanceType"] = instance_type
+ request["InstanceInitiatedShutdownBehavior"] = self.config.get(
+ "ec2", "shutdown_behavior"
+ )
- if not self.config.has_option('ec2', 'aws_ami'):
- exit('aws_ami property must be set!')
- image_id = self.config.get('ec2', 'aws_ami')
+ if not self.config.has_option("ec2", "aws_ami"):
+ exit("aws_ami property must be set!")
+ image_id = self.config.get("ec2", "aws_ami")
if not image_id:
- exit('aws_ami property was not properly')
+ exit("aws_ami property was not properly")
- request['ImageId'] = image_id
- request['BlockDeviceMappings'] = get_block_device_map(instance_type)
+ request["ImageId"] = image_id
+ request["BlockDeviceMappings"] = get_block_device_map(instance_type)
- if self.config.has_option('ec2', 'key_name'):
- request['KeyName'] = self.config.get('ec2', 'key_name')
+ if self.config.has_option("ec2", "key_name"):
+ request["KeyName"] = self.config.get("ec2", "key_name")
return request
def launch(self):
if self.active_nodes():
- exit('ERROR - There are already instances running for {0} cluster'.format(self.config.cluster_name))
+ exit(
+ "ERROR - There are already instances "
+ "running for {0} cluster".format(self.config.cluster_name)
+ )
- if isfile(self.config.hosts_path):
- exit("ERROR - A hosts file already exists at {0}. Please delete before running launch again"
- .format(self.config.hosts_path))
+ if path.isfile(self.config.hosts_path):
+ exit(
+ "ERROR - A hosts file already exists at {0}. "
+ "Please delete before running launch again".format(
+ self.config.hosts_path
+ )
+ )
self.config.verify_launch()
print("Launching {0} cluster".format(self.config.cluster_name))
- if self.config.has_option('ec2', 'security_group_id'):
- sg_id = self.config.get('ec2', 'security_group_id')
+ if self.config.has_option("ec2", "security_group_id"):
+ sg_id = self.config.get("ec2", "security_group_id")
else:
sg_id = self.create_security_group()
instance_d = {}
for (hostname, services) in list(self.config.nodes().items()):
instance = self.launch_node(hostname, services, sg_id)
- instance_d[instance['InstanceId']] = hostname
+ instance_d[instance["InstanceId"]] = hostname
- num_running = len(self.get_status(['running']))
+ num_running = len(self.get_status(["running"]))
num_expected = len(self.config.nodes())
while num_running != num_expected:
- print("{0} of {1} nodes have started. Waiting another 5 sec..".format(num_running, num_expected))
+ print(
+ "{0} of {1} nodes have started. "
+ "Waiting another 5 sec..".format(num_running, num_expected)
+ )
time.sleep(5)
- num_running = len(self.get_status(['running']))
+ num_running = len(self.get_status(["running"]))
- with open(self.config.hosts_path, 'w') as hosts_file:
- for instance in self.get_status(['running']):
- public_ip = ''
- if 'PublicIpAddress' in instance:
- public_ip = instance['PublicIpAddress']
- private_ip = instance['PrivateIpAddress']
- hostname = instance_d[instance['InstanceId']]
- print("{0} {1} {2}".format(hostname, private_ip, public_ip), file=hosts_file)
+ with open(self.config.hosts_path, "w") as hosts_file:
+ for instance in self.get_status(["running"]):
+ public_ip = ""
+ if "PublicIpAddress" in instance:
+ public_ip = instance["PublicIpAddress"]
+ private_ip = instance["PrivateIpAddress"]
+ hostname = instance_d[instance["InstanceId"]]
+ print(
+ "{0} {1} {2}".format(hostname, private_ip, public_ip),
+ file=hosts_file,
+ )
- print("All {0} nodes have started. Created hosts file at {1}".format(num_expected, self.config.hosts_path))
+ print(
+ "All {0} nodes have started. Created hosts file at {1}".format(
+ num_expected, self.config.hosts_path
+ )
+ )
def status(self):
- nodes = self.get_status(['running'])
- print("Found {0} nodes in {1} cluster".format(len(nodes), self.config.cluster_name))
+ nodes = self.get_status(["running"])
+ print(
+ "Found {0} nodes in {1} cluster".format(
+ len(nodes), self.config.cluster_name
+ )
+ )
self.print_nodes(nodes)
def get_status(self, states):
- ec2 = boto3.client('ec2')
- response = ec2.describe_instances(Filters=[{'Name': 'tag:Muchos', 'Values': [self.config.cluster_name]}])
+ ec2 = boto3.client("ec2")
+ response = ec2.describe_instances(
+ Filters=[
+ {"Name": "tag:Muchos", "Values": [self.config.cluster_name]}
+ ]
+ )
nodes = []
- for res in response['Reservations']:
- for inst in res['Instances']:
- if inst['State']['Name'] in states:
+ for res in response["Reservations"]:
+ for inst in res["Instances"]:
+ if inst["State"]["Name"] in states:
nodes.append(inst)
return nodes
def active_nodes(self):
- return self.get_status(['pending', 'running', 'stopping', 'stopped'])
+ return self.get_status(["pending", "running", "stopping", "stopped"])
@staticmethod
def print_nodes(nodes):
for node in nodes:
- name = 'Unknown'
- for tag in node['Tags']:
- if tag['Key'] == 'Name':
- name = tag['Value']
- print(" ", name, node['InstanceId'], node['PrivateIpAddress'], node.get('PublicIpAddress', ''))
+ name = "Unknown"
+ for tag in node["Tags"]:
+ if tag["Key"] == "Name":
+ name = tag["Value"]
+ print(
+ " ",
+ name,
+ node["InstanceId"],
+ node["PrivateIpAddress"],
+ node.get("PublicIpAddress", ""),
+ )
def terminate(self):
nodes = self.active_nodes()
- print("The following {0} nodes in {1} cluster will be terminated:".format(len(nodes), self.config.cluster_name))
+ print(
+ "The following {0} nodes in {1} cluster "
+ "will be terminated:".format(len(nodes), self.config.cluster_name)
+ )
self.print_nodes(nodes)
response = input("Do you want to continue? (y/n) ")
if response == "y":
- ec2 = boto3.client('ec2')
+ ec2 = boto3.client("ec2")
for node in nodes:
- ec2.terminate_instances(InstanceIds=[node['InstanceId']])
+ ec2.terminate_instances(InstanceIds=[node["InstanceId"]])
print("Terminated nodes.")
- if not self.config.has_option('ec2', 'security_group_id'):
+ if not self.config.has_option("ec2", "security_group_id"):
self.delete_security_group()
- if isfile(self.config.hosts_path):
+ if path.isfile(self.config.hosts_path):
os.remove(self.config.hosts_path)
print("Removed hosts file at ", self.config.hosts_path)
else:
@@ -242,19 +328,27 @@
def wipe(self):
super().wipe()
-class Ec2ClusterTemplate(Ec2Cluster):
+class Ec2ClusterTemplate(Ec2Cluster):
def __init__(self, config):
Ec2Cluster.__init__(self, config)
def launch(self):
- print("Using cluster template '{0}' to launch nodes".format(self.config.cluster_template_d['id']))
+ print(
+ "Using cluster template '{0}' to launch nodes".format(
+ self.config.cluster_template_d["id"]
+ )
+ )
super().launch()
def init_request(self, hostname, services, sg_id):
# the first service in the list denotes the node's target template
print("Template '{0}' selected for {1}".format(services[0], hostname))
# interpolate any values from the ec2 config section and create request
- ec2_d = dict(self.config.items('ec2'))
- ec2_d['security_group_id'] = sg_id
- return json.loads(Template(self.config.cluster_template_d[services[0]]).substitute(ec2_d))
+ ec2_d = dict(self.config.items("ec2"))
+ ec2_d["security_group_id"] = sg_id
+ return json.loads(
+ Template(self.config.cluster_template_d[services[0]]).substitute(
+ ec2_d
+ )
+ )
diff --git a/lib/muchos/existing.py b/lib/muchos/existing.py
index 310afba..31d7fc9 100644
--- a/lib/muchos/existing.py
+++ b/lib/muchos/existing.py
@@ -19,31 +19,40 @@
import shutil
import subprocess
import time
-from os.path import isfile, join
+from os import path
from sys import exit
from os import listdir
-class ExistingCluster:
+class ExistingCluster:
def __init__(self, config):
self.config = config
def launch(self):
- exit('ERROR - Attempting to launch when cluster_type is set to "existing"')
+ exit(
+ "ERROR - 'launch' command cannot be used "
+ "when cluster_type is set to 'existing'"
+ )
def sync(self):
config = self.config
- print('Syncing ansible directory on {0} cluster proxy node'.format(config.cluster_name))
+ print(
+ "Syncing ansible directory on {0} cluster proxy node".format(
+ config.cluster_name
+ )
+ )
host_vars = config.ansible_host_vars()
play_vars = config.ansible_play_vars()
- for k,v in host_vars.items():
+ for k, v in host_vars.items():
host_vars[k] = self.config.resolve_value(k, default=v)
- for k,v in play_vars.items():
+ for k, v in play_vars.items():
play_vars[k] = self.config.resolve_value(k, default=v)
- with open(join(config.deploy_path, "ansible/site.yml"), 'w') as site_file:
+ with open(
+ path.join(config.deploy_path, "ansible/site.yml"), "w"
+ ) as site_file:
print("- import_playbook: common.yml", file=site_file)
print("- import_playbook: zookeeper.yml", file=site_file)
@@ -55,52 +64,86 @@
if config.has_service("metrics"):
print("- import_playbook: metrics.yml", file=site_file)
print("- import_playbook: accumulo.yml", file=site_file)
- if config.has_service('fluo'):
+ if config.has_service("fluo"):
print("- import_playbook: fluo.yml", file=site_file)
- if config.has_service('fluo_yarn'):
+ if config.has_service("fluo_yarn"):
print("- import_playbook: fluo_yarn.yml", file=site_file)
if config.has_service("mesosmaster"):
print("- import_playbook: mesos.yml", file=site_file)
if config.has_service("swarmmanager"):
print("- import_playbook: docker.yml", file=site_file)
- ansible_conf = join(config.deploy_path, "ansible/conf")
- with open(join(ansible_conf, "hosts"), 'w') as hosts_file:
- print("[proxy]\n{0}".format(config.proxy_hostname()), file=hosts_file)
+ ansible_conf = path.join(config.deploy_path, "ansible/conf")
+ with open(path.join(ansible_conf, "hosts"), "w") as hosts_file:
+ print(
+ "[proxy]\n{0}".format(config.proxy_hostname()), file=hosts_file
+ )
print("\n[accumulomaster]", file=hosts_file)
- for (index, accu_host) in enumerate(config.get_service_hostnames("accumulomaster"), start=1):
- print("{0}".format(accu_host,index), file=hosts_file)
- print("\n[namenode]",file=hosts_file)
- for (index, nn_host) in enumerate(config.get_service_hostnames("namenode"), start=1):
- print("{0}".format(nn_host,index), file=hosts_file)
- print("\n[journalnode]",file=hosts_file)
- for (index, jn_host) in enumerate(config.get_service_hostnames("journalnode"), start=1):
- print("{0}".format(jn_host,index), file=hosts_file)
- print("\n[zkfc]",file=hosts_file)
- for (index, zkfc_host) in enumerate(config.get_service_hostnames("zkfc"), start=1):
- print("{0}".format(zkfc_host,index), file=hosts_file)
- print("\n[resourcemanager]",file=hosts_file)
- for (index, rm_host) in enumerate(config.get_service_hostnames("resourcemanager"), start=1):
- print("{0}".format(rm_host,index), file=hosts_file)
+ for (index, accu_host) in enumerate(
+ config.get_service_hostnames("accumulomaster"), start=1
+ ):
+ print("{0}".format(accu_host, index), file=hosts_file)
+ print("\n[namenode]", file=hosts_file)
+ for (index, nn_host) in enumerate(
+ config.get_service_hostnames("namenode"), start=1
+ ):
+ print("{0}".format(nn_host, index), file=hosts_file)
+ print("\n[journalnode]", file=hosts_file)
+ for (index, jn_host) in enumerate(
+ config.get_service_hostnames("journalnode"), start=1
+ ):
+ print("{0}".format(jn_host, index), file=hosts_file)
+ print("\n[zkfc]", file=hosts_file)
+ for (index, zkfc_host) in enumerate(
+ config.get_service_hostnames("zkfc"), start=1
+ ):
+ print("{0}".format(zkfc_host, index), file=hosts_file)
+ print("\n[resourcemanager]", file=hosts_file)
+ for (index, rm_host) in enumerate(
+ config.get_service_hostnames("resourcemanager"), start=1
+ ):
+ print("{0}".format(rm_host, index), file=hosts_file)
if config.has_service("spark"):
- print("\n[spark]\n{0}".format(config.get_service_hostnames("spark")[0]), file=hosts_file)
+ print(
+ "\n[spark]\n{0}".format(
+ config.get_service_hostnames("spark")[0]
+ ),
+ file=hosts_file,
+ )
if config.has_service("mesosmaster"):
- print("\n[mesosmaster]\n{0}".format(config.get_service_hostnames("mesosmaster")[0]), file=hosts_file)
+ print(
+ "\n[mesosmaster]\n{0}".format(
+ config.get_service_hostnames("mesosmaster")[0]
+ ),
+ file=hosts_file,
+ )
if config.has_service("metrics"):
- print("\n[metrics]\n{0}".format(config.get_service_hostnames("metrics")[0]), file=hosts_file)
+ print(
+ "\n[metrics]\n{0}".format(
+ config.get_service_hostnames("metrics")[0]
+ ),
+ file=hosts_file,
+ )
if config.has_service("swarmmanager"):
- print("\n[swarmmanager]\n{0}".format(config.get_service_hostnames("swarmmanager")[0]), file=hosts_file)
+ print(
+ "\n[swarmmanager]\n{0}".format(
+ config.get_service_hostnames("swarmmanager")[0]
+ ),
+ file=hosts_file,
+ )
print("\n[zookeepers]", file=hosts_file)
- for (index, zk_host) in enumerate(config.get_service_hostnames("zookeeper"), start=1):
+ for (index, zk_host) in enumerate(
+ config.get_service_hostnames("zookeeper"), start=1
+ ):
print("{0} id={1}".format(zk_host, index), file=hosts_file)
- if config.has_service('fluo'):
+ if config.has_service("fluo"):
print("\n[fluo]", file=hosts_file)
for host in config.get_service_hostnames("fluo"):
print(host, file=hosts_file)
- if config.has_service('fluo_yarn'):
+ if config.has_service("fluo_yarn"):
print("\n[fluo_yarn]", file=hosts_file)
for host in config.get_service_hostnames("fluo_yarn"):
print(host, file=hosts_file)
@@ -109,143 +152,239 @@
for worker_host in config.get_service_hostnames("worker"):
print(worker_host, file=hosts_file)
- print("\n[accumulo:children]\naccumulomaster\nworkers", file=hosts_file)
- print("\n[hadoop:children]\nnamenode\nresourcemanager\nworkers", file=hosts_file)
+ print(
+ "\n[accumulo:children]\naccumulomaster\nworkers",
+ file=hosts_file,
+ )
+ print(
+ "\n[hadoop:children]\nnamenode\nresourcemanager"
+ "\nworkers\nzkfc\njournalnode",
+ file=hosts_file,
+ )
print("\n[nodes]", file=hosts_file)
for (private_ip, hostname) in config.get_private_ip_hostnames():
- print("{0} ansible_ssh_host={1} node_type={2}".format(hostname, private_ip,
- config.node_type(hostname)), file=hosts_file)
+ print(
+ "{0} ansible_ssh_host={1} node_type={2}".format(
+ hostname, private_ip, config.node_type(hostname)
+ ),
+ file=hosts_file,
+ )
print("\n[all:vars]", file=hosts_file)
for (name, value) in sorted(host_vars.items()):
print("{0} = {1}".format(name, value), file=hosts_file)
- with open(join(config.deploy_path, "ansible/group_vars/all"), 'w') as play_vars_file:
+ with open(
+ path.join(config.deploy_path, "ansible/group_vars/all"), "w"
+ ) as play_vars_file:
for (name, value) in sorted(play_vars.items()):
print("{0}: {1}".format(name, value), file=play_vars_file)
# copy keys file to ansible/conf (if it exists)
- conf_keys = join(config.deploy_path, "conf/keys")
- ansible_keys = join(ansible_conf, "keys")
- if isfile(conf_keys):
+ conf_keys = path.join(config.deploy_path, "conf/keys")
+ ansible_keys = path.join(ansible_conf, "keys")
+ if path.isfile(conf_keys):
shutil.copyfile(conf_keys, ansible_keys)
else:
- open(ansible_keys, 'w').close()
+ open(ansible_keys, "w").close()
cmd = "rsync -az --delete -e \"ssh -o 'StrictHostKeyChecking no'\""
- subprocess.call("{cmd} {src} {usr}@{ldr}:{tdir}".format(cmd=cmd, src=join(config.deploy_path, "ansible"),
- usr=config.get('general', 'cluster_user'),
- ldr=config.get_proxy_ip(), tdir=config.user_home()),
- shell=True)
+ subprocess.call(
+ "{cmd} {src} {usr}@{ldr}:{tdir}".format(
+ cmd=cmd,
+ src=path.join(config.deploy_path, "ansible"),
+ usr=config.get("general", "cluster_user"),
+ ldr=config.get_proxy_ip(),
+ tdir=config.user_home(),
+ ),
+ shell=True,
+ )
- self.exec_on_proxy_verified("{0}/ansible/scripts/install_ansible.sh".format(config.user_home()), opts='-t')
+ self.exec_on_proxy_verified(
+ "{0}/ansible/scripts/install_ansible.sh".format(
+ config.user_home()
+ ),
+ opts="-t",
+ )
def setup(self):
config = self.config
- print('Setting up {0} cluster'.format(config.cluster_name))
+ print("Setting up {0} cluster".format(config.cluster_name))
self.sync()
- conf_upload = join(config.deploy_path, "conf/upload")
+ conf_upload = path.join(config.deploy_path, "conf/upload")
cluster_tarballs = "{0}/tarballs".format(config.user_home())
self.exec_on_proxy_verified("mkdir -p {0}".format(cluster_tarballs))
for f in listdir(conf_upload):
- tarball_path = join(conf_upload, f)
- if isfile(tarball_path) and tarball_path.endswith("gz"):
+ tarball_path = path.join(conf_upload, f)
+ if path.isfile(tarball_path) and tarball_path.endswith("gz"):
self.send_to_proxy(tarball_path, cluster_tarballs)
self.execute_playbook("site.yml")
@staticmethod
def status():
- exit("ERROR - 'status' command cannot be used when cluster_type=existing")
+ exit(
+ "ERROR - 'status' command cannot be used "
+ "when cluster_type is set to 'existing'"
+ )
@staticmethod
def terminate():
- exit("ERROR - 'terminate' command cannot be used when cluster_type=existing")
+ exit(
+ "ERROR - 'terminate' command cannot be used "
+ "when cluster_type is set to 'existing'"
+ )
def ssh(self):
self.wait_until_proxy_ready()
- fwd = ''
- if self.config.has_option('general', 'proxy_socks_port'):
- fwd = "-D " + self.config.get('general', 'proxy_socks_port')
- ssh_command = "ssh -C -A -o 'StrictHostKeyChecking no' {fwd} {usr}@{ldr}".format(
- usr=self.config.get('general', 'cluster_user'), ldr=self.config.get_proxy_ip(), fwd=fwd)
+ fwd = ""
+ if self.config.has_option("general", "proxy_socks_port"):
+ fwd = "-D " + self.config.get("general", "proxy_socks_port")
+ ssh_command = (
+ "ssh -C -A -o 'StrictHostKeyChecking no' " "{fwd} {usr}@{ldr}"
+ ).format(
+ usr=self.config.get("general", "cluster_user"),
+ ldr=self.config.get_proxy_ip(),
+ fwd=fwd,
+ )
print("Logging into proxy using: {0}".format(ssh_command))
retcode = subprocess.call(ssh_command, shell=True)
if retcode != 0:
- exit("ERROR - Command failed with return code of {0}: {1}".format(retcode, ssh_command))
+ exit(
+ "ERROR - Command failed with return code of {0}: {1}".format(
+ retcode, ssh_command
+ )
+ )
- def exec_on_proxy(self, command, opts=''):
- ssh_command = "ssh -A -o 'StrictHostKeyChecking no' {opts} {usr}@{ldr} '{cmd}'".format(
- usr=self.config.get('general', 'cluster_user'),
- ldr=self.config.get_proxy_ip(), cmd=command, opts=opts)
+ def exec_on_proxy(self, command, opts=""):
+ ssh_command = (
+ "ssh -A -o 'StrictHostKeyChecking no' "
+ "{opts} {usr}@{ldr} '{cmd}'"
+ ).format(
+ usr=self.config.get("general", "cluster_user"),
+ ldr=self.config.get_proxy_ip(),
+ cmd=command,
+ opts=opts,
+ )
return subprocess.call(ssh_command, shell=True), ssh_command
- def exec_on_proxy_verified(self, command, opts=''):
+ def exec_on_proxy_verified(self, command, opts=""):
(retcode, ssh_command) = self.exec_on_proxy(command, opts)
if retcode != 0:
- exit("ERROR - Command failed with return code of {0}: {1}".format(retcode, ssh_command))
+ exit(
+ "ERROR - Command failed with return code of {0}: {1}".format(
+ retcode, ssh_command
+ )
+ )
def wait_until_proxy_ready(self):
- cluster_user = self.config.get('general', 'cluster_user')
- print("Checking if '{0}' proxy can be reached using: ssh {1}@{2}"
- .format(self.config.proxy_hostname(), cluster_user, self.config.get_proxy_ip()))
+ cluster_user = self.config.get("general", "cluster_user")
+ print(
+ "Checking if '{0}' proxy can be reached using: "
+ "ssh {1}@{2}".format(
+ self.config.proxy_hostname(),
+ cluster_user,
+ self.config.get_proxy_ip(),
+ )
+ )
while True:
- (retcode, ssh_command) = self.exec_on_proxy('pwd > /dev/null')
+ (retcode, ssh_command) = self.exec_on_proxy("pwd > /dev/null")
if retcode == 0:
print("Connected to proxy using SSH!")
time.sleep(1)
break
- print("Proxy could not be accessed using SSH. Will retry in 5 sec...")
+ print(
+ "Proxy could not be accessed using SSH. "
+ "Will retry in 5 sec..."
+ )
time.sleep(5)
def execute_playbook(self, playbook):
print("Executing '{0}' playbook".format(playbook))
- azure_proxy_host = self.config.get("azure","azure_proxy_host")
- var_azure_proxy_host = "_" if (azure_proxy_host==None or azure_proxy_host.strip()=='') else azure_proxy_host
- self.exec_on_proxy_verified("time -p ansible-playbook {base}/ansible/{playbook} --extra-vars \"azure_proxy_host={var_azure_proxy_host}\""
- .format(base=self.config.user_home(), playbook=playbook, var_azure_proxy_host=var_azure_proxy_host), opts='-t')
+ azure_proxy_host = self.config.get("azure", "azure_proxy_host")
+ var_azure_proxy_host = (
+ "_"
+ if (azure_proxy_host is None or azure_proxy_host.strip() == "")
+ else azure_proxy_host
+ )
+ self.exec_on_proxy_verified(
+ "time -p ansible-playbook {base}/ansible/{playbook} "
+ "--extra-vars 'azure_proxy_host={var_azure_proxy_host}'".format(
+ base=self.config.user_home(),
+ playbook=playbook,
+ var_azure_proxy_host=var_azure_proxy_host,
+ ),
+ opts="-t",
+ )
def send_to_proxy(self, path, target, skip_if_exists=True):
print("Copying to proxy: ", path)
cmd = "scp -o 'StrictHostKeyChecking no'"
if skip_if_exists:
- cmd = "rsync --update --progress -e \"ssh -q -o 'StrictHostKeyChecking no'\""
- subprocess.call("{cmd} {src} {usr}@{ldr}:{tdir}".format(
- cmd=cmd, src=path, usr=self.config.get('general', 'cluster_user'), ldr=self.config.get_proxy_ip(),
- tdir=target), shell=True)
+ cmd = (
+ 'rsync --update --progress -e "ssh -q -o '
+ "'StrictHostKeyChecking no'\""
+ )
+ subprocess.call(
+ "{cmd} {src} {usr}@{ldr}:{tdir}".format(
+ cmd=cmd,
+ src=path,
+ usr=self.config.get("general", "cluster_user"),
+ ldr=self.config.get_proxy_ip(),
+ tdir=target,
+ ),
+ shell=True,
+ )
def wipe(self):
- if not isfile(self.config.hosts_path):
- exit("Hosts file does not exist for cluster: " + self.config.hosts_path)
- print("Killing all processes started by Muchos and wiping Muchos data from {0} cluster"
- .format(self.config.cluster_name))
+ if not path.isfile(self.config.hosts_path):
+ exit(
+ "Hosts file does not exist for cluster: "
+ + self.config.hosts_path
+ )
+ print(
+ "Killing all processes started by Muchos and "
+ "wiping Muchos data from {0} cluster".format(
+ self.config.cluster_name
+ )
+ )
self.execute_playbook("wipe.yml")
def perform(self, action):
- if action == 'launch':
+ if action == "launch":
self.launch()
- elif action == 'status':
+ elif action == "status":
self.status()
- elif action == 'sync':
+ elif action == "sync":
self.sync()
- elif action == 'setup':
+ elif action == "setup":
self.setup()
- elif action == 'ssh':
+ elif action == "ssh":
self.ssh()
- elif action == 'wipe':
+ elif action == "wipe":
self.wipe()
- elif action in ('kill', 'cancel_shutdown'):
- if not isfile(self.config.hosts_path):
- exit("Hosts file does not exist for cluster: " + self.config.hosts_path)
- elif action == 'kill':
- print("Killing all processes started by Muchos on {0} cluster".format(self.config.cluster_name))
- elif action == 'cancel_shutdown':
- print("Cancelling automatic shutdown of {0} cluster".format(self.config.cluster_name))
+ elif action in ("kill", "cancel_shutdown"):
+ if not path.isfile(self.config.hosts_path):
+ exit(
+ "Hosts file does not exist for cluster: "
+ + self.config.hosts_path
+ )
+ elif action == "kill":
+ print(
+ "Killing all processes started by Muchos "
+ "on {0} cluster".format(self.config.cluster_name)
+ )
+ elif action == "cancel_shutdown":
+ print(
+ "Cancelling automatic shutdown of {0} cluster".format(
+ self.config.cluster_name
+ )
+ )
self.execute_playbook(action + ".yml")
- elif action == 'terminate':
+ elif action == "terminate":
self.terminate()
else:
- print('ERROR - Unknown action:', action)
+ print("ERROR - Unknown action:", action)
diff --git a/lib/muchos/util.py b/lib/muchos/util.py
index 0a665d0..158ea92 100644
--- a/lib/muchos/util.py
+++ b/lib/muchos/util.py
@@ -31,6 +31,7 @@
self.ephemeral = ephemeral
self.has_nvme = has_nvme
+
AMI_HELP_MSG = """PLEASE NOTE - If you have accepted the software terms for CentOS 7 and still get an error,
this could be due to CentOS releasing new images of CentOS 7. When this occurs, the old images
are no longer available to new users. If you think this is the case, go to the CentOS 7 product
@@ -40,7 +41,7 @@
On the product page, find the latest AMI ID for your EC2 region. This should be used to set the 'aws_ami'
property in your muchos.props. After setting the 'aws_ami' property, run the launch command again.
-"""
+""" # noqa
instance_types = {
"c1.medium": EC2Type("pvm"),
@@ -88,13 +89,16 @@
"d2.xlarge": EC2Type("hvm", 3),
"d2.2xlarge": EC2Type("hvm", 6),
"d2.4xlarge": EC2Type("hvm", 12),
- "d2.8xlarge": EC2Type("hvm", 24)
+ "d2.8xlarge": EC2Type("hvm", 24),
}
def verify_type(instance_type):
if instance_type not in instance_types:
- print("ERROR - EC2 instance type '%s' is currently not supported!" % instance_type)
+ print(
+ "ERROR - EC2 instance type '{}' is currently "
+ "not supported!".format(instance_type)
+ )
print("This is probably due to the instance type being EBS-only.")
print("Below is a list of supported instance types:")
for key in instance_types:
@@ -106,6 +110,7 @@
verify_type(instance_type)
return instance_types.get(instance_type).arch
+
def get_ephemeral_devices(instance_type):
verify_type(instance_type)
devices = []
@@ -117,46 +122,60 @@
for i in range(start, ec2_type.ephemeral + start):
if ec2_type.has_nvme:
- devices.append('/dev/nvme' + str(i) + 'n1')
+ devices.append("/dev/nvme" + str(i) + "n1")
else:
- devices.append('/dev/xvd' + chr(ord('b') + i))
+ devices.append("/dev/xvd" + chr(ord("b") + i))
return devices
+
def get_block_device_map(instance_type):
verify_type(instance_type)
- bdm = [{'DeviceName': '/dev/sda1',
- 'Ebs': {'DeleteOnTermination': True}}]
+ bdm = [{"DeviceName": "/dev/sda1", "Ebs": {"DeleteOnTermination": True}}]
ec2_type = instance_types.get(instance_type)
- if not ec2_type.has_nvme :
+ if not ec2_type.has_nvme:
for i in range(0, ec2_type.ephemeral):
- device = {'DeviceName': '/dev/xvd' + chr(ord('b') + i),
- 'VirtualName': 'ephemeral' + str(i)}
+ device = {
+ "DeviceName": "/dev/xvd" + chr(ord("b") + i),
+ "VirtualName": "ephemeral" + str(i),
+ }
bdm.append(device)
return bdm
+
def parse_args(hosts_dir, input_args=None):
parser = OptionParser(
- usage="muchos [options] <action>\n\n"
- + "where <action> can be:\n"
- + " launch Launch cluster in Azure or EC2\n"
- + " status Check status of Azure or EC2 cluster\n"
- + " setup Set up cluster\n"
- + " sync Sync ansible directory on cluster proxy node\n"
- + " config Print configuration for that cluster. Requires '-p'. Use '-p all' for all config.\n"
- + " ssh SSH to cluster proxy node\n"
- + " kill Kills processes on cluster started by Muchos\n"
- + " wipe Wipes cluster data and kills processes\n"
- + " terminate Terminate EC2 cluster\n"
- + " cancel_shutdown Cancels automatic shutdown of EC2 cluster",
- add_help_option=False)
- parser.add_option("-c", "--cluster", dest="cluster", help="Specifies cluster")
- parser.add_option("-p", "--property", dest="property", help="Specifies property to print (if using 'config' action)"
- ". Set to 'all' to print every property")
- parser.add_option("-h", "--help", action="help", help="Show this help message and exit")
+ usage="muchos [options] <action>\n\n"
+ + "where <action> can be:\n"
+ + " launch Launch cluster in Azure or EC2\n"
+ + " status Check status of Azure or EC2 cluster\n"
+ + " setup Set up cluster\n"
+ + " sync Sync ansible directory on cluster proxy node\n"
+ + " config Print configuration for that cluster. "
+ "Requires '-p'. Use '-p all' for all config.\n"
+ + " ssh SSH to cluster proxy node\n"
+ + " kill Kills processes on cluster started by Muchos\n"
+ + " wipe Wipes cluster data and kills processes\n"
+ + " terminate Terminate EC2 cluster\n"
+ + " cancel_shutdown Cancels automatic shutdown of EC2 cluster",
+ add_help_option=False,
+ )
+ parser.add_option(
+ "-c", "--cluster", dest="cluster", help="Specifies cluster"
+ )
+ parser.add_option(
+ "-p",
+ "--property",
+ dest="property",
+ help="Specifies property to print (if using 'config' action)"
+ ". Set to 'all' to print every property",
+ )
+ parser.add_option(
+ "-h", "--help", action="help", help="Show this help message and exit"
+ )
if input_args:
(opts, args) = parser.parse_args(input_args)
@@ -168,7 +187,7 @@
return
action = args[0]
- if action == 'launch' and not opts.cluster:
+ if action == "launch" and not opts.cluster:
print("ERROR - You must specify a cluster if using launch command")
return
@@ -176,17 +195,24 @@
if not opts.cluster:
if len(clusters) == 0:
- print("ERROR - No clusters found in conf/hosts or specified by --cluster option")
+ print(
+ "ERROR - No clusters found in conf/hosts "
+ "or specified by --cluster option"
+ )
return
elif len(clusters) == 1:
opts.cluster = clusters[0]
else:
- print("ERROR - Multiple clusters {0} found in conf/hosts/. " \
- "Please pick one using --cluster option".format(clusters))
+ print(
+ "ERROR - Multiple clusters {0} found in conf/hosts/. "
+ "Please pick one using --cluster option".format(clusters)
+ )
return
- if action == 'config' and not opts.property:
- print("ERROR - For config action, you must set -p to a property or 'all'")
+ if action == "config" and not opts.property:
+ print(
+ "ERROR - For config action, you must set -p to a property or 'all'"
+ )
return
return opts, action, args[1:]
diff --git a/lib/tests/__init__.py b/lib/tests/__init__.py
index 555eccf..cce3aca 100644
--- a/lib/tests/__init__.py
+++ b/lib/tests/__init__.py
@@ -14,4 +14,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
diff --git a/lib/tests/azure/__init__.py b/lib/tests/azure/__init__.py
index 555eccf..cce3aca 100644
--- a/lib/tests/azure/__init__.py
+++ b/lib/tests/azure/__init__.py
@@ -14,4 +14,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
diff --git a/lib/tests/azure/test_config.py b/lib/tests/azure/test_config.py
index 1b7e4c7..873c09e 100644
--- a/lib/tests/azure/test_config.py
+++ b/lib/tests/azure/test_config.py
@@ -19,66 +19,105 @@
def test_azure_cluster():
- c = AzureDeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', '../conf/templates', 'mycluster')
+ c = AzureDeployConfig(
+ "muchos",
+ "../conf/muchos.props.example",
+ "../conf/hosts/example/example_cluster",
+ "../conf/checksums",
+ "../conf/templates",
+ "mycluster",
+ )
- # since we are sharing a single muchos.props.example file, we need
- # to stub the cluster type to be azure (as the file itself has a default of ec2)
+ # since we are sharing a single muchos.props.example file, we need to stub
+ # the cluster type to be azure (as the file itself has a default of ec2)
- c.cluster_type = 'azure'
+ c.cluster_type = "azure"
- assert c.checksum_ver('accumulo', '1.9.0') == 'sha256:f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe'
- assert c.checksum('accumulo') == 'sha256:df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47'
- assert c.get('azure', 'vm_sku') == 'Standard_D8s_v3'
- assert c.get('azure', 'managed_disk_type') == 'Standard_LRS'
- assert c.user_home() == '/home/centos'
- assert c.mount_root() == '/var/data'
- assert c.worker_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
- assert c.default_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
- assert c.metrics_drive_ids() == ['var-data1', 'var-data2', 'var-data3']
- assert c.shutdown_delay_minutes() == '0'
- assert c.mounts(2) == ['/var/data0', '/var/data1']
- assert c.node_type('worker1') == 'worker'
- assert c.node_type('leader1') == 'default'
- assert c.has_option('azure', 'resource_group')
- assert c.has_option('azure', 'vnet')
- assert c.has_option('azure', 'vnet_cidr')
- assert c.has_option('azure', 'subnet')
- assert c.has_option('azure', 'subnet_cidr')
- assert c.has_option('azure', 'numnodes')
- assert c.has_option('azure', 'location')
+ assert c.checksum_ver("accumulo", "1.9.0") == (
+ "sha256:"
+ "f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe"
+ )
+ assert c.checksum("accumulo") == (
+ "sha256:"
+ "df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47"
+ )
+ assert c.get("azure", "vm_sku") == "Standard_D8s_v3"
+ assert c.get("azure", "managed_disk_type") == "Standard_LRS"
+ assert c.user_home() == "/home/centos"
+ assert c.mount_root() == "/var/data"
+ assert c.worker_data_dirs() == ["/var/data1", "/var/data2", "/var/data3"]
+ assert c.default_data_dirs() == ["/var/data1", "/var/data2", "/var/data3"]
+ assert c.metrics_drive_ids() == ["var-data1", "var-data2", "var-data3"]
+ assert c.shutdown_delay_minutes() == "0"
+ assert c.mounts(2) == ["/var/data0", "/var/data1"]
+ assert c.node_type("worker1") == "worker"
+ assert c.node_type("leader1") == "default"
+ assert c.has_option("azure", "resource_group")
+ assert c.has_option("azure", "vnet")
+ assert c.has_option("azure", "vnet_cidr")
+ assert c.has_option("azure", "subnet")
+ assert c.has_option("azure", "subnet_cidr")
+ assert c.has_option("azure", "numnodes")
+ assert c.has_option("azure", "location")
assert len(c.nodes()) == 6
- assert c.get_node('leader1') == ['namenode', 'resourcemanager', 'accumulomaster', 'zookeeper']
- assert c.get_node('leader2') == ['metrics']
- assert c.get_node('worker1') == ['worker', 'swarmmanager']
- assert c.get_node('worker2') == ['worker']
- assert c.get_node('worker3') == ['worker']
- assert c.has_service('accumulomaster')
- assert not c.has_service('fluo')
- assert c.get_service_hostnames('worker') == ['worker1', 'worker2', 'worker3', 'worker4']
- assert c.get_service_hostnames('zookeeper') == ['leader1']
- assert c.get_hosts() == {'leader2': ('10.0.0.1', None), 'leader1': ('10.0.0.0', '23.0.0.0'),
- 'worker1': ('10.0.0.2', None), 'worker3': ('10.0.0.4', None),
- 'worker2': ('10.0.0.3', None), 'worker4': ('10.0.0.5', None)}
- assert c.get_public_ip('leader1') == '23.0.0.0'
- assert c.get_private_ip('leader1') == '10.0.0.0'
- assert c.cluster_name == 'mycluster'
- assert c.get_cluster_type() == 'azure'
- assert c.version("accumulo").startswith('2.')
- assert c.version("fluo").startswith('1.')
- assert c.version("hadoop").startswith('3.')
- assert c.version("zookeeper").startswith('3.')
- assert c.get_service_private_ips("worker") == ['10.0.0.2', '10.0.0.3', '10.0.0.4', '10.0.0.5']
- assert c.get('general', 'proxy_hostname') == "leader1"
+ assert c.get_node("leader1") == [
+ "namenode",
+ "resourcemanager",
+ "accumulomaster",
+ "zookeeper",
+ ]
+ assert c.get_node("leader2") == ["metrics"]
+ assert c.get_node("worker1") == ["worker", "swarmmanager"]
+ assert c.get_node("worker2") == ["worker"]
+ assert c.get_node("worker3") == ["worker"]
+ assert c.has_service("accumulomaster")
+ assert not c.has_service("fluo")
+ assert c.get_service_hostnames("worker") == [
+ "worker1",
+ "worker2",
+ "worker3",
+ "worker4",
+ ]
+ assert c.get_service_hostnames("zookeeper") == ["leader1"]
+ assert c.get_hosts() == {
+ "leader2": ("10.0.0.1", None),
+ "leader1": ("10.0.0.0", "23.0.0.0"),
+ "worker1": ("10.0.0.2", None),
+ "worker3": ("10.0.0.4", None),
+ "worker2": ("10.0.0.3", None),
+ "worker4": ("10.0.0.5", None),
+ }
+ assert c.get_public_ip("leader1") == "23.0.0.0"
+ assert c.get_private_ip("leader1") == "10.0.0.0"
+ assert c.cluster_name == "mycluster"
+ assert c.get_cluster_type() == "azure"
+ assert c.version("accumulo").startswith("2.")
+ assert c.version("fluo").startswith("1.")
+ assert c.version("hadoop").startswith("3.")
+ assert c.version("zookeeper").startswith("3.")
+ assert c.get_service_private_ips("worker") == [
+ "10.0.0.2",
+ "10.0.0.3",
+ "10.0.0.4",
+ "10.0.0.5",
+ ]
+ assert c.get("general", "proxy_hostname") == "leader1"
assert c.proxy_public_ip() == "23.0.0.0"
assert c.proxy_private_ip() == "10.0.0.0"
- assert c.get('general', 'cluster_user') == "centos"
- assert c.get('general', 'cluster_group') == "centos"
- assert c.get_non_proxy() == [('10.0.0.1', 'leader2'), ('10.0.0.2', 'worker1'), ('10.0.0.3', 'worker2'),
- ('10.0.0.4', 'worker3'), ('10.0.0.5', 'worker4')]
- assert c.get_host_services() == [('leader1', 'namenode resourcemanager accumulomaster zookeeper'),
- ('leader2', 'metrics'),
- ('worker1', 'worker swarmmanager'),
- ('worker2', 'worker'),
- ('worker3', 'worker'),
- ('worker4', 'worker')]
+ assert c.get("general", "cluster_user") == "centos"
+ assert c.get("general", "cluster_group") == "centos"
+ assert c.get_non_proxy() == [
+ ("10.0.0.1", "leader2"),
+ ("10.0.0.2", "worker1"),
+ ("10.0.0.3", "worker2"),
+ ("10.0.0.4", "worker3"),
+ ("10.0.0.5", "worker4"),
+ ]
+ assert c.get_host_services() == [
+ ("leader1", "namenode resourcemanager accumulomaster zookeeper"),
+ ("leader2", "metrics"),
+ ("worker1", "worker swarmmanager"),
+ ("worker2", "worker"),
+ ("worker3", "worker"),
+ ("worker4", "worker"),
+ ]
diff --git a/lib/tests/ec2/__init__.py b/lib/tests/ec2/__init__.py
index 555eccf..cce3aca 100644
--- a/lib/tests/ec2/__init__.py
+++ b/lib/tests/ec2/__init__.py
@@ -14,4 +14,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
diff --git a/lib/tests/ec2/test_config.py b/lib/tests/ec2/test_config.py
index 640f1ff..52b619a 100644
--- a/lib/tests/ec2/test_config.py
+++ b/lib/tests/ec2/test_config.py
@@ -19,91 +19,150 @@
def test_ec2_cluster():
- c = Ec2DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', '../conf/templates', 'mycluster')
- assert c.checksum_ver('accumulo', '1.9.0') == 'sha256:f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe'
- assert c.checksum('accumulo') == 'sha256:df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47'
- assert c.get('ec2', 'default_instance_type') == 'm5d.large'
- assert c.get('ec2', 'worker_instance_type') == 'm5d.large'
- assert c.get('ec2', 'aws_ami') == 'ami-9887c6e7'
- assert c.user_home() == '/home/centos'
+ c = Ec2DeployConfig(
+ "muchos",
+ "../conf/muchos.props.example",
+ "../conf/hosts/example/example_cluster",
+ "../conf/checksums",
+ "../conf/templates",
+ "mycluster",
+ )
+ assert c.checksum_ver("accumulo", "1.9.0") == (
+ "sha256:"
+ "f68a6145029a9ea843b0305c90a7f5f0334d8a8ceeea94734267ec36421fe7fe"
+ )
+ assert c.checksum("accumulo") == (
+ "sha256:"
+ "df172111698c7a73aa031de09bd5589263a6b824482fbb9b4f0440a16602ed47"
+ )
+ assert c.get("ec2", "default_instance_type") == "m5d.large"
+ assert c.get("ec2", "worker_instance_type") == "m5d.large"
+ assert c.get("ec2", "aws_ami") == "ami-9887c6e7"
+ assert c.user_home() == "/home/centos"
assert c.max_ephemeral() == 1
- assert c.mount_root() == '/media/ephemeral'
- assert c.fstype() == 'ext3'
- assert c.force_format() == 'no'
- assert c.worker_data_dirs() == ['/media/ephemeral0']
- assert c.default_data_dirs() == ['/media/ephemeral0']
- assert c.metrics_drive_ids() == ['media-ephemeral0']
- assert c.shutdown_delay_minutes() == '0'
- assert c.mounts(2) == ['/media/ephemeral0', '/media/ephemeral1']
- assert c.node_type_map() == {'default': {'mounts': ['/media/ephemeral0', ], 'devices': ['/dev/nvme1n1', ]},
- 'worker': {'mounts': ['/media/ephemeral0', ], 'devices': ['/dev/nvme1n1', ]}}
- assert c.node_type('worker1') == 'worker'
- assert c.node_type('leader1') == 'default'
- assert not c.has_option('ec2', 'vpc_id')
- assert not c.has_option('ec2', 'subnet_id')
- assert c.get('ec2', 'key_name') == 'my_aws_key'
+ assert c.mount_root() == "/media/ephemeral"
+ assert c.fstype() == "ext3"
+ assert c.force_format() == "no"
+ assert c.worker_data_dirs() == ["/media/ephemeral0"]
+ assert c.default_data_dirs() == ["/media/ephemeral0"]
+ assert c.metrics_drive_ids() == ["media-ephemeral0"]
+ assert c.shutdown_delay_minutes() == "0"
+ assert c.mounts(2) == ["/media/ephemeral0", "/media/ephemeral1"]
+ assert c.node_type_map() == {
+ "default": {
+ "mounts": ["/media/ephemeral0"],
+ "devices": ["/dev/nvme1n1"],
+ },
+ "worker": {
+ "mounts": ["/media/ephemeral0"],
+ "devices": ["/dev/nvme1n1"],
+ },
+ }
+ assert c.node_type("worker1") == "worker"
+ assert c.node_type("leader1") == "default"
+ assert not c.has_option("ec2", "vpc_id")
+ assert not c.has_option("ec2", "subnet_id")
+ assert c.get("ec2", "key_name") == "my_aws_key"
assert c.instance_tags() == {}
assert len(c.nodes()) == 6
- assert c.get_node('leader1') == ['namenode', 'resourcemanager', 'accumulomaster', 'zookeeper']
- assert c.get_node('leader2') == ['metrics']
- assert c.get_node('worker1') == ['worker', 'swarmmanager']
- assert c.get_node('worker2') == ['worker']
- assert c.get_node('worker3') == ['worker']
- assert c.has_service('accumulomaster')
- assert not c.has_service('fluo')
- assert c.get_service_hostnames('worker') == ['worker1', 'worker2', 'worker3', 'worker4']
- assert c.get_service_hostnames('zookeeper') == ['leader1']
- assert c.get_hosts() == {'leader2': ('10.0.0.1', None), 'leader1': ('10.0.0.0', '23.0.0.0'),
- 'worker1': ('10.0.0.2', None), 'worker3': ('10.0.0.4', None),
- 'worker2': ('10.0.0.3', None), 'worker4': ('10.0.0.5', None)}
- assert c.get_public_ip('leader1') == '23.0.0.0'
- assert c.get_private_ip('leader1') == '10.0.0.0'
- assert c.cluster_name == 'mycluster'
- assert c.get_cluster_type() == 'ec2'
- assert c.version("accumulo").startswith('2.')
- assert c.version("fluo").startswith('1.')
- assert c.version("hadoop").startswith('3.')
- assert c.version("zookeeper").startswith('3.')
- assert c.get_service_private_ips("worker") == ['10.0.0.2', '10.0.0.3', '10.0.0.4', '10.0.0.5']
- assert c.get('general', 'proxy_hostname') == "leader1"
+ assert c.get_node("leader1") == [
+ "namenode",
+ "resourcemanager",
+ "accumulomaster",
+ "zookeeper",
+ ]
+ assert c.get_node("leader2") == ["metrics"]
+ assert c.get_node("worker1") == ["worker", "swarmmanager"]
+ assert c.get_node("worker2") == ["worker"]
+ assert c.get_node("worker3") == ["worker"]
+ assert c.has_service("accumulomaster")
+ assert not c.has_service("fluo")
+ assert c.get_service_hostnames("worker") == [
+ "worker1",
+ "worker2",
+ "worker3",
+ "worker4",
+ ]
+ assert c.get_service_hostnames("zookeeper") == ["leader1"]
+ assert c.get_hosts() == {
+ "leader2": ("10.0.0.1", None),
+ "leader1": ("10.0.0.0", "23.0.0.0"),
+ "worker1": ("10.0.0.2", None),
+ "worker3": ("10.0.0.4", None),
+ "worker2": ("10.0.0.3", None),
+ "worker4": ("10.0.0.5", None),
+ }
+ assert c.get_public_ip("leader1") == "23.0.0.0"
+ assert c.get_private_ip("leader1") == "10.0.0.0"
+ assert c.cluster_name == "mycluster"
+ assert c.get_cluster_type() == "ec2"
+ assert c.version("accumulo").startswith("2.")
+ assert c.version("fluo").startswith("1.")
+ assert c.version("hadoop").startswith("3.")
+ assert c.version("zookeeper").startswith("3.")
+ assert c.get_service_private_ips("worker") == [
+ "10.0.0.2",
+ "10.0.0.3",
+ "10.0.0.4",
+ "10.0.0.5",
+ ]
+ assert c.get("general", "proxy_hostname") == "leader1"
assert c.proxy_public_ip() == "23.0.0.0"
assert c.proxy_private_ip() == "10.0.0.0"
- assert c.get('general', 'cluster_user') == "centos"
- assert c.get('general', 'cluster_group') == "centos"
- assert c.get_non_proxy() == [('10.0.0.1', 'leader2'), ('10.0.0.2', 'worker1'), ('10.0.0.3', 'worker2'),
- ('10.0.0.4', 'worker3'), ('10.0.0.5', 'worker4')]
- assert c.get_host_services() == [('leader1', 'namenode resourcemanager accumulomaster zookeeper'),
- ('leader2', 'metrics'),
- ('worker1', 'worker swarmmanager'),
- ('worker2', 'worker'),
- ('worker3', 'worker'),
- ('worker4', 'worker')]
+ assert c.get("general", "cluster_user") == "centos"
+ assert c.get("general", "cluster_group") == "centos"
+ assert c.get_non_proxy() == [
+ ("10.0.0.1", "leader2"),
+ ("10.0.0.2", "worker1"),
+ ("10.0.0.3", "worker2"),
+ ("10.0.0.4", "worker3"),
+ ("10.0.0.5", "worker4"),
+ ]
+ assert c.get_host_services() == [
+ ("leader1", "namenode resourcemanager accumulomaster zookeeper"),
+ ("leader2", "metrics"),
+ ("worker1", "worker swarmmanager"),
+ ("worker2", "worker"),
+ ("worker3", "worker"),
+ ("worker4", "worker"),
+ ]
def test_case_sensitive():
- c = Ec2DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', '../conf/templates', 'mycluster')
- assert c.has_option('ec2', 'default_instance_type') == True
- assert c.has_option('ec2', 'Default_instance_type') == False
- c.set('nodes', 'CamelCaseWorker', 'worker,fluo')
+ c = Ec2DeployConfig(
+ "muchos",
+ "../conf/muchos.props.example",
+ "../conf/hosts/example/example_cluster",
+ "../conf/checksums",
+ "../conf/templates",
+ "mycluster",
+ )
+ assert c.has_option("ec2", "default_instance_type")
+ assert not c.has_option("ec2", "Default_instance_type")
+ c.set("nodes", "CamelCaseWorker", "worker,fluo")
c.init_nodes()
- assert c.get_node('CamelCaseWorker') == ['worker', 'fluo']
+ assert c.get_node("CamelCaseWorker") == ["worker", "fluo"]
def test_ec2_cluster_template():
- c = Ec2DeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', '../conf/templates', 'mycluster')
+ c = Ec2DeployConfig(
+ "muchos",
+ "../conf/muchos.props.example",
+ "../conf/hosts/example/example_cluster",
+ "../conf/checksums",
+ "../conf/templates",
+ "mycluster",
+ )
- c.set('ec2', 'cluster_template', 'example')
- c.init_template('../conf/templates')
+ c.set("ec2", "cluster_template", "example")
+ c.init_template("../conf/templates")
# init_template already calls validate_template, so just ensure that
# we've loaded all the expected dictionary items from the example
- assert 'accumulomaster' in c.cluster_template_d
- assert 'client' in c.cluster_template_d
- assert 'metrics' in c.cluster_template_d
- assert 'namenode' in c.cluster_template_d
- assert 'resourcemanager' in c.cluster_template_d
- assert 'worker' in c.cluster_template_d
- assert 'zookeeper' in c.cluster_template_d
- assert 'devices' in c.cluster_template_d
+ assert "accumulomaster" in c.cluster_template_d
+ assert "client" in c.cluster_template_d
+ assert "metrics" in c.cluster_template_d
+ assert "namenode" in c.cluster_template_d
+ assert "resourcemanager" in c.cluster_template_d
+ assert "worker" in c.cluster_template_d
+ assert "zookeeper" in c.cluster_template_d
+ assert "devices" in c.cluster_template_d
diff --git a/lib/tests/existing/__init__.py b/lib/tests/existing/__init__.py
index 555eccf..cce3aca 100644
--- a/lib/tests/existing/__init__.py
+++ b/lib/tests/existing/__init__.py
@@ -14,4 +14,3 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
diff --git a/lib/tests/existing/test_config.py b/lib/tests/existing/test_config.py
index ef9a29a..1be2a86 100644
--- a/lib/tests/existing/test_config.py
+++ b/lib/tests/existing/test_config.py
@@ -19,13 +19,19 @@
def test_existing_cluster():
- c = ExistingDeployConfig("muchos", '../conf/muchos.props.example', '../conf/hosts/example/example_cluster',
- '../conf/checksums', '../conf/templates', 'mycluster')
- c.cluster_type = 'existing'
- assert c.get_cluster_type() == 'existing'
+ c = ExistingDeployConfig(
+ "muchos",
+ "../conf/muchos.props.example",
+ "../conf/hosts/example/example_cluster",
+ "../conf/checksums",
+ "../conf/templates",
+ "mycluster",
+ )
+ c.cluster_type = "existing"
+ assert c.get_cluster_type() == "existing"
assert c.node_type_map() == {}
- assert c.mount_root() == '/var/data'
- assert c.worker_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
- assert c.default_data_dirs() == ['/var/data1', '/var/data2', '/var/data3']
- assert c.metrics_drive_ids() == ['var-data1', 'var-data2', 'var-data3']
- assert c.shutdown_delay_minutes() == '0'
\ No newline at end of file
+ assert c.mount_root() == "/var/data"
+ assert c.worker_data_dirs() == ["/var/data1", "/var/data2", "/var/data3"]
+ assert c.default_data_dirs() == ["/var/data1", "/var/data2", "/var/data3"]
+ assert c.metrics_drive_ids() == ["var-data1", "var-data2", "var-data3"]
+ assert c.shutdown_delay_minutes() == "0"
diff --git a/lib/tests/test_decorators.py b/lib/tests/test_decorators.py
index 95a17cb..06b902e 100644
--- a/lib/tests/test_decorators.py
+++ b/lib/tests/test_decorators.py
@@ -24,35 +24,35 @@
@property
@decorators.ansible_host_var
def host_var1(self):
- return 'host_var1'
+ return "host_var1"
@property
- @decorators.ansible_host_var(name='named_host_var2')
+ @decorators.ansible_host_var(name="named_host_var2")
def host_var2(self):
- return 'named_host_var2'
+ return "named_host_var2"
@property
@decorators.ansible_play_var
def play_var1(self):
- return 'play_var1'
+ return "play_var1"
@property
- @decorators.ansible_play_var(name='named_play_var2')
+ @decorators.ansible_play_var(name="named_play_var2")
def play_var2(self):
- return 'named_play_var2'
+ return "named_play_var2"
@property
@decorators.ansible_extra_var
def extra_var1(self):
- return 'extra_var1'
+ return "extra_var1"
@property
- @decorators.ansible_extra_var(name='named_extra_var2')
+ @decorators.ansible_extra_var(name="named_extra_var2")
def extra_var2(self):
- return 'named_extra_var2'
+ return "named_extra_var2"
@property
- @decorators.default('default_val')
+ @decorators.default("default_val")
def default_val(self):
return None
@@ -74,7 +74,7 @@
@property
@decorators.required
def required_val(self):
- return 'required_val'
+ return "required_val"
@property
@decorators.required
@@ -89,44 +89,74 @@
def test_decorators(self):
thing = DecoratedThing()
- actual_host_vars = decorators.get_ansible_vars('host', type(thing))
- actual_play_vars = decorators.get_ansible_vars('play', type(thing))
- actual_extra_vars = decorators.get_ansible_vars('extra', type(thing))
+ actual_host_vars = decorators.get_ansible_vars("host", type(thing))
+ actual_play_vars = decorators.get_ansible_vars("play", type(thing))
+ actual_extra_vars = decorators.get_ansible_vars("extra", type(thing))
expected_host_vars = [
- decorators._ansible_var('host_var1', 'DecoratedThing', 'host_var1', 'tests.test_decorators'),
- decorators._ansible_var('named_host_var2', 'DecoratedThing', 'host_var2', 'tests.test_decorators')
+ decorators._ansible_var(
+ "host_var1",
+ "DecoratedThing",
+ "host_var1",
+ "tests.test_decorators",
+ ),
+ decorators._ansible_var(
+ "named_host_var2",
+ "DecoratedThing",
+ "host_var2",
+ "tests.test_decorators",
+ ),
]
expected_play_vars = [
- decorators._ansible_var('play_var1', 'DecoratedThing', 'play_var1', 'tests.test_decorators'),
- decorators._ansible_var('named_play_var2', 'DecoratedThing', 'play_var2', 'tests.test_decorators')
+ decorators._ansible_var(
+ "play_var1",
+ "DecoratedThing",
+ "play_var1",
+ "tests.test_decorators",
+ ),
+ decorators._ansible_var(
+ "named_play_var2",
+ "DecoratedThing",
+ "play_var2",
+ "tests.test_decorators",
+ ),
]
expected_extra_vars = [
- decorators._ansible_var('extra_var1', 'DecoratedThing', 'extra_var1', 'tests.test_decorators'),
- decorators._ansible_var('named_extra_var2', 'DecoratedThing', 'extra_var2', 'tests.test_decorators')
+ decorators._ansible_var(
+ "extra_var1",
+ "DecoratedThing",
+ "extra_var1",
+ "tests.test_decorators",
+ ),
+ decorators._ansible_var(
+ "named_extra_var2",
+ "DecoratedThing",
+ "extra_var2",
+ "tests.test_decorators",
+ ),
]
self.assertEquals(
set([str(v) for v in expected_host_vars]),
- set([str(v) for v in actual_host_vars])
+ set([str(v) for v in actual_host_vars]),
)
self.assertEquals(
set([str(v) for v in expected_play_vars]),
- set([str(v) for v in actual_play_vars])
+ set([str(v) for v in actual_play_vars]),
)
self.assertEquals(
set([str(v) for v in expected_extra_vars]),
- set([str(v) for v in actual_extra_vars])
+ set([str(v) for v in actual_extra_vars]),
)
- self.assertEquals(thing.default_val, 'default_val')
+ self.assertEquals(thing.default_val, "default_val")
self.assertEquals(thing.default_boolean_val_True, True)
self.assertEquals(thing.default_boolean_val_False, False)
self.assertEquals(thing.default_missing_boolean_val, True)
- self.assertEquals(thing.required_val, 'required_val')
+ self.assertEquals(thing.required_val, "required_val")
with self.assertRaises(decorators.ConfigMissingError):
- thing.missing_required_val
\ No newline at end of file
+ thing.missing_required_val
diff --git a/lib/tests/test_util.py b/lib/tests/test_util.py
index 3a7cb55..3225a63 100644
--- a/lib/tests/test_util.py
+++ b/lib/tests/test_util.py
@@ -19,21 +19,25 @@
def test_util():
- assert set(get_ephemeral_devices('m3.large')) == set(['/dev/xvdb'])
- assert set(get_ephemeral_devices('m3.xlarge')) == set(['/dev/xvdb', '/dev/xvdc'])
+ assert set(get_ephemeral_devices("m3.large")) == set(["/dev/xvdb"])
+ assert set(get_ephemeral_devices("m3.xlarge")) == set(
+ ["/dev/xvdb", "/dev/xvdc"]
+ )
- assert set(get_ephemeral_devices('i3.xlarge')) == set(['/dev/nvme0n1'])
- assert set(get_ephemeral_devices('i3.4xlarge')) == set(['/dev/nvme0n1','/dev/nvme1n1'])
+ assert set(get_ephemeral_devices("i3.xlarge")) == set(["/dev/nvme0n1"])
+ assert set(get_ephemeral_devices("i3.4xlarge")) == set(
+ ["/dev/nvme0n1", "/dev/nvme1n1"]
+ )
- assert get_arch('m1.large') == 'pvm'
- assert get_arch('m3.large') == 'hvm'
+ assert get_arch("m1.large") == "pvm"
+ assert get_arch("m3.large") == "hvm"
- hosts_dir = '../conf/hosts'
- assert parse_args(hosts_dir, ['launch']) is None
- assert parse_args(hosts_dir, ['launch', 'mycluster']) is None
- assert parse_args(hosts_dir, ['-c', 'mycluster', 'launch']) is not None
+ hosts_dir = "../conf/hosts"
+ assert parse_args(hosts_dir, ["launch"]) is None
+ assert parse_args(hosts_dir, ["launch", "mycluster"]) is None
+ assert parse_args(hosts_dir, ["-c", "mycluster", "launch"]) is not None
- hosts_dir = '../conf/hosts/example'
- assert parse_args(hosts_dir, ['setup']) is not None
- assert parse_args(hosts_dir, ['config']) is None
- assert parse_args(hosts_dir, ['-p', 'all', 'config']) is not None
+ hosts_dir = "../conf/hosts/example"
+ assert parse_args(hosts_dir, ["setup"]) is not None
+ assert parse_args(hosts_dir, ["config"]) is None
+ assert parse_args(hosts_dir, ["-p", "all", "config"]) is not None
diff --git a/lib/tests/test_validators.py b/lib/tests/test_validators.py
index 0785717..ad3e843 100644
--- a/lib/tests/test_validators.py
+++ b/lib/tests/test_validators.py
@@ -70,7 +70,7 @@
@property
@is_valid(validators.contains(5))
def containsFive(self):
- return [4,5,6]
+ return [4, 5, 6]
@property
@is_valid(validators.contains(5))
@@ -100,7 +100,7 @@
@property
@is_valid(validators.is_type(str))
def stringIsString(self):
- return 'some string'
+ return "some string"
class ValidationTests(TestCase):
@@ -131,7 +131,7 @@
with self.assertRaises(Exception):
thing.sixeNotEqualFive
- self.assertEqual(thing.containsFive, [4,5,6])
+ self.assertEqual(thing.containsFive, [4, 5, 6])
with self.assertRaises(Exception):
thing.notContainsFive
@@ -147,4 +147,4 @@
with self.assertRaises(Exception):
thing.intIsNotString
- self.assertEqual(thing.stringIsString, 'some string')
\ No newline at end of file
+ self.assertEqual(thing.stringIsString, "some string")
diff --git a/scripts/checkMissingEOF b/scripts/checkMissingEOF
new file mode 100755
index 0000000..bf479cd
--- /dev/null
+++ b/scripts/checkMissingEOF
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+if grep --quiet --files-with-matches --binary-files=without-match --max-count=1 --regexp='.*' "$1"; then
+ if [ "$(tail --bytes=1 "$1")" ]; then
+ echo "ERROR: No new line at end of $1."; false;
+ fi
+fi
diff --git a/scripts/cibuild b/scripts/cibuild
new file mode 100755
index 0000000..b78d3e9
--- /dev/null
+++ b/scripts/cibuild
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -e
+echo "Running nose tests..."
+nosetests -w lib/
+echo "SUCCESS: All Nose tests completed."
+echo "Running Ansible-lint..."
+ansible-lint -x 204,301
+echo "SUCCESS: Ansible-lint completed."
+echo "Running flake8 to check Python code..."
+flake8
+echo "SUCCESS: flake8 completed."
+echo "Checking for files which have no newline at end of file..."
+git ls-files | tr '\n' '\0' | xargs -0 -L1 -n1 "$(dirname "$0")/checkMissingEOF"
+echo "SUCCESS: All files have newline at the end."
+echo "All tests passed!"
