METRON-2321 Remove Legacy AWS Deployment Path (nickwallen) closes apache/metron#1565
diff --git a/Upgrading.md b/Upgrading.md
index 66350c3..d5a848b 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -22,6 +22,9 @@
## 0.7.2 to 0.7.3
+### [METRON-2321: Remove Legacy AWS Deployment Path](https://issues.apache.org/jira/browse/METRON-2321)
+The automated Amazon AWS deployment mechanism (previously located at `metron-deployment/amazon-ec2`) has been removed. It is not the preferred installation path for deploying to AWS. Using Ambari and the Metron MPack is the preferred installation path. To deploy Metron to AWS, provision EC2 nodes, install Ambari, install the Metron MPack, then use Ambari to deploy Metron.
+
### [METRON-614: Eliminate use of the default Charset](https://issues.apache.org/jira/browse/METRON-614)
The use of the system default Charset is being dropped throughout the code in favor or explicit Charsets, by default UTF-8. As part of this change, individual parsers may now set a configuration property `readCharset` to allow each parser to set the Charset used by its data. This has potential upgrade implications:
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 73cf602..33e46a0 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -22,7 +22,6 @@
* [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm)
* [How do I build RPM packages?](#how-do-i-build-rpm-packages)
* [How do I build DEB packages?](#how-do-i-build-deb-packages)
- * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws)
* [How do I build Metron with Docker?](#how-do-i-build-metron-with-docker)
@@ -149,30 +148,6 @@
To build the DEB packages, follow the instructions at [packaging/docker/deb-docker](packaging/docker/deb-docker).
-How do I deploy Metron within AWS?
-----------------------------------
-
-This deploys Apache Metron on an automatically provisioned 10-node cluster running in Amazon Web Service's EC2 platform.
-
-This installs real sources of telemetry like Bro, Snort, and YAF, but feeds those sensors with canned pcap data.
-
-#### What is this good for?
-
-* If you are a Metron developer wanting to test at-scale on a multi-node cluster, then this is the right option for you.
-
-#### What is this **not** good for?
-
-* If you want to run Metron in AWS with real data for either testing or production, then this is NOT the right option for you.
-
-* **WARNING** This is only intended for creating an ephemeral cluster for brief periods of testing. This deployment method has the following severe limitations.
- * The cluster is not secured in any way. It is up to you to manually secure it.
- * The cluster will not survive a reboot.
-
-#### How?
-
-Follow the instructions available at [amazon-ec2](amazon-ec2).
-
-
How do I build Metron with Docker?
----------------------------------
diff --git a/metron-deployment/amazon-ec2/.gitignore b/metron-deployment/amazon-ec2/.gitignore
deleted file mode 100644
index 34c24d2..0000000
--- a/metron-deployment/amazon-ec2/.gitignore
+++ /dev/null
@@ -1,5 +0,0 @@
-*.pem
-*.secret
-*.log
-*.retry
-.metron-env
diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md
deleted file mode 100644
index ed1c73b..0000000
--- a/metron-deployment/amazon-ec2/README.md
+++ /dev/null
@@ -1,322 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-Apache Metron on Amazon EC2
-===========================
-
-This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.
-
-Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges.
-
-AWS Defaults
-------------
-
-The Ansible playbook uses the following defaults for AWS deployment:
-
-- Instances: 10
-- Region: us-west-2
-- Instance type: m4.xlarge
-
-Getting Started
----------------
-
-### Prerequisites
-
-The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.
-
- - Ansible 2.4.0+
- - Python 2.7.11
- - Maven 3.3.9
-
-Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
-
-1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions.
-
- ```
- /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
- ```
-
-2. With Homebrew installed, run the following command in a terminal to install all of the required tools and dependencies.
-
- ```
- brew update
- brew tap caskroom/versions
- brew cask install java8 vagrant virtualbox
- brew install maven git node
- ```
-
-3. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
-
-4. Ensure that a public SSH key is located at `~/.ssh/id_rsa.pub`.
-
- ```
- $ cat ~/.ssh/id_rsa.pub
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost
- ```
-
- If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.
-
- ```
- ssh-keygen -t rsa
- ```
-
-5. Ensure the JAVA_HOME environment variable is set
-
- ```
- export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home"
- ```
-
- Notice: You must replace the path with the installed JDK version path
-
-### Amazon Web Services
-
-If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.
-
-1. Head over to [Amazon Web Services](http://aws.amazon.com/) and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.
-
-2. Create a set of user credentials through [Amazon's Identity and Access Management (IAM) ](https://console.aws.amazon.com/iam/) dashboard. On the IAM dashboard menu click "Users" and then "Create New User". Provide a name and ensure that "Generate an access key for each user" remains checked. Download the credentials and keep them for later use.
-
-3. While still in [Amazon's Identity and Access Management (IAM) ](https://console.aws.amazon.com/iam/) dashboard, click on the user that was previously created. Click the "Permissions" tab and then the "Attach Policy" button. Attach the following policies to the user.
-
- - AmazonEC2FullAccess
- - AmazonVPCFullAccess
-
-4. Apache Metron uses the [official, open source CentOS 6](https://aws.amazon.com/marketplace/pp/B00NQAYLWO) Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon's terms and conditions. Navigate to the [web page for this AMI](https://aws.amazon.com/marketplace/pp/B00NQAYLWO) and click the "Continue" button. Choose the "Manual Launch" tab then click the "Accept Software Terms" button.
-
-Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us.
-
-### Deploy Metron
-
-1. Use the Amazon access key by exporting its values via the shell's environment. This allows Ansible to authenticate with Amazon EC2. For example:
-
- ```
- export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
- export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
- ```
-
- Notice: You must replace the access key values above with values from your own access key.
-
-2. Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default.
-
- ```
- $ ./run.sh
- Metron Environment [metron-test]: my-metron-env
- ...
- ```
-
- The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.
-
-### Explore Metron
-
-1. After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.
-
- ```
- TASK [debug] *******************************************************************
- ok: [localhost] => {
- "Success": [
- "Apache Metron deployed successfully",
- " Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
- " Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
- " Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
- "For additional information, see https://metron.apache.org/'"
- ]
- }
- ```
-
-2. Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user `centos` will not require a password as it will authenticate with the pre-defined SSH key.
-
- ```
- ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
- ```
-
-Advanced Usage
---------------
-
-### Multiple Environments
-
-This process can support provisioning of multiple, isolated environments. Simply change the `env` settings in `conf/defaults.yml`. For example, you might provision separate development, test, and production environments.
-
-```
-env: metron-test
-```
-
-### Selective Provisioning
-
-To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command:
-
-```
-./run.sh --tags="ec2,sensors"
-```
-
-### Setting REST API Profile
-
-By default, EC2 is deployed with the `dev` Spring profile. This simply instructs the REST API to automatically setup default test users `[user, user1, user2, admin]` with password "`password`" as opposed to requiring the user to manually
- create users in the REST database. You can change this default functionality by removing the `metron_spring_profiles_active` setting in `metron-deployment/roles/ambari_config/vars/small_cluster.yml`.
- You can view more detail on the REST Spring profiles [here](../../metron-interface/metron-rest/README.md#spring-profiles).
-
-### Custom SSH Key
-
-
-By default, the playbook will attempt to register your public SSH key `~/.ssh/id_rsa.pub` with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the `key_file` property in `conf/defaults.yml`.
-
-For example, generate a new SSH key for Metron that will be stored at `~/.ssh/my-metron-key`.
-
-```
-$ ssh-keygen -q -f ~/.ssh/my-metron-key
-Enter passphrase (empty for no passphrase):
-Enter same passphrase again:
-```
-
-Add the path to the newly created SSH public key to `conf/defaults.yml`.
-
-```
-key_file: ~/.ssh/metron-private-key.pub
-```
-
-Common Errors
--------------
-
-### Error: [unsupported_operation_exception] custom format isn't supported
-
-This error might be seen within Metron's default dashboard in Kibana 4. This occurs when the index templates do not exist for the Snort, Bro or YAF indices in Elasticsearch.
-
-The dashboard expects fields to be of a certain type. If the index templates have not been loaded correctly, the data types for the fields in these indices will be incorrect and the dashboard will display this error.
-
-#### Solution
-
-If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.
-
-1. Define which Elasticsearch host to interact with. Any Elasticsearch host should work.
- ```
- export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
- ```
-
-1. Confirm the index templates are in fact missing.
- ```
- curl -s -XGET $ES_HOST/_template
- ```
-
-1. Manually load the index templates.
- ```
- cd metron-deployment
- curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
- curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
- curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
- ```
-
-1. Delete the existing indexes. Only a new index will use the templates defined in the previous step.
- ```
- curl -s -XDELETE "$ES_HOST/yaf_index*"
- curl -s -XDELETE "$ES_HOST/bro_index*"
- curl -s -XDELETE "$ES_HOST/snort_index*"
- ```
-
-1. Open up Kibana and wait for the new indexes to be created. The dashboard should now work.
-
-### Error: 'No handler was ready to authenticate...Check your credentials'
-
-```
-TASK [Define keypair] **********************************************************
-failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
- r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
- sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
- "msg": "No handler was ready to authenticate. 1 handlers were checked.
- ['HmacAuthV4Handler'] Check your credentials"}
-```
-
-#### Solution 1
-
-This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon's [Identity and Access Management](https://console.aws.amazon.com/iam/) console.
-
-```
-$ echo $AWS_ACCESS_KEY_ID
-AKIAI6NRFEO27E5FFELQ
-
-$ echo $AWS_SECRET_ACCESS_KEY
-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
-```
-
-#### Solution 2
-
-This error can occur if you have exported the correct AWS access key, but you are using `sudo` to run the Ansible playbook. Do not use the `sudo` command when running the Ansible playbook.
-
-### Error: 'OptInRequired: ... you need to accept terms and subscribe'
-
-```
-TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
-product you need to accept terms and subscribe. To do so please visit
-http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-Apache Metron uses the [official CentOS 6 Amazon Machine Image](https://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2) when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook.
-
-### Error: 'PendingVerification: Your account is currently being verified'
-
-```
-TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => PendingVerification: Your account is currently
-being verified. Verification normally takes less than 2 hours. Until your
-account is verified, you may not be able to launch additional instances or
-create additional volumes. If you are still receiving this message after more
-than 2 hours, please let us know by writing to aws-verification@amazon.com. We
-appreciate your patience."}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon's verification process is complete. Amazon has some additional [advice for dealing with this error and more](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html).
-
-> Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact [AWS Support](http://console.aws.amazon.com/support/home#/).
-
-### Error: 'Instance creation failed => InstanceLimitExceeded'
-
-```
-TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => InstanceLimitExceeded: You have requested more
-instances (11) than your current instance limit of 10 allows for the specified
-instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
-request an adjustment to this limit."}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing `deployment/amazon-ec/playbook.yml`. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional [advice for dealing with this error and more](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html).
-
-> You've reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see [How many instances can I run in Amazon EC2](http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2). If you need additional instances, complete the [Amazon EC2 Instance Request Form](https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-ec2-instances).
-
-### Error: 'SSH encountered an unknown error during the connection'
-
-```
-TASK [setup] *******************************************************************
-fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
- "changed": false, "msg": "SSH encountered an unknown error during the
- connection. We recommend you re-run the command using -vvvv, which will enable
- SSH debugging output to help diagnose the issue", "unreachable": true}
-```
-
-#### Solution
-
-This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the `key_file` variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.
diff --git a/metron-deployment/amazon-ec2/ansible.cfg b/metron-deployment/amazon-ec2/ansible.cfg
deleted file mode 100644
index 4291cc8..0000000
--- a/metron-deployment/amazon-ec2/ansible.cfg
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = False
-library = ../ansible/extra_modules
-roles_path = ../ansible/roles
-pipelining = True
-remote_user = centos
-forks = 20
-log_path = ./ansible.log
-callback_plugins = ../ansible/callback_plugins
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-#[ssh_connection]
-#control_path = ~/.ssh/ansible-ssh-%%C
-
-[ssh_connection]
-control_path = ~/.ssh/ansbile-ssh-%%h-%%r
diff --git a/metron-deployment/amazon-ec2/conf/defaults.yml b/metron-deployment/amazon-ec2/conf/defaults.yml
deleted file mode 100644
index b23ea8b..0000000
--- a/metron-deployment/amazon-ec2/conf/defaults.yml
+++ /dev/null
@@ -1,97 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-# the ami for cent6 by region
-amis_by_region:
- us-east-1: ami-1c221e76 # US East (N. Virginia)
- us-west-2: ami-05cf2265 # US West (Oregon)
- us-west-1: ami-ac5f2fcc # US West (N. California)
- eu-central-1: ami-2bf11444 # EU (Frankfurt)
- eu-west-1: ami-edb9069e # EU (Ireland)
- ap-south-1: ami-9b1c76f4 # Asia Pacific (Mumbai)
- ap-southeast-1: ami-106aa373 # Asia Pacific (Singapore)
- ap-southeast-2: ami-87d2f4e4 # Asia Pacific (Sydney)
- ap-northeast-1: ami-fa3d3f94 # Asia Pacific (Tokyo)
- ap-northeast-2: ami-56478938 # Asia Pacific (Seoul)
- sa-east-1: ami-03b93b6f # South America (Sao Paulo)
-
-# ec2
-env: metron-test
-region: us-west-2
-instance_type: m4.xlarge
-volume_type: standard
-key_name: metron-key
-xvda_vol_size: 50
-xvdb_vol_size: 100
-xvdc_vol_size: 100
-image: "{{ amis_by_region[region] }}"
-
-# ambari
-ambari_host: "{{ groups.ambari_master[0] }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: small_cluster
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# kafka
-num_partitions: 3
-retention_in_gb: 25
-
-# metron variables
-metron_version: 0.7.2
-metron_directory: /usr/metron/{{ metron_version }}
-
-# sensors
-install_snort: True
-install_pycapa: True
-install_bro: True
-install_yaf: True
-install_pcap_replay: True
-install_tap: True
-sensor_test_mode: True
-pcap_replay_interface: tap0
-pcap_replay_home: /opt/pcap-replay
-sniff_interface: tap0
-
-# data directories
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data2/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-# search
-install_elasticsearch: True
-install_solr: False
-elasticsearch_transport_port: 9300
-## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores.
-elasticsearch_network_interface: _eth0_
-elasticsearch_web_port: 9200
diff --git a/metron-deployment/amazon-ec2/conf/ec2.ini b/metron-deployment/amazon-ec2/conf/ec2.ini
deleted file mode 100755
index 646ffaf..0000000
--- a/metron-deployment/amazon-ec2/conf/ec2.ini
+++ /dev/null
@@ -1,105 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Ansible EC2 external inventory script settings.
-#
-# Refer to https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.ini
-# for additional information on available settings
-#
-
-[ec2]
-
-# AWS regions to make calls to. Set this to 'all' to make request to all regions
-# in AWS and merge the results together. Alternatively, set this to a comma
-# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'
-regions = all
-
-# When generating inventory, Ansible needs to know how to address a server.
-# Each EC2 instance has a lot of variables associated with it. Here is the list:
-# http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance
-# Below are 2 variables that are used as the address of a server:
-# - destination_variable
-# - vpc_destination_variable
-
-# This is the normal destination variable to use. If you are running Ansible
-# from outside EC2, then 'public_dns_name' makes the most sense. If you are
-# running Ansible from within EC2, then perhaps you want to use the internal
-# address, and should set this to 'private_dns_name'. The key of an EC2 tag
-# may optionally be used; however the boto instance variables hold precedence
-# in the event of a collision.
-destination_variable = public_dns_name
-
-# For server inside a VPC, using DNS names may not make sense. When an instance
-# has 'subnet_id' set, this variable is used. If the subnet is public, setting
-# this to 'ip_address' will return the public IP address. For instances in a
-# private subnet, this should be set to 'private_ip_address', and Ansible must
-# be run from within EC2. The key of an EC2 tag may optionally be used; however
-# the boto instance variables hold precedence in the event of a collision.
-# WARNING: - instances that are in the private vpc, _without_ public ip address
-# will not be listed in the inventory until You set:
-# vpc_destination_variable = private_ip_address
-#vpc_destination_variable = ip_address
-vpc_destination_variable = public_dns_name
-
-# To exclude RDS instances from the inventory, uncomment and set to False.
-#rds = False
-
-# To exclude ElastiCache instances from the inventory, uncomment and set to False.
-#elasticache = False
-
-# API calls to EC2 are slow. For this reason, we cache the results of an API
-# call. Set this to the path you want cache files to be written to. Two files
-# will be written to this directory:
-# - ansible-ec2.cache
-# - ansible-ec2.index
-cache_path = ~/.ansible/tmp
-
-# The number of seconds a cache file is considered valid. After this many
-# seconds, a new API call will be made, and the cache file will be updated.
-# To disable the cache, set this value to 0
-cache_max_age = 300
-
-# Organize groups into a nested/hierarchy instead of a flat namespace.
-nested_groups = False
-
-# Replace - tags when creating groups to avoid issues with ansible
-replace_dash_in_groups = True
-
-# If set to true, any tag of the form "a,b,c" is expanded into a list
-# and the results are used to create additional tag_* inventory groups.
-expand_csv_tags = False
-
-# The EC2 inventory output can become very large. To manage its size,
-# configure which groups should be created.
-group_by_instance_id = True
-group_by_region = True
-group_by_availability_zone = True
-group_by_ami_id = True
-group_by_instance_type = True
-group_by_key_pair = True
-group_by_vpc_id = True
-group_by_security_group = True
-group_by_tag_keys = True
-group_by_tag_none = True
-group_by_route53_names = True
-group_by_rds_engine = True
-group_by_rds_parameter_group = True
-group_by_elasticache_engine = True
-group_by_elasticache_cluster = True
-group_by_elasticache_parameter_group = True
-group_by_elasticache_replication_group = True
diff --git a/metron-deployment/amazon-ec2/playbook.yml b/metron-deployment/amazon-ec2/playbook.yml
deleted file mode 100644
index 4e8880a..0000000
--- a/metron-deployment/amazon-ec2/playbook.yml
+++ /dev/null
@@ -1,70 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-#
-# instantiate the hosts on amazon ec2
-#
-- hosts: localhost
- vars_files:
- - conf/defaults.yml
- tasks:
- - include: tasks/create-keypair.yml
- - include: tasks/create-vpc.yml
- - include: tasks/create-open-inbound-security-group.yml
- - include: tasks/create-open-outbound-security-group.yml
- - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,ec2,monit
- - include: tasks/create-hosts.yml host_count=4 host_type=ambari_slave,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=monit,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- - include: tasks/create-hosts.yml host_count=2 host_type=ambari_slave,search,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,web,ec2
- tags:
- - ec2
-
-#
-# wait for all ec2 hosts to come up
-#
-- hosts: ec2
- become: True
- vars_files:
- - conf/defaults.yml
- gather_facts: False
- tasks:
- - include: tasks/check-hosts.yml
- tags:
- - ec2
- - wait
-
-#
-# mount additional data volumes on all ec2 hosts
-#
-- hosts: ec2
- become: True
- vars_files:
- - conf/defaults.yml
- tasks:
- - include: tasks/mount-volume.yml vol_src=/dev/xvdb vol_mnt=/data1
- - include: tasks/mount-volume.yml vol_src=/dev/xvdc vol_mnt=/data2
- - include: tasks/check-volume.yml vol_name=xvda vol_src=/dev/xvda vol_size={{ xvda_vol_size }}
- tags:
- - ec2
- - mount
-
-#
-# build the metron cluster
-#
-- include: ../ansible/playbooks/metron_full_install.yml
diff --git a/metron-deployment/amazon-ec2/run.sh b/metron-deployment/amazon-ec2/run.sh
deleted file mode 100755
index 96c07af..0000000
--- a/metron-deployment/amazon-ec2/run.sh
+++ /dev/null
@@ -1,67 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Builds Metron platform jars, instantiates hosts, and deploys Metron to those
-# hosts on Amazon EC2
-#
-LOGFILE="./ansible.log"
-DEPLOYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-EXTRA_ARGS="-v ${@:1}"
-NOW=`date`
-DEFAULT_ENV="metron-test"
-DEFAULT_ENV_FILE="./.metron-env"
-
-# ensure aws access key is defined
-if [ -z "$AWS_ACCESS_KEY_ID" ]; then
- echo "Error: missing AWS_ACCESS_KEY_ID"
- exit 1
-fi
-
-# ensure aws access key is defined
-if [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
- echo "Error: missing AWS_SECRET_ACCESS_KEY"
- exit 1
-fi
-
-# retrieve environment name from previous run
-if [ -f $DEFAULT_ENV_FILE ]; then
- ENV=`cat $DEFAULT_ENV_FILE`
-else
- ENV=$DEFAULT_ENV
-fi
-
-# prompt the user for an environment name
-read -p "Metron Environment [$ENV]: " INPUT
-[ -n "$INPUT" ] && ENV=$INPUT
-
-# store the environment name for the next run
-echo "$ENV" > $DEFAULT_ENV_FILE
-
-# log information about the host platform
-echo "=============================================================" >> $LOGFILE
-echo "Launching Metron[$ENV] @ $NOW"... >> $LOGFILE
-$DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
-
-# deploy metron
-cd $DEPLOYDIR
-export EC2_INI_PATH=conf/ec2.ini
-ansible-playbook -i ec2.py playbook.yml \
- --skip-tags="sensor-stubs" \
- --extra-vars="env=$ENV" \
- $EXTRA_ARGS
diff --git a/metron-deployment/amazon-ec2/tasks/check-hosts.yml b/metron-deployment/amazon-ec2/tasks/check-hosts.yml
deleted file mode 100644
index a20a319..0000000
--- a/metron-deployment/amazon-ec2/tasks/check-hosts.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Wait for connectivity to host(s)
- local_action: wait_for host={{ inventory_hostname }} state=started port=22 timeout=300 delay=10
- become: False
diff --git a/metron-deployment/amazon-ec2/tasks/check-volume.yml b/metron-deployment/amazon-ec2/tasks/check-volume.yml
deleted file mode 100644
index 5757260..0000000
--- a/metron-deployment/amazon-ec2/tasks/check-volume.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "Check size of volume {{ vol_src }}"
- shell: "lsblk | grep part | grep {{ vol_name }} | awk '{ print $4}' | sed 's/[^0-9]//g'"
- register: current_size
-
-- name: "Status of {{ vol_src }} volume"
- debug: msg="volume={{ vol_src }} current={{ current_size.stdout|int }} expected={{ vol_size|int }}"
-
-- include: expand-volume.yml vol_src_1={{ vol_src }}
- when: current_size.stdout|int < vol_size|int
diff --git a/metron-deployment/amazon-ec2/tasks/create-hosts.yml b/metron-deployment/amazon-ec2/tasks/create-hosts.yml
deleted file mode 100644
index 39bae3a..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-hosts.yml
+++ /dev/null
@@ -1,54 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Instantiate {{ host_count }} host(s) as {{ host_type }}"
- ec2:
- region: "{{ region }}"
- instance_type: "{{ instance_type }}"
- image: "{{ image }}"
- key_name: "{{ env }}-{{ key_name }}"
- assign_public_ip: True
- group: ["{{ env }}-vpc-all-inbound","{{ env }}-vpc-all-outbound"]
- vpc_subnet_id: "{{ vpc.subnets[0].id }}"
- instance_tags:
- Name: "[{{ env }}] {{ host_type }}"
- type: "{{ host_type }}"
- env: "{{ env }}"
- exact_count: "{{ host_count }}"
- count_tag:
- type: "{{ host_type }}"
- env: "{{ env }}"
- volumes:
- - device_name: /dev/sda1
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvda_vol_size }}"
- delete_on_termination: true
- - device_name: /dev/xvdb
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvdb_vol_size }}"
- delete_on_termination: true
- - device_name: /dev/xvdc
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvdc_vol_size }}"
- delete_on_termination: true
- wait: yes
- register: ec2
-
-- name: Add host(s) to a hostgroup
- add_host: hostname={{ item.public_dns_name }} groups={{ host_type }}
- with_items: "{{ ec2.tagged_instances }}"
- when: item.public_dns_name is defined
diff --git a/metron-deployment/amazon-ec2/tasks/create-keypair.yml b/metron-deployment/amazon-ec2/tasks/create-keypair.yml
deleted file mode 100644
index 693039e..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-keypair.yml
+++ /dev/null
@@ -1,29 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- set_fact:
- the_key_file: "{{ key_file | default('~/.ssh/id_rsa.pub') }}"
-
-- name: Define keypair
- ec2_key:
- name: "{{ env }}-{{ key_name }}"
- region: "{{ region }}"
- key_material: "{{ item }}"
- with_file: "{{ the_key_file }}"
-
-- debug: msg="Created keypair '{{ env }}-{{ key_name }}' from '{{ the_key_file }}'"
-
diff --git a/metron-deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml b/metron-deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml
deleted file mode 100644
index 67e89c8..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define open inbound security group"
- ec2_group:
- name: "{{ env }}-vpc-all-inbound"
- description: WARNING allow all inbound connections from the internet
- region: "{{ region }}"
- vpc_id: "{{ vpc_id }}"
- rules:
- - proto: all
- cidr_ip: 0.0.0.0/0
diff --git a/metron-deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml b/metron-deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml
deleted file mode 100644
index 53f505f..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define open outbound security group"
- ec2_group:
- name: "{{ env }}-vpc-all-outbound"
- description: allow all outbound connections to the internet
- region: "{{ region }}"
- vpc_id: "{{ vpc_id }}"
- rules_egress:
- - proto: all
- cidr_ip: 0.0.0.0/0
diff --git a/metron-deployment/amazon-ec2/tasks/create-security-group.yml b/metron-deployment/amazon-ec2/tasks/create-security-group.yml
deleted file mode 100644
index 1c9b909..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-security-group.yml
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define the {{ name }} security group"
- ec2_group:
- name: "{{ env }}-{{ name }}"
- region: "{{ region }}"
- description: "[{{env}}] {{ name }}/{{ proto }}/{{ port }}"
- vpc_id: "{{ vpc_id }}"
- rules:
- - proto: "{{ proto }}"
- from_port: "{{ port }}"
- to_port: "{{ port }}"
- cidr_ip: 0.0.0.0/0
diff --git a/metron-deployment/amazon-ec2/tasks/create-vpc.yml b/metron-deployment/amazon-ec2/tasks/create-vpc.yml
deleted file mode 100644
index 7fc31e7..0000000
--- a/metron-deployment/amazon-ec2/tasks/create-vpc.yml
+++ /dev/null
@@ -1,50 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
- - name: "{{ env }}: Create virtual private cloud"
- ec2_vpc:
- region: "{{ region }}"
- internet_gateway: True
- resource_tags:
- Name: "{{ env }}-virtual-private-cloud"
- env: "{{ env }}"
- cidr_block: 10.0.0.0/16
- dns_hostnames: yes
- dns_support: yes
- subnets:
- - cidr: 10.0.0.0/24
- resource_tags:
- tier: web
- - cidr: 10.0.1.0/24
- resource_tags:
- tier: hdp
- - cidr: 10.0.2.0/24
- resource_tags:
- tier: sensors
- route_tables:
- - subnets:
- - 10.0.0.0/24
- - 10.0.1.0/24
- - 10.0.2.0/24
- routes:
- - dest: 0.0.0.0/0
- gw: igw
- register: vpc
-
- - name: "[{{ env }}] Created vpc with id={{ vpc.vpc_id }}"
- set_fact:
- vpc_id: "{{ vpc.vpc_id }}"
diff --git a/metron-deployment/amazon-ec2/tasks/expand-volume.yml b/metron-deployment/amazon-ec2/tasks/expand-volume.yml
deleted file mode 100644
index 35314df..0000000
--- a/metron-deployment/amazon-ec2/tasks/expand-volume.yml
+++ /dev/null
@@ -1,30 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "Expand {{ vol_src_1 }} volume"
-# sectors delete 1 new primary first past mbr to end bootable write and exit
- shell: "(echo u s; echo d 1; echo n; echo p; echo 1; echo 2048 ; echo ;echo a; echo 1; echo w) | fdisk {{ vol_src_1 }} || true"
- args:
- executable: /bin/bash
-
-- name: Restart host(s)
- command: shutdown -r now "Trigger volume changes"
- async: 0
- poll: 0
- ignore_errors: True
-
-- include: tasks/check-hosts.yml
diff --git a/metron-deployment/amazon-ec2/tasks/mount-volume.yml b/metron-deployment/amazon-ec2/tasks/mount-volume.yml
deleted file mode 100644
index 11259a5..0000000
--- a/metron-deployment/amazon-ec2/tasks/mount-volume.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Install xfsprogs
- yum:
- name: xfsprogs
- state: present
- update_cache: yes
- register: result
- until: result.rc == 0
- retries: 5
- delay: 10
-
-- name: Format data volume(s)
- filesystem: fstype=xfs dev={{ vol_src }}
-
-- name: Mount the volume
- mount: name={{ vol_mnt }} src={{ vol_src }} opts=noatime fstype=xfs state=mounted
diff --git a/metron-deployment/ansible/playbooks/ambari_install.yml b/metron-deployment/ansible/playbooks/ambari_install.yml
index 5b802b3..9c67589 100644
--- a/metron-deployment/ansible/playbooks/ambari_install.yml
+++ b/metron-deployment/ansible/playbooks/ambari_install.yml
@@ -15,13 +15,6 @@
# limitations under the License.
#
---
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
- hosts: packer
become: true
tasks:
diff --git a/metron-deployment/ansible/playbooks/metron_install.yml b/metron-deployment/ansible/playbooks/metron_install.yml
index 56f7a7a..4ed6b6e 100644
--- a/metron-deployment/ansible/playbooks/metron_install.yml
+++ b/metron-deployment/ansible/playbooks/metron_install.yml
@@ -15,13 +15,6 @@
# limitations under the License.
#
---
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
- hosts: packer
become: true
tasks:
diff --git a/metron-deployment/ansible/playbooks/sensor_install.yml b/metron-deployment/ansible/playbooks/sensor_install.yml
index a683ccb..0684a66 100644
--- a/metron-deployment/ansible/playbooks/sensor_install.yml
+++ b/metron-deployment/ansible/playbooks/sensor_install.yml
@@ -15,13 +15,6 @@
# limitations under the License.
#
---
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
- hosts: packer
become: true
tasks:
diff --git a/metron-deployment/ansible/playbooks/solr_install.yml b/metron-deployment/ansible/playbooks/solr_install.yml
index 464ec80..8ae3ba0 100644
--- a/metron-deployment/ansible/playbooks/solr_install.yml
+++ b/metron-deployment/ansible/playbooks/solr_install.yml
@@ -15,13 +15,6 @@
# limitations under the License.
#
---
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
- hosts: packer
become: true
tasks:
diff --git a/metron-deployment/ansible/roles/sensor-stubs/README.md b/metron-deployment/ansible/roles/sensor-stubs/README.md
index 015f1d1..9a1b55f 100644
--- a/metron-deployment/ansible/roles/sensor-stubs/README.md
+++ b/metron-deployment/ansible/roles/sensor-stubs/README.md
@@ -31,7 +31,7 @@
Using the default playbooks, this role can be installed by using the Ansible tag `sensor-stubs`. This service is installed on the same hosts where the sensors would be; defined by the `sensors` host group.
-The defaults for the "Full Dev" environment have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.
+The defaults for the "Full Dev" environment have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves.
### (Q) How do I use the sensor stubs?
diff --git a/metron-deployment/ansible/roles/sensor-test-mode/README.md b/metron-deployment/ansible/roles/sensor-test-mode/README.md
index 37afad8..a5e8f9f 100644
--- a/metron-deployment/ansible/roles/sensor-test-mode/README.md
+++ b/metron-deployment/ansible/roles/sensor-test-mode/README.md
@@ -37,7 +37,7 @@
- role: sensor-test-mode
```
-The role has also been added to the default `metron_install.yml` playbook so that it can be turned on/off with a property in both the local Virtualbox and the remote EC2 deployments.
+The role has also been added to the default `metron_install.yml` playbook so that it can be turned on/off with a property in the development environment.
```
sensor_test_mode: True
