This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.
The host that will drive the provisioning process will need to have Ansible, Python and PIP installed. In most cases, a development laptop serves this purpose just fine. Also, install the Python library boto and its dependencies.
pip install boto six
Ensure that an SSH key has been generated and stored at ~/.ssh/id_rsa.pub. In most cases this key will already exist and no further action will be needed.
Use Amazon's Identity and Access Management tool to create a user account by navigating to Users > Create New User.
Grant the user permission by clicking on Permissions > Attach Policy and add the following policies.
AmazonEC2FullAccess AmazonVPCFullAccess
Create an access key for the user by clicking on Security Credentials > Create Access Key. Save the provided access key values in a safe place. These values cannot be retrieved from the web console at a later time.
Use the access key by exporting its values to the shell's environment. This allows Ansible to authenticate with Amazon EC2. For example:
export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ" export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
cd ../../metron-streaming mvn clean package -DskipTests
export EC2_INI_PATH=conf/ec2.ini ansible-playbook -i ec2.py playbook.yml
TASK [debug] *******************************************************************
ok: [localhost] => {
"Success": [
"Apache Metron deployed successfully",
" Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
" Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
" Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
"For additional information, see https://metron.incubator.apache.org/'"
]
}
centos will not require a password as it will authenticate with the pre-defined SSH key.ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
This process can support provisioning of multiple, isolated environments. Simply change the env settings in conf/defaults.yml. For example, you might provision separate development, test, and production environments.
env: metron-test
To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command.
ansible-playbook -i ec2.py playbook.yml --tags "ec2,sensors"
By default, the playbook will attempt to register your public SSH key ~/.ssh/id_rsa.pub with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the key_file property in conf/defaults.yml.
For example, generate a new SSH key for Metron that will be stored at ~/.ssh/my-metron-key.
$ ssh-keygen -q -f ~/.ssh/my-metron-key Enter passphrase (empty for no passphrase): Enter same passphrase again:
Add the path to the newly created SSH public key to conf/defaults.yml.
key_file: ~/.ssh/metron-private-key.pub
TASK [Define keypair] **********************************************************
failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
"msg": "No handler was ready to authenticate. 1 handlers were checked.
['HmacAuthV4Handler'] Check your credentials"}
This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon's Identity and Access Management console.
$ echo $AWS_ACCESS_KEY_ID AKIAI6NRFEO27E5FFELQ $ echo $AWS_SECRET_ACCESS_KEY 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
This error can occur if you have exported the correct AWS access key, but you are using sudo to run the Ansible playbook. Do not use the sudo command when running the Ansible playbook.
TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
product you need to accept terms and subscribe. To do so please visit
http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
to retry, use: --limit @playbook.retry
Apache Metron uses the official CentOS 6 Amazon Machine Image when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook.
TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => PendingVerification: Your account is currently
being verified. Verification normally takes less than 2 hours. Until your
account is verified, you may not be able to launch additional instances or
create additional volumes. If you are still receiving this message after more
than 2 hours, please let us know by writing to aws-verification@amazon.com. We
appreciate your patience."}
to retry, use: --limit @playbook.retry
This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon's verification process is complete. Amazon has some additional advice for dealing with this error and more.
Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact AWS Support.
TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
"Instance creation failed => InstanceLimitExceeded: You have requested more
instances (11) than your current instance limit of 10 allows for the specified
instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
request an adjustment to this limit."}
to retry, use: --limit @playbook.retry
This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing deployment/amazon-ec/playbook.yml. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional advice for dealing with this error and more.
You've reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see How many instances can I run in Amazon EC2. If you need additional instances, complete the Amazon EC2 Instance Request Form.
TASK [setup] *******************************************************************
fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
"changed": false, "msg": "SSH encountered an unknown error during the
connection. We recommend you re-run the command using -vvvv, which will enable
SSH debugging output to help diagnose the issue", "unreachable": true}
This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the key_file variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.