AWS has an “access key” and a “secret key”, which correspond to Brooklyn's identity and credential respectively.
These keys are the way for any programmatic mechanism to access the AWS API.
To generate an access key and a secret key, see jclouds instructions and AWS IAM instructions.
An example of the expected format is shown below:
location:
jclouds:aws-ec2:
region: us-east-1
identity: ABCDEFGHIJKLMNOPQRST
credential: abcdefghijklmnopqrstu+vwxyzabcdefghijklm
Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.
Below are examples of configuration options that use values specific to AWS EC2:
The region is the AWS region code. For example, region: us-east-1. You can in-line the region name using the following format: jclouds:aws-ec2:us-east-1. A specific availability zone within the region can be specified by including its letter identifier as a suffix. For example, region: us-east-1a.
The hardwareId is the instance type. For example, hardwareId: m4.large.
The imageId is the region-specific AMI id. For example, imageId: us-east-1/ami-05ebd06c.
The securityGroups option takes one or more names of pre-existing security groups. For example, securityGroups: mygroup1 or securityGroups: [ mygroup1, mygroup2 ].
The size of the EBS boot volume for a VM can be specified using mapNewVolumeToDeviceName in templateOptions as shown below:
location:
jclouds:aws-ec2:
region: ...
templateOptions:
mapNewVolumeToDeviceName:
- /dev/sda1
- 123
- true
Where /dev/sda1 is the device name of the boot volume on the selected AMI, 123 is the size of the volume in gigabytes and true is a boolean indicating the volume should be deleted on VM termination.
You can specify a keyPair to use for initial provisioning as a configuration option. If this is omitted, Brooklyn will use jclouds to create a new ad hoc key pair at AWS for that machine, and it will delete it afterwards. This is usually seamless and occurs behind the scenes, with the post-provision user set up and configured as normal for all locations. However using AWS heavily or optimizing creation, using a known key pairs can
make some images more reliable and speed things up.
First, in the AWS Console, open the EC2 service in the region you are interested in, then click “Key Pairs” at the left. For us-east-1, the link is here. Click “Create Key Pair” (or “Import Key Pair” if you want to provide a public key) and follow the instructions.
Then define your location as follows for aws-us-east-1. Make sure to replace the XXXX sections with the key-pair name defined above and the corresponding private key data.
brooklyn.catalog: version: "1.0" itemType: location items: - id: aws-base item: type: jclouds:aws-ec2 brooklyn.config: identity: XXXXXXXXXXXXXXXX credential: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx - id: aws-us-east-1 item: type: aws-base brooklyn.config: region: us-east-1 keyPair: XXXXXXXXX loginUser.privateKeyData: | -----BEGIN RSA PRIVATE KEY----- XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -----END RSA PRIVATE KEY-----
Apache Brooklyn can run with AWS VPC and both public and private subnets. Simply provide the subnet-a1b2c3d4 as the networkName when deploying:
location:
jclouds:aws-ec2:
region: us-west-1
networkName: subnet-a1b2c3d4 # use your subnet ID
Subnets are typically used in conjunction with security groups. Brooklyn does not attempt to open additional ports when private subnets or security groups are supplied, so the subnet and ports must be configured appropriately for the blueprints being deployed. You can configure a default security group with appropriate (or all) ports opened for access from the appropriate (or all) CIDRs and security groups, or you can define specific securityGroups on the location or as provisioning.properties on the entities.
Make sure that Brooklyn has access to the machines under management. This includes SSH, which might be done with a public IP created with inbound access on port 22 permitted for a CIDR range including the IP from which Brooklyn contacts it. Alternatively you can run Brooklyn on a machine in that same subnet, or set up a VPN or jumphost which Brooklyn will use.
If you have a pre-2014 Amazon account, it is likely configured in some regions to run in “EC2 Classic” mode by default, instead of the more modern “VPC” default mode. This can cause failures when requesting certain hardware configurations because many of the more recent hardware “instance types” only run in “VPC” mode. For instance when requesting an instance with minRam: 8gb, Brooklyn may opt for an m4.large, which is a VPC-only instance type. If you are in a region configured to use “EC2 Classic” mode, you may see a message such as this:
400 VPCResourceNotSpecified: The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.
This is a limitation of “legacy” accounts. The easiest fixes are either:
m3.xlarge (see below)eu-central-1 should work as it only offers VPC mode, irrespective of the age of your AWS account)To understand the situation, the following resources may be useful:
If you want to solve this problem with your existing account, you can create a VPC and instruct Brooklyn to use it:
default security group for that VPC. Modify its “Inbound Rules” to allow “All traffic” from “Anywhere”. (Or for more secure options, see the instructions in the previous section, “Using Subnets”.)subnet-a1b2c3d4) for use in Brooklyn.You can then deploy blueprints to the subnet, allowing VPC hardware instance types, by specifying the subnet ID as the networkName in your YAML blueprint. This is covered in the previous section, “Using Subnets”.
Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group from being deleted until there are no VMs using it. However, there is eventual consistency for recording which VMs still reference those security groups: after deleting the VM, it can sometimes take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but does not block for longer.
Whilst there is eventual consistency for recording which VMs still reference security groups, after deleting a VM, it can sometimes take several minutes before a security group can be deleted
There is utility written by Cloudsoft for deleting these unused resources: http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-jclouds.html.