Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
This vagrant recipe is based on the vagrant recipe from vagrant-puppet-vm
with added feature of vagrant-openstack-provider plugin. The plugin allows us to deploy a Hadoop cluster on an actual virtual environment as if we are deploying on local vagrant vms. It will spin up and provision the vm(s) for us.
The Vagrantfile creates a BigTop virtual Hadoop cluster on OpenStack by using BigTop puppet recipes and pulling from existing bigtop repositories
When the configuration is correctly set up in vagrantconfig.yaml, we should be able to deploy a cluster with on single command vagrant up
This can be use:
Install vagrant from official website
If you want to provision machines in parallel, install gnu-parallel
# for centos yum install parallel # for mac brew install parallel # for debian/ubuntu apt-get install parallel
/etc/hosts
vagrant plugin install vagrant-hostmanager
vagrant plugin install vagrant-openstack-provider
For now this is partically handled by openstack rc file and partically handled by vagrantconfig.yaml file
Download rc file from Openstack Horizon dashbord Access & Security and run
source projectname-openrc.sh
You will also need to specify flavor, image_id, keypair_name, and FQDN (fully qualified domain name of your openstack environment) in vagrantconfig.yaml to successfully spin up a vm
flavor: "name of your choice of flavor" # e.g. m1.small image_id: "UUID of your choice of image" # e.g. 8fddf8aa-1809-414d-b478-f93b8415f5f4 keypair_name: "your key pair name on openstack" # e.g. cloud-key FQDN: "the fully qualified domain name of the environment" key_path: "location of your private key" #e.g. ~/.ssh/cloud-key.pem
There are other options in vagrantconfig.yaml that you can specify such as set number of vms in the cluster, and automatically run smoke tests
The run_in_parallel
options should be set to true if want to provision machins in parallel. see below for how to.
num_instance: 1 run_smoke_tests: true run_in_parallel: false
You can also determine what components are being installed and tested
components: [hadoop, yarn] smoke_test_components: [mapredcue, pig]
For deployment in sequence
run_in_parallel
option in vagrantconfig.yaml
to falsevagrant up --provider=openstack
For parallel provisioning:
run_in_parallel
option in vagrantconfig.yaml
to true./para-provision.sh
This script is based on Joe Miller's para-vagrant.sh script please see NOTICE for more information
Script reads parameter num_instance
, run_smoke_tests
and smoke_test_components
from vagrantconfig.yaml
to determine how many machines to spin up and whether or not to run smoke tests and what components will be tested
This script will spin up vms on openstack sequentially first, and then do the provisioning in parallel. Each guest machine will have it's own log file. And will generate a log file for smoke tests if run_smoke_tests
set to true
There are some sketchy places in the code...(cuz I suck in bash) such as:
sed
: so OS X hates me, and it won't let me use sed -r
so again I create another temporary file for the smoke tests logNOTE:
Vagrantfile
has to match the name of the vim in para-provision.sh
, you can change them in line 66for ((i=1; i<=$NUM_INSTANCE; i++)); do cat <<EOF name_of_your_vm$i EOF
Vagrantfile
the run smoke test part is currently commented out, in order to run smoke test after all the machines are done provisioning. The smoke tests are currently being handled by para-provision.sh
.