Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
#BigTop Docker provisioner
The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories. This cluster can be used:
These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster.
This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6.
Install VirtualBox
Install Vagrant. Need version 1.6.5 or higher.
NOTE : SELinux can PREVENT you from ssh'ing into your docker container. As a brute force way to disable it - remove it from vi /etc/sysconfig/docker arguments (fedora and centos may by default launch docker daemon with the --selinux-enabled option)! In the future, lets update this README with the RIGHT way to allow selinux without breaking ssh into a docker container!
service docker restart docker pull bigtop/deploy:centos-6
Now, you can start your cluster:
In case of errors you can attempt running as root, or else, ping the mailing list.
./docker-hadoop.sh --create 3
./docker-hadoop.sh --destroy
./docker-hadoop.sh --provision
./docker-hadoop.sh --smoke-tests
./docker-hadoop.sh --create 5 --smoke-tests --destroy
Commands will be executed by following order:
create 5 node cluster => run smoke tests => destroy the cluster
./docker-hadoop.sh -h usage: docker-hadoop.sh [options] -c NUM_INSTANCES, --create=NUM_INSTANCES Create a docker based Bigtop Hadoop cluster -p, --provision Deploy configuration changes -s, --smoke-tests Run Bigtop smoke tests -d, --destroy Destroy the cluster -h, --help
##Configurations
docker: memory_size: "2048"
boot2docker: memory_size: "4096" number_cpus: "2"
namenode_ui_port: "50070" yarn_ui_port: "8088" hbase_ui_port: "60010"
Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed
##Configure Apache Hadoop ecosystem components
components: "hadoop,hbase,yarn,..."
By default, Apache Hadoop, YARN, and Apache HBase will be installed. See bigtop-deploy/puppet/config/site.csv.example
for more details.
##Notes