docs/how_to/cloud.md - mxnet-test - Git at Google

 # Cloud Setup for MXNet

 ## Set Up an AWS GPU Cluster from Scratch

 In this topic, we provide a step-by-step tutorial on how to set up an AWS cluster with
 MXNet. We show how to:

 - [Use Amazon S3 to host data](#use-amazon-s3-to-host-data)
 - [Set up an EC2 GPU instance with all dependencies installed](#set-up-an-ec2-gpu-instance)
 - [Build and run MXNet on a single computer](#build-and-run-mxnet-on-a-gpu-instance)
 - [Set up an EC2 GPU cluster for distributed training](#set-up-an-ec2-gpu-cluster-for-distributed-training)

 ### Use Amazon S3 to Host Data

 Amazon S3 provides distributed data storage, which is convenient for hosting large datasets. To use S3, you need [AWS credentials](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html),
 which include an `ACCESS_KEY_ID` and a `SECRET_ACCESS_KEY`.

 To use MXNet with S3, set the environment variables `AWS_ACCESS_KEY_ID` and
 `AWS_SECRET_ACCESS_KEY` by adding the following two lines in
 `~/.bashrc` (replacing the strings with the correct ones):

 ```bash
 export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
 export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
 ```

 There are several ways to upload data to S3. A simple way is using
 [s3cmd](http://s3tools.org/s3cmd). For example:

 ```bash
 wget http://data.mxnet.io/mxnet/data/mnist.zip
 unzip mnist.zip && s3cmd put t*-ubyte s3://dmlc/mnist/
 ```

 ### Set Up an EC2 GPU Instance

 MXNet requires the following libraries:

 - C++ compiler with C++11 support, such as `gcc >= 4.8`
 - `CUDA` (`CUDNN` in optional) for GPU linear algebra
 - `BLAS` (cblas, open-blas, atblas, mkl, or others) for CPU linear algebra
 - `opencv` for image augmentations
 - `curl` and `openssl` for the ability to read/write to Amazon S3

 Installing `CUDA` on EC2 instances requires some effort. Caffe has a good
 [tutorial](https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN-3))
 on how to install CUDA 7.0 on Ubuntu 14.04.

 ***Note:*** We tried CUDA 7.5 on Nov 7,
 2015, but found it problematic.

 You can install the rest using the package manager. For example, on Ubuntu:

 ```
 sudo apt-get update
 sudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy
 ```

 The Amazon Machine Image [ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178) has the  packages listed above installed.


 ### Build and Run MXNet on a GPU Instance

 The following commands build MXNet with CUDA/CUDNN, Amazon S3, and distributed
 training.

 ```bash
 git clone --recursive https://github.com/dmlc/mxnet
 cd mxnet; cp make/config.mk .
 echo "USE_CUDA=1" >>config.mk
 echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
 echo "USE_CUDNN=1" >>config.mk
 echo "USE_BLAS=atlas" >> config.mk
 echo "USE_DIST_KVSTORE = 1" >>config.mk
 echo "USE_S3=1" >>config.mk
 make -j$(nproc)
 ```

 To test whether everything is installed properly, train a convolutional neural network (CNN) on MNIST using a GPU:

 ```bash
 python tests/python/gpu/test_conv.py
 ```

 If you've placed the MNIST data on `s3://dmlc/mnist`, you can read the data stored on Amazon S3 directly with the following command:

 ```bash
 sed -i.bak "s!data_dir = 'data'!data_dir = 's3://dmlc/mnist'!" tests/python/gpu/test_conv.py
 ```

 ***Note:*** You can use `sudo ln /dev/null /dev/raw1394` to fix the opencv error `libdc1394 error: Failed to initialize libdc1394`.

 ### Set Up an EC2 GPU Cluster for Distributed Training

 A cluster consists of multiple computers. You can use the computer with MXNet
 installed as the root computer for submitting jobs, and then launch several
 slave computers to run the jobs. For example, launch multiple instances using an
 AMI, e.g.,
 [ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178),
 with dependencies installed. There are two options:

 - Make all slaves' ports accessible (same for the root) by setting type: All TCP,
    Source: Anywhere in Configure Security Group.

 - Use the same `pem` as the root computer to access all slave computers, and
    then copy the `pem` file into the root computer's `~/.ssh/id_rsa`. If you do this, all slave computers can be accessed with SSH from the root.

 Now, run the CNN on multiple computers. Assume that we are on a working
 directory of the root computer, such as `~/train`, and MXNet is built as `~/mxnet`.

 1. Pack the MXNet Python library into this working directory for easy
   synchronization:

   ```bash
   cp -r ~/mxnet/python/mxnet .
   cp ~/mxnet/lib/libmxnet.so mxnet/
   ```

   And then copy the training program:

   ```bash
   cp ~/mxnet/example/image-classification/*.py .
   cp -r ~/mxnet/example/image-classification/common .
   ```

 2. Prepare a host file with all slaves private IPs. For example, `cat hosts`:

   ```bash
   172.30.0.172
   172.30.0.171
   ```

 3. Assuming that there are two computers, train the CNN using two workers:

   ```bash
   ../../tools/launch.py -n 2 -H hosts --sync-dir /tmp/mxnet python train_mnist.py --kv-store dist_sync
   ```

 ***Note:*** Sometimes the jobs linger at the slave computers even you've pressed `Ctrl-c`
 at the root node. To terminate them, use the following command:

 ```bash
 cat hosts | xargs -I{} ssh -o StrictHostKeyChecking=no {} 'uname -a; pgrep python | xargs kill -9'
 ```

 ***Note:*** The preceding example is very simple to train, and, therefore, isn't a good
 benchmark for distributed training. Consider using other [examples](https://github.com/dmlc/mxnet/tree/master/example/image-classification).

 ### More Options
 #### Use Multiple Data Shards
 It is common to pack a dataset into multiple files, especially when working in a distributed environment. MXNet supports direct loading from multiple data shards. Put all of the record files into a folder, and point the data path to the folder.

 #### Use YARN and SGE
 Although using SSH can be simple when you don't have a cluster scheduling framework, MXNet is designed to be portable to various platforms.  We provide other scripts in [tracker](https://github.com/dmlc/dmlc-core/tree/master/tracker) to allow running on other cluster frameworks, including Hadoop (YARN) and SGE. We welcome your contribution of examples of running MXNet on your favorite distributed platform.
	# Cloud Setup for MXNet

	## Set Up an AWS GPU Cluster from Scratch

	In this topic, we provide a step-by-step tutorial on how to set up an AWS cluster with
	MXNet. We show how to:

	- [Use Amazon S3 to host data](#use-amazon-s3-to-host-data)
	- [Set up an EC2 GPU instance with all dependencies installed](#set-up-an-ec2-gpu-instance)
	- [Build and run MXNet on a single computer](#build-and-run-mxnet-on-a-gpu-instance)
	- [Set up an EC2 GPU cluster for distributed training](#set-up-an-ec2-gpu-cluster-for-distributed-training)

	### Use Amazon S3 to Host Data

	Amazon S3 provides distributed data storage, which is convenient for hosting large datasets. To use S3, you need [AWS credentials](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html),
	which include an `ACCESS_KEY_ID` and a `SECRET_ACCESS_KEY`.

	To use MXNet with S3, set the environment variables `AWS_ACCESS_KEY_ID` and
	`AWS_SECRET_ACCESS_KEY` by adding the following two lines in
	`~/.bashrc` (replacing the strings with the correct ones):

	```bash
	export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
	export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
	```

	There are several ways to upload data to S3. A simple way is using
	[s3cmd](http://s3tools.org/s3cmd). For example:

	```bash
	wget http://data.mxnet.io/mxnet/data/mnist.zip
	unzip mnist.zip && s3cmd put t*-ubyte s3://dmlc/mnist/
	```

	### Set Up an EC2 GPU Instance

	MXNet requires the following libraries:

	- C++ compiler with C++11 support, such as `gcc >= 4.8`
	- `CUDA` (`CUDNN` in optional) for GPU linear algebra
	- `BLAS` (cblas, open-blas, atblas, mkl, or others) for CPU linear algebra
	- `opencv` for image augmentations
	- `curl` and `openssl` for the ability to read/write to Amazon S3

	Installing `CUDA` on EC2 instances requires some effort. Caffe has a good
	[tutorial](https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN-3))
	on how to install CUDA 7.0 on Ubuntu 14.04.

	*Note:* We tried CUDA 7.5 on Nov 7,
	2015, but found it problematic.

	You can install the rest using the package manager. For example, on Ubuntu:

	```
	sudo apt-get update
	sudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy
	```

	The Amazon Machine Image [ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178) has the packages listed above installed.


	### Build and Run MXNet on a GPU Instance

	The following commands build MXNet with CUDA/CUDNN, Amazon S3, and distributed
	training.

	```bash
	git clone --recursive https://github.com/dmlc/mxnet
	cd mxnet; cp make/config.mk .
	echo "USE_CUDA=1" >>config.mk
	echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
	echo "USE_CUDNN=1" >>config.mk
	echo "USE_BLAS=atlas" >> config.mk
	echo "USE_DIST_KVSTORE = 1" >>config.mk
	echo "USE_S3=1" >>config.mk
	make -j$(nproc)
	```

	To test whether everything is installed properly, train a convolutional neural network (CNN) on MNIST using a GPU:

	```bash
	python tests/python/gpu/test_conv.py
	```

	If you've placed the MNIST data on `s3://dmlc/mnist`, you can read the data stored on Amazon S3 directly with the following command:

	```bash
	sed -i.bak "s!data_dir = 'data'!data_dir = 's3://dmlc/mnist'!" tests/python/gpu/test_conv.py
	```

	*Note:* You can use `sudo ln /dev/null /dev/raw1394` to fix the opencv error `libdc1394 error: Failed to initialize libdc1394`.

	### Set Up an EC2 GPU Cluster for Distributed Training

	A cluster consists of multiple computers. You can use the computer with MXNet
	installed as the root computer for submitting jobs, and then launch several
	slave computers to run the jobs. For example, launch multiple instances using an
	AMI, e.g.,
	[ami-12fd8178](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-12fd8178),
	with dependencies installed. There are two options:

	- Make all slaves' ports accessible (same for the root) by setting type: All TCP,
	Source: Anywhere in Configure Security Group.

	- Use the same `pem` as the root computer to access all slave computers, and
	then copy the `pem` file into the root computer's `~/.ssh/id_rsa`. If you do this, all slave computers can be accessed with SSH from the root.

	Now, run the CNN on multiple computers. Assume that we are on a working
	directory of the root computer, such as `~/train`, and MXNet is built as `~/mxnet`.

	1. Pack the MXNet Python library into this working directory for easy
	synchronization:

	```bash
	cp -r ~/mxnet/python/mxnet .
	cp ~/mxnet/lib/libmxnet.so mxnet/
	```

	And then copy the training program:

	```bash
	cp ~/mxnet/example/image-classification/*.py .
	cp -r ~/mxnet/example/image-classification/common .
	```

	2. Prepare a host file with all slaves private IPs. For example, `cat hosts`:

	```bash
	172.30.0.172
	172.30.0.171
	```

	3. Assuming that there are two computers, train the CNN using two workers:

	```bash
	../../tools/launch.py -n 2 -H hosts --sync-dir /tmp/mxnet python train_mnist.py --kv-store dist_sync
	```

	*Note:* Sometimes the jobs linger at the slave computers even you've pressed `Ctrl-c`
	at the root node. To terminate them, use the following command:

	```bash
	cat hosts \| xargs -I{} ssh -o StrictHostKeyChecking=no {} 'uname -a; pgrep python \| xargs kill -9'
	```

	*Note:* The preceding example is very simple to train, and, therefore, isn't a good
	benchmark for distributed training. Consider using other [examples](https://github.com/dmlc/mxnet/tree/master/example/image-classification).

	### More Options
	#### Use Multiple Data Shards
	It is common to pack a dataset into multiple files, especially when working in a distributed environment. MXNet supports direct loading from multiple data shards. Put all of the record files into a folder, and point the data path to the folder.

	#### Use YARN and SGE
	Although using SSH can be simple when you don't have a cluster scheduling framework, MXNet is designed to be portable to various platforms. We provide other scripts in [tracker](https://github.com/dmlc/dmlc-core/tree/master/tracker) to allow running on other cluster frameworks, including Hadoop (YARN) and SGE. We welcome your contribution of examples of running MXNet on your favorite distributed platform.