This fold contains examples for image classification. The goal of image classifcation is to identify the objects contained in images. The following example shows recognized object classes with corresponding probabilities using a pre-traind model.
Both python and R training programs are provided. Use
train_*.R to train a network on a particular dataset. For example:
train a multilayer perception on the mnist dataset
python train_mnist.py --network mlp
train a 110-layer resnet on the cifar10 dataset with batch size 128 and GPU 0 and 1
python train_cifar10.py --network resnet --num-layers 110 --batch-size 128 --gpus 0,1
There is a rich set of options, one can list them by passing
--help. Some commonly used options are listed as following:
|The network to train, which is defined in symbol/. Some networks may accept additional arguments, such as |
|The data for training and validation. It can be either a filename or a directory. For the latter, all files in the directory will be used. But if |
|The list of GPUs to use, such as |
|The batch size for SGD training. It specifies the number of examples used for each SGD iteration. If we use k GPUs, then each GPU will compute batch_size/k examples in each time.|
|The model name to save (and load). A model will be saved into two parts: |
|The maximal number of epochs to train.|
|If given integer k, then resume the training starting from epoch k with the model saved at the end of epoch k-1. Note that the training starts from epoch 0, and the model saved at the end of this epoch will be |
|The initial learning rate, namely for epoch 0.|
|Reduce the learning rate on give epochs. For example, |
The recommended data format is RecordIO, which concatenates multiple examples into seekable binary files for better read efficiency. We provide a tool
im2rec.py located in
tools/ to convert individual images into
For a simple tutorial, assume all images are stored as individual image files such as
.jpg, and images belonging to the same class are placed in the same directory. All these class directories are then in the same root
img_data directory. Our goal is to generate two files,
mydata_train.rec for training and
mydata_val.rec for validation, and the former contains 95% images.
We first prepare two
.lst files, which consist of the labels and image paths can be used for generating
python tools/im2rec.py --list True --recursive True --train-ratio 0.95 mydata img_data
Then we generate the
.rec files. We resize the images such that the short edge is at least 480px and save them with 95/100 quality. We also use 16 threads to accelerate the packing.
python tools/im2rec.py --resize 480 --quality 95 --num-thread 16 mydata img_data
We can use a cloud storage instance to prepare the data. For example, AWS
i2.4xlarge provides 4 x 800 GB SSDs.
We can make a software RAID over multiple disks. For example, the following command create a RAID0 on 4 disks:
sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=4 \ /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 sudo mkfs /dev/md0
*.shscripts in the
data/folder for more examples
im2rec.py --helpto see more options.
We provide multiple pre-trained models on various datasets. Use common/modelzone.py to download these models. These models can be used in any front-end language MXNet supports. For example, the tutorial shows how to classify an image with jupyter notebook.
It is first used by ImageNet challenge 2012, which contains about 1.2M images with 1000 classes. To test these models, one can use data/imagenet1k-val.sh to prepare the validation dataset and score.py to calculate the accuracy.
Single K80 GPU with batch size 32.
It is generated from the complete Imagenet dataset, namely
fall11_whole.tar from http://www.image-net.org/download-images. In addition, we removed classes which have less than 500 images, and then randomly picked 50 images from each class as the validation set. As a result, this dataset contains 11221 classes, with 11,797,630 images for training.
This dataset combine the Imagenet 11K dataset with the Place 365 challenge dataset. The latter contains 365 classes with 8 millions images. It results in a dataset with around 20 million images.
Fine-tune refers training with parameters partially intialized with pre-trained model. One can use fine-tune.py to train another dataset with pre-trained models listed above. For example, first run data/caltech256.sh to download and prepare the Caltech-256 dataset, then fine tune it with
imagenet11k-resnet-152 by using 8 GPUs:
python fine-tune.py --pretrained-model imagenet11k-resnet-152 --gpus 0,1,2,3,4,5,6,7 \ --data-train data/caltech256-train.rec --data-val data/caltech256-val.rec \ --batch-size 128 --num-classes 256 --num-examples 15240
The simplest way for distributing training is that both programs and data are placed on the a shared filesystem such as NFS and AWS EFS, and there is one machine, we call it the root machine, can ssh to all others. Assume we save the hostnames (or IPs) of all machines will be used for training (might include the root machine) into a file named
hosts. The outputs of
cat hosts may be
Now we can run the previous cifar10 training on two machines:
python ../../tools/launch.py -n 2 -H hosts \ python train_cifar10.py --network resnet --num-layers 110 --batch-size 128 --gpus 0,1 \ --kv-store dist_device_sync
It differs the previous command in two aspects. First, we use
launch.py to start the program, which creates two workers (given by
-n) on the two machines specified in
hosts . Second, we change the
--kv-store from the default
device, which means try to use GPU P2P, to
dist_device_sync. The latter uses distributed synchronized communication.
For more usages:
To run benchmark on imagenet networks, use
--benchmark 1 as the argument to
train_imagenet.py, An example is shown below:
python train_imagenet.py --benchmark 1 --gpus 0,1 --network inception-v3 --batch-size 64 \ --image-shape 3,299,299 --num-epochs 1 --kv-store device
When running in benchmark mode, the script generates synthetic data of the given data shape and batch size.
benchmark.py can be used to run a series of benchmarks against different image networks on a given set of workers and takes the following arguments:
--worker_file: file that contains a list of worker hostnames or list of worker ip addresses that have passwordless ssh enabled.
--worker_count: number of workers to run benchmark on.
--gpu_count: number of gpus on each worker to use.
--networks: one or more networks in the format network_name:batch_size:image_size.
benchmark.py script runs benchmarks on variable number of gpus upto gpu_count starting from 1 gpu doubling the number of gpus in each run using
kv-store=device and after that running on variable number of nodes on all gpus starting with 1 node upto
worker_count doubling the number of nodes used in each run using
An example to run the benchmark script is shown below with 8 workers and 16 gpus on each worker:
python benchmark.py --worker_file /opt/deeplearning/workers --worker_count 8 \ --gpu_count 16 --networks 'inception-v3:32:299'
We fixed the batch size per GPU and then increase the number of GPUs. Synchronized SGD is used, namely
--kv-store dist_device_sync. The following three CNNs (located in symbol/) are used
|batch per GPU||512||32||32|
|model size (MB)||203||95||240|
Number of images proccessed per second is shown in the following table:
The following figure shows the speedup against a single GPU compared to the ideal scalability.
We show the convergence of training ResNet-152 on Imagenet 1K. The single machine with 8 GPUs results are from Wei Wu. We then trained the model using 10 machines, each machine has 8 GPUs, with the same hyper-parameters, except for we changed the total batch size from 8*32 to 80*32 and the initial learning rate to 0.5 instead of 0.1. The validation accuracy versus data epoch is shown as following. Both models have almost identical convergence rate.
It is often straightforward to achieve a reasonable validation accuracy, but sometimes matching the state-of-the-art numbers reported in the papers is extremely hard. Here we list some aspects you may check to improve the validation accuracy:
lr=0.1for 200 epochs and then reduce to 0.01.
First check the workload is not too small (e.g. LeNet on MNIST) and also batch size is reasonable large. The performance bottleneck often happens in three aspects:
--test-io 1flag to check how many images can be pre-processed per second
--data-nthreads(default is 4) to use more threads for data augmentation can help.
opencv. If opencv is compiled from source codes, check if it is configured correctly.
--benchmark 1to use randomly generated data rather than real data.
Refer to how_to/performance for more details about CPU, GPU and multi-device performance.
An over sized batch size may result in out of GPU memory. The common error message is
cudaMalloc failed: out of memory. Now we can
MXNET_BACKWARD_DO_MIRRORto 1. It trades off computation for memory consumption. For example, with batch size 64, inception-v3 uses 10G memory and trains 30 image/sec on a single K80 GPU. When mirroring is enabled, with 10G GPU memory consumption, we can run inception-v3 using batch size 128. The cost is that the speed reduces to 27 images/sec.