tree: dbab803dde99eecfad1a10647ded9db5ad571d82 [path history] [tgz]
  1. common/
  2. data/
  3. predict-cpp/
  4. symbols/
  11. symbol_alexnet.R
  12. symbol_googlenet.R
  13. symbol_inception-bn-28-small.R
  14. symbol_inception-bn.R
  15. symbol_inception-resnet-v1.R
  16. symbol_inception-resnet-v2.R
  17. symbol_lenet.R
  18. symbol_mlp.R
  19. symbol_resnet-28-small.R
  20. symbol_resnet-v2.R
  21. symbol_resnet.R
  22. symbol_vgg.R
  25. train_cifar10.R
  27. train_imagenet.R
  29. train_mnist.R
  30. train_model.R

Image Classification

This fold contains examples for image classification. The goal of image classifcation is to identify the objects contained in images. The following example shows recognized object classes with corresponding probabilities using a pre-traind model.


  1. Basic usages
  2. How to prepare datasets
  3. A List of pre-trained models
  4. How to fine-tune a dataset with a pre-trained model
  5. How to train with multiple machines
  6. Frequently asked questions

Basic Usages

Both python and R training programs are provided. Use train_*.py or train_*.R to train a network on a particular dataset. For example:

  • train a multilayer perception on the mnist dataset

    python --network mlp
  • train a 110-layer resnet on the cifar10 dataset with batch size 128 and GPU 0 and 1

    python --network resnet --num-layers 110 --batch-size 128 --gpus 0,1

There is a rich set of options, one can list them by passing --help. Some commonly used options are listed as following:

network                    The network to train, which is defined in symbol/. Some networks may accept additional arguments, such as --num-layers is used to specify the number of layers in ResNet.
data-train, data-valThe data for training and validation. It can be either a filename or a directory. For the latter, all files in the directory will be used. But if --benchmark 1 is used, then there two arguments will be ignored.
gpusThe list of GPUs to use, such as 0 or 0,3,4,7. If an empty string '' is given, then we will use CPU.
batch-sizeThe batch size for SGD training. It specifies the number of examples used for each SGD iteration. If we use k GPUs, then each GPU will compute batch_size/k examples in each time.
modelThe model name to save (and load). A model will be saved into two parts: model-symbol.json for the network definition and model-n.params for the parameters saved on epoch n.
num-epochsThe maximal number of epochs to train.
load-epochIf given integer k, then resume the training starting from epoch k with the model saved at the end of epoch k-1. Note that the training starts from epoch 0, and the model saved at the end of this epoch will be model-0001.params.
lrThe initial learning rate, namely for epoch 0.
lr-factor, lr-step-epochsReduce the learning rate on give epochs. For example, --lr-factor .1 --lr-step-epochs 30,60 will reduce the learning rate by 0.1 on epoch 30, and then reduce it by 0.1 again on epoch 60.

Prepare Datasets

The recommended data format is RecordIO, which concatenates multiple examples into seekable binary files for better read efficiency. We provide a tool located in tools/ to convert individual images into .rec files.

For a simple tutorial, assume all images are stored as individual image files such as .png or .jpg, and images belonging to the same class are placed in the same directory. All these class directories are then in the same root img_data directory. Our goal is to generate two files, mydata_train.rec for training and mydata_val.rec for validation, and the former contains 95% images.

We first prepare two .lst files, which consist of the labels and image paths can be used for generating rec files.

python tools/ --list True --recursive True --train-ratio 0.95 mydata img_data

Then we generate the .rec files. We resize the images such that the short edge is at least 480px and save them with 95/100 quality. We also use 16 threads to accelerate the packing.

python tools/ --resize 480 --quality 95 --num-thread 16 mydata img_data


  • SSD is much faster than HDD when dealing with a large number of small files. (but HDD is good enough to read rec files).
    • We can use a cloud storage instance to prepare the data. For example, AWS i2.4xlarge provides 4 x 800 GB SSDs.

    • We can make a software RAID over multiple disks. For example, the following command create a RAID0 on 4 disks:

      sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=4 \
        /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
      sudo mkfs /dev/md0
  • Check *.sh scripts in the data/ folder for more examples
  • Use --help to see more options.

Pre-trained Models

We provide multiple pre-trained models on various datasets. Use common/ to download these models. These models can be used in any front-end language MXNet supports. For example, the tutorial shows how to classify an image with jupyter notebook.

ImageNet 1K

It is first used by ImageNet challenge 2012, which contains about 1.2M images with 1000 classes. To test these models, one can use data/ to prepare the validation dataset and to calculate the accuracy.

Single Center Crop Accuracy



  • our Resnet does not need to specify the RGB mean due the data batch normalization layer. While the inception models needs --rgb-mean 123.68,116.779,103.939
  • Resnet training logs are available at tornadomeet/ResNet
  • We warm up our Resnext-101-64x4d by training it with 1/100 and 1/10 of the base learning rate for the 1st and 2nd epoch. We use 3 p2.16xlarge instances with a batch size of 384 on each node with base lr set to 0.45, and decay step set at 50, 80, 110 epoch. After 133 epoch, we use one node to finetune, and turn off color and scale data augmentation, with lr reduced to 1.5e-04.

Speed and Memory Footprint:

Single K80 GPU with batch size 32.

Modelmemory (MB)images/sec

Imagenet 11K

It is generated from the complete Imagenet dataset, namely  fall11_whole.tar from In addition, we removed classes which have less than 500 images, and then randomly picked 50 images from each class as the validation set. As a result, this dataset contains 11221 classes, with 11,797,630 images for training.

Single Center Crop Accuracy


Imagenet 11K + Place365 Challenge

This dataset combine the Imagenet 11K dataset with the Place 365 challenge dataset. The latter contains 365 classes with 8 millions images. It results in a dataset with around 20 million images.

Single Center Crop Accuracy


Fine-tune another Dataset

Fine-tune refers training with parameters partially intialized with pre-trained model. One can use to train another dataset with pre-trained models listed above. For example, first run data/ to download and prepare the Caltech-256 dataset, then fine tune it with imagenet11k-resnet-152 by using 8 GPUs:

python --pretrained-model imagenet11k-resnet-152 --gpus 0,1,2,3,4,5,6,7 \
    --data-train data/caltech256-train.rec --data-val data/caltech256-val.rec \
    --batch-size 128 --num-classes 256 --num-examples 15240

We obtained 87.3% top-1 validation accuracy, and the training log is available here. See the python notebook for more explanations.

Distributed Training

The simplest way for distributing training is that both programs and data are placed on the a shared filesystem such as NFS and AWS EFS, and there is one machine, we call it the root machine, can ssh to all others. Assume we save the hostnames (or IPs) of all machines will be used for training (might include the root machine) into a file named hosts. The outputs of cat hosts may be

Now we can run the previous cifar10 training on two machines:

python ../../tools/ -n 2 -H hosts \
    python --network resnet --num-layers 110 --batch-size 128 --gpus 0,1 \
    --kv-store dist_device_sync

It differs the previous command in two aspects. First, we use to start the program, which creates two workers (given by -n) on the two machines specified in hosts . Second, we change the --kv-store from the default device, which means try to use GPU P2P, to dist_device_sync. The latter uses distributed synchronized communication.

For more usages:

  • One can use to run distributed benchmarks (also for multiple GPUs with single machine)
  • A how-to tutorial with more explanation.
  • A blog about setuping up a GPU cluster on AWS with cloud formation.


To run benchmark on imagenet networks, use --benchmark 1 as the argument to, An example is shown below:

python --benchmark 1 --gpus 0,1 --network inception-v3 --batch-size 64 \
  --image-shape 3,299,299 --num-epochs 1 --kv-store device

When running in benchmark mode, the script generates synthetic data of the given data shape and batch size.

The can be used to run a series of benchmarks against different image networks on a given set of workers and takes the following arguments:

  • --worker_file: file that contains a list of worker hostnames or list of worker ip addresses that have passwordless ssh enabled.
  • --worker_count: number of workers to run benchmark on.
  • --gpu_count: number of gpus on each worker to use.
  • --networks: one or more networks in the format network_name:batch_size:image_size.

The script runs benchmarks on variable number of gpus upto gpu_count starting from 1 gpu doubling the number of gpus in each run using kv-store=device and after that running on variable number of nodes on all gpus starting with 1 node upto worker_count doubling the number of nodes used in each run using kv-store=dist_sync_device.

An example to run the benchmark script is shown below with 8 workers and 16 gpus on each worker:

python --worker_file /opt/deeplearning/workers --worker_count 8 \
  --gpu_count 16 --networks 'inception-v3:32:299'

Scalability Results

We fixed the batch size per GPU and then increase the number of GPUs. Synchronized SGD is used, namely --kv-store dist_device_sync. The following three CNNs (located in symbol/) are used

batch per GPU5123232
model size (MB)20395240

Number of images proccessed per second is shown in the following table:


The following figure shows the speedup against a single GPU compared to the ideal scalability.


We show the convergence of training ResNet-152 on Imagenet 1K. The single machine with 8 GPUs results are from Wei Wu. We then trained the model using 10 machines, each machine has 8 GPUs, with the same hyper-parameters, except for we changed the total batch size from 8*32 to 80*32 and the initial learning rate to 0.5 instead of 0.1. The validation accuracy versus data epoch is shown as following. Both models have almost identical convergence rate.


Validation Accuracy

It is often straightforward to achieve a reasonable validation accuracy, but sometimes matching the state-of-the-art numbers reported in the papers is extremely hard. Here we list some aspects you may check to improve the validation accuracy:

  • Add more data argumentations, which often reduces the gap between training accuracy and validation accuracy. You may reduce the data argumentation close to end.
  • Increase the learning rate and keep large learning rate for a long time. For example, in CIFAR10 we keep lr=0.1 for 200 epochs and then reduce to 0.01.
  • Do not use too large batch size, especially for batch size >> number of classes.


First check the workload is not too small (e.g. LeNet on MNIST) and also batch size is reasonable large. The performance bottleneck often happens in three aspects:

  • Reading data. Use the --test-io 1 flag to check how many images can be pre-processed per second
    • Increase --data-nthreads (default is 4) to use more threads for data augmentation can help.
    • Data preprocessing is done by opencv. If opencv is compiled from source codes, check if it is configured correctly.
    • Use --benchmark 1 to use randomly generated data rather than real data.

Refer to how_to/performance for more details about CPU, GPU and multi-device performance.


An over sized batch size may result in out of GPU memory. The common error message is cudaMalloc failed: out of memory. Now we can

  • Reduce the batch size
  • Set the environment variable MXNET_BACKWARD_DO_MIRROR to 1. It trades off computation for memory consumption. For example, with batch size 64, inception-v3 uses 10G memory and trains 30 image/sec on a single K80 GPU. When mirroring is enabled, with 10G GPU memory consumption, we can run inception-v3 using batch size 128. The cost is that the speed reduces to 27 images/sec.


  • Nov 9, 2015: major refactor.
    • Organize files into sub-directories
    • Add Resnet, pretrained models, and fine-tune scripts.
    • Update documents.
    • Move ../cpp/image-classification into ./predict-cpp/
  • Oct 15, 2016: add R examples
  • Nov 19, 2015: major refactor.
    • Various networks (Alex/VGG/Inception) on multiple dataset (MNIST/Cifar10/Imagenet)
    • Distributed training