| # Neural Net Partition |
| |
| --- |
| |
| The purposes of partitioning neural network is to distribute the partitions onto |
| different working units (e.g., threads or nodes, called workers in this article) |
| and parallelize the processing. |
| Another reason for partition is to handle large neural network which cannot be |
| hold in a single node. For instance, to train models against images with high |
| resolution we need large neural networks (in terms of training parameters). |
| |
| Since *Layer* is the first class citizen in SIGNA, we do the partition against |
| layers. Specifically, we support partitions at two levels. First, users can configure |
| the location (i.e., worker ID) of each layer. In this way, users assign one worker |
| for each layer. Secondly, for one layer, we can partition its neurons or partition |
| the instances (e.g, images). They are called layer partition and data partition |
| respectively. We illustrate the two types of partitions using an simple convolutional neural network. |
| |
| <img src="../_static/images/conv-mnist.png" style="width: 220px"/> |
| |
| The above figure shows a convolutional neural network without any partition. It |
| has 8 layers in total (one rectangular represents one layer). The first layer is |
| DataLayer (data) which reads data from local disk files/databases (or HDFS). The second layer |
| is a MnistLayer which parses the records from MNIST data to get the pixels of a batch |
| of 8 images (each image is of size 28x28). The LabelLayer (label) parses the records to get the label |
| of each image in the batch. The ConvolutionalLayer (conv1) transforms the input image to the |
| shape of 8x27x27. The ReLULayer (relu1) conducts elementwise transformations. The PoolingLayer (pool1) |
| sub-samples the images. The fc1 layer is fully connected with pool1 layer. It |
| mulitplies each image with a weight matrix to generate a 10 dimension hidden feature which |
| is then normalized by a SoftmaxLossLayer to get the prediction. |
| |
| <img src="../_static/images/conv-mnist-datap.png" style="width: 1000px"/> |
| |
| The above figure shows the convolutional neural network after partitioning all layers |
| except the DataLayer and ParserLayers, into 3 partitions using data partition. |
| The read layers process 4 images of the batch, the black and blue layers process 2 images |
| respectively. Some helper layers, i.e., SliceLayer, ConcateLayer, BridgeSrcLayer, |
| BridgeDstLayer and SplitLayer, are added automatically by our partition algorithm. |
| Layers of the same color resident in the same worker. There would be data transferring |
| across different workers at the boundary layers (i.e., BridgeSrcLayer and BridgeDstLayer), |
| e.g., between s-slice-mnist-conv1 and d-slice-mnist-conv1. |
| |
| <img src="../_static/images/conv-mnist-layerp.png" style="width: 1000px"/> |
| |
| The above figure shows the convolutional neural network after partitioning all layers |
| except the DataLayer and ParserLayers, into 2 partitions using layer partition. We can |
| see that each layer processes all 8 images from the batch. But different partitions process |
| different part of one image. For instance, the layer conv1-00 process only 4 channels. The other |
| 4 channels are processed by conv1-01 which residents in another worker. |
| |
| |
| Since the partition is done at the layer level, we can apply different partitions for |
| different layers to get a hybrid partition for the whole neural network. Moreover, |
| we can also specify the layer locations to locate different layers to different workers. |
