

``mx.symbol.BatchNorm``
==============================================

Description
----------------------

Batch normalization.

Normalizes a data batch by mean and variance, and applies a scale ``gamma`` as
well as offset ``beta``.

Assume the input has more than one dimension and we normalize along axis 1.
We first compute the mean and variance along this axis:

.. math::

  data\_mean[i] = mean(data[:,i,:,...]) \\
  data\_var[i] = var(data[:,i,:,...])

Then compute the normalized output, which has the same shape as input, as following:

.. math::

  out[:,i,:,...] = \frac{data[:,i,:,...] - data\_mean[i]}{\sqrt{data\_var[i]+\epsilon}} * gamma[i] + beta[i]

Both *mean* and *var* returns a scalar by treating the input as a vector.

Assume the input has size *k* on axis 1, then both ``gamma`` and ``beta``
have shape *(k,)*. If ``output_mean_var`` is set to be true, then outputs both ``data_mean`` and
the inverse of ``data_var``, which are needed for the backward pass. Note that gradient of these
two outputs are blocked.

Besides the inputs and the outputs, this operator accepts two auxiliary
states, ``moving_mean`` and ``moving_var``, which are *k*-length
vectors. They are global statistics for the whole dataset, which are updated
by::

	 moving_mean = moving_mean * momentum + data_mean * (1 - momentum)
	 moving_var = moving_var * momentum + data_var * (1 - momentum)
	 
	 If ``use_global_stats`` is set to be true, then ``moving_mean`` and
	 ``moving_var`` are used instead of ``data_mean`` and ``data_var`` to compute
	 the output. It is often used during inference.
	 
	 The parameter ``axis`` specifies which axis of the input shape denotes
	 the 'channel' (separately normalized groups).  The default is 1.  Specifying -1 sets the channel
	 axis to be the last item in the input shape.
	 
	 Both ``gamma`` and ``beta`` are learnable parameters. But if ``fix_gamma`` is true,
	 then set ``gamma`` to 1 and its gradient to 0.
	 

.. note::
	 When ``fix_gamma`` is set to True, no sparse support is provided. If ``fix_gamma is`` set to False,
	 the sparse tensors will fallback.
	 
	 
Usage
----------

.. code:: r

	mx.symbol.BatchNorm(...)

Arguments
------------------

+----------------------------------------+------------------------------------------------------------+
| Argument                               | Description                                                |
+========================================+============================================================+
| ``data``                               | NDArray-or-Symbol.                                         |
|                                        |                                                            |
|                                        | Input data to batch normalization                          |
+----------------------------------------+------------------------------------------------------------+
| ``gamma``                              | NDArray-or-Symbol                                          |
|                                        | gamma array                                                |
+----------------------------------------+------------------------------------------------------------+
| ``beta``                               | NDArray-or-Symbol                                          |
|                                        | beta array                                                 |
+----------------------------------------+------------------------------------------------------------+
| ``moving.mean``                        | NDArray-or-Symbol                                          |
|                                        | running mean of input                                      |
+----------------------------------------+------------------------------------------------------------+
| ``moving.var``                         | NDArray-or-Symbol                                          |
|                                        | running variance of input                                  |
+----------------------------------------+------------------------------------------------------------+
| ``eps``                                | double, optional, default=0.0010000000474974513.           |
|                                        |                                                            |
|                                        | Epsilon to prevent div 0. Must be no less than             |
|                                        | CUDNN_BN_MIN_EPSILON defined in cudnn.h when using cudnn   |
|                                        | (usually                                                   |
|                                        | 1e-5)                                                      |
+----------------------------------------+------------------------------------------------------------+
| ``momentum``                           | float, optional, default=0.899999976.                      |
|                                        |                                                            |
|                                        | Momentum for moving average                                |
+----------------------------------------+------------------------------------------------------------+
| ``fix.gamma``                          | boolean, optional, default=1.                              |
|                                        |                                                            |
|                                        | Fix gamma while training                                   |
+----------------------------------------+------------------------------------------------------------+
| ``use.global.stats``                   | boolean, optional, default=0.                              |
|                                        |                                                            |
|                                        | Whether use global moving statistics instead of local      |
|                                        | batch-norm. This will force change batch-norm into a scale |
|                                        | shift                                                      |
|                                        | operator.                                                  |
+----------------------------------------+------------------------------------------------------------+
| ``output.mean.var``                    | boolean, optional, default=0.                              |
|                                        |                                                            |
|                                        | Output the mean and inverse std                            |
+----------------------------------------+------------------------------------------------------------+
| ``axis``                               | int, optional, default='1'.                                |
|                                        |                                                            |
|                                        | Specify which shape axis the channel is specified          |
+----------------------------------------+------------------------------------------------------------+
| ``cudnn.off``                          | boolean, optional, default=0.                              |
|                                        |                                                            |
|                                        | Do not select CUDNN operator, if available                 |
+----------------------------------------+------------------------------------------------------------+
| ``min.calib.range``                    | float or None, optional, default=None.                     |
|                                        |                                                            |
|                                        | The minimum scalar value in the form of float32 obtained   |
|                                        | through calibration. If present, it will be used to by     |
|                                        | quantized batch norm op to calculate primitive scale.Note: |
|                                        | this calib_range is to calib bn                            |
|                                        | output.                                                    |
+----------------------------------------+------------------------------------------------------------+
| ``max.calib.range``                    | float or None, optional, default=None.                     |
|                                        |                                                            |
|                                        | The maximum scalar value in the form of float32 obtained   |
|                                        | through calibration. If present, it will be used to by     |
|                                        | quantized batch norm op to calculate primitive scale.Note: |
|                                        | this calib_range is to calib bn                            |
|                                        | output.                                                    |
+----------------------------------------+------------------------------------------------------------+
| ``name``                               | string, optional.                                          |
|                                        |                                                            |
|                                        | Name of the resulting symbol.                              |
+----------------------------------------+------------------------------------------------------------+

Value
----------

``out`` The result mx.symbol


Link to Source Code: http://github.com/apache/incubator-mxnet/blob/1.6.0/src/operator/nn/batch_norm.cc#L571