| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "id": "7ad28566", |
| "metadata": {}, |
| "source": [ |
| "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n", |
| "<!--- or more contributor license agreements. See the NOTICE file -->\n", |
| "<!--- distributed with this work for additional information -->\n", |
| "<!--- regarding copyright ownership. The ASF licenses this file -->\n", |
| "<!--- to you under the Apache License, Version 2.0 (the -->\n", |
| "<!--- \"License\"); you may not use this file except in compliance -->\n", |
| "<!--- with the License. You may obtain a copy of the License at -->\n", |
| "\n", |
| "<!--- http://www.apache.org/licenses/LICENSE-2.0 -->\n", |
| "\n", |
| "<!--- Unless required by applicable law or agreed to in writing, -->\n", |
| "<!--- software distributed under the License is distributed on an -->\n", |
| "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n", |
| "<!--- KIND, either express or implied. See the License for the -->\n", |
| "<!--- specific language governing permissions and limitations -->\n", |
| "<!--- under the License. -->\n", |
| "\n", |
| "\n", |
| "# Gluon2.0: Migration Guide\n", |
| "\n", |
| "## Overview\n", |
| "Since the introduction of the Gluon API in MXNet 1.x, it has superseded commonly used symbolic, module and model APIs for model development. In fact, Gluon was the first in the deep learning community to unify the flexibility of imperative programming with the performance benefits of symbolic programming, through just-in-time compilation.\n", |
| "\n", |
| "In Gluon2.0, we extend the support to MXNet NumPy and NumPy extension with simplified interface and new functionalities:\n", |
| "\n", |
| "- **Simplified hybridization with deferred compute and tracing**: Deferred compute allows the imperative execution to be used for graph construction, which allows us to unify the historic divergence of NDArray and Symbol. Hybridization now works in a simplified hybrid forward interface. Users only need to specify the computation through imperative programming. Hybridization also works through tracing, i.e. tracing the data flow of the first input data to create a graph.\n", |
| "\n", |
| "- **Data 2.0**: The new design for data loading in Gluon allows hybridizing and deploying data processing pipeline in the same way as model hybridization. The new C++ data loader improves data loading efficiency on CIFAR 10 by 50%.\n", |
| "\n", |
| "- **Distributed 2.0**: The new distributed-training design in Gluon 2.0 provides a unified distributed data parallel interface across native Parameter Server, BytePS, and Horovod, and is extensible for supporting custom distributed training libraries.\n", |
| "\n", |
| "- **Gluon Probability**: parameterizable probability distributions and sampling functions to facilitate more areas of research such as Baysian methods and AutoML.\n", |
| "\n", |
| "- **Gluon Metrics** and **Optimizers**: refactored with MXNet NumPy interface and addressed legacy issues.\n", |
| "\n", |
| "Adopting these new functionalities may or may not require modifications on your models. But don't worry, this migration guide will go through a high-level mapping from old functionality to new APIs and make Gluon2.0 migration a hassle-free experience.\n", |
| "\n", |
| "## Data Pipeline\n", |
| "**What's new**: In Gluon2.0, `MultithreadingDataLoader` is introduced to speed up the data loading pipeline. It will use the pure MXNet C++ implementation of dataloader, datasets and batchify functions. So, you can use either MXNet internal multithreading mode dataloader or python multiprocessing mode dataloader in Gluon2.0.\n", |
| "\n", |
| "**Migration Guide**: Users can continue with the traditional gluon.data.Dataloader and the C++ backend will be applied automatically.\n", |
| "\n", |
| "[Gluon2.0 dataloader](../../api/gluon/data/index.rst#mxnet.gluon.data.DataLoader) will provide a new parameter called `try_nopython`. This parameter takes a default value of None; when set to `True` the dataloader will compile the python dataloading pipeline into pure MXNet C++ implementation. The compilation is not guaranteed to support all use cases, but it will fallback to python in case of failure:\n", |
| "\n", |
| "- The dataset is not fully [supported by the backend](../../api/gluon/data/index.rst#mxnet.gluon.data.Dataset) (e.g., there are custom python datasets).\n", |
| "\n", |
| "- Transform is not fully hybridizable.\n", |
| "\n", |
| "- Bachify is not fully [supported by the backend](https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/data/batchify.py).\n", |
| "\n", |
| "\n", |
| "You can refer to [Step 5 in Crash Course](https://mxnet.apache.org/versions/master/api/python/docs/tutorials/getting-started/crash-course/5-datasets.html#New-in-MXNet-2.0:-faster-C++-backend-dataloaders) for a detailed performance increase with C++ backend.\n", |
| "\n", |
| "## Modeling\n", |
| "In Gluon2.0, users will have a brand new modeling experience with NumPy-compatible APIs and the deferred compute mechanism.\n", |
| "\n", |
| "- **NumPy-compatible programing experience**: users can build their models with MXNet implementation with NumPy array library, NumPy-compatible math operators and some neural network extension operators.\n", |
| "\n", |
| "- **Imperative-only coding experience**: with the deferred compute and tracing being introduced, users only need to specify the computation through imperative coding but can still make hybridization work. Users will no longer need to interact with symbol APIs.\n", |
| "\n", |
| "To help users migrate smoothly to use these simplified interfaces, we will provide the following guidance on how to replace legacy operators with NumPy-compatible operators, how to build models with `forward` instead of `hybrid_forward` and how to use `Parameter` class to register your parameters.\n", |
| "\n", |
| "\n", |
| "### NumPy-compatible Programming Experience\n", |
| "#### NumPy Arrays\n", |
| "MXNet [NumPy ndarray (i.e. mx.np.ndarray)](../../api/np/arrays.ndarray.rst) is a multidimensional container of items of the same type and size. Most of its properties and attributes are the same as legacy NDArrays (i.e. `mx.nd.ndarray`), so users can use the NumPy array library just as they did with legacy NDArrays. But, there are still some changes and deprecations that need attention, as mentioned below.\n", |
| "\n", |
| "**Migration Guide**:\n", |
| "\n", |
| "1. Currently, NumPy ndarray only supports `default` storage type, other storage types, like `row_sparse`, `csr` are not supported. Also, `tostype()` attribute is deprecated.\n", |
| "\n", |
| "2. Users can use `as_np_ndarray` attribute to switch from a legacy NDArray to NumPy ndarray just like this:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "e439b668", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "import mxnet as mx\n", |
| "nd_array = mx.ones((5,3))\n", |
| "np_array = nd_array.as_np_ndarray()\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "f370cd7d", |
| "metadata": {}, |
| "source": [ |
| "3. Compared with legacy NDArray, some attributes are deprecated in NumPy ndarray. Listed below are some of the deprecated APIs and their corresponding replacements in NumPy ndarray, others can be found in [**Appendix/NumPy Array Deprecated Attributes**](#NumPy-Array-Deprecated-Attributes).\n", |
| "\n", |
| "| Deprecated Attributes | NumPy ndarray Equivalent |\n", |
| "| ----------------------------------------------------- | ------------------------------ |\n", |
| "| `a.asscalar()` | `a.item()` |\n", |
| "| `a.as_in_context()` | `a.to_device()` |\n", |
| "| `a.context` | `a.device` |\n", |
| "| `a.reshape_like(b)` | `a.reshape(b.shape)` |\n", |
| "| `a.zeros_like(b)` | `mx.np.zeros_like(b)` |\n", |
| "| `a.ones_like(b)` | `mx.np.ones_like(b)` |\n", |
| "\n", |
| "\n", |
| "**NOTE**\n", |
| "\n", |
| "`Context` class has also been deprecated in MXNet2.0, it is renamed to `Device` and some related methods and attributes are also renamed as above. All the creation functions inside MXNet NumPy package will take `device` as keyword instead of `ctx`.\n", |
| "\n", |
| "\n", |
| "4. Compared with legacy NDArray, some attributes will have different behaviors and take different inputs. \n", |
| "\n", |
| "+--------------------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+\n", |
| "| Attribute | Legacy Inputs | NumPy Inputs |\n", |
| "+==================================================+==============================================================+==================================================================+\n", |
| "| a.reshape(*args, **kwargs) | **shape**: Some dimensions of the shape can take special | **shape**: shape parameter will be **positional argument** rather|\n", |
| "| | values from the set {0, -1, -2, -3, -4}. | than key-word argument. Some dimensions of the shape |\n", |
| "| | The significance of each is explained below: | can take special values from the set {-1, -2, -3, -4, |\n", |
| "| | 0 copy this dimension from the input to the output shape. | -5, -6}. |\n", |
| "| | -1 infers the dimension of the output shape by using the | The significance of each is explained below: |\n", |
| "| | remainder of the input dimensions. | -1 infers the dimension of the output shape by using the |\n", |
| "| | -2 copy all/remainder of the input dimensions to the | remainder of the input dimensions. |\n", |
| "| | output shape. | -2 copy this dimension from the input to the output shape. |\n", |
| "| | -3 use the product of two consecutive dimensions of the | -3 skip the current dimension if and only if the current dim size|\n", |
| "| | input shape as the output dimension. | is one. |\n", |
| "| | -4 split one dimension of the input into two dimensions | -4 copy all the remaining the input dimensions to the output |\n", |
| "| | passed subsequent to -4 in shape (can contain -1). | shape. |\n", |
| "| | **reverse**: If set to 1, then the special values are | -5 use the product of two consecutive dimensions of the input |\n", |
| "| | inferred from right to left | shape as the output. |\n", |
| "| | | -6 split one dimension of the input into two dimensions passed |\n", |
| "| | | subsequent to -6 in the new shape. |\n", |
| "| | | **reverse**: No **reverse** parameter for `np.reshape` but for |\n", |
| "| | | `npx.reshape`. |\n", |
| "| | | **order**: Read the elements of `a` using this index order, and |\n", |
| "| | | place the elements into the reshaped array using this |\n", |
| "| | | index order. |\n", |
| "+--------------------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+\n", |
| "\n", |
| "\n", |
| "\n", |
| "#### NumPy and NumPy-extension Operators\n", |
| "Most of the legacy NDArray operators (`mx.nd.op`) have the equivalent ones in np/npx namespace. Users can just replace them with `mx.np.op` or `mx.npx.op` to migrate. Some of the operators will have different inputs and behaviors as listed in the table below.\n", |
| "\n", |
| "**Migration Guide**:\n", |
| "\n", |
| "1. Operators migration with name/inputs changes\n", |
| "\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| Legacy Operators | NumPy Operators Equivalent | Changes |\n", |
| "+==================================================+=========================================================+=======================================================================+\n", |
| "| mx.nd.flatten(*args, **kwargs) | mx.npx.batch_flatten(*args, **kwargs) | moved to npx namespace with new name batch_flatten |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.concat(a, b, c) | mx.np.concatenate([a, b, c]) | - moved to np namespace with new name concatenate. |\n", |
| "| | | - use list of ndarrays as input rather than positional ndarrays |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.stack(a, b, c) | mx.np.stack([a, b, c]) | - moved to np namespace. |\n", |
| "| | | - use list of ndarrays as input rather than positional ndarrays |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.SliceChannel(*args, **kwargs) | mx.npx.slice_channel(*args, **kwargs) | moved to npx namespace with new name slice_channel. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.FullyConnected(*args, **kwargs) | mx.npx.fully_connected(*args, **kwargs) | moved to npx namespace with new name fully_connected. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.Activation(*args, **kwargs) | mx.npx.activation(*args, **kwargs) | moved to npx namespace with new name activation. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.elemwise_add(a, b) | a + b | Just use ndarray python operator. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| mx.nd.elemwise_mul(a, b) | mx.np.multiply(a, b) | Use multiply operator in np namespace. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "\n", |
| "2. Operators migration with multiple steps: `mx.nd.mean` -> `mx.np.mean`:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "aeb2f67a", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "import mxnet as mx\n", |
| "# Legacy: calculate mean value with reduction on axis 1\n", |
| "# with `exclude` option on \n", |
| "nd_mean = mx.nd.mean(data, axis=1, exclude=1)\n", |
| "\n", |
| "# Numpy: no exclude option to users, but user can perform steps as follow\n", |
| "axes = list(range(data.ndim))\n", |
| "del axes[1]\n", |
| "np_mean = mx.np.mean(data, axis=axes)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "453cd6b4", |
| "metadata": {}, |
| "source": [ |
| "3. Random Operators\n", |
| "\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| Legacy Operators | NumPy Operators Equivalent | Changes |\n", |
| "+==================================================+=========================================================+=======================================================================+\n", |
| "| `mx.random.uniform(-1.0, 1.0, shape=(2, 3))` | `mx.np.random.uniform(-1.0, 1.0, size=(2, 3))` | For all the NumPy random operators, use **size** keyword instead of |\n", |
| "| `mx.nd.random.uniform(-1.0, 1.0, shape=(2, 3))` | | **shape** |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| `mx.nd.random.multinomial(*args, **kwargs)` | `mx.npx.random.categorical(*args, **kwargs)` | use `npx.random.categorical` to have the behavior of drawing 1 sample from multiple distributions. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "\n", |
| "4. Control Flow Operators\n", |
| "\n", |
| "+----------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n", |
| "| Legacy Operators | NumPy Operators Equivalent | Changes |\n", |
| "+============================================================================+===========================================================================+===============================================================================================================+\n", |
| "| `mx.nd.contrib.foreach(body, data, init_states, name)` | `mx.npx.foreach(body, data, init_states, name)` | - moved to `npx` namespace. |\n", |
| "| | | - Will not support global variables as body's inputs(body's inputs must be either data or states or both) |\n", |
| "+----------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n", |
| "| `mx.nd.contrib.while_loop(cond, func, loop_vars, max_iterations, name)` | `mx.npx.while_loop(cond, func, loop_vars, max_iterations, name)` | - moved to `npx` namespace. |\n", |
| "| | | - Will not support global variables as cond or func's inputs(cond or func's inputs must be in loop_vars) |\n", |
| "+----------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n", |
| "| `mx.nd.contrib.cond(pred, then_func, else_func, inputs, name)` | `mx.npx.cond(pred, then_func, else_func, name)` | - moved to `npx` namespace. |\n", |
| "| | | - users needs to provide the inputs of pred, then_func and else_func as inputs |\n", |
| "| | | - Will not support global variables as pred, then_func or else_func's |\n", |
| "| | | inputs(pred, then_func or else_func's inputs must be in inputs) |\n", |
| "+----------------------------------------------------------------------------+---------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+\n", |
| "\n", |
| "5. Functionalities\n", |
| "\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| Legacy Operators | NumPy Operators Equivalent | Changes |\n", |
| "+==================================================+=========================================================+=======================================================================+\n", |
| "| `mx.nd.save(*args, **kwargs)` | `mx.npx.savez(*args, **kwargs)` | - moved to `npx` namespace. |\n", |
| "| | | - Only accept positional arguments, try to flatten the list/dict |\n", |
| "| | | before feed in |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| `mx.nd.load(*args, **kwargs)` | `mx.npx.load(*args, **kwargs)` | - moved to `npx` namespace. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "| `mx.nd.waitall()` | `mx.npx.waitall()` | - moved to `npx` namespace. |\n", |
| "+--------------------------------------------------+---------------------------------------------------------+-----------------------------------------------------------------------+\n", |
| "\n", |
| "Other operator changes are included in [**Appendix/NumPy and NumPy-extension Operators**](#NumPy-and-NumPy-extension-Operators1) \n", |
| "\n", |
| "\n", |
| "\n", |
| "### Layers and Blocks\n", |
| "With the deferred compute and tracing being introduced in Gluon2.0, users do not need to interact with symbols any more. There are a lot of changes in building a model with Gluon API, including parameter management and naming, forward pass computing and parameter shape inferencing. We provide step-by-step migration guidance on how to build a model with new APIs. \n", |
| "\n", |
| "#### Parameter Management and Block Naming\n", |
| "In Gluon, each Parameter or Block has a name (and prefix). Parameter names are specified by users and Block names can be either specified by users or automatically created. In Gluon 1.x, parameters are accessed via the `params` variable of the `ParameterDict` in `Block`. Users will need to manually use `with self.name_scope():` for children blocks and specify prefix for the top level block. Otherwise, it will lead to wrong name scopes and can return parameters of children blocks that are not in the current name scope. An example for initializing the Block and Parameter in Gluon 1.x:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "67c01c9e", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "from mxnet.gluon import Parameter, Constant, HybridBlock\n", |
| "class SampleBlock(HybridBlock):\n", |
| " def __init__(self):\n", |
| " super(SampleBlock, self).__init__()\n", |
| " with self.name_scope():\n", |
| " # Access parameters, which are iterated during training\n", |
| " self.weight = self.params.get('weight')\n", |
| " # Access constant parameters, which are not iterated during training\n", |
| " self.weight = self.params.get_constant('const', const_arr)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "04011860", |
| "metadata": {}, |
| "source": [ |
| "Now in Gluon 2.0, Block/HybridBlock objects will not maintain the parameter dictionary (`ParameterDict`). Instead, users can access these parameters via `Parameter` class and `Constant` class. These parameters will be registered automatically as part of the Block. Users will no longer need to manage the name scope for children blocks and hence can remove `with self.name_scope():` this statement. For example:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "a53210f7", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "class SampleBlock(HybridBlock):\n", |
| " def __init__(self):\n", |
| " super(SampleBlock, self).__init__()\n", |
| " # Access parameters, which are iterated during training\n", |
| " self.weight = Parameter('weight')\n", |
| " # Access constant parameters, which are not iterated during training\n", |
| " self.weight = Constant('const', const_arr)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "9f432aa0", |
| "metadata": {}, |
| "source": [ |
| "Also, there will be new mechanisms for parameter loading, sharing and setting device. \n", |
| "\n", |
| "1. Parameter loading in Gluon 1.x vs Gluon 2.0:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "f41a5277", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# in Gluon 1.x\n", |
| "net = nn.Dense(8, activation='relu')\n", |
| "net.collect_params().load_dict(arg_dict, ctx=ctx)\n", |
| "# in Gluon 2.0\n", |
| "net = nn.Dense(8, activation='relu')\n", |
| "net.load_dict(arg_dict, device=device)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "02fc1f0a", |
| "metadata": {}, |
| "source": [ |
| "2. Parameter sharing in Gluon 1.x vs Gluon 2.0:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "e6bcdd88", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# in Gluon 1.x\n", |
| "shared = nn.Dense(8, activation='relu')\n", |
| "net = nn.Dense(8, activation='relu', params=shared.params)\n", |
| "# in Gluon 2.0\n", |
| "shared = nn.Dense(8, activation='relu')\n", |
| "net = nn.Dense(8, activation='relu').share_parameters(shared.params)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "12d04ba1", |
| "metadata": {}, |
| "source": [ |
| "3. Parameter setting device in Gluon 1.x vs Gluon 2.0:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "7294c5c9", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# in Gluon 1.x\n", |
| "net = nn.Dense(8, activation='relu')\n", |
| "net.collect_params().reset_ctx(devices)\n", |
| "# in Gluon 2.0\n", |
| "net = nn.Dense(8, activation='relu')\n", |
| "net.reset_device(devices)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "8a220445", |
| "metadata": {}, |
| "source": [ |
| "#### Forward Interface\n", |
| "`hybrid_forward` interface in Gluon1.x provides the user with a unified imperative and symbolic programming interface to do graph construction and imperative execution. For the inputs of `hybrid_forward`, `F` can be either mx.symbol or mx.ndarray depending on the running mode(symbolic or imperative) of variable recording. Apart from `F` and input arrays, the parameters registered when Block is initialized are also required as part of the inputs. Take `nn.Dense` as an example:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "a0bb32a6", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# hybrid_forward interface, F can be either symbol or ndarray, weights\n", |
| "# and bias are part of inputs\n", |
| "def hybrid_forward(self, F, x, weight, bias=None):\n", |
| " fc = F.npx.fully_connected if is_np_array() else F.FullyConnected\n", |
| " act = fc(x, weight, bias, no_bias=bias is None, num_hidden=self._units,\n", |
| " flatten=self._flatten, name='fwd')\n", |
| " if self.act is not None:\n", |
| " act = self.act(act)\n", |
| " return act\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "3602dd07", |
| "metadata": {}, |
| "source": [ |
| "Now, in deferred computation mode of Gluon2.0, the divergence of NDArray and Symbol is unified, which means users no longer need to define `F` with specific running mode. One can easily specify the computation through imperative programming, hybridization will work through the tracing mechanism(data flow of the first input batch). What's more, users can implement the forward interface with `npx/npx` operators instead of `nd` and `symbol`." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "776d3fb1", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# forward interface, no F any more\n", |
| "def forward(self, x):\n", |
| " # get the device information of input array and make parameters run on the same device\n", |
| " device = x.device\n", |
| " # use np/npx interfaces instead of F\n", |
| " act = npx.fully_connected(x, self.weight.data(device),\n", |
| " self.bias.data(device) if self.bias is not None else None,\n", |
| " no_bias=self.bias is None,\n", |
| " num_hidden=self._units, flatten=self._flatten, name='fwd')\n", |
| " if self.act is not None:\n", |
| " act = self.act(act)\n", |
| " return act\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "e9755034", |
| "metadata": {}, |
| "source": [ |
| "#### Implement Infer Shape\n", |
| "In Gluon1.x, parameter shape inference happens in MXNet backend. Now in Gluon2.0, shape inference is disabled in the case of deferred parameter initialization. So, users should now always implement `infer_shape` method to set the parameter shapes if the parameter shape was not set during HybridBlock initialization." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "99a15570", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "def infer_shape(self, x, *args):\n", |
| " # if true, self.weight.shape[1] will be flattened of input's shape\n", |
| " if self._flatten:\n", |
| " num_input = 1\n", |
| " for i in range(1, x.ndim):\n", |
| " num_input *= x.shape[i]\n", |
| " self.weight.shape = (self.weight.shape[0], num_input)\n", |
| " # if false, self.weight.shape[1] = x.shape[-1]\n", |
| " else:\n", |
| " self.weight.shape = (self.weight.shape[0], x.shape[x.ndim - 1])\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "9c441d4e", |
| "metadata": {}, |
| "source": [ |
| "Now, in Gluon2.0, users can implement a Dense Block like this:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "9d6a7214", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "class Dense(HybridBlock):\n", |
| " def __init__(self, units, activation=None, use_bias=True, flatten=True,\n", |
| " dtype='float32', weight_initializer=None, bias_initializer='zeros',\n", |
| " in_units=0, **kwargs):\n", |
| " super(Dense, self).__init__(**kwargs)\n", |
| " self._flatten = flatten\n", |
| " self._units = units\n", |
| " self._in_units = in_units\n", |
| " self.weight = Parameter('weight', shape=(units, in_units),\n", |
| " init=weight_initializer, dtype=dtype,\n", |
| " allow_deferred_init=True)\n", |
| " if use_bias:\n", |
| " self.bias = Parameter('bias', shape=(units,),\n", |
| " init=bias_initializer, dtype=dtype,\n", |
| " allow_deferred_init=True)\n", |
| " else:\n", |
| " self.bias = None\n", |
| " if activation is not None:\n", |
| " self.act = Activation(activation)\n", |
| " else:\n", |
| " self.act = None\n", |
| "\n", |
| " def forward(self, x):\n", |
| " device = x.device\n", |
| " act = npx.fully_connected(x, self.weight.data(device),\n", |
| " self.bias.data(device) if self.bias is not None else None,\n", |
| " no_bias=self.bias is None,\n", |
| " num_hidden=self._units, flatten=self._flatten, name='fwd')\n", |
| " if self.act is not None:\n", |
| " act = self.act(act)\n", |
| " return act\n", |
| "\n", |
| " def infer_shape(self, x, *args):\n", |
| " if self._flatten:\n", |
| " num_input = 1\n", |
| " for i in range(1, x.ndim):\n", |
| " num_input *= x.shape[i]\n", |
| " self.weight.shape = (self.weight.shape[0], num_input)\n", |
| " else:\n", |
| " self.weight.shape = (self.weight.shape[0], x.shape[x.ndim - 1])\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "b2c0b571", |
| "metadata": {}, |
| "source": [ |
| "## Optimizers\n", |
| "Optimizer module in MXNet provides a lot of optimization algorithms to reduce the training error. In Gluon 2.0, optimizers will also switch to use MXNet NumPy-compatible interface. Some important changes that needs attention are: \n", |
| "\n", |
| "1. AdaGrad: \n", |
| " - use `epsilon` instead of `eps`\n", |
| " - e.g. `adagrad_optimizer = optimizer.AdaGrad(learning_rate=0.1, epsilon=1e-07)`\n", |
| "\n", |
| "2. RMSProp:\n", |
| " - use `rho` instead of `gamma1` and use `momentum` instead of `gamma2`\n", |
| " - e.g. `rmsprop_optimizer = optimizer.RMSProp(learning_rate=0.001, rho=0.9, momentum=0.9, epsilon=1e-07, centered=False)`\n", |
| "\n", |
| "3. `optimizer.ccSGD` and `optimizer.LBSGD` are deprecated.\n", |
| "\n", |
| "## Metrics\n", |
| "Metrics module in MXNet provides different methods for users to judge the performance of models. In Gluon 2.0, metrics will use MXNet NumPy-compatible interface and also introduce a lot of new evaluation metrics.\n", |
| "**Changes**:\n", |
| "1. metric module has been moved to gluon namespace\n", |
| " - `mxnet.metric` -> `mxnet.gluon.metric`\n", |
| "\n", |
| "2. Add new evaluation metrics: \n", |
| " - `Class BinaryAccuracy(threshold=0.5)`\n", |
| " - `Class MeanCosineSimilarity(axis=-1, eps=1e-12)`\n", |
| " - `Class MeanPairwiseDistance(p=2)`\n", |
| " - `Class Fbeta(class_type=\"binary\", beta=1, threshold=0.5, average=\"micro\")`\n", |
| "\n", |
| "3. Improve Class F1\n", |
| " - `Class F1(name='f1',output_names=None, label_names=None, average=\"macro\")` to\n", |
| " `Class F1(name='f1',output_names=None, label_names=None, class_type=\"binary\", threshold=0.5, average=\"micro\")`\n", |
| " - **average**: Strategy to be used for aggregating across mini-batches.\n", |
| " - \"macro\": Calculate metrics for each label and return unweighted mean of f1.\n", |
| " - \"micro\": Calculate metrics globally by counting the total TP, FN and FP.\n", |
| " - None: Return f1 scores for each class (numpy.ndarray).\n", |
| " - **class_type**:\n", |
| " - \"binary\": f1 for binary classification.\n", |
| " - \"multiclass\": f1 for multiclassification problem.\n", |
| " - \"multilabel\": f1 for multilabel classification.\n", |
| " - **threshold**: threshold for postive confidence value.\n", |
| "\n", |
| "\n", |
| "## Key-Value Store\n", |
| "Gluon 2.0 will provide a new and unified low level API for data parallel training. These unified APIs can support different communication backends, including native Parameter Server, Horovod and BytePS. \n", |
| "Example:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "e86d508a", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "import mxnet as mx\n", |
| "# create key-value store with horovod backend\n", |
| "kv = mx.kv.create('horovod') # or choose 'kvstore', 'byteps' as backend\n", |
| "device = mx.gpu(kv.local_rank) if mx.device.num_gpus() > 0 else mx.cpu(kv.local_rank)\n", |
| "val = mx.np.zeros((2, 3), device=device)\n", |
| "# broadcast the value at rank 0 to all ranks\n", |
| "kv.broadcast('0', mx.np.zeros((2, 3), device=device), out=val)\n", |
| "scale = kv.rank + 1\n", |
| "# performs allreduce on a single array\n", |
| "kv.pushpull('3', val * scale)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "75cdc58b", |
| "metadata": {}, |
| "source": [ |
| "## Probability\n", |
| "A new module called `mxnet.gluon.probability` has been introduced in Gluon 2.0. It is analogous to pytorch distribution and the main difference is that `mxnet.gluon.probability` will use MXNet NumPy compatible operators and will allow hybridization. It has three parts: \n", |
| "\n", |
| "1. [Distribution Objects](https://github.com/apache/incubator-mxnet/tree/master/python/mxnet/gluon/probability/distributions): `gluon.probability.Bernoulli`, `gluon.probability.Beta` ...\n", |
| "\n", |
| "2. [StochasticBlock](https://github.com/apache/incubator-mxnet/tree/master/python/mxnet/gluon/probability/block): support accumulating loss in the forward phase, which is useful in building Bayesian Neural Network. \n", |
| "\n", |
| "3. [Transformation](https://github.com/apache/incubator-mxnet/tree/master/python/mxnet/gluon/probability/transformation): implement invertible transformation with computable log det jacobians.\n", |
| "\n", |
| "## oneDNN Integration\n", |
| "### Operator Fusion\n", |
| "In versions 1.x of MXNet pattern fusion in execution graph was enabled by default when using MXNet built with oneDNN library support and could have been disabled by setting 'MXNET_SUBGRAPH_BACKEND' environment flag to `None`. MXNet 2.0 introduced changes in forward inference flow which led to refactor of fusion mechanism. To fuse model in MXNet 2.0 there are two requirements:\n", |
| "\n", |
| " - the model must be defined as a subclass of HybridBlock or Symbol,\n", |
| "\n", |
| " - the model must have specific operator patterns which can be fused.\n", |
| "\n", |
| "Both HybridBlock and Symbol classes provide API to easily run fusion of operators. Adding only one line of code is needed to run fusion passes on model:" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "3902bfd2", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "# on HybridBlock\n", |
| "net.optimize_for(data, backend='ONEDNN')\n", |
| "# on Symbol\n", |
| "optimized_symbol = sym.optimize_for(backend='ONEDNN')\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "7a833368", |
| "metadata": {}, |
| "source": [ |
| "Controling which patterns should be fused still can be done by setting proper environment variables. See [**oneDNN Environment Variables**](#oneDNN-Environment-Variables)\n", |
| "\n", |
| "### INT8 Quantization / Precision reduction\n", |
| "Quantization API was also refactored to be consistent with other new features and mechanisms. In comparison to MXNet 1.x releases, in MXNet 2.0 `quantize_net_v2` function has been removed and development focused mainly on `quantize_net` function to make it easier to use for end user and ultimately give him more flexibility.\n", |
| "Quantization can be performed on either subclass of HybridBlock with `quantize_net` or Symbol with deprecated `quantize_model` (`quantize_model` is left only to provide backward compatibility and its usage is strongly discouraged)." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "1b1a6913", |
| "metadata": {}, |
| "source": [ |
| "```{.python}\n", |
| "import mxnet as mx\n", |
| "from mxnet.contrib.quantization import quantize_net\n", |
| "from mxnet.gluon.model_zoo.vision import resnet50_v1\n", |
| "\n", |
| "# load model\n", |
| "net = resnet50_v1(pretrained=True)\n", |
| "\n", |
| "# prepare calibration data\n", |
| "dummy_data = mx.nd.random.uniform(-1.0, 1.0, (batch_size, 3, 224, 224))\n", |
| "calib_data_loader = mx.gluon.data.DataLoader(dummy_data, batch_size=batch_size)\n", |
| "\n", |
| "# quantization\n", |
| "qnet = quantize_net(net, calib_mode='naive', calib_data=calib_data_loader)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "06099335", |
| "metadata": {}, |
| "source": [ |
| "`quantize_net` can be much more complex - all function attributes can be found in the [API](../../api/contrib/quantization/index.rst).\n", |
| "\n", |
| "### oneDNN Environment Variables\n", |
| "In version 2.0 of MXNet all references to MKLDNN (former name of oneDNN) were replaced by ONEDNN. Below table lists all environment variables:\n", |
| "\n", |
| "| MXNet 1.x | MXNet 2.0 |\n", |
| "| ------------------------------------ | ---------------------------------------|\n", |
| "| MXNET_MKLDNN_ENABLED | MXNET_ONEDNN_ENABLED |\n", |
| "| MXNET_MKLDNN_CACHE_NUM | MXNET_ONEDNN_CACHE_NUM |\n", |
| "| MXNET_MKLDNN_FORCE_FC_AB_FORMAT | MXNET_ONEDNN_FORCE_FC_AB_FORMAT |\n", |
| "| MXNET_MKLDNN_ENABLED | MXNET_ONEDNN_ENABLED |\n", |
| "| MXNET_MKLDNN_DEBUG | MXNET_ONEDNN_DEBUG |\n", |
| "| MXNET_USE_MKLDNN_RNN | MXNET_USE_ONEDNN_RNN |\n", |
| "| MXNET_DISABLE_MKLDNN_CONV_OPT | MXNET_DISABLE_ONEDNN_CONV_OPT |\n", |
| "| MXNET_DISABLE_MKLDNN_FUSE_CONV_BN | MXNET_DISABLE_ONEDNN_FUSE_CONV_BN |\n", |
| "| MXNET_DISABLE_MKLDNN_FUSE_CONV_RELU | MXNET_DISABLE_ONEDNN_FUSE_CONV_RELU |\n", |
| "| MXNET_DISABLE_MKLDNN_FUSE_CONV_SUM | MXNET_DISABLE_ONEDNN_FUSE_CONV_SUM |\n", |
| "| MXNET_DISABLE_MKLDNN_FC_OPT | MXNET_DISABLE_ONEDNN_FC_OPT |\n", |
| "| MXNET_DISABLE_MKLDNN_FUSE_FC_ELTWISE | MXNET_DISABLE_ONEDNN_FUSE_FC_ELTWISE |\n", |
| "| MXNET_DISABLE_MKLDNN_TRANSFORMER_OPT | MXNET_DISABLE_ONEDNN_TRANSFORMER_OPT |\n", |
| "| n/a | MXNET_DISABLE_ONEDNN_BATCH_DOT_FUSE |\n", |
| "| n/a | MXNET_ONEDNN_FUSE_REQUANTIZE |\n", |
| "| n/a | MXNET_ONEDNN_FUSE_DEQUANTIZE |\n", |
| "\n", |
| "## Appendix\n", |
| "### NumPy Array Deprecated Attributes\n", |
| "| Deprecated Attributes | NumPy ndarray Equivalent |\n", |
| "| ----------------------------------------------------- | ------------------------------ |\n", |
| "| `a.abs()` | `mx.np.abs(a)` |\n", |
| "| `a.sign()` | `mx.np.sign(a)` |\n", |
| "| `a.split_v2(2, axis=1)` | `mx.np.split(a, 2, axis=1)` |\n", |
| "| `a.flip(*args, **kwargs)` | `mx.np.flip(a, *args, **kwargs)` |\n", |
| "| `a.diag(*args, **kwargs)` | `mx.np.diag(a, *args, **kwargs)` |\n", |
| "| `a.nansum(*args, **kwargs)` | `mx.np.nan_to_num(a, *args, **kwargs).sum()` |\n", |
| "| `a.nanprod(*args, **kwargs)` | `mx.np.nan_to_num(a, *args, **kwargs).prod()` |\n", |
| "| `a.diag(*args, **kwargs)` | `mx.np.diag(a, *args, **kwargs)` |\n", |
| "| `a.norm()` | `mx.npx.norm(a)` |\n", |
| "| `a.rint(*args, **kwargs)` | `mx.np.rint(a, *args, **kwargs)` |\n", |
| "| `a.fix(*args, **kwargs)` | `mx.np.fix(a, *args, **kwargs)` |\n", |
| "| `a.floor(*args, **kwargs)` | `mx.np.floor(a, *args, **kwargs)` |\n", |
| "| `a.ceil(*args, **kwargs)` | `mx.np.ceil(a, *args, **kwargs)` |\n", |
| "| `a.trunc(*args, **kwargs)` | `mx.np.trunc(a, *args, **kwargs)` |\n", |
| "| `a.sin(*args, **kwargs)` | `mx.np.sin(a, *args, **kwargs)` |\n", |
| "| `a.cos(*args, **kwargs)` | `mx.np.cos(a, *args, **kwargs)` |\n", |
| "| `a.tan(*args, **kwargs)` | `mx.np.tan(a, *args, **kwargs)` |\n", |
| "| `a.arcsin(*args, **kwargs)` | `mx.np.arcsin(a, *args, **kwargs)` |\n", |
| "| `a.arccos(*args, **kwargs)` | `mx.np.arccos(a, *args, **kwargs)` |\n", |
| "| `a.arctan(*args, **kwargs)` | `mx.np.arctan(a, *args, **kwargs)` |\n", |
| "| `a.degrees(*args, **kwargs)` | `mx.np.degrees(a, *args, **kwargs)` |\n", |
| "| `a.radians(*args, **kwargs)` | `mx.np.radians(a, *args, **kwargs)` |\n", |
| "| `a.sinh(*args, **kwargs)` | `mx.np.sinh(a, *args, **kwargs)` |\n", |
| "| `a.cosh(*args, **kwargs)` | `mx.np.cosh(a, *args, **kwargs)` |\n", |
| "| `a.tanh(*args, **kwargs)` | `mx.np.tanh(a, *args, **kwargs)` |\n", |
| "| `a.arcsinh(*args, **kwargs)` | `mx.np.arcsinh(a, *args, **kwargs)` |\n", |
| "| `a.arccosh(*args, **kwargs)` | `mx.np.arccosh(a, *args, **kwargs)` |\n", |
| "| `a.arctanh(*args, **kwargs)` | `mx.np.arctanh(a, *args, **kwargs)` |\n", |
| "| `a.exp(*args, **kwargs)` | `mx.np.exp(a, *args, **kwargs)` |\n", |
| "| `a.expm1(*args, **kwargs)` | `mx.np.expm1(a, *args, **kwargs)` |\n", |
| "| `a.log(*args, **kwargs)` | `mx.np.log(a, *args, **kwargs)` |\n", |
| "| `a.log10(*args, **kwargs)` | `mx.np.log10(a, *args, **kwargs)` |\n", |
| "| `a.log2(*args, **kwargs)` | `mx.np.log2(a, *args, **kwargs)` |\n", |
| "| `a.log1p(*args, **kwargs)` | `mx.np.log1p(a, *args, **kwargs)` |\n", |
| "| `a.sqrt(*args, **kwargs)` | `mx.np.sqrt(a, *args, **kwargs)` |\n", |
| "| `a.rsqrt(*args, **kwargs)` | `1 / mx.np.sqrt(a, *args, **kwargs)` |\n", |
| "| `a.cbrt(*args, **kwargs)` | `mx.np.cbrt(a, *args, **kwargs)` |\n", |
| "| `a.rcbrt(*args, **kwargs)` | `1 / mx.np.cbrt(a, *args, **kwargs)` |\n", |
| "| `a.square(*args, **kwargs)` | `mx.np.square(a, *args, **kwargs)` |\n", |
| "| `a.pad(*args, **kwargs)` | `mx.npx.pad(a, *args, **kwargs)` |\n", |
| "| `a.split(axis=1, num_outputs=2)` | `mx.np.split(a, 2, axis=1)` |\n", |
| "| `a.slice(*args, **kwargs)` | `mx.npx.slice(a, *args, **kwargs)` |\n", |
| "| `a.one_hot(*args, **kwargs)` | `mx.npx.one_hot(a, *args, **kwargs)` |\n", |
| "| `a.pick(*args, **kwargs)` | `mx.npx.pick(a, *args, **kwargs)` |\n", |
| "| `a.topk(*args, **kwargs)` | `mx.npx.topk(a, *args, **kwargs)` |\n", |
| "| `a.shape_array()` | `mx.np.array(a.shape)` |\n", |
| "| `a.size_array()` | `mx.np.array(a.size)` |\n", |
| "| `a.expand_dims(*args, **kwargs)` | `mx.np.expand_dims(a, *args, **kwargs)` |\n", |
| "| `a.relu(*args, **kwargs)` | `mx.npx.relu(a, *args, **kwargs)` |\n", |
| "| `a.sigmoid(*args, **kwargs)` | `mx.npx.sigmoid(a, *args, **kwargs)` |\n", |
| "| `a.softmax(*args, **kwargs)` | `mx.npx.softmax(a, *args, **kwargs)` |\n", |
| "| `a.log_softmax(*args, **kwargs)` | `mx.npx.log_softmax(a, *args, **kwargs)` |\n", |
| "| `a.broadcast_like(*args, **kwargs)` | `mx.npx.broadcast_like(a, *args, **kwargs)` |\n", |
| "| `a.reciprocal(*args, **kwargs)` | `mx.np.reciprocal(a, *args, **kwargs)` |\n", |
| "\n", |
| "### NumPy and NumPy-extension Operators\n", |
| "| Legacy Operators | NumPy Operators Equivalent | Changes |\n", |
| "| ----------------------------------------------------- | ------------------------------ | ------------------- |\n", |
| "| `mx.nd.softmax(*args, **kwargs)` | `mx.npx.softmax(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.log_softmax(*args, **kwargs)` | `mx.npx.log_softmax(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.masked_softmax(*args, **kwargs)` | `mx.npx.masked_softmax(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.masked_log_softmax(*args, **kwargs)` | `mx.npx.masked_log_softmax(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.pick(*args, **kwargs)` | `mx.npx.pick(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.topk(*args, **kwargs)` | `mx.npx.topk(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.batch_dot(*args, **kwargs)` | `mx.npx.batch_dot(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.broadcast_like(*args, **kwargs)` | `mx.npx.broadcast_like(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.arange_like(*args, **kwargs)` | `mx.npx.arange_like(*args, **kwargs)` | moved to `npx` namespace |\n", |
| "| `mx.nd.BatchNorm(*args, **kwargs)` | `mx.npx.batch_norm(*args, **kwargs)` | - moved to `npx` namespace with new name `batch_norm`. |\n", |
| "| `mx.nd.Convolution(*args, **kwargs)` | `mx.npx.convolution(*args, **kwargs)` | - moved to `npx` namespace with new name `convolution`. |\n", |
| "| `mx.nd.Deconvolution(*args, **kwargs)` | `mx.npx.deconvolution(*args, **kwargs)` | - moved to `npx` namespace with new name `deconvolution`. |\n", |
| "| `mx.nd.Pooling(*args, **kwargs)` | `mx.npx.pooling(*args, **kwargs)` | - moved to `npx` namespace with new name `pooling`. |\n", |
| "| `mx.nd.Dropout(*args, **kwargs)` | `mx.npx.dropout(*args, **kwargs)` | - moved to `npx` namespace with new name `dropout`. |\n", |
| "| `mx.nd.RNN(*args, **kwargs)` | `mx.npx.rnn(*args, **kwargs)` | - moved to `npx` namespace with new name `rnn`. |\n", |
| "| `mx.nd.Embedding(*args, **kwargs)` | `mx.npx.embedding(*args, **kwargs)` | - moved to `npx` namespace with new name `embedding`. |\n", |
| "| `mx.nd.LayerNorm(*args, **kwargs)` | `mx.npx.layer_norm(*args, **kwargs)` | - moved to `npx` namespace with new name `layer_norm`. |\n", |
| "| `mx.nd.LeakyReLU(*args, **kwargs)` | `mx.npx.leaky_relu(*args, **kwargs)` | - moved to `npx` namespace with new name `leaky_relu`. |\n", |
| "| `mx.nd.GroupNorm(*args, **kwargs)` | `mx.npx.group_norm(*args, **kwargs)` | - moved to `npx` namespace with new name `group_norm`. |\n", |
| "\n", |
| "## Reference\n", |
| "\n", |
| "1. [Next Generation of GluonNLP](https://github.com/dmlc/gluon-nlp/tree/master)\n", |
| "2. [MXNet NumPy-compatible coding experience](https://github.com/apache/incubator-mxnet/issues/14253)\n", |
| "3. [Gluon Data API Extension](https://github.com/apache/incubator-mxnet/issues/17269)\n", |
| "4. [Simplifying MXNet Gluon APIs](https://github.com/apache/incubator-mxnet/issues/18412)\n", |
| "5. [Deferred Compute and Tracing](https://github.com/apache/incubator-mxnet/issues/16376)\n", |
| "6. [MXNet Metrics Improvements](https://github.com/apache/incubator-mxnet/issues/18046)\n", |
| "7. [Gluon Distribution Module](https://github.com/apache/incubator-mxnet/issues/17240)" |
| ] |
| } |
| ], |
| "metadata": { |
| "language_info": { |
| "name": "python" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 5 |
| } |
