MXNet Change Log
New Features - Gluon RNN layers are now HybridBlocks
- In this release, Gluon RNN layers such as
HybridBlocks as part of gluon.rnn improvements project (#11482).
- This is the result of newly available fused RNN operators added for CPU: LSTM(#10104), vanilla RNN(#11399), GRU(#10311)
- Now many dynamic networks that are based on Gluon RNN layers can now be completely hybridized, exported, and used in the inference APIs in other language bindings such as R, Scala, etc.
- Introducing more functionality support for MKL-DNN as follows:
- Added support for more activation functions like, “sigmoid”, “tanh”, “softrelu”. (#10336)
- Added Debugging functionality: Result check (#12069) and Backend switch (#12058).
New Features - Gluon Model Zoo Pre-trained Models
- Gluon Vision Model Zoo now provides MobileNetV2 pre-trained models (#10879) in addition to AlexNet, DenseNet, Inception V3, MobileNetV1, ResNet V1 and V2, SqueezeNet 1.0 and 1.1, and VGG pretrained models.
- Updated pre-trained models provide state-of-the-art performance on all resnetv1, resnetv2, and vgg16, vgg19, vgg16_bn, vgg19_bn models (#11327 #11860 #11830).
New Features - Clojure package (experimental)
- MXNet now supports the Clojure programming language. The MXNet Clojure package brings flexible and efficient GPU computing and state-of-art deep learning to Clojure. It enables you to write seamless tensor/matrix computation with multiple GPUs in Clojure. It also lets you construct and customize the state-of-art deep learning models in Clojure, and apply them to tasks, such as image classification and data science challenges.(#11205)
- Checkout examples and API documentation here.
New Features - Synchronized Cross-GPU Batch Norm (experimental)
- Gluon now supports Synchronized Batch Normalization (#11502).
- This enables stable training on large-scale networks with high memory consumption such as FCN for image segmentation.
New Features - Sparse Tensor Support for Gluon (experimental)
- Sparse gradient support is added to
sparse_grad=True to enable when constructing the Embedding block. (#10924)
- Gluon Parameter now supports “row_sparse” storage type, which reduces communication cost and memory consumption for multi-GPU training for large models.
gluon.contrib.nn.SparseEmbedding is an example empowered by this. (#11001, #11429)
- Gluon HybridBlock now supports hybridization with sparse operators (#11306).
New Features - Control flow operators (experimental)
- This is the first step towards optimizing dynamic neural networks with variable computation graphs, by adding symbolic and imperative control flow operators. Proposal.
- New operators introduced: foreach(#11531), while_loop(#11566), cond(#11760).
New Features - Scala API Improvements (experimental)
- Improvements to MXNet Scala API usability(#10660, #10787, #10991)
- Symbol.api and NDArray.api would bring new set of functions that have complete definition for all arguments.
- Please see this Type safe API design document for more details.
New Features - Rounding GPU Memory Pool for dynamic networks with variable-length inputs and outputs (experimental)
- MXNet now supports a new memory pool type for GPU memory (#11041).
- Unlike the default memory pool requires exact size match to reuse released memory chunks, this new memory pool uses exponential-linear rounding so that similar sized memory chunks can all be reused, which is more suitable for all the workloads with dynamic-shape inputs and outputs. Set environment variable
MXNET_GPU_MEM_POOL_TYPE=Round to enable.
New Features - Topology-aware AllReduce (experimental)
- This features uses trees to perform the Reduce and Broadcast. It uses the idea of minimum spanning trees to do a binary tree Reduce communication pattern to improve it. This topology aware approach reduces the existing limitations for single machine communication shown by mehods like parameter server and NCCL ring reduction. It is an experimental feature (#11591).
- Paper followed for implementation: Optimal message scheduling for aggregation.
- Set environment variable
MXNET_KVSTORE_USETREE=1 to enable.
New Features - Export MXNet models to ONNX format (experimental)
- With this feature, now MXNet models can be exported to ONNX format(#11213). Currently, MXNet supports ONNX v1.2.1. API documentation.
- Checkout this tutorial which shows how to use MXNet to ONNX exporter APIs. ONNX protobuf so that those models can be imported in other frameworks for inference.
New Features - TensorRT Runtime Integration (experimental)
- TensorRT provides significant acceleration of model inference on NVIDIA GPUs compared to running the full graph in MxNet using unfused GPU operators. In addition to faster fp32 inference, TensorRT optimizes fp16 inference, and is capable of int8 inference (provided the quantization steps are performed). Besides increasing throughput, TensorRT significantly reduces inference latency, especially for small batches.
- This feature in MXNet now introduces runtime integration of TensorRT into MXNet, in order to accelerate inference.(#11325)
- Currently, its in contrib package.
New Examples - Scala
- Refurnished Scala Examples with improved API, documentation and CI test coverage. (#11753, #11621 )
- Now all Scala examples have:
- No bugs block in the middle
- Good Readme to start with
- with Type-safe API usage inside
- monitored in CI in each PR runs
Maintenance - Flaky Tests improvement effort
- Fixed 130 flaky tests on CI. Tracked progress of the project here.
- Add flakiness checker (#11572)
Maintenance - MXNet Model Backwards Compatibility Checker
- This tool (#11626) helps in ensuring consistency and sanity while performing inference on the latest version of MXNet using models trained on older versions of MXNet.
- This tool will help in detecting issues earlier in the development cycle which break backwards compatibility on MXNet and would contribute towards ensuring a healthy and stable release of MXNet.
Maintenance - Integrated testing for “the Straight Dope”
- “Deep Learning - The Straight Dope” is a deep learning book based on Apache MXNet Gluon that are contributed by many Gluon users.
- Now the testing of this book is integrated in the nightly tests.
- Fix gperftools/jemalloc and lapack warning bug. (#11110)
- Fix mkldnn performance regression + improve test logging (#11262)
- Fix row_sparse_param.save() (#11266)
- Fix trainer init_kvstore (#11266)
- Fix axis Bug in MKLDNN Softmax (#11335)
- Fix ‘AttributeError: ‘_thread._local’ object has no attribute ‘value’’ on distributed processing applications (#11332)
- Fix recordfile dataset with multi worker (#11370)
- Manually check node existence in CachedOp (#11545)
- Javadoc fix (#11239)
- Fix bugs in MKLDNN operators to handle the kAddTo request (#11129)
- Fix InferStorage for sparse fallback in FullyConnected (#11498)
- Fix batchnorm problem with sparse matrices when fix_gamma=True (#11656)
- Fix rnn layer save (#11776)
- Fix BucketSentenceIter bug related to #11430 (#11580)
- Fix for _backward_softsign activation (#11827)
- Fix a bug in CachedOp. (#11675)
- Fix quantization divide by zero errors (#11833)
- Refactor R optimizers to fix memory leak (#11374)
- Avoid use of troublesome cudnnFind() results when grad_req=‘add’ (#11338)
- Fix shared memory with gluon dataloader, add option pin_memory (#11908)
- Fix quantized graph pass bug (#11937)
- Fix MXPredReshape in the c_predict_api (#11493)
- Fix the topk regression issue (#12197)
- Fix image-classification example and add missing optimizers w/ momentum support (#11826)
- Added static allocation and static shape for HybridBloc gluon (#11320)
- Fix RecordIO augmentation speed (#11474)
- Improve sparse pull performance for gluon trainer (#11429)
- CTC operator performance improvement from HawkAaron/MXNet-CTC (#11834)
- Improve performance of broadcast ops backward pass (#11252)
- Improved numerical stability as a result of using stable L2 norm (#11573)
- Accelerate the performance of topk for GPU and CPU side (#12085 #10997 ; This changes the behavior of topk when nan values occur in the input)
- Support for dot(dns, csr) = dns and dot(dns, csr.T) = dns on CPU (#11113)
- Performance improvement for Batch Dot on CPU from mshadow (mshadow PR#342)
- Allow Scala users to specify data/label names for NDArrayIter (#11256)
- Allow user to define unknown token symbol to rnn encode_sentences() (#10461)
- Added count_include_pad argument for Avg Pooling (#11021)
- Add standard ResNet data augmentation for ImageRecordIter (#11027)
- Add seed_aug parameter for ImageRecordIter to fix random seed for default augmentation (#11247)
- Add support for accepting MXNet NDArrays in ColorNormalizeAug (#11606)
- Enhancement of take operator (#11326)
- Add temperature parameter in Softmax operator (#11466)
- Add support for 1D inputs in leaky relu (#11850)
- Add verify_ssl option to gluon.utils.download (#11546)
- Added ccache reporting to CI (#11322)
- Restructure dockcross dockerfiles to fix caching (#11302)
- Added tests for MKLDNN backward operators (#11232)
- Add elemwise_add/sub between rsp and rsp on GPU (#11179)
- Add clip_global_norm(row_sparse_grad) (#11266)
- Add subgraph storage type inference to CachedOp (#11306)
- Enable support for dense weight and sparse grad Adagrad updates (#11355)
- Added Histogram Operator (#10931)
- Added Matthew's Correlation Coefficient to metrics (#10524)
- Added support for add_n(dense, csr, dense) = dense on CPU & GPU (#11330)
- Added support for add_n(any combination longer than 4 with at least one dense storage) = dense on CPU & GPU (#11330)
- L1 Normalization (#11229)
- Add support for int64 data type in CSVIter (#11446)
- Add test for new int64 type in CSVIter (#11499)
- Add sample ratio for ROI Align (#11145)
- Shape and Size Operator (#10889)
- Add HybidSequentialRNNCell, which can be nested in HybridBlock (#11003)
- Support for a bunch of unary functions for csr matrices (#11559)
- Added NDArrayCollector to dispose intermediate allocated NDArrays automatically (#11751)
- Added the diag() operator (#11643)
- Added broadcast_like operator (#11820)
- Allow Partial shape infer for Slice (#11406)
- Added support to profile kvstore server during distributed training (#11215)
- Add function for GPU Memory Query to C API (#12083)
- Generalized reshape_like operator to be more flexible (#11928)
- Add support for selu activation function (#12059)
- Add support for accepting NDArray as input to Module predict API (#12166)
- Add DataDesc type for the Scala Package (#11844)
- Added NDArray auto-collector for Scala (#11751, #12232)
- Added docs for mx.initializer.Constant (#10637)
- Added build from souce instructions on windows (#11276)
- Added a tutorial explaining how to use the profiler (#11274)
- Added two tutorials on Learning Rate Schedules (#11296)
- Added a tutorial for mixed precision training with float16 (#10391)
- Create CPP test for concat MKLDNN operator (#11371)
- Update large word language model example (#11405)
- MNIST Examples for Scala new API (#11250)
- Updated installation info to have latest packages and more clarity (#11503)
- GAN MNIST Examples for Scala new API (#11547)
- Added Learning Rate Finder tutorial (#11304)
- Fix Installation instructions for R bindings on Linux systems. (#11590)
- Integration Test for Scala (#11596)
- Documentation enhancement for optimizers (#11657)
- Update rcnn example (#11373)
- Gluon ModelZoo, Gluon examples for Perl APIs (#11642)
- Fix R installation in CI (#11761,#11755, #11768, #11805, #11954, #11976)
- CNN Examples for Scala new API (#11292)
- Custom Operator Example for Scala (#11401)
- Added detailed doc about global pool layers in Gluon (#11832)
- Updated MultiTask example to use new infer api (#11605)
- Added logistic regression tutorial (#11651)
- Added Support for integer type in ImageIter (#11864)
- Added depth_to_space and space_to_depth operators (#11587)
- Increased operator support for ONNX to MXNet importer (#11856)
- Add linux and macos MKLDNN Building Instruction (#11049)
- Add download utility for Scala APIs (#11866)
- Improving documentation and error messages for Async distributed training with Gluon (#11910)
- Added NeuralStyle Example for Scala (#11621)
For more information and examples, see full release notes
New Features - Added Scala Inference APIs
- Implemented new Scala Inference APIs which offer an easy-to-use, Scala Idiomatic and thread-safe high level APIs for performing predictions with deep learning models trained with MXNet (#9678). Implemented a new ImageClassifier class which provides APIs for classification tasks on a Java BufferedImage using a pre-trained model you provide (#10054). Implemented a new ObjectDetector class which provides APIs for object and boundary detections on a Java BufferedImage using a pre-trained model you provide (#10229).
New Features - Added a Module to Import ONNX models into MXNet
- Implemented a new ONNX module in MXNet which offers an easy to use API to import ONNX models into MXNet's symbolic interface (#9963). Checkout the example on how you could use this API to import ONNX models and perform inference on MXNet. Currently, the ONNX-MXNet Import module is still experimental. Please use it with caution.
New Features - Added Support for Model Quantization with Calibration
- Implemented model quantization by adopting the TensorFlow approach with calibration by borrowing the idea from Nvidia's TensorRT. The focus of this work is on keeping quantized models (ConvNets for now) inference accuracy loss under control when compared to their corresponding FP32 models. Please see the example on how to quantize a FP32 model with or without calibration (#9552). Currently, the Quantization support is still experimental. Please use it with caution.
New Features - MKL-DNN Integration
- MXNet now integrates with Intel MKL-DNN to accelerate neural network operators: Convolution, Deconvolution, FullyConnected, Pooling, Batch Normalization, Activation, LRN, Softmax, as well as some common operators: sum and concat (#9677). This integration allows NDArray to contain data with MKL-DNN layouts and reduces data layout conversion to get the maximal performance from MKL-DNN. Currently, the MKL-DNN integration is still experimental. Please use it with caution.
New Features - Added Exception Handling Support for Operators
- Implemented Exception Handling Support for Operators in MXNet. MXNet now transports backend C++ exceptions to the different language front-ends and prevents crashes when exceptions are thrown during operator execution (#9681).
New Features - Enhanced FP16 support
- Added support for distributed mixed precision training with FP16. It supports storing of master copy of weights in float32 with the multi_precision mode of optimizers (#10183). Improved speed of float16 operations on x86 CPU by 8 times through F16C instruction set. Added support for more operators to work with FP16 inputs (#10125, #10078, #10169). Added a tutorial on using mixed precision with FP16 (#10391).
New Features - Added Profiling Enhancements
- Enhanced built-in profiler to support native Intel:registered: VTune:tm: Amplifier objects such as Task, Frame, Event, Counter and Marker from both C++ and Python -- which is also visible in the Chrome tracing view(#8972). Added Runtime tracking of symbolic and imperative operators as well as memory and API calls. Added Tracking and dumping of aggregate profiling data. Profiler also no longer affects runtime performance when not in use.
- Changed Namespace for MXNet scala from
- Changed API for the Pooling operator from
mxnet.symbol.Pooling(data=None, global_pool=_Null, cudnn_off=_Null, kernel=_Null, pool_type=_Null, pooling_convention=_Null, stride=_Null, pad=_Null, name=None, attr=None, out=None, **kwargs) to
mxnet.symbol.Pooling(data=None, kernel=_Null, pool_type=_Null, global_pool=_Null, cudnn_off=_Null, pooling_convention=_Null, stride=_Null, pad=_Null, name=None, attr=None, out=None, **kwargs). This is a breaking change when kwargs are not provided since the new api expects the arguments starting from
global_pool at the fourth position instead of the second position. (#10000).
- Fixed tests - Flakiness/Bugs - (#9598, #9951, #10259, #10197, #10136, #10422). Please see: Tests Improvement Project
cudnn_deconv deadlock (#10392).
- Fixed a race condition in
io.LibSVMIter when batch size is large (#10124).
- Fixed a race condition in converting data layouts in MKL-DNN (#9862).
- Fixed MKL-DNN sigmoid/softrelu issue (#10336).
- Fixed incorrect indices generated by device row sparse pull (#9887).
- Fixed cast storage support for same stypes (#10400).
- Fixed uncaught exception for bucketing module when symbol name not specified (#10094).
- Fixed regression output layers (#9848).
- Fixed crash with
sample_multinomial crash when
- Fixed buggy type inference in correlation (#10135).
- Fixed race condition for
CPUSharedStorageManager->Free and launched workers at iter init stage to avoid frequent relaunch (#10096).
- Fixed DLTensor Conversion for int64 (#10083).
- Fixed issues where hex symbols of the profiler were not being recognized by chrome tracing tool(#9932)
- Fixed crash when profiler was not enabled (#10306)
- Fixed ndarray assignment issues (#10022, #9981, #10468).
- Fixed incorrect indices generated by device row sparse pull (#9887).
print_summary bug in visualization module (#9492).
- Fixed shape mismatch in accuracy metrics (#10446).
- Fixed random samplers from uniform and random distributions in R bindings (#10450).
- Fixed a bug that was causing training metrics to be printed as NaN sometimes (#10437).
- Fixed a crash with non positive reps for tile ops (#10417).
- On average, after the MKL-DNN change, the inference speed of MXNet + MKLDNN outperforms MXNet + OpenBLAS by a factor of 32, outperforms MXNet + MKLML by 82% and outperforms MXNet + MKLML with the experimental flag by 8%. The experiments were run for the image classifcation example, for different networks and different batch sizes.
- Improved sparse SGD, sparse AdaGrad and sparse Adam optimizer speed on GPU by 30x (#9561, #10312, #10293, #10062).
sparse.retain performance on CPU by 2.5x (#9722)
std::swap_ranges with memcpy (#10351)
- Implemented DepthwiseConv2dBackwardFilterKernel which is over 5x faster (#10098)
- Implemented CPU LSTM Inference (#9977)
- Added Layer Normalization in C++ (#10029)
- Optimized Performance for rtc (#10018)
- Improved CPU performance of ROIpooling operator by using OpenMP (#9958)
- Accelerated the calculation of F1 (#9833)
Block.save_params now match parameters according to model structure instead of names to avoid prefix mismatching problems during saving and loading (#10511).
- Added an optional argument
mx.random.seed. Seeding with
ctx option produces random number sequence independent of device id. (#10367).
- Added copy flag for astype (#10347).
- Added context parameter to Scala Infer API - ImageClassifier and ObjectDetector (#10252).
- Added axes support for dropout in gluon (#10032).
- Added default
ctx to cpu for
- Added support for variable sequence length in gluon.RecurrentCell (#9934).
- Added convenience fluent method for squeeze op (#9734).
array.reshape compatible with numpy (#9790).
- Added axis support and gradient for L2norm (#9740).
- Added support for multi-GPU training with
row_sparse weights using
device KVStore (#9987).
Module.prepare API for multi-GPU and multi-machine training with row_sparse weight (#10285).
deterministic option for
contrib.SparseEmbedding operator (#9846).
sparse.broadcast_div with CSRNDArray and 1-D dense NDArray on CPU (#10208).
- Added sparse support for Custom Operator (#10374).
- Added Sparse feature for Perl (#9988).
force_deterministic option for sparse embedding (#9882).
sparse.where with condition being csr ndarray (#9481).
- Added constant parameter for gluon (#9893).
- Added Gluon PreLU, ELU, SELU, Swish activation layers for Gluon (#9662)
- Added Squeeze Op (#9700).
- Added multi-proposal operator (CPU version) and fixed bug in multi-proposal operator (GPU version) (#9939).
- Added in Large-Batch SGD with a warmup, and a LARS startegy (#8918).
- Added Language Modelling datasets and Sampler (#9514).
- Added instance norm and reflection padding to Gluon (#7938).
- Added micro-averaging strategy for F1 metric (#9777).
- Added Softsign Activation Function (#9851).
- Added eye operator, for default storage type (#9770).
- Added TVM bridge support to JIT NDArray Function by TVM (#9880).
- Added float16 support for correlation operator and L2Normalization operator (#10125, #10078).
- Added random shuffle implementation for NDArray (#10048).
- Added load from buffer functions for CPP package (#10261).
- Added embedding learning example for Gluon (#9165).
- Added tutorial on how to use data augmenters (#10055).
- Added tutorial for Data Augmentation with Masks (#10178).
- Added LSTNet example (#9512).
- Added MobileNetV2 example (#9614).
- Added tutorial for Gluon Datasets and DataLoaders (#10251).
- Added Language model with Google's billion words dataset (#10025).
- Added example for custom operator using RTC (#9870).
- Improved image classification examples (#9799, #9633).
- Added reshape predictor function to c_predict_api (#9984).
- Added guide for implementing sparse ops (#10081).
- Added naming tutorial for gluon blocks and parameters (#10511).
- MXNet crash when built with
USE_GPERFTOOLS = 1 (#8968).
- DevGuide.md in the 3rdparty submodule googletest licensed under CC-BY-2.5.
- Incompatibility in the behavior of MXNet Convolution operator for certain unsupported use cases: Raises an exception when MKLDNN is enabled, fails silently when it is not.
- MXNet convolution generates wrong results for 1-element strides (#10689).
- Tutorial on fine-tuning an ONNX model fails when using cpu context.
- CMake build ignores the
USE_MKLDNN flag and doesn't build with MKLDNN support even with
-DUSE_MKLDNN=1. To workaround the issue please see: #10801.
- Linking the dmlc-core library fails with CMake build when building with
USE_OPENMP=OFF. To workaround the issue, please use the updated CMakeLists in dmlc-core unit tests directory: https://github.com/dmlc/dmlc-core/pull/396. You can also workaround the issue by using make instead of cmake when building with
For more information and examples, see full release notes
- Improved the usability of examples and tutorials
- Fixed I/O multiprocessing for too many open file handles (#8904), race condition (#8995), deadlock (#9126).
- Fixed image IO integration with OpenCV 3.3 (#8757).
- Fixed Gluon block printing (#8956).
- Fixed float16 argmax when there is negative input. (#9149)
- Fixed random number generator to ensure sufficient randomness. (#9119, #9256, #9300)
- Fixed custom op multi-GPU scaling (#9283)
- Fixed gradient of gather_nd when duplicate entries exist in index. (#9200)
- Fixed overriden contexts in Module
group2ctx option when using multiple contexts (#8867)
swap_axes operator with “add_to” gradient req (#9541)
- Added experimental API in
contrib.text for building vocabulary, and loading pre-trained word embeddings, with built-in support for 307 GloVe and FastText pre-trained embeddings. (#8763)
- Added experimental structural blocks in
sparse.dot(dense, csr) operator (#8938)
Khatri-Rao operator (#7781)
Signum optimizer (#9220, #9262)
ENABLE_CUDA_RTC build option (#9428)
- Added zero gradients to rounding operators including
axis argument to
SequenceReverse operators (#9306)
lazy_update option for standard
Adam optimizer with
row_sparse gradients (#9468, #9189)
select option in
Block.collect_params to support regex (#9348)
- Added support for (one-to-one and sequence-to-one) inference on explicit unrolled RNN models in R (#9022)
- The Scala API name space is still called
ml.dmlc. The name space is likely be changed in a future release to
org.apache and might brake existing applications and scripts (#9579, #9324)
- Improved GPU inference speed by 20% when batch size is 1 (#9055)
SequenceLast operator speed (#9306)
- Added multithreading for the class of broadcast_reduce operators on CPU (#9444)
- Improved batching for GEMM/TRSM operators with large matrices on GPU (#8846)
- “Predict with pre-trained models” tutorial is broken
- “example/numpy-ops/ndarray_softmax.py” is broken
For more information and examples, see full release notes
- Enhanced the performance of
- MXNet now automatically set OpenMP to use all available CPU cores to maximize CPU utilization when
NUM_OMP_THREADS is not set.
- Unary and binary operators now avoid using OpenMP on small arrays if using OpenMP actually hurts performance due to multithreading overhead.
- Significantly improved performance of
broadcast_mul, etc on CPU.
- Added bulk execution to imperative mode. You can control segment size with
mxnet.engine.bulk. As a result, the speed of Gluon in hybrid mode is improved, especially on small networks and multiple GPUs.
- Improved speed for
ctypes invocation from Python frontend.
New Features - Gradient Compression [Experimental]
- Speed up multi-GPU and distributed training by compressing communication of gradients. This is especially effective when training networks with large fully-connected layers. In Gluon this can be activated with
compression_params in Trainer.
New Features - Support of NVIDIA Collective Communication Library (NCCL) [Experimental]
kvstore=’nccl’ for (in some cases) faster training on multiple GPUs.
- Significantly faster than kvstore=’device’ when batch size is small.
- It is recommended to set environment variable
PARALLEL when using NCCL version 2.1 or newer.
New Features - Advanced Indexing [General Availability]
New Features - Gluon [General Availability]
- Performance optimizations discussed above.
- Added support for loading data in parallel with multiple processes to
gluon.data.DataLoader. The number of workers can be set with
num_worker. Does not support windows yet.
- Added Block.cast to support networks with different data types, e.g.
- Added Lambda block for wrapping a user defined function as a block.
gluon.data.ArrayDataset to support arbitrary number of arrays.
New Features - ARM / Raspberry Pi support [Experimental]
New Features - NVIDIA Jetson support [Experimental]
- MXNet now compiles and runs on NVIDIA Jetson TX2 boards with GPU acceleration.
- You can install the python MXNet package on a Jetson board by running -
$ pip install mxnet-jetson-tx2.
New Features - Sparse Tensor Support [General Availability]
- Added more sparse operators:
asscipy() for easier conversion to scipy.
check_format() for sparse ndarrays to check if the array format is valid.
- Fixed a[-1] indexing doesn't work on
expand_dims if axis < 0.
- Fixed a bug that causes topk to produce incorrect result on large arrays.
- Improved numerical precision of unary and binary operators for
- Fixed derivatives of log2 and log10. They used to be the same with log.
- Fixed a bug that causes MXNet to hang after fork. Note that you still cannot use GPU in child processes after fork due to limitations of CUDA.
- Fixed a bug that causes
CustomOp to fail when using auxiliary states.
- Fixed a security bug that is causing MXNet to listen on all available interfaces when running training in distributed mode.
- Added a security best practices document under FAQ section.
- Fixed License Headers including restoring copyright attributions.
- Documentation updates.
- Links for viewing source.
For more information and examples, see full release notes
- Added GPU support for the
syevd operator which ensures that there is GPU support for all linalg-operators.
- Bugfix for
syevd on CPU such that it works for
- Fixed API call when
OMP_NUM_THREADS environment variable is set.
- Fixed bug related to passing
- Fixed some minor bugs for sparse distributed training.
- Fixed a bug on
Slice accessing uninitialized memory in
param.begin in the file
- Fixed a bug that caused
autograd to crash on some networks.
- Added full support for NVIDIA Volta GPU Architecture and CUDA 9. Training CNNs is up to 3.5x faster than Pascal when using float16 precision.
- Enabled JIT compilation. Autograd and Gluon hybridize now use less memory and has faster speed. Performance is almost the same with old symbolic style code.
- Improved ImageRecordIO image loading performance and added indexed RecordIO support.
- Added better openmp thread management to improve CPU performance.
New Features - Gluon
- Added enhancements to the Gluon package, a high-level interface designed to be easy to use while keeping most of the flexibility of low level API. Gluon supports both imperative and symbolic programming, making it easy to train complex models imperatively with minimal impact on performance. Neural networks (and other machine learning models) can be defined and trained with
- Added new loss functions -
gluon.Trainer now allows reading and setting learning rate with
- Added API
HybridBlock.export for exporting gluon models to MXNet format.
- Convolutional recurrent network cells for RNN, LSTM and GRU.
New Features - Autograd
- Added enhancements to
autograd package, which enables automatic differentiation of NDArray operations.
autograd.Function allows defining both forward and backward computation for custom operators.
mx.autograd.grad and experimental second order gradient support (most operators don't support second order gradient yet).
- Autograd now supports cross-device graphs. Use
x.copyto(mx.cpu()) to do computation on multiple devices.
New Features - Sparse Tensor Support
- Added support for sparse matrices.
- Added limited cpu support for two sparse formats in
- Added a sparse dot product operator and many element-wise sparse operators.
- Added a data iterator for sparse data input -
- Added three optimizers for sparse gradient updates:
RowSparseNDArray in distributed kvstore.
Other New Features
- Added limited support for fancy indexing, which allows you to very quickly access and modify complicated subsets of an array's values.
x[idx_arr0, idx_arr1, ..., idx_arrn] is now supported. Features such as combining and slicing are planned for the next release. Checkout master to get a preview.
- Random number generators in
mx.sym.random.* now support both CPU and GPU.
Symbol now supports “fluent” methods. You can now use
x.exp() etc instead of
mx.rtc.CudaModule for writing and running CUDA kernels from python.
multi_precision option to optimizer for easier float16 training.
- Better support for IDE auto-completion. IDEs like PyCharm can now correctly parse mxnet operators.
- Operators like
mx.sym.random_* are now moved to
mx.sym.random.*. The old names are still available but deprecated.
random_* are now merged as
random.*, which supports both scalar and
NDArray distribution parameters.
- Fixed a bug that causes
argsort operator to fail on large tensors.
- Fixed numerical stability issues when summing large tensors.
- Fixed a bug that causes arange operator to output wrong results for large ranges.
- Improved numerical precision for unary and binary operators on
For more information and examples, see full release notes
- Apple Core ML model converter
- Support for Keras v1.2.2
- For more information see full release notes
CachedOp. You can now cache the operators that’s called frequently with the same set of arguments to reduce overhead.
- Added sample_multinomial for sampling from multinomial distributions.
trunc operator for rounding towards zero.
- Added linalg_gemm, linalg_potrf, ... operators for lapack support.
- Added verbose option to Initializer for printing out initialization details.
- Added DeformableConvolution to contrib from the Deformable Convolutional Networks paper.
- Added float64 support for dot and batch_dot operator.
allow_extra is added to Module.set_params to ignore extra parameters.
mod operator for modulo.
multi_precision option to SGD optimizer to improve training with float16. Resnet50 now achieves the same accuracy when trained with float16 and gives 50% speedup on Titan XP.
- ImageRecordIter now stores data in pinned memory to improve GPU memcopy speed.
- Cython interface is fixed.
make cython and
python setup.py install --with-cython should install the cython interface and reduce overhead in applications that use imperative/bucketing.
- Fixed various bugs in Faster-RCNN example: https://github.com/dmlc/mxnet/pull/6486
- Fixed various bugs in SSD example.
out argument not working for
expand_dims now supports backward shape inference.
- Fixed a bug in rnn. BucketingSentenceIter that causes incorrect layout handling on multi-GPU.
- Fixed context mismatch when loading optimizer states.
- Fixed a bug in ReLU activation when using MKL.
- Fixed a few race conditions that causes crashes on shutdown.
- Refactored TShape/TBlob to use int64 dimensions and DLTensor as internal storage. Getting ready for migration to DLPack. As a result TBlob::dev_mask_ and TBlob::stride_ are removed.
- Overhauled documentation for commonly used Python APIs, Installation instructions, Tutorials, HowTos and MXNet Architecture.
- Updated mxnet.io for improved readability.
- Pad operator now support reflection padding.
- Fixed a memory corruption error in threadedengine.
- Added CTC loss layer to contrib package. See mx.contrib.sym.ctc_loss.
- Added new sampling operators for several distributions (normal,uniform,gamma,exponential,negative binomial).
- Added documentation for experimental RNN APIs.
- Move symbolic API to NNVM @tqchen
- Most front-end C API are backward compatible
- Removed symbolic API in MXNet and relies on NNVM
- New features:
- MXNet profiler for profiling operator-level executions
- mxnet.image package for fast image loading and processing
- Change of JSON format
- param and attr field are merged to attr
- New code is backward-compatible can load old json format
- OpProperty registration now is deprecated
- New operators are encouraged to register their property to NNVM op registry attribute
- Known features removed limitations to be fixed
- Bulk segment execution not yet added.
This is the last release before the NNVM refactor.
- CaffeOp and CaffeIter for interfacing with Caffe by @HrWangChengdu @cjolivier01
- WrapCTC plugin for sequence learning by @xlvector
- Improved Multi-GPU performance by @mli
- CuDNN RNN support by @sbodenstein
- OpenCV plugin for parallel image IO by @piiswrong
- More operators as simple op
- Simple OP @tqchen
- element wise op with axis and broadcast @mli @sxjscience
- Cudnn auto tuning for faster convolution by @piiswrong
- More applications
- Faster RCNN by @precedenceguo
- 0.6 is skipped because there are a lot of improvements since initial release
- More math operators
- elementwise ops and binary ops
- Attribute support in computation graph
- Now user can use attributes to give various hints about specific learning rate, allocation plans etc
- MXNet is more memory efficient
- Support user defined memory optimization with attributes
- Support mobile applications by @antinucleon
- Refreshed update of new documents
- Model parallel training of LSTM by @tqchen
- Simple operator refactor by @tqchen
- add operator_util.h to enable quick registration of both ndarray and symbolic ops
- Distributed training by @mli
- Support Torch Module by @piiswrong
- MXNet now can use any of the modules from Torch.
- Support custom native operator by @piiswrong
- Support data types including fp16, fp32, fp64, int32, and uint8 by @piiswrong
- Support monitor for easy printing and debugging by @piiswrong
- Support new module API by @pluskid
- Module API is a middle level API that can be used in imperative manner like Torch-Module
- Support bucketing API for variable length input by @pluskid
- Support CuDNN v5 by @antinucleon
- More applications
v0.5 (initial release)