| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements; and to You under the Apache License, Version 2.0. " |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "![Apache Singa](http://singa.apache.org/en/_static/singa.png)\n", |
| "\n", |
| "# A Tutorial of SINGA V1\n", |
| "\n", |
| "Welcome to this tutorial for Apache Incubator-singa.\n", |
| "Please install [conda and SINGA](./installation.ipynb) before running the notebooks.\n", |
| "\n", |
| "## Outline\n", |
| "\n", |
| "* Introduction and installation\n", |
| "* SINGA classes\n", |
| " * [Core classes](./core.ipynb) - Tensor and Device\n", |
| " * [Model classes](./model.ipynb) - Layer, Loss, Metric, FeedForwardNet, Optimzier, Initializer\n", |
| "* Simple models \n", |
| " * [Linear Regression](./regression.ipynb)\n", |
| " * [Multi-layer Perceptron](./mlp.ipynb)\n", |
| "* Deep learning models\n", |
| " * [Convolutional Neural Network (CNN)](./cnn.ipynb)\n", |
| " * [Recurrent Neural Networks (RNN)](./rnn.ipynb)\n", |
| " * [Restricted Boltzmann Machine (RBM)](./rbm.ipynb)\n", |
| "* [Distributed training](./distributed.ipynb) (WIP)\n", |
| "* [Rafiki](./rafiki.ipynb) (WIP)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Introduction \n", |
| "### Training and Inference of deep learning models\n", |
| "\n", |
| "#### Training objective\n", |
| "\n", |
| "$\\min_{\\theta} \\sum_{<x, y>\\in D}L(f(x|\\theta), y)$\n", |
| "\n", |
| "Training procedure\n", |
| "1. randomly initialize $\\theta$\n", |
| "2. get a mini-batch of data $\\{<x, y>\\}$ and compute $\\frac{\\partial L}{\\partial \\theta}$\n", |
| "3. $\\theta \\leftarrow \\theta - \\alpha \\frac{\\partial L}{\\partial \\theta}$\n", |
| "4. repeat 2-3 until converge, i.e., $||\\frac{\\partial L}{\\partial \\theta}|| < \\epsilon$ \n", |
| "\n", |
| "<img src=\"./static/sgd.png\" width=\"400px\"/>\n", |
| "\n", |
| "#### Back-propagation for gradient computation\n", |
| "\n", |
| "<img src=\"./static/bp.PNG\" width=\"300px\"/> \n", |
| "\n", |
| "#### Popular deep learning models\n", |
| "\n", |
| "<img src=\"./static/models.PNG\" width=\"200px\"/>\n", |
| "\n", |
| "\n", |
| "\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "### SINGA V1\n", |
| "#### Software Stack\n", |
| "\n", |
| "<img src=\"http://singa.apache.org/en/_static/images/singav1-sw.png\" width=\"500px\"/>\n", |
| "\n", |
| "#### Progress\n", |
| "\n", |
| "|Components|Feature|Status|\n", |
| "|:---------|:------|:-----|\n", |
| "|Core |Tensor implementation|V1.0|\n", |
| "| |Device implementation (CppCPU, CudaGPU, OpenclGPU) |V1.1|\n", |
| "|Model|Basic layers for CNN and RNN| V1.0|\n", |
| "| |Basic loss and metric functions|V1.0|\n", |
| "| |Basic optimizer and initializers |V1.0|\n", |
| "|IO |Message paasing via socket| V1.0|\n", |
| "|Installation| From source on Linux and Mac|V1.0|\n", |
| "| | From source on Windows|V1.1|\n", |
| "| | From wheel on Linux and Mac|V1.0|\n", |
| "| | From debian package on Ubuntu |V1.1|\n", |
| "| | Docker images | V1.1|\n", |
| "| | AWS AMI | V1.1|\n", |
| "\n", |
| "#### Schedule\n", |
| "|Components|Feature|Status|\n", |
| "|:---------|:------|:-----|\n", |
| "|Core |Tensor API = Numpy API| V1.2|\n", |
| "| |Operation scheduling and memory optimization| V1.3|\n", |
| "|Model | To support more vison, NLP examples| V1.2|\n", |
| "| | General neural net API| V1.2|\n", |
| "| | To support reinforcement learning examples| V1.3|\n", |
| "| | Auto-grad + Functional API | TBD|\n", |
| "|IO | HDFS integration |V1.3|\n", |
| "|Server| Consistency and communication optimization |V1.2| \n", |
| "| |Message passing via Infiniband |V1.2| \n", |
| "|Installation | From wheel on Windows| V1.2|\n", |
| "| | From debian on Linux | V1.2|\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Next: [SINGA core classes](./core.ipynb)" |
| ] |
| } |
| ], |
| "metadata": { |
| "anaconda-cloud": {}, |
| "kernelspec": { |
| "display_name": "py3", |
| "language": "python", |
| "name": "py3" |
| }, |
| "language_info": { |
| "codemirror_mode": { |
| "name": "ipython", |
| "version": 3 |
| }, |
| "file_extension": ".py", |
| "mimetype": "text/x-python", |
| "name": "python", |
| "nbconvert_exporter": "python", |
| "pygments_lexer": "ipython3", |
| "version": "3.5.3" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 2 |
| } |