blob: fe93aae1c9315b40dc5e771c8559bb4ae1acf14e [file] [log] [blame]
{
"cells": [
{
"cell_type": "markdown",
"id": "24596ffd",
"metadata": {},
"source": [
"<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
"<!--- or more contributor license agreements. See the NOTICE file -->\n",
"<!--- distributed with this work for additional information -->\n",
"<!--- regarding copyright ownership. The ASF licenses this file -->\n",
"<!--- to you under the Apache License, Version 2.0 (the -->\n",
"<!--- \"License\"); you may not use this file except in compliance -->\n",
"<!--- with the License. You may obtain a copy of the License at -->\n",
"\n",
"<!--- http://www.apache.org/licenses/LICENSE-2.0 -->\n",
"\n",
"<!--- Unless required by applicable law or agreed to in writing, -->\n",
"<!--- software distributed under the License is distributed on an -->\n",
"<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
"<!--- KIND, either express or implied. See the License for the -->\n",
"<!--- specific language governing permissions and limitations -->\n",
"<!--- under the License. -->\n",
"\n",
"# Real-time Object Detection with MXNet On The Raspberry Pi \n",
"\n",
"This tutorial shows developers who work with the Raspberry Pi or similar embedded ARM-based devices how to compile MXNet for those devices and run a pretrained deep network model. It also shows how to use AWS IoT to manage and monitor MXNet models running on your devices.\n",
"\n",
"## What's In This Tutorial?\n",
"\n",
"This tutorial shows how to:\n",
"\n",
"1. Use MXNet to set up a real-time object classifier on a Raspberry Pi 3 device.\n",
"\n",
"2. Connect the local Raspberry Pi model to the AWS Cloud with AWS IoT to get real-time updates from the device.\n",
"\n",
"### Who's This Tutorial For?\n",
"\n",
"It assumes that you are familiar with the Raspbian operating system and the [Raspberry Pi ecosystem](https://www.raspberrypi.org/) and are somewhat familiar with machine learning, MXNet, and [AWS IoT](https://aws.amazon.com/iot/). All code is written in Python 2.7.\n",
"\n",
"### How to Use This Tutorial\n",
"\n",
"To follow this tutorial, you must set up your Pi as instructed (preferably from a fresh Raspbian install), and then create the files and run the bash commands described below. All instructions described are can be executed on the Raspberry Pi directly or via SSH.\n",
"\n",
"You will accomplish the following:\n",
"\n",
"- Build and Install MXNet with Python bindings on your Raspbian Based Raspberry Pi\n",
"- Fetch and run a pre-trained MXNet model on your Pi\n",
"- Create a real-time video analysis application for the Pi\n",
"- Connect the application to the AWS IoT service\n",
"\n",
"## Prerequisites\n",
"\n",
"To complete this tutorial, you need:\n",
"\n",
"* Raspbian Wheezy or later, which can be downloaded [here](https://www.raspberrypi.org/downloads/raspbian/), loaded onto a 8GB+ micro SD card (with at least 4GB+ free)\n",
"* A [Raspberry Pi 3](https://www.raspberrypi.org/blog/raspberry-pi-3-on-sale/) or equivalent Raspberry Pi with 1GB+ of RAM\n",
"* A [Raspberry Pi Camera Module](https://www.raspberrypi.org/products/camera-module-v2/) [activated and running with the corresponding Python module](http://www.pyimagesearch.com/2015/02/23/install-opencv-and-python-on-your-raspberry-pi-2-and-b/) (for the real-time video analysis with the deep network model)\n",
"* An AWS account With AWS IoT enabled and the [AWS IoT Python SDK](https://github.com/aws/aws-iot-device-sdk-python) (for remote, real-time managing and monitoring of the model running on the Pi)\n",
"* The [cv2 Python library](http://www.pyimagesearch.com/2015/02/23/install-opencv-and-python-on-your-raspberry-pi-2-and-b/) for the Pi\n",
"\n",
"## Building MXNet for The Pi\n",
"\n",
"The first step is to get MXNet with the Python bindings running on your Raspberry Pi 3. There is a tutorial for that provided [here](https://mxnet.io/get_started). The linked tutorial walks you through downloading the dependencies, and building the full MXNet library for the Pi with the ARM specific compile flags. Be sure to build the library with open CV as we will be using a model that requires it to process images. Then you will register the Python bindings to MXNet. After this is done you should test that your installation works by opening a python REPL on your Pi and typing the following commands:"
]
},
{
"cell_type": "markdown",
"id": "50340f57",
"metadata": {},
"source": [
"```bash\n",
"python\n",
">>> import mxnet as mx\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "5bcef9fc",
"metadata": {},
"source": [
"*Note: If you are getting memory allocation failed errors at this point (or at any point in this tutorial) it is likely because the full MXNet library takes up a large amount of RAM when loaded. You might want to kill the GUI and other processes that are occupying memory.*\n",
"\n",
"\n",
"## Running A Pre-Trained Inception Model on The Pi\n",
"\n",
"We are now ready to load a pre-trained model and run inference on the Pi. We will be using a simple object recognition model trained on the ImageNet data set. The model is called batch normalized Inception network (or Inception_BN for short) and it is found in the MXNet model zoo.\n",
"\n",
"### Getting the Model\n",
"\n",
"The first step is to download, unzip, and set up the pre-trained deep network model files that we will be using to classify images. To do this run the following commands in your home directory:"
]
},
{
"cell_type": "markdown",
"id": "4e5c1061",
"metadata": {},
"source": [
"```bash\n",
"curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L\n",
"\n",
"tar -xvzf inception-bn.tar.gz\n",
"\n",
"mv Inception_BN-0039.params Inception_BN-0000.params\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "b4ac25bb",
"metadata": {},
"source": [
"### Running the Model\n",
"\n",
"The next step is to create a python script to load the model, and run inference on local image files. To do this create a new file in your home directory called inception_predict.py and add the following code to it:"
]
},
{
"cell_type": "markdown",
"id": "0220af88",
"metadata": {},
"source": [
"```python\n",
"# inception_predict.py\n",
"\n",
"import mxnet as mx\n",
"import numpy as np\n",
"import time\n",
"import cv2, os, urllib\n",
"from collections import namedtuple\n",
"Batch = namedtuple('Batch', ['data'])\n",
"\n",
"# Load the symbols for the networks\n",
"with open('synset.txt', 'r') as f:\n",
" synsets = [l.rstrip() for l in f]\n",
"\n",
"# Load the network parameters\n",
"sym, arg_params, aux_params = mx.model.load_checkpoint('Inception-BN', 126)\n",
"\n",
"\n",
"# Load the network into an MXNet module and bind the corresponding parameters\n",
"mod = mx.mod.Module(symbol=sym, context=mx.cpu())\n",
"mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))])\n",
"mod.set_params(arg_params, aux_params)\n",
"\n",
"'''\n",
"Function to predict objects by giving the model a pointer to an image file and running a forward pass through the model.\n",
"\n",
"inputs:\n",
"filename = jpeg file of image to classify objects in\n",
"mod = the module object representing the loaded model\n",
"synsets = the list of symbols representing the model\n",
"N = Optional parameter denoting how many predictions to return (default is top 5)\n",
"\n",
"outputs:\n",
"python list of top N predicted objects and corresponding probabilities\n",
"'''\n",
"def predict(filename, mod, synsets, N=5):\n",
" tic = time.time()\n",
" img = cv2.cvtColor(cv2.imread(filename), cv2.COLOR_BGR2RGB)\n",
" if img is None:\n",
" return None\n",
" img = cv2.resize(img, (224, 224))\n",
" img = np.swapaxes(img, 0, 2)\n",
" img = np.swapaxes(img, 1, 2)\n",
" img = img[np.newaxis, :]\n",
" print \"pre-processed image in \"+str(time.time()-tic)\n",
"\n",
" toc = time.time()\n",
" mod.forward(Batch([mx.nd.array(img)]))\n",
" prob = mod.get_outputs()[0].asnumpy()\n",
" prob = np.squeeze(prob)\n",
" print \"forward pass in \"+str(time.time()-toc)\n",
"\n",
"\n",
" topN = []\n",
" a = np.argsort(prob)[::-1]\n",
" for i in a[0:N]:\n",
" print('probability=%f, class=%s' %(prob[i], synsets[i]))\n",
" topN.append((prob[i], synsets[i]))\n",
" return topN\n",
"\n",
"\n",
"# Code to download an image from the internet and run a prediction on it\n",
"def predict_from_url(url, N=5):\n",
" filename = url.split(\"/\")[-1]\n",
" urllib.urlretrieve(url, filename)\n",
" img = cv2.imread(filename)\n",
" if img is None:\n",
" print \"Failed to download\"\n",
" else:\n",
" return predict(filename, mod, synsets, N)\n",
"\n",
"# Code to predict on a local file\n",
"def predict_from_local_file(filename, N=5):\n",
" return predict(filename, mod, synsets, N)\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "235406c8",
"metadata": {},
"source": [
"Now that we have defined inception_predict.py we can test that the model is running correctly. Open a Python REPL in your home directory and enter the following:"
]
},
{
"cell_type": "markdown",
"id": "66bb4ea2",
"metadata": {},
"source": [
"```bash\n",
"python\n",
">>> from inception_predict import *\n",
">>> predict_from_url(\"https://i.imgur.com/HzafyBA.jpg\")\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "83420d07",
"metadata": {},
"source": [
"This should give a reasonable prediction for the fluffy cow in this [image](http://imgur.com/HzafyBA)."
]
},
{
"cell_type": "markdown",
"id": "7065d39c",
"metadata": {},
"source": [
"```\n",
"pre-processed image in 0.20366191864\n",
"forward pass in 63.2164611816\n",
"probability=0.718524, class=n02403003 ox\n",
"probability=0.176381, class=n02389026 sorrel\n",
"probability=0.095558, class=n03868242 oxcart\n",
"probability=0.002765, class=n02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis\n",
"probability=0.001262, class=n03935335 piggy bank, penny bank\n",
"[(0.71852392, 'n02403003 ox'), (0.17638102, 'n02389026 sorrel'), (0.09555836, 'n03868242 oxcart'),\n",
"(0.0027645244, 'n02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis'),\n",
"(0.0012616422, 'n03935335 piggy bank, penny bank')]\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "9cf9a756",
"metadata": {},
"source": [
"## Running an Inception on Real-Time Video From PiCamera\n",
"\n",
"We can now move on to using this network for object detection in real-time video from the PiCamera.\n",
"\n",
"Doing this requires sending the images that the camera is capturing to the prediction code that we created in the previous step. To do this, create a new file in your home directory called camera_test.py and add the following code to it:"
]
},
{
"cell_type": "markdown",
"id": "2daf5dad",
"metadata": {},
"source": [
"```python\n",
"# camera_test.py\n",
"\n",
"import picamera\n",
"import inception_predict\n",
"\n",
"# Create camera interface\n",
"camera = picamera.PiCamera()\n",
"while True:\n",
" # Take the jpg image from camera\n",
" print \"Capturing\"\n",
" filename = '/home/pi/cap.jpg'\n",
" # Show quick preview of what's being captured\n",
" camera.start_preview()\n",
" camera.capture(filename)\n",
" camera.stop_preview()\n",
"\n",
" # Run inception prediction on image\n",
" print \"Predicting\"\n",
" topn = inception_predict.predict_from_local_file(filename, N=5)\n",
"\n",
" # Print the top N most likely objects in image (default set to 5, change this in the function call above)\n",
" print topn\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "26703d2c",
"metadata": {},
"source": [
"You can then run this file by entering the following command:"
]
},
{
"cell_type": "markdown",
"id": "abd581c5",
"metadata": {},
"source": [
"```bash\n",
"python camera_test.py\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "f10e862b",
"metadata": {},
"source": [
"If camera_test.py is working you should see a preview every few seconds of the image that is being captured and fed to the model, as well as predicted classes for objects in the image being written to the terminal.\n",
"\n",
"Try pointing the PiCamera at a few different objects and see what predictions the network comes out with.\n",
"\n",
"## Connecting Our Model To The AWS Cloud\n",
"\n",
"We can now move on to adding the code to send the predictions that this real-time model is making locally to the AWS cloud if certain conditions are met.\n",
"\n",
"The first step is to set up an AWS account if you don't have one yet. Then go to the [AWS IoT dashboard](https://us-west-2.console.aws.amazon.com/iotv2/home?region=us-west-2#/thinghub) and register a new device.\n",
"\n",
"After the device is registered, download and copy the corresponding rootCA, Certificate, and Private key to your home directory. Note the unique endpoint of your device shadow on the AWS IoT Dashboard.\n",
"\n",
"We will now build an application, based off the code in camera_test.py, which will send a message to the cloud whenever a wine bottle is detected in a frame by the PiCamera.\n",
"\n",
"To do this create a new file in your home directory called wine_alerter.py and add the following code to it:"
]
},
{
"cell_type": "markdown",
"id": "730bfe08",
"metadata": {},
"source": [
"```python\n",
"# wine_alerter.py\n",
"\n",
"import AWSIoTPythonSDK\n",
"from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient\n",
"import sys\n",
"import logging\n",
"import time\n",
"import getopt\n",
"import picamera\n",
"import inception_predict\n",
"\n",
"# Custom MQTT message callback\n",
"def customCallback(client, userdata, message):\n",
" print(\"Received a new message: \")\n",
" print(message.payload)\n",
" print(\"from topic: \")\n",
" print(message.topic)\n",
" print(\"--------------\\n\\n\")\n",
"\n",
"# Usage\n",
"usageInfo = \"\"\"Usage:\n",
"\n",
"Use certificate based mutual authentication:\n",
"python wine_alerter.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>\n",
"\n",
"Use MQTT over WebSocket:\n",
"python wine_alerter.py -e <endpoint> -r <rootCAFilePath> -w\n",
"\n",
"Type \"python wine_alerter.py -h\" for available options.\n",
"\"\"\"\n",
"\n",
"# Help info\n",
"helpInfo = \"\"\"-e, --endpoint\n",
" Your AWS IoT custom endpoint\n",
"-r, --rootCA\n",
" Root CA file path\n",
"-c, --cert\n",
" Certificate file path\n",
"-k, --key\n",
" Private key file path\n",
"-w, --websocket\n",
" Use MQTT over WebSocket\n",
"-h, --help\n",
" Help information\n",
"\"\"\"\n",
"\n",
"# Read in command-line parameters\n",
"useWebsocket = False\n",
"host = \"\"\n",
"rootCAPath = \"\"\n",
"certificatePath = \"\"\n",
"privateKeyPath = \"\"\n",
"try:\n",
" opts, args = getopt.getopt(sys.argv[1:], \"hwe:k:c:r:\", [\"help\", \"endpoint=\", \"key=\",\"cert=\",\"rootCA=\", \"websocket\"])\n",
" if len(opts) == 0:\n",
" raise getopt.GetoptError(\"No input parameters!\")\n",
" for opt, arg in opts:\n",
" if opt in (\"-h\", \"--help\"):\n",
" print(helpInfo)\n",
" exit(0)\n",
" if opt in (\"-e\", \"--endpoint\"):\n",
" host = arg\n",
" if opt in (\"-r\", \"--rootCA\"):\n",
" rootCAPath = arg\n",
" if opt in (\"-c\", \"--cert\"):\n",
" certificatePath = arg\n",
" if opt in (\"-k\", \"--key\"):\n",
" privateKeyPath = arg\n",
" if opt in (\"-w\", \"--websocket\"):\n",
" useWebsocket = True\n",
"except getopt.GetoptError:\n",
" print(usageInfo)\n",
" exit(1)\n",
"\n",
"# Missing configuration notification\n",
"missingConfiguration = False\n",
"if not host:\n",
" print(\"Missing '-e' or '--endpoint'\")\n",
" missingConfiguration = True\n",
"if not rootCAPath:\n",
" print(\"Missing '-r' or '--rootCA'\")\n",
" missingConfiguration = True\n",
"if not useWebsocket:\n",
" if not certificatePath:\n",
" print(\"Missing '-c' or '--cert'\")\n",
" missingConfiguration = True\n",
" if not privateKeyPath:\n",
" print(\"Missing '-k' or '--key'\")\n",
" missingConfiguration = True\n",
"if missingConfiguration:\n",
" exit(2)\n",
"\n",
"\n",
"# Configure logging\n",
"logger = logging.getLogger(\"AWSIoTPythonSDK.core\")\n",
"logger.setLevel(logging.DEBUG)\n",
"streamHandler = logging.StreamHandler()\n",
"formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')\n",
"streamHandler.setFormatter(formatter)\n",
"logger.addHandler(streamHandler)\n",
"\n",
"\n",
"# Init AWSIoTMQTTClient For Publish/Subscribe Communication With Server\n",
"myAWSIoTMQTTClient = None\n",
"if useWebsocket:\n",
" myAWSIoTMQTTClient = AWSIoTMQTTClient(\"basicPubSub\", useWebsocket=True)\n",
" myAWSIoTMQTTClient.configureEndpoint(host, 443)\n",
" myAWSIoTMQTTClient.configureCredentials(rootCAPath)\n",
"else:\n",
" myAWSIoTMQTTClient = AWSIoTMQTTClient(\"basicPubSub\")\n",
" myAWSIoTMQTTClient.configureEndpoint(host, 8883)\n",
" myAWSIoTMQTTClient.configureCredentials(rootCAPath, privateKeyPath, certificatePath)\n",
"\n",
"\n",
"# AWSIoTMQTTClient connection configuration\n",
"myAWSIoTMQTTClient.configureAutoReconnectBackoffTime(1, 32, 20)\n",
"myAWSIoTMQTTClient.configureOfflinePublishQueueing(-1) # Infinite offline Publish queueing\n",
"myAWSIoTMQTTClient.configureDrainingFrequency(2) # Draining: 2 Hz\n",
"myAWSIoTMQTTClient.configureConnectDisconnectTimeout(10) # 10 sec\n",
"myAWSIoTMQTTClient.configureMQTTOperationTimeout(5) # 5 sec\n",
"\n",
"\n",
"# Connect and subscribe to AWS IoT\n",
"myAWSIoTMQTTClient.connect()\n",
"myAWSIoTMQTTClient.subscribe(\"sdk/test/Python\", 1, customCallback)\n",
"time.sleep(2)\n",
"\n",
"\n",
"# Start the Camera and tell the Server we are alive\n",
"print \"Running camera\"\n",
"myAWSIoTMQTTClient.publish(\"sdk/test/Python\", \"New Message: Starting Camera\", 0)\n",
"camera = picamera.PiCamera()\n",
"\n",
"# Capture forever (this is a modified version of camera_test.py)\n",
"while True:\n",
" filename = '/home/pi/cap.jpg'\n",
" camera.start_preview()\n",
" camera.capture(filename)\n",
" camera.stop_preview()\n",
" topn = inception_predict.predict_from_local_file(filename, N=5)\n",
"\n",
" # Check if either of the top two predictions are wine related and publish a message if it is\n",
" # you can change 'wine' here to anything you want to alert the server about detecting\n",
" if 'wine' in topn[0][1] or 'wine' in topn[1][1]:\n",
" myAWSIoTMQTTClient.publish(\"sdk/test/Python\", \"New Message: WINE DETECTED!\", 0)\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "e7b3d1ef",
"metadata": {},
"source": [
"You can then run this file by entering the following command"
]
},
{
"cell_type": "markdown",
"id": "f39dd17f",
"metadata": {},
"source": [
"```bash\n",
"python wine_alerter.py -e <endpointURL> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "43246fed",
"metadata": {},
"source": [
"If this is working you should see the same kind of image preview you did with camera_test.py every few seconds, however the console will only print a message now when a wine bottle is detected in the shot (you can edit the bottom lines in the wine_alerter.py code to make this alert for any object label from the [ImageNet-11k dataset](http://image-net.org/index) that you specify).\n",
"\n",
"You can open up the activity tab for the thing that you registered on the AWS IoT Dashboard and see the corresponding messages pushed to the server whenever a wine bottle is detected in a camera shot. Even if network connectivity periodically fails, AWS IoT will push updates out to the server when possible, allowing this system to reliably let you know when there is wine around.\n",
"\n",
"## Summary\n",
"You now have a Raspberry Pi running a pre-trained MXNet model fully locally. This model is also linked to the cloud via AWS IoT and can reliably alert AWS whenever it sees a wine bottle.\n",
"\n",
"You can now extend this demo to create more interesting applications, such as using AWS IoT to push model updates to your Pi, loading different pre-trained models from the MXNet Model Zoo onto the Pi, or even training full end-to-end models on the Pi."
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}