{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Toree Magics<a name=\"top\"></a>\n",
    "Magics are special \"functions\" which enable features or execute some special code. Magics can receive input arguments when they are invoked. There are two types of magics: `cell` magics and `line` magics. Magics invocations are not case sensitive.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "1. [Line Magics](#line-magics)\n",
    "    1. [LsMagic](#lsmagic)\n",
    "    1. [Truncation](#truncation)\n",
    "    1. [ShowTypes](#showtypes)\n",
    "    1. [AddJar](#addjar)\n",
    "    1. [AddDeps](#adddeps)\n",
    "1. [Cell Magics](#cell-magics)\n",
    "    1. [DataFrame](#dataframe)\n",
    "    1. [Html](#html)\n",
    "    1. [JavaScript](#javascript)\n",
    "    1. [PySpark](#pyspark)\n",
    "    1. [SparkR](#sparkr)\n",
    "    1. [SparkSQL](#sparksql)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Line Magics<a name=\"line-magics\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "Line magics are run on a single line and can have other code and line magics within the same cell. Line magics use the following syntax: \n",
    "\n",
    "```\n",
    "%magicname [args]\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### %LsMagic<a name=\"lsmagic\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `LsMagic` is a magic to list all the available magics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available line magics:\n",
      "%addjar %lsmagic %showtypes %adddeps %truncation\n",
      "\n",
      "Available cell magics:\n",
      "%%sql %%html %%javascript %%rdd %%scala %%sparkr %%pyspark\n",
      "\n",
      "Type %<magic_name> for usage info.\n",
      "         \n"
     ]
    }
   ],
   "source": [
    "%LsMagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### %Truncation<a name=\"truncation\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "Toree will, by default, truncate results from statements. This can be managed through the `%Truncation` magic. To see the current state of the truncation setting you can invoke the magic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Truncation is currently on \n"
     ]
    }
   ],
   "source": [
    "// invoke the truncation magic to see if truncation is on or off\n",
    "%Truncation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// return a value to see the truncation\n",
    "(1 to 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output will NOT be truncated\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%Truncation off\n",
    "(1 to 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output WILL be truncated.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%Truncation on\n",
    "(1 to 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### %ShowTypes<a name=\"showtypes\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The type information for a result is hidden by default. This behavior can be changed by using the `%ShowTypes` magic. You can view the current state of `%ShowTypes` by invoking it with no arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ShowTypes is currently off \n"
     ]
    }
   ],
   "source": [
    "%ShowTypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Hello types!"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Hello types!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types will be printed.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "String = Hello types!"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%ShowTypes on\n",
    "\"Hello types!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Int, String) = (1,Hello types!)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1, \"Hello types!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types will not be printed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Hello types!"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%ShowTypes off\n",
    "\"Hello types!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### %AddJar<a name=\"addjar\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "`AddJar` is a magic that allows the addition of jars to Torree's environment. You can see the arguments for `AddJar` by invoking it with no arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: %AddJar <jar_url>\n",
      "\n",
      "Option   Description                        \n",
      "------   -----------                        \n",
      "-f       forces re-download of specified jar\n",
      "--magic  loads jar as a magic extension     \n"
     ]
    }
   ],
   "source": [
    "%AddJar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting download from https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.0.0b/lwjgl-3.0.0b.jar\n",
      "Finished download of lwjgl-3.0.0b.jar\n"
     ]
    }
   ],
   "source": [
    "%AddJar https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.0.0b/lwjgl-3.0.0b.jar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0.0b SNAPSHOT"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "org.lwjgl.Version.getVersion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## %AddDeps<a name=\"adddeps\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "`AddDeps` is a magic to add dependencies from a maven repository. You can see the arguments for `AddDeps` by invoking it with no arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: %AddDeps my.company artifact-id version\n",
      "\n",
      "Option                        Description                          \n",
      "------                        -----------                          \n",
      "--abort-on-resolution-errors  Abort (no downloads) when resolution \n",
      "                                fails                              \n",
      "--repository                  Adds an additional repository to     \n",
      "                                available list                     \n",
      "--trace                       Prints out trace of download progress\n",
      "--transitive                  Retrieve dependencies recursively    \n",
      "--verbose                     Prints out additional information    \n"
     ]
    }
   ],
   "source": [
    "%AddDeps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that by default the `AddDeps` magic will only retrieve the specified dependency. If you want the transitive dependencies provide the `--transitive` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Marking org.joda:joda-money:0.11 for download\n",
      "Preparing to fetch from:\n",
      "-> file:/tmp/toree_add_deps5662724810625125387/\n",
      "-> https://repo1.maven.org/maven2\n",
      "=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom.sha1\n",
      "=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom\n",
      "===> 1 (joda-money-0.11.pom.sha1): Is 40 total bytes\n",
      "===> 1 (joda-money-0.11.pom.sha1): Downloaded 40 bytes (100.00%)\n",
      "=> 1 (joda-money-0.11.pom.sha1): Finished downloading\n",
      "===> 1 (joda-money-0.11.pom): Is 22792 total bytes\n",
      "===> 1 (joda-money-0.11.pom): Downloaded 15727 bytes (69.00%)\n",
      "===> 1 (joda-money-0.11.pom): Downloaded 22792 bytes (100.00%)\n",
      "=> 1 (joda-money-0.11.pom): Finished downloading\n",
      "=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar.sha1\n",
      "=> 2 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar\n",
      "===> 1 (joda-money-0.11.jar.sha1): Is 40 total bytes\n",
      "===> 1 (joda-money-0.11.jar.sha1): Downloaded 40 bytes (100.00%)\n",
      "=> 1 (joda-money-0.11.jar.sha1): Finished downloading\n",
      "===> 2 (joda-money-0.11.jar): Is 63725 total bytes\n",
      "===> 2 (joda-money-0.11.jar): Downloaded 15713 bytes (24.66%)\n",
      "===> 2 (joda-money-0.11.jar): Downloaded 32097 bytes (50.37%)\n",
      "===> 2 (joda-money-0.11.jar): Downloaded 48481 bytes (76.08%)\n",
      "===> 2 (joda-money-0.11.jar): Downloaded 63725 bytes (100.00%)\n",
      "=> 2 (joda-money-0.11.jar): Finished downloading\n",
      "-> New file at /tmp/toree_add_deps5662724810625125387/https/repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar\n"
     ]
    }
   ],
   "source": [
    "%AddDeps org.joda joda-money 0.11 --transitive --trace --verbose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AUD"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "org.joda.money.CurrencyUnit.AUD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Cell Magics<a name=\"cell-magics\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "Cell magics are magics which take the whole cell as their argument. They take the following form:\n",
    "\n",
    "```\n",
    "%%magicname\n",
    "line1\n",
    "line2\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### %%DataFrame<a name=\"dataframe\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%DataFrame` magic is used to convert a Spark SQL DataFrame into various formats. Currently, `json`, `html`, and `csv` are supported. The magic takes an expression, which evauluates to a dataframe, to perform the conversion. So, we first need to create a DataFrame object for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "case class DFRecord(key: String, value: Int)\n",
    "val sqlc = spark\n",
    "import sqlc.implicits._\n",
    "val df = sc.parallelize(1 to 10).map(x => DFRecord(x.toString, x)).toDF()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default output is `html`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "%%dataframe [arguments]\n",
       "DATAFRAME_CODE\n",
       "\n",
       "DATAGRAME_CODE can be any numbered lines of code, as long as the\n",
       "last line is a reference to a variable which is a DataFrame.\n",
       "    Option    Description                           \n",
       "------    -----------                           \n",
       "--help    Displays the help and usage text for  \n",
       "            this magic.                         \n",
       "--limit   The type of the output: html          \n",
       "            (default), csv, json (default: 10)  \n",
       "--output  The type of the output: html          \n",
       "            (default), csv, json (default: html)\n"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>key</th><th>value</th></tr><tr><td>1</td><td>1</td></tr><tr><td>2</td><td>2</td></tr><tr><td>3</td><td>3</td></tr><tr><td>4</td><td>4</td></tr><tr><td>5</td><td>5</td></tr><tr><td>6</td><td>6</td></tr><tr><td>7</td><td>7</td></tr><tr><td>8</td><td>8</td></tr><tr><td>9</td><td>9</td></tr><tr><td>10</td><td>10</td></tr></table>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify the `--output` argument to change the output type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key,value\n",
       "1,1\n",
       "2,2\n",
       "3,3\n",
       "4,4\n",
       "5,5\n",
       "6,6\n",
       "7,7\n",
       "8,8\n",
       "9,9\n",
       "10,10"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%dataframe --output=csv\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also an option to limit the number of records returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>key</th><th>value</th></tr><tr><td>1</td><td>1</td></tr><tr><td>2</td><td>2</td></tr><tr><td>3</td><td>3</td></tr></table>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%dataframe --limit=3\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### %%Html<a name=\"html\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%HTML` magic allows you to return HTML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>\n",
       "Hello, <strong>Magics</strong>!\n",
       "</p>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%html\n",
    "<p>\n",
    "Hello, <strong>Magics</strong>!\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### %%JavaScript<a name=\"javascript\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%JavaScript` magic allows to return JavaScript. The JavaScript code will run in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {},
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%JavaScript\n",
    "alert(\"Hello, Magics!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### %%PySpark<a name=\"pyspark\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%PySpark` exposes an environment with and a python interpreter and a shared `SparkContext`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4950\n"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%PySpark\n",
    "from operator import add\n",
    "print(sc.parallelize(range(1, 100)).reduce(add))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### %%SparkR<a name=\"sparkr\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%SparkR` exposes an environment with and an R interpreter and a shared `SparkContext`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* installing *binary* package ‘SparkR’ ...\n",
      "* DONE (SparkR)\n",
      "Loading required package: methods\n",
      "\n",
      "Attaching package: ‘SparkR’\n",
      "\n",
      "The following objects are masked from ‘package:stats’:\n",
      "\n",
      "    filter, na.omit\n",
      "\n",
      "The following objects are masked from ‘package:base’:\n",
      "\n",
      "    intersect, rbind, sample, subset, summary, table, transform\n",
      "\n",
      "NULL\n",
      "[1] \"Received Id 680be9df-a384-455d-b548-405560772cc1 Code df <- createDataFrame(spark, faithful)\\nhead(df)\"\n",
      "[1] \"Code expr df <- createDataFrame(spark, faithful)\"\n",
      "[2] \"Code expr head(df)\"                                   \n",
      "[1] \"Result type character 7\"\n",
      "[1] \"Success 680be9df-a384-455d-b548-405560772cc1   eruptions waiting\"\n",
      "[2] \"Success 680be9df-a384-455d-b548-405560772cc1 1     3.600      79\"\n",
      "[3] \"Success 680be9df-a384-455d-b548-405560772cc1 2     1.800      54\"\n",
      "[4] \"Success 680be9df-a384-455d-b548-405560772cc1 3     3.333      74\"\n",
      "[5] \"Success 680be9df-a384-455d-b548-405560772cc1 4     2.283      62\"\n",
      "[6] \"Success 680be9df-a384-455d-b548-405560772cc1 5     4.533      85\"\n",
      "[7] \"Success 680be9df-a384-455d-b548-405560772cc1 6     2.883      55\"\n",
      "[1] \"Marking success with output: eruptions waiting\\n1     3.600      79\\n2     1.800      54\\n3     3.333      74\\n4     2.283      62\\n5     4.533      85\\n6     2.883      55\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "eruptions waiting\n",
       "1     3.600      79\n",
       "2     1.800      54\n",
       "3     3.333      74\n",
       "4     2.283      62\n",
       "5     4.533      85\n",
       "6     2.883      55"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%SparkR\n",
    "df <- createDataFrame(spark, faithful)\n",
    "head(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### %%SparkSQL<a name=\"sparksql\"></a><span style=\"float: right; font-size: 0.5em\"><a href=\"#top\">Top</a></span>\n",
    "The `%%SparkSQL` magic allows for SQL queries to be performed against tables saved in spark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val sqlc = spark\n",
    "import sqlc.implicits._\n",
    "case class Record(key: String, value: Int)\n",
    "val df = sc.parallelize(1 to 10).map(x => Record(x.toString, x)).toDF()\n",
    "df.registerTempTable(\"MYTABLE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+---+-----+\n",
       "|key|value|\n",
       "+---+-----+\n",
       "|  6|    6|\n",
       "|  7|    7|\n",
       "|  8|    8|\n",
       "|  9|    9|\n",
       "| 10|   10|\n",
       "+---+-----+\n",
       "\n"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%SQL\n",
    "SELECT * FROM MYTABLE WHERE value >= 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+---+-----+\n",
       "|key|value|\n",
       "+---+-----+\n",
       "|  4|    4|\n",
       "|  5|    5|\n",
       "|  6|    6|\n",
       "|  7|    7|\n",
       "|  8|    8|\n",
       "|  9|    9|\n",
       "| 10|   10|\n",
       "+---+-----+\n",
       "\n"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%SQL\n",
    "SELECT * FROM MYTABLE WHERE value >= 4"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Apache Toree - Scala",
   "language": "scala",
   "name": "apache_toree_scala"
  },
  "language_info": {
   "file_extension": ".scala",
   "name": "scala",
   "version": "2.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}