Google Git
Sign in
apache / spark-website / 28656f2c00baf1bc2d26570a16b4f96f9a890723 / . / site / docs / 1.0.0 / mllib-linear-methods.html
blob: 87cf5f0412ba53e06c04895e2535bb2285cf526f [file] [log] [blame]
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Linear Methods - MLlib - Spark 1.0.0 Documentation</title>
<meta name="description" content="">
<link rel="stylesheet" href="css/bootstrap.min.css">
<style>
body {
padding-top: 60px;
padding-bottom: 40px;
}
</style>
<meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="css/bootstrap-responsive.min.css">
<link rel="stylesheet" href="css/main.css">
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
<link rel="stylesheet" href="css/pygments-default.css">
<!-- Google analytics script -->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-32518208-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
<![endif]-->
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
<div class="navbar navbar-fixed-top" id="topbar">
<div class="navbar-inner">
<div class="container">
<div class="brand"><a href="index.html">
<img src="img/spark-logo-hd.png" style="height:50px;"/></a><span class="version">1.0.0</span>
</div>
<ul class="nav">
<!--TODO(andyk): Add class="active" attribute to li some how.-->
<li><a href="index.html">Overview</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="quick-start.html">Quick Start</a></li>
<li><a href="programming-guide.html">Spark Programming Guide</a></li>
<li class="divider"></li>
<li><a href="streaming-programming-guide.html">Spark Streaming</a></li>
<li><a href="sql-programming-guide.html">Spark SQL</a></li>
<li><a href="mllib-guide.html">MLlib (Machine Learning)</a></li>
<li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>
<li><a href="bagel-programming-guide.html">Bagel (Pregel on Spark)</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="api/scala/index.html#org.apache.spark.package">Scaladoc</a></li>
<li><a href="api/java/index.html">Javadoc</a></li>
<li><a href="api/python/index.html">Python API</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="cluster-overview.html">Overview</a></li>
<li><a href="submitting-applications.html">Submitting Applications</a></li>
<li class="divider"></li>
<li><a href="ec2-scripts.html">Amazon EC2</a></li>
<li><a href="spark-standalone.html">Standalone Mode</a></li>
<li><a href="running-on-mesos.html">Mesos</a></li>
<li><a href="running-on-yarn.html">YARN</a></li>
</ul>
</li>
<li class="dropdown">
<a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="configuration.html">Configuration</a></li>
<li><a href="monitoring.html">Monitoring</a></li>
<li><a href="tuning.html">Tuning Guide</a></li>
<li><a href="job-scheduling.html">Job Scheduling</a></li>
<li><a href="security.html">Security</a></li>
<li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
<li><a href="hadoop-third-party-distributions.html">3<sup>rd</sup>-Party Hadoop Distros</a></li>
<li class="divider"></li>
<li><a href="building-with-maven.html">Building Spark with Maven</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark">Contributing to Spark</a></li>
</ul>
</li>
</ul>
<!--<p class="navbar-text pull-right"><span class="version-text">v1.0.0</span></p>-->
</div>
</div>
</div>
<div class="container" id="content">
<h1 class="title"><a href="mllib-guide.html">MLlib</a> - Linear Methods</h1>
<ul id="markdown-toc">
<li><a href="#mathematical-formulation">Mathematical formulation</a> <ul>
<li><a href="#loss-functions">Loss functions</a></li>
<li><a href="#regularizers">Regularizers</a></li>
</ul>
</li>
<li><a href="#binary-classification">Binary classification</a> <ul>
<li><a href="#linear-support-vector-machine-svm">Linear support vector machine (SVM)</a></li>
<li><a href="#logistic-regression">Logistic regression</a></li>
<li><a href="#evaluation-metrics">Evaluation metrics</a></li>
<li><a href="#examples">Examples</a></li>
</ul>
</li>
<li><a href="#linear-least-squares-lasso-and-ridge-regression">Linear least squares, Lasso, and ridge regression</a> <ul>
<li><a href="#examples-1">Examples</a></li>
</ul>
</li>
<li><a href="#implementation-developer">Implementation (developer)</a></li>
</ul>
<p><code>\[
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\x}{\mathbf{x}}
\newcommand{\y}{\mathbf{y}}
\newcommand{\wv}{\mathbf{w}}
\newcommand{\av}{\mathbf{\alpha}}
\newcommand{\bv}{\mathbf{b}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\id}{\mathbf{I}}
\newcommand{\ind}{\mathbf{1}}
\newcommand{\0}{\mathbf{0}}
\newcommand{\unit}{\mathbf{e}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\zero}{\mathbf{0}}
\]</code></p>
<h2 id="mathematical-formulation">Mathematical formulation</h2>
<p>Many standard <em>machine learning</em> methods can be formulated as a convex optimization problem, i.e.
the task of finding a minimizer of a convex function <code>$f$</code> that depends on a variable vector
<code>$\wv$</code> (called <code>weights</code> in the code), which has <code>$d$</code> entries.
Formally, we can write this as the optimization problem <code>$\min_{\wv \in\R^d} \; f(\wv)$</code>, where
the objective function is of the form
<code>\begin{equation}
f(\wv) :=
\frac1n \sum_{i=1}^n L(\wv;\x_i,y_i) +
\lambda\, R(\wv_i)
\label{eq:regPrimal}
\ .
\end{equation}</code>
Here the vectors <code>$\x_i\in\R^d$</code> are the training data examples, for <code>$1\le i\le n$</code>, and
<code>$y_i\in\R$</code> are their corresponding labels, which we want to predict.
We call the method <em>linear</em> if $L(\wv; \x, y)$ can be expressed as a function of $\wv^T x$ and $y$.
Several MLlib&#8217;s classification and regression algorithms fall into this category,
and are discussed here.</p>
<p>The objective function <code>$f$</code> has two parts:
the loss that measures the error of the model on the training data,
and the regularizer that measures the complexity of the model.
The loss function <code>$L(\wv;.)$</code> must be a convex function in <code>$\wv$</code>.
The fixed regularization parameter <code>$\lambda \ge 0$</code> (<code>regParam</code> in the code) defines the trade-off
between the two goals of small loss and small model complexity.</p>
<h3 id="loss-functions">Loss functions</h3>
<p>The following table summarizes the loss functions and their gradients or sub-gradients for the
methods MLlib supports:</p>
<table class="table">
<thead>
<tr><th></th><th>loss function $L(\wv; \x, y)$</th><th>gradient or sub-gradient</th></tr>
</thead>
<tbody>
<tr>
<td>hinge loss</td><td>$\max \{0, 1-y \wv^T \x \}, \quad y \in \{-1, +1\}$</td>
<td>$\begin{cases}-y \cdot \x &amp; \text{if $y \wv^T \x &lt;1$}, \\ 0 &amp;
\text{otherwise}.\end{cases}$</td>
</tr>
<tr>
<td>logistic loss</td><td>$\log(1+\exp( -y \wv^T \x)), \quad y \in \{-1, +1\}$</td>
<td>$-y \left(1-\frac1{1+\exp(-y \wv^T \x)} \right) \cdot \x$</td>
</tr>
<tr>
<td>squared loss</td><td>$\frac{1}{2} (\wv^T \x - y)^2, \quad y \in \R$</td>
<td>$(\wv^T \x - y) \cdot \x$</td>
</tr>
</tbody>
</table>
<h3 id="regularizers">Regularizers</h3>
<p>The purpose of the <a href="http://en.wikipedia.org/wiki/Regularization_(mathematics)">regularizer</a> is to
encourage simple models, by punishing the complexity of the model <code>$\wv$</code>, in order to e.g. avoid
over-fitting.
We support the following regularizers in MLlib:</p>
<table class="table">
<thead>
<tr><th></th><th>regularizer $R(\wv)$</th><th>gradient or sub-gradient</th></tr>
</thead>
<tbody>
<tr>
<td>zero (unregularized)</td><td>0</td><td>$\0$</td>
</tr>
<tr>
<td>L2</td><td>$\frac{1}{2}\|\wv\|_2^2$</td><td>$\wv$</td>
</tr>
<tr>
<td>L1</td><td>$\|\wv\|_1$</td><td>$\mathrm{sign}(\wv)$</td>
</tr>
</tbody>
</table>
<p>Here <code>$\mathrm{sign}(\wv)$</code> is the vector consisting of the signs (<code>$\pm1$</code>) of all the entries
of <code>$\wv$</code>.</p>
<p>L2-regularized problems are generally easier to solve than L1-regularized due to smoothness.
However, L1 regularization can help promote sparsity in weights, leading to simpler models, which is
also used for feature selection. It is not recommended to train models without any regularization,
especially when the number of training examples is small.</p>
<h2 id="binary-classification">Binary classification</h2>
<p><a href="http://en.wikipedia.org/wiki/Binary_classification">Binary classification</a> is to divide items into
two categories: positive and negative. MLlib supports two linear methods for binary classification:
linear support vector machine (SVM) and logistic regression. The training data set is represented
by an RDD of <a href="mllib-data-types.html">LabeledPoint</a> in MLlib. Note that, in the mathematical
formulation, a training label $y$ is either $+1$ (positive) or $-1$ (negative), which is convenient
for the formulation. <em>However</em>, the negative label is represented by $0$ in MLlib instead of $-1$,
to be consistent with multiclass labeling.</p>
<h3 id="linear-support-vector-machine-svm">Linear support vector machine (SVM)</h3>
<p>The <a href="http://en.wikipedia.org/wiki/Support_vector_machine#Linear_SVM">linear SVM</a>
has become a standard choice for large-scale classification tasks.
The name &#8220;linear SVM&#8221; is actually ambiguous.
By &#8220;linear SVM&#8221;, we mean specifically the linear method with the loss function in formulation
<code>$\eqref{eq:regPrimal}$</code> given by the hinge loss
<code>\[
L(\wv;\x,y) := \max \{0, 1-y \wv^T \x \}.
\]</code>
By default, linear SVMs are trained with an L2 regularization.
We also support alternative L1 regularization. In this case,
the problem becomes a <a href="http://en.wikipedia.org/wiki/Linear_programming">linear program</a>.</p>
<p>Linear SVM algorithm outputs a SVM model, which makes predictions based on the value of $\wv^T \x$.
By the default, if $\wv^T \x \geq 0$, the outcome is positive, or negative otherwise.
However, quite often in practice, the default threshold $0$ is not a good choice.
The threshold should be determined via model evaluation.</p>
<h3 id="logistic-regression">Logistic regression</h3>
<p><a href="http://en.wikipedia.org/wiki/Logistic_regression">Logistic regression</a> is widely used to predict a
binary response. It is a linear method with the loss function in formulation
<code>$\eqref{eq:regPrimal}$</code> given by the logistic loss
<code>\[
L(\wv;\x,y) := \log(1+\exp( -y \wv^T \x)).
\]</code></p>
<p>Logistic regression algorithm outputs a logistic regression model, which makes predictions by
applying the logistic function
<code>\[
\mathrm{logit}(z) = \frac{1}{1 + e^{-z}}
\]</code>
$\wv^T \x$.
By default, if $\mathrm{logit}(\wv^T x) &gt; 0.5$, the outcome is positive, or negative otherwise.
For the same reason mentioned above, quite often in practice, this default threshold is not a good choice.
The threshold should be determined via model evaluation.</p>
<h3 id="evaluation-metrics">Evaluation metrics</h3>
<p>MLlib supports common evaluation metrics for binary classification (not available in Python). This
includes precision, recall, <a href="http://en.wikipedia.org/wiki/F1_score">F-measure</a>,
<a href="http://en.wikipedia.org/wiki/Receiver_operating_characteristic">receiver operating characteristic (ROC)</a>,
precision-recall curve, and
<a href="http://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve">area under the curves (AUC)</a>.
Among the metrics, area under ROC is commonly used to compare models and precision/recall/F-measure
can help determine the threshold to use.</p>
<h3 id="examples">Examples</h3>
<div class="codetabs">
<div data-lang="scala">
<p>The following code snippet illustrates how to load a sample dataset, execute a
training algorithm on this training data using a static method in the algorithm
object, and make predictions with the resulting model to compute the training
error.</p>
<div class="highlight"><pre><code class="scala"><span class="k">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.classification.SVMWithSGD</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.evaluation.BinaryClassificationMetrics</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.regression.LabeledPoint</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.linalg.Vectors</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.util.MLUtils</span>
<span class="c1">// Load training data in LIBSVM format.</span>
<span class="k">val</span> <span class="n">data</span> <span class="k">=</span> <span class="nc">MLUtils</span><span class="o">.</span><span class="n">loadLibSVMFile</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="s">&quot;mllib/data/sample_libsvm_data.txt&quot;</span><span class="o">)</span>
<span class="c1">// Split data into training (60%) and test (40%).</span>
<span class="k">val</span> <span class="n">splits</span> <span class="k">=</span> <span class="n">data</span><span class="o">.</span><span class="n">randomSplit</span><span class="o">(</span><span class="nc">Array</span><span class="o">(</span><span class="mf">0.6</span><span class="o">,</span> <span class="mf">0.4</span><span class="o">),</span> <span class="n">seed</span> <span class="k">=</span> <span class="mi">11L</span><span class="o">)</span>
<span class="k">val</span> <span class="n">training</span> <span class="k">=</span> <span class="n">splits</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="n">cache</span><span class="o">()</span>
<span class="k">val</span> <span class="n">test</span> <span class="k">=</span> <span class="n">splits</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>
<span class="c1">// Run training algorithm to build the model</span>
<span class="k">val</span> <span class="n">numIterations</span> <span class="k">=</span> <span class="mi">100</span>
<span class="k">val</span> <span class="n">model</span> <span class="k">=</span> <span class="nc">SVMWithSGD</span><span class="o">.</span><span class="n">train</span><span class="o">(</span><span class="n">training</span><span class="o">,</span> <span class="n">numIterations</span><span class="o">)</span>
<span class="c1">// Clear the default threshold.</span>
<span class="n">model</span><span class="o">.</span><span class="n">clearThreshold</span><span class="o">()</span>
<span class="c1">// Compute raw scores on the test set. </span>
<span class="k">val</span> <span class="n">scoreAndLabels</span> <span class="k">=</span> <span class="n">test</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">point</span> <span class="k">=&gt;</span>
<span class="k">val</span> <span class="n">score</span> <span class="k">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="o">(</span><span class="n">point</span><span class="o">.</span><span class="n">features</span><span class="o">)</span>
<span class="o">(</span><span class="n">score</span><span class="o">,</span> <span class="n">point</span><span class="o">.</span><span class="n">label</span><span class="o">)</span>
<span class="o">}</span>
<span class="c1">// Get evaluation metrics.</span>
<span class="k">val</span> <span class="n">metrics</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">BinaryClassificationMetrics</span><span class="o">(</span><span class="n">scoreAndLabels</span><span class="o">)</span>
<span class="k">val</span> <span class="n">auROC</span> <span class="k">=</span> <span class="n">metrics</span><span class="o">.</span><span class="n">areaUnderROC</span><span class="o">()</span>
<span class="n">println</span><span class="o">(</span><span class="s">&quot;Area under ROC = &quot;</span> <span class="o">+</span> <span class="n">auROC</span><span class="o">)</span>
</code></pre></div>
<p>The <code>SVMWithSGD.train()</code> method by default performs L2 regularization with the
regularization parameter set to 1.0. If we want to configure this algorithm, we
can customize <code>SVMWithSGD</code> further by creating a new object directly and
calling setter methods. All other MLlib algorithms support customization in
this way as well. For example, the following code produces an L1 regularized
variant of SVMs with regularization parameter set to 0.1, and runs the training
algorithm for 200 iterations.</p>
<div class="highlight"><pre><code class="scala"><span class="k">import</span> <span class="nn">org.apache.spark.mllib.optimization.L1Updater</span>
<span class="k">val</span> <span class="n">svmAlg</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">SVMWithSGD</span><span class="o">()</span>
<span class="n">svmAlg</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span>
<span class="n">setNumIterations</span><span class="o">(</span><span class="mi">200</span><span class="o">).</span>
<span class="n">setRegParam</span><span class="o">(</span><span class="mf">0.1</span><span class="o">).</span>
<span class="n">setUpdater</span><span class="o">(</span><span class="k">new</span> <span class="n">L1Updater</span><span class="o">)</span>
<span class="k">val</span> <span class="n">modelL1</span> <span class="k">=</span> <span class="n">svmAlg</span><span class="o">.</span><span class="n">run</span><span class="o">(</span><span class="n">training</span><span class="o">)</span>
</code></pre></div>
<p>Similarly, you can use replace <code>SVMWithSGD</code> by
<a href="api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithSGD"><code>LogisticRegressionWithSGD</code></a>.</p>
</div>
<div data-lang="java">
<p>All of MLlib&#8217;s methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate <code>JavaRDD</code> class. You can convert a Java RDD to a Scala one by
calling <code>.rdd()</code> on your <code>JavaRDD</code> object.</p>
</div>
<div data-lang="python">
<p>The following example shows how to load a sample dataset, build Logistic Regression model,
and make predictions with the resulting model to compute the training error.</p>
<div class="highlight"><pre><code class="python"><span class="kn">from</span> <span class="nn">pyspark.mllib.classification</span> <span class="kn">import</span> <span class="n">LogisticRegressionWithSGD</span>
<span class="kn">from</span> <span class="nn">pyspark.mllib.regression</span> <span class="kn">import</span> <span class="n">LabeledPoint</span>
<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">array</span>
<span class="c"># Load and parse the data</span>
<span class="k">def</span> <span class="nf">parsePoint</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="nb">float</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&#39; &#39;</span><span class="p">)]</span>
<span class="k">return</span> <span class="n">LabeledPoint</span><span class="p">(</span><span class="n">values</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">values</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">textFile</span><span class="p">(</span><span class="s">&quot;mllib/data/sample_svm_data.txt&quot;</span><span class="p">)</span>
<span class="n">parsedData</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">parsePoint</span><span class="p">)</span>
<span class="c"># Build the model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">LogisticRegressionWithSGD</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">parsedData</span><span class="p">)</span>
<span class="c"># Evaluating the model on training data</span>
<span class="n">labelsAndPreds</span> <span class="o">=</span> <span class="n">parsedData</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">p</span><span class="p">:</span> <span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">features</span><span class="p">)))</span>
<span class="n">trainErr</span> <span class="o">=</span> <span class="n">labelsAndPreds</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">p</span><span class="p">):</span> <span class="n">v</span> <span class="o">!=</span> <span class="n">p</span><span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span> <span class="o">/</span> <span class="nb">float</span><span class="p">(</span><span class="n">parsedData</span><span class="o">.</span><span class="n">count</span><span class="p">())</span>
<span class="k">print</span><span class="p">(</span><span class="s">&quot;Training Error = &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">trainErr</span><span class="p">))</span>
</code></pre></div>
</div>
</div>
<h2 id="linear-least-squares-lasso-and-ridge-regression">Linear least squares, Lasso, and ridge regression</h2>
<p>Linear least squares is a family of linear methods with the loss function in formulation
<code>$\eqref{eq:regPrimal}$</code> given by the squared loss</p>
<p><code>\[
L(\wv;\x,y) := \frac{1}{2} (\wv^T \x - y)^2.
\]</code></p>
<p>Depending on the regularization type, we call the method
<a href="http://en.wikipedia.org/wiki/Ordinary_least_squares"><em>ordinary least squares</em></a> or simply
<a href="http://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)"><em>linear least squares</em></a> if there
is no regularization, <a href="http://en.wikipedia.org/wiki/Ridge_regression"><em>ridge regression</em></a> if L2
regularization is used, and <a href="http://en.wikipedia.org/wiki/Lasso_(statistics)"><em>Lasso</em></a> if L1
regularization is used. This average loss $\frac{1}{n} \sum_{i=1}^n (\wv^T x_i - y_i)^2$ is also
known as the <a href="http://en.wikipedia.org/wiki/Mean_squared_error">mean squared error</a>.</p>
<p>Note that the squared loss is sensitive to outliers.
Regularization or a robust alternative (e.g., $\ell_1$ regression) is usually necessary in practice.</p>
<h3 id="examples-1">Examples</h3>
<div class="codetabs">
<div data-lang="scala">
<p>The following example demonstrate how to load training data, parse it as an RDD of LabeledPoint.
The example then uses LinearRegressionWithSGD to build a simple linear model to predict label
values. We compute the Mean Squared Error at the end to evaluate
<a href="http://en.wikipedia.org/wiki/Goodness_of_fit">goodness of fit</a>.</p>
<div class="highlight"><pre><code class="scala"><span class="k">import</span> <span class="nn">org.apache.spark.mllib.regression.LinearRegressionWithSGD</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.regression.LabeledPoint</span>
<span class="k">import</span> <span class="nn">org.apache.spark.mllib.linalg.Vectors</span>
<span class="c1">// Load and parse the data</span>
<span class="k">val</span> <span class="n">data</span> <span class="k">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">textFile</span><span class="o">(</span><span class="s">&quot;mllib/data/ridge-data/lpsa.data&quot;</span><span class="o">)</span>
<span class="k">val</span> <span class="n">parsedData</span> <span class="k">=</span> <span class="n">data</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">line</span> <span class="k">=&gt;</span>
<span class="k">val</span> <span class="n">parts</span> <span class="k">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="sc">&#39;,&#39;</span><span class="o">)</span>
<span class="nc">LabeledPoint</span><span class="o">(</span><span class="n">parts</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="n">toDouble</span><span class="o">,</span> <span class="nc">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="o">(</span><span class="n">parts</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="n">split</span><span class="o">(</span><span class="sc">&#39; &#39;</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toDouble</span><span class="o">)))</span>
<span class="o">}</span>
<span class="c1">// Building the model</span>
<span class="k">val</span> <span class="n">numIterations</span> <span class="k">=</span> <span class="mi">100</span>
<span class="k">val</span> <span class="n">model</span> <span class="k">=</span> <span class="nc">LinearRegressionWithSGD</span><span class="o">.</span><span class="n">train</span><span class="o">(</span><span class="n">parsedData</span><span class="o">,</span> <span class="n">numIterations</span><span class="o">)</span>
<span class="c1">// Evaluate model on training examples and compute training error</span>
<span class="k">val</span> <span class="n">valuesAndPreds</span> <span class="k">=</span> <span class="n">parsedData</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">point</span> <span class="k">=&gt;</span>
<span class="k">val</span> <span class="n">prediction</span> <span class="k">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="o">(</span><span class="n">point</span><span class="o">.</span><span class="n">features</span><span class="o">)</span>
<span class="o">(</span><span class="n">point</span><span class="o">.</span><span class="n">label</span><span class="o">,</span> <span class="n">prediction</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">val</span> <span class="nc">MSE</span> <span class="k">=</span> <span class="n">valuesAndPreds</span><span class="o">.</span><span class="n">map</span><span class="o">{</span><span class="k">case</span><span class="o">(</span><span class="n">v</span><span class="o">,</span> <span class="n">p</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">math</span><span class="o">.</span><span class="n">pow</span><span class="o">((</span><span class="n">v</span> <span class="o">-</span> <span class="n">p</span><span class="o">),</span> <span class="mi">2</span><span class="o">)}.</span><span class="n">mean</span><span class="o">()</span>
<span class="n">println</span><span class="o">(</span><span class="s">&quot;training Mean Squared Error = &quot;</span> <span class="o">+</span> <span class="nc">MSE</span><span class="o">)</span>
</code></pre></div>
<p>Similarly you can use
<a href="api/scala/index.html#org.apache.spark.mllib.regression.RidgeRegressionWithSGD"><code>RidgeRegressionWithSGD</code></a>
and <a href="api/scala/index.html#org.apache.spark.mllib.regression.LassoWithSGD"><code>LassoWithSGD</code></a>.</p>
</div>
<div data-lang="java">
<p>All of MLlib&#8217;s methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate <code>JavaRDD</code> class. You can convert a Java RDD to a Scala one by
calling <code>.rdd()</code> on your <code>JavaRDD</code> object.</p>
</div>
<div data-lang="python">
<p>The following example demonstrate how to load training data, parse it as an RDD of LabeledPoint.
The example then uses LinearRegressionWithSGD to build a simple linear model to predict label
values. We compute the Mean Squared Error at the end to evaluate
<a href="http://en.wikipedia.org/wiki/Goodness_of_fit">goodness of fit</a>.</p>
<div class="highlight"><pre><code class="python"><span class="kn">from</span> <span class="nn">pyspark.mllib.regression</span> <span class="kn">import</span> <span class="n">LabeledPoint</span><span class="p">,</span> <span class="n">LinearRegressionWithSGD</span>
<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">array</span>
<span class="c"># Load and parse the data</span>
<span class="k">def</span> <span class="nf">parsePoint</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="nb">float</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">line</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">&#39;,&#39;</span><span class="p">,</span> <span class="s">&#39; &#39;</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&#39; &#39;</span><span class="p">)]</span>
<span class="k">return</span> <span class="n">LabeledPoint</span><span class="p">(</span><span class="n">values</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">values</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">textFile</span><span class="p">(</span><span class="s">&quot;mllib/data/ridge-data/lpsa.data&quot;</span><span class="p">)</span>
<span class="n">parsedData</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">parsePoint</span><span class="p">)</span>
<span class="c"># Build the model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">LinearRegressionWithSGD</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">parsedData</span><span class="p">)</span>
<span class="c"># Evaluate the model on training data</span>
<span class="n">valuesAndPreds</span> <span class="o">=</span> <span class="n">parsedData</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">p</span><span class="p">:</span> <span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">features</span><span class="p">)))</span>
<span class="n">MSE</span> <span class="o">=</span> <span class="n">valuesAndPreds</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">p</span><span class="p">):</span> <span class="p">(</span><span class="n">v</span> <span class="o">-</span> <span class="n">p</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span> <span class="o">/</span> <span class="n">valuesAndPreds</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="s">&quot;Mean Squared Error = &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">MSE</span><span class="p">))</span>
</code></pre></div>
</div>
</div>
<h2 id="implementation-developer">Implementation (developer)</h2>
<p>Behind the scene, MLlib implements a simple distributed version of stochastic gradient descent
(SGD), building on the underlying gradient descent primitive (as described in the <a href="mllib-optimization.html">optimization</a> section). All provided algorithms take as input a
regularization parameter (<code>regParam</code>) along with various parameters associated with stochastic
gradient descent (<code>stepSize</code>, <code>numIterations</code>, <code>miniBatchFraction</code>). For each of them, we support
all three possible regularizations (none, L1 or L2).</p>
<p>Algorithms are all implemented in Scala:</p>
<ul>
<li><a href="api/scala/index.html#org.apache.spark.mllib.classification.SVMWithSGD">SVMWithSGD</a></li>
<li><a href="api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithSGD">LogisticRegressionWithSGD</a></li>
<li><a href="api/scala/index.html#org.apache.spark.mllib.regression.LinearRegressionWithSGD">LinearRegressionWithSGD</a></li>
<li><a href="api/scala/index.html#org.apache.spark.mllib.regression.RidgeRegressionWithSGD">RidgeRegressionWithSGD</a></li>
<li><a href="api/scala/index.html#org.apache.spark.mllib.regression.LassoWithSGD">LassoWithSGD</a></li>
</ul>
<p>Python calls the Scala implementation via
<a href="api/scala/index.html#org.apache.spark.mllib.api.python.PythonMLLibAPI">PythonMLLibAPI</a>.</p>
</div> <!-- /container -->
<script src="js/vendor/jquery-1.8.0.min.js"></script>
<script src="js/vendor/bootstrap.min.js"></script>
<script src="js/main.js"></script>
<!-- MathJax Section -->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: { equationNumbers: { autoNumber: "AMS" } }
});
</script>
<script type="text/javascript"
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
MathJax.Hub.Config({
tex2jax: {
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
processEscapes: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
}
});
</script>
</body>
</html>