docs/v1.2/group__grp__mlogreg.html - madlib-site - Git at Google

 <!-- HTML header for doxygen 1.8.4-->
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
 <meta http-equiv="X-UA-Compatible" content="IE=9"/>
 <meta name="generator" content="Doxygen 1.8.4"/>
 <meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/>
 <title>MADlib: Multinomial Logistic Regression</title>
 <link href="tabs.css" rel="stylesheet" type="text/css"/>
 <script type="text/javascript" src="jquery.js"></script>
 <script type="text/javascript" src="dynsections.js"></script>
 <link href="navtree.css" rel="stylesheet" type="text/css"/>
 <script type="text/javascript" src="resize.js"></script>
 <script type="text/javascript" src="navtree.js"></script>
 <script type="text/javascript">
   $(document).ready(initResizable);
   $(window).load(resizeHeight);
 </script>
 <link href="search/search.css" rel="stylesheet" type="text/css"/>
 <script type="text/javascript" src="search/search.js"></script>
 <script type="text/javascript">
   $(document).ready(function() { searchBox.OnSelectItem(0); });
 </script>
 <script type="text/x-mathjax-config">
   MathJax.Hub.Config({
     extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
     jax: ["input/TeX","output/HTML-CSS"],
 });
 </script><script src="../mathjax/MathJax.js"></script>
 <link href="doxygen.css" rel="stylesheet" type="text/css" />
 <link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
 <!-- google analytics -->
 <script>
   (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
   (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
   m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
   })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
   ga('create', 'UA-45382226-1', 'auto');
   ga('send', 'pageview');
 </script>
 </head>
 <body>
 <div id="top"><!-- do not remove this div, it is closed by doxygen! -->
 <div id="titlearea">
 <table cellspacing="0" cellpadding="0">
  <tbody>
  <tr style="height: 56px;">
   <td style="padding-left: 0.5em;">
    <div id="projectname">MADlib
    &#160;<span id="projectnumber">1.2</span> <span style="font-size:10pt; font-style:italic"><a href="../latest/./group__grp__mlogreg.html"> A newer version is available</a></span>
    </div>
    <div id="projectbrief">User Documentation</div>
   </td>
   <!--BEGIN VERSIONS LINKS-->
     <td style="padding-left: 0.5em;">
       <div class="versionlist"><ul>
         <li class="head">More versions:</li>
         <li><a href="../v1.1/index.html">v1.1</li>
         <li><a href="../v1.0/index.html">v1.0</li>
         <li><a href="../v0.7/index.html">v0.7</li>
         <li><a href="../v0.5/index.html">v0.5</li></ul>
       </div>
     </td>
    <td>        <div id="MSearchBox" class="MSearchBoxInactive">
         <span class="left">
           <img id="MSearchSelect" src="search/mag_sel.png"
                onmouseover="return searchBox.OnSearchSelectShow()"
                onmouseout="return searchBox.OnSearchSelectHide()"
                alt=""/>
           <input type="text" id="MSearchField" value="Search" accesskey="S"
                onfocus="searchBox.OnSearchFieldFocus(true)"
                onblur="searchBox.OnSearchFieldFocus(false)"
                onkeyup="searchBox.OnSearchFieldChange(event)"/>
           </span><span class="right">
             <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
           </span>
         </div>
 </td>
  </tr>
  </tbody>
 </table>
 </div>
 <!-- end header part -->
 <!-- Generated by Doxygen 1.8.4 -->
 <script type="text/javascript">
 var searchBox = new SearchBox("searchBox", "search",false,'Search');
 </script>
 </div><!-- top -->
 <div id="side-nav" class="ui-resizable side-nav-resizable">
   <div id="nav-tree">
     <div id="nav-tree-contents">
       <div id="nav-sync" class="sync"></div>
     </div>
   </div>
   <div id="splitbar" style="-moz-user-select:none;"
        class="ui-resizable-handle">
   </div>
 </div>
 <script type="text/javascript">
 $(document).ready(function(){initNavTree('group__grp__mlogreg.html','');});
 </script>
 <div id="doc-content">
 <!-- window showing the filter options -->
 <div id="MSearchSelectWindow"
      onmouseover="return searchBox.OnSearchSelectShow()"
      onmouseout="return searchBox.OnSearchSelectHide()"
      onkeydown="return searchBox.OnSearchSelectKey(event)">
 <a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(0)"><span class="SelectionMark">&#160;</span>All</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(1)"><span class="SelectionMark">&#160;</span>Files</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(2)"><span class="SelectionMark">&#160;</span>Functions</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(3)"><span class="SelectionMark">&#160;</span>Variables</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(4)"><span class="SelectionMark">&#160;</span>Groups</a></div>

 <!-- iframe showing the search results (closed by default) -->
 <div id="MSearchResultsWindow">
 <iframe src="javascript:void(0)" frameborder="0"
         name="MSearchResults" id="MSearchResults">
 </iframe>
 </div>

 <div class="header">
   <div class="headertitle">
 <div class="title">Multinomial Logistic Regression<div class="ingroups"><a class="el" href="group__grp__glm.html">Generalized Linear Models</a></div></div>  </div>
 </div><!--header-->
 <div class="contents">
 <div class="toc"><b>Contents</b> </p>
 <ul>
 <li class="level1">
 <a href="#about">About</a> </li>
 <li class="level1">
 <a href="#train">Training Function</a> </li>
 <li class="level1">
 <a href="#output">Output Format</a> </li>
 <li class="level1">
 <a href="#examples">Examples</a> </li>
 <li class="level1">
 <a href="#seealso">See Also</a> </li>
 <li class="level1">
 <a href="#background">Technical Background</a> </li>
 <li class="level1">
 <a href="#literature">Literature</a> </li>
 </ul>
 </div><p><a class="anchor" id="about"></a></p>
 <dl class="section user"><dt>About</dt><dd></dd></dl>
 <p>Multinomial logistic regression is a widely used regression analysis tool that models the outcomes of categorical dependent random variables. The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables. Multinomial logistic regression finds the vector of coefficients that maximizes the likelihood of the observations.</p>
 <p><a class="anchor" id="train"></a></p>
 <dl class="section user"><dt>Training Function</dt><dd>The multinomial logistic regression training function has the following syntax: <pre class="fragment">mlogregr(
     source,
     depvar,
     indepvar,
     max_num_iterations,
     optimizer,
     precision,
     ref_category)
 </pre></dd></dl>
 <dl class="arglist">
 <dt>source </dt>
 <dd><p class="startdd">TEXT. The name of the table containing the input data.</p>
 <p class="enddd">The training data is expected to be of the following form:<br/>
  </p>
 <pre class="fragment">{TABLE|VIEW} source (
     ...
     dependentVariable INTEGER,
     independentVariables FLOAT8[],
     ...
 )
 </pre>  </dd>
 <dt>depvar </dt>
 <dd>TEXT. The name of the column containing the dependent variable. </dd>
 <dt>indepvar </dt>
 <dd>TEXT. Expression list to evaluate for the independent variables. An intercept variable is not assumed. The number of independent variables cannot exceed 65535. </dd>
 <dt>max_num_iterations (optional) </dt>
 <dd>INTEGER, default: 20. The maximum number of iterations that are allowed. </dd>
 <dt>optimizer (optional) </dt>
 <dd>TEXT, default: 'irls'. The name of the optimizer to use, 'newton', or 'irls' (iteratively reweighted least squares).  </dd>
 <dt>precision (optional) </dt>
 <dd>FLOAT8, default: 0.0001. The difference between log-likelihood values in successive iterations that should indicate convergence. Note that a non-positive value here disables the convergence criterion, and execution will only stop after <code>max_num_iterations</code> iterations. </dd>
 <dt>ref_category (optional) </dt>
 <dd>INTEGER, default: 0. The reference category ranges from [0, <em>numCategories</em> &ndash; 1]. </dd>
 </dl>
 <p><a class="anchor" id="notes"></a></p>
 <dl class="section note"><dt>Note</dt><dd>All table names can be optionally schema qualified and all table and column names should follow the same case-sensitivity and quoting rules as in the database.</dd></dl>
 <p><a class="anchor" id="output"></a></p>
 <dl class="section user"><dt>Output Format</dt><dd>The output of the <a class="el" href="multilogistic_8sql__in.html#a7be20ccb465d47808e18149140fc666f">mlogregr()</a> function is a composite type with the following fields: <dl class="arglist">
 <dt>ref_category </dt>
 <dd>INTEGER. The reference category. Categories are encoded as integers with values from {0, 1, 2,..., <em>numCategories</em> &ndash; 1} </dd>
 <dt>coef </dt>
 <dd>FLOAT8[]. An array of coefficients, \( \boldsymbol c \).  </dd>
 <dt>log_likelihood </dt>
 <dd>FLOAT8. The log-likelihood, \( l(\boldsymbol c) \). </dd>
 <dt>std_err </dt>
 <dd>FLOAT8[]. An array of the standard errors. </dd>
 <dt>z_stats </dt>
 <dd>FLOAT8[]. An array of the Wald z-statistics. </dd>
 <dt>p_values </dt>
 <dd>FLOAT8[]. An array of the Wald p-values. </dd>
 <dt>odds_ratios </dt>
 <dd>FLOAT8[]. An array of the odds ratios. </dd>
 <dt>condition_no </dt>
 <dd>FLOAT8. The condition number of the matrix, computed using the coefficients of the iteration immediately preceding convergence. </dd>
 <dt>num_iterations </dt>
 <dd>INTEGER. The number of iterations executed before the algorithm completed. </dd>
 </dl>
 </dd></dl>
 <p><a class="anchor" id="examples"></a></p>
 <dl class="section user"><dt>Examples</dt><dd><ol type="1">
 <li>Create the training data table. <pre class="fragment">sql&gt; DROP TABLE IF EXISTS patients;
 sql&gt; CREATE TABLE patients (id INTEGER NOT NULL, second_attack INTEGER,
     treatment INTEGER, trait_anxiety INTEGER);
 sql&gt; COPY patients FROM STDIN WITH DELIMITER '|';
   1 |             1 |         1 |            70
   3 |             1 |         1 |            50
   5 |             1 |         0 |            40
   7 |             1 |         0 |            75
   9 |             1 |         0 |            70
  11 |             0 |         1 |            65
  13 |             0 |         1 |            45
  15 |             0 |         1 |            40
  17 |             0 |         0 |            55
  19 |             0 |         0 |            50
   2 |             1 |         1 |            80
   4 |             1 |         0 |            60
   6 |             1 |         0 |            65
   8 |             1 |         0 |            80
  10 |             1 |         0 |            60
  12 |             0 |         1 |            50
  14 |             0 |         1 |            35
  16 |             0 |         1 |            50
  18 |             0 |         0 |            45
  20 |             0 |         0 |            60
 \.
 </pre></li>
 <li>Run the multilogistic regression function. <pre class="fragment">sql&gt; \x on
 Expanded display is on.
 sql&gt; SELECT * FROM madlib.mlogregr('patients', 'second_attack',
                    'ARRAY[1, treatment, trait_anxiety]', 20,
                    'irls', 0.0001, 1);
 -[ RECORD 1 ]--+----------------------------------------------------------
 ref_category   | 1
 coef           | {6.3634699417818,1.02410605239327,-0.119044916668605}
 log_likelihood | -9.41018298388876
 std_err        | {3.21389766375092,1.17107844860319,0.0549790458269306}
 z_stats        | {1.97998524145757,0.874498248699546,-2.16527796868917}
 p_values       | {0.0477051870698143,0.381846973530451,0.0303664045046178}
 odds_ratios    | {580.256322802414,2.78460505656035,0.887767924288744}
 condition_no   | 106329.420371447
 num_iterations | 5
 </pre></li>
 </ol>
 </dd></dl>
 <p><a class="anchor" id="seealso"></a></p>
 <dl class="section see"><dt>See Also</dt><dd>File <a class="el" href="multilogistic_8sql__in.html" title="SQL functions for multinomial logistic regression. ">multilogistic.sql_in</a> documenting the multinomial <a class="el" href="logistic_8sql__in.html#a4ded9be5c8b111dbb3109efaad83d69e" title="Evaluate the usual logistic function in an under-/overflow-safe way. ">logistic</a> regression functions </dd>
 <dd>
 grp_logregr</dd></dl>
 <p><a class="anchor" id="background"></a></p>
 <dl class="section user"><dt>Technical Background</dt><dd>Multinomial logistic regression models the outcomes of categorical dependent random variables (denoted \( Y \in \{ 0,1,2 \ldots k \} \)). The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables (usually denoted \( \boldsymbol x \)). That is, <p class="formulaDsp">
 \[ E[Y \mid \boldsymbol x] = \sigma(\boldsymbol c^T \boldsymbol x) \]
 </p>
  for some unknown vector of coefficients \( \boldsymbol c \) and where \( \sigma(x) = \frac{1}{1 + \exp(-x)} \) is the logistic function. Multinomial logistic regression finds the vector of coefficients \( \boldsymbol c \) that maximizes the likelihood of the observations.</dd></dl>
 <p>Let</p>
 <ul>
 <li>\( \boldsymbol y \in \{ 0,1 \}^{n \times k} \) denote the vector of observed dependent variables, with \( n \) rows and \( k \) columns, containing the observed values of the dependent variable,</li>
 <li>\( X \in \mathbf R^{n \times k} \) denote the design matrix with \( k \) columns and \( n \) rows, containing all observed vectors of independent variables \( \boldsymbol x_i \) as rows.</li>
 </ul>
 <p>By definition, </p>
 <p class="formulaDsp">
 \[ P[Y = y_i | \boldsymbol x_i] = \sigma((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i) \,. \]
 </p>
 <p> Maximizing the likelihood \( \prod_{i=1}^n \Pr(Y = y_i \mid \boldsymbol x_i) \) is equivalent to maximizing the log-likelihood \( \sum_{i=1}^n \log \Pr(Y = y_i \mid \boldsymbol x_i) \), which simplifies to </p>
 <p class="formulaDsp">
 \[ l(\boldsymbol c) = -\sum_{i=1}^n \log(1 + \exp((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i)) \,. \]
 </p>
 <p> The Hessian of this objective is \( H = -X^T A X \) where \( A = \text{diag}(a_1, \dots, a_n) \) is the diagonal matrix with \( a_i = \sigma(\boldsymbol c^T \boldsymbol x) \cdot \sigma(-\boldsymbol c^T \boldsymbol x) \,. \) Since \( H \) is non-positive definite, \( l(\boldsymbol c) \) is convex. There are many techniques for solving convex optimization problems. Currently, logistic regression in MADlib can use:</p>
 <ul>
 <li>Iteratively Reweighted Least Squares</li>
 </ul>
 <p>We estimate the standard error for coefficient \( i \) as </p>
 <p class="formulaDsp">
 \[ \mathit{se}(c_i) = \left( (X^T A X)^{-1} \right)_{ii} \,. \]
 </p>
 <p> The Wald z-statistic is </p>
 <p class="formulaDsp">
 \[ z_i = \frac{c_i}{\mathit{se}(c_i)} \,. \]
 </p>
 <p>The Wald \( p \)-value for coefficient \( i \) gives the probability (under the assumptions inherent in the Wald test) of seeing a value at least as extreme as the one observed, provided that the null hypothesis ( \( c_i = 0 \)) is true. Letting \( F \) denote the cumulative density function of a standard normal distribution, the Wald \( p \)-value for coefficient \( i \) is therefore </p>
 <p class="formulaDsp">
 \[ p_i = \Pr(|Z| \geq |z_i|) = 2 \cdot (1 - F( |z_i| )) \]
 </p>
 <p> where \( Z \) is a standard normally distributed random variable.</p>
 <p>The odds ratio for coefficient \( i \) is estimated as \( \exp(c_i) \).</p>
 <p>The condition number is computed as \( \kappa(X^T A X) \) during the iteration immediately <em>preceding</em> convergence (i.e., \( A \) is computed using the coefficients of the previous iteration). A large condition number (say, more than 1000) indicates the presence of significant multicollinearity.</p>
 <p>The multinomial logistic regression uses a default reference category of zero, and the regression coefficients in the output are in the order described below. For a problem with \( K \) dependent variables \( (1, ..., K) \) and \( J \) categories \( (0, ..., J-1) \), let \( {m_{k,j}} \) denote the coefficient for dependent variable \( k \) and category \( j \). The output is \( {m_{k_1, j_0}, m_{k_1, j_1} \ldots m_{k_1, j_{J-1}}, m_{k_2, j_0}, m_{k_2, j_1}, \ldots m_{k_2, j_{J-1}} \ldots m_{k_K, j_{J-1}}} \). The order is NOT CONSISTENT with the multinomial regression marginal effect calculation with function <em>marginal_mlogregr</em>. This is deliberate because the interfaces of all multinomial regressions (robust, clustered, ...) will be moved to match that used in marginal.</p>
 <p><a class="anchor" id="literature"></a></p>
 <dl class="section user"><dt>Literature</dt><dd></dd></dl>
 <p>A collection of nice write-ups, with valuable pointers into further literature:</p>
 <p>[1] Annette J. Dobson: An Introduction to Generalized Linear Models, Second Edition. Nov 2001</p>
 <p>[2] Cosma Shalizi: Statistics 36-350: Data Mining, Lecture Notes, 18 November 2009, <a href="http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf">http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf</a></p>
 <p>[3] Scott A. Czepiel: Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation, Retrieved Jul 12 2012, <a href="http://czep.net/stat/mlelr.pdf">http://czep.net/stat/mlelr.pdf</a> </p>
 </div><!-- contents -->
 </div><!-- doc-content -->
 <!-- start footer part -->
 <div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
   <ul>
     <li class="footer">Generated on Thu Jan 9 2014 20:35:40 for MADlib by
     <a href="http://www.doxygen.org/index.html">
     <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.4 </li>
   </ul>
 </div>
 </body>
 </html>
	<!-- HTML header for doxygen 1.8.4-->
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	<head>
	<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
	<meta http-equiv="X-UA-Compatible" content="IE=9"/>
	<meta name="generator" content="Doxygen 1.8.4"/>
	<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/>
	<title>MADlib: Multinomial Logistic Regression</title>
	<link href="tabs.css" rel="stylesheet" type="text/css"/>
	<script type="text/javascript" src="jquery.js"></script>
	<script type="text/javascript" src="dynsections.js"></script>
	<link href="navtree.css" rel="stylesheet" type="text/css"/>
	<script type="text/javascript" src="resize.js"></script>
	<script type="text/javascript" src="navtree.js"></script>
	<script type="text/javascript">
	$(document).ready(initResizable);
	$(window).load(resizeHeight);
	</script>
	<link href="search/search.css" rel="stylesheet" type="text/css"/>
	<script type="text/javascript" src="search/search.js"></script>
	<script type="text/javascript">
	$(document).ready(function() { searchBox.OnSelectItem(0); });
	</script>
	<script type="text/x-mathjax-config">
	MathJax.Hub.Config({
	extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
	jax: ["input/TeX","output/HTML-CSS"],
	});
	</script><script src="../mathjax/MathJax.js"></script>
	<link href="doxygen.css" rel="stylesheet" type="text/css" />
	<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
	<!-- google analytics -->
	<script>
	(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]\|\|function(){
	(i[r].q=i[r].q\|\|[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
	m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
	})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
	ga('create', 'UA-45382226-1', 'auto');
	ga('send', 'pageview');
	</script>
	</head>
	<body>
	<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
	<div id="titlearea">
	<table cellspacing="0" cellpadding="0">
	<tbody>
	<tr style="height: 56px;">
	<td style="padding-left: 0.5em;">
	<div id="projectname">MADlib
	<span id="projectnumber">1.2</span> <span style="font-size:10pt; font-style:italic"><a href="../latest/./group__grp__mlogreg.html"> A newer version is available</a></span>
	</div>
	<div id="projectbrief">User Documentation</div>
	</td>
	<!--BEGIN VERSIONS LINKS-->
	<td style="padding-left: 0.5em;">
	<div class="versionlist"><ul>
	<li class="head">More versions:</li>
	<li><a href="../v1.1/index.html">v1.1</li>
	<li><a href="../v1.0/index.html">v1.0</li>
	<li><a href="../v0.7/index.html">v0.7</li>
	<li><a href="../v0.5/index.html">v0.5</li></ul>
	</div>
	</td>
	<td> <div id="MSearchBox" class="MSearchBoxInactive">
	<span class="left">
	<img id="MSearchSelect" src="search/mag_sel.png"
	onmouseover="return searchBox.OnSearchSelectShow()"
	onmouseout="return searchBox.OnSearchSelectHide()"
	alt=""/>
	<input type="text" id="MSearchField" value="Search" accesskey="S"
	onfocus="searchBox.OnSearchFieldFocus(true)"
	onblur="searchBox.OnSearchFieldFocus(false)"
	onkeyup="searchBox.OnSearchFieldChange(event)"/>
	</span><span class="right">
	<a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
	</span>
	</div>
	</td>
	</tr>
	</tbody>
	</table>
	</div>
	<!-- end header part -->
	<!-- Generated by Doxygen 1.8.4 -->
	<script type="text/javascript">
	var searchBox = new SearchBox("searchBox", "search",false,'Search');
	</script>
	</div><!-- top -->
	<div id="side-nav" class="ui-resizable side-nav-resizable">
	<div id="nav-tree">
	<div id="nav-tree-contents">
	<div id="nav-sync" class="sync"></div>
	</div>
	</div>
	<div id="splitbar" style="-moz-user-select:none;"
	class="ui-resizable-handle">
	</div>
	</div>
	<script type="text/javascript">
	$(document).ready(function(){initNavTree('group__grp__mlogreg.html','');});
	</script>
	<div id="doc-content">
	<!-- window showing the filter options -->
	<div id="MSearchSelectWindow"
	onmouseover="return searchBox.OnSearchSelectShow()"
	onmouseout="return searchBox.OnSearchSelectHide()"
	onkeydown="return searchBox.OnSearchSelectKey(event)">
	<a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(0)"><span class="SelectionMark"> </span>All</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(1)"><span class="SelectionMark"> </span>Files</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(2)"><span class="SelectionMark"> </span>Functions</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(3)"><span class="SelectionMark"> </span>Variables</a><a class="SelectItem" href="javascript:void(0)" onclick="searchBox.OnSelectItem(4)"><span class="SelectionMark"> </span>Groups</a></div>

	<!-- iframe showing the search results (closed by default) -->
	<div id="MSearchResultsWindow">
	<iframe src="javascript:void(0)" frameborder="0"
	name="MSearchResults" id="MSearchResults">
	</iframe>
	</div>

	<div class="header">
	<div class="headertitle">
	<div class="title">Multinomial Logistic Regression<div class="ingroups"><a class="el" href="group__grp__glm.html">Generalized Linear Models</a></div></div> </div>
	</div><!--header-->
	<div class="contents">
	<div class="toc"><b>Contents</b> </p>
	<ul>
	<li class="level1">
	<a href="#about">About</a> </li>
	<li class="level1">
	<a href="#train">Training Function</a> </li>
	<li class="level1">
	<a href="#output">Output Format</a> </li>
	<li class="level1">
	<a href="#examples">Examples</a> </li>
	<li class="level1">
	<a href="#seealso">See Also</a> </li>
	<li class="level1">
	<a href="#background">Technical Background</a> </li>
	<li class="level1">
	<a href="#literature">Literature</a> </li>
	</ul>
	</div><p><a class="anchor" id="about"></a></p>
	<dl class="section user"><dt>About</dt><dd></dd></dl>
	<p>Multinomial logistic regression is a widely used regression analysis tool that models the outcomes of categorical dependent random variables. The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables. Multinomial logistic regression finds the vector of coefficients that maximizes the likelihood of the observations.</p>
	<p><a class="anchor" id="train"></a></p>
	<dl class="section user"><dt>Training Function</dt><dd>The multinomial logistic regression training function has the following syntax: <pre class="fragment">mlogregr(
	source,
	depvar,
	indepvar,
	max_num_iterations,
	optimizer,
	precision,
	ref_category)
	</pre></dd></dl>
	<dl class="arglist">
	<dt>source </dt>
	<dd><p class="startdd">TEXT. The name of the table containing the input data.</p>
	<p class="enddd">The training data is expected to be of the following form:<br/>
	</p>
	<pre class="fragment">{TABLE\|VIEW} source (
	...
	dependentVariable INTEGER,
	independentVariables FLOAT8[],
	...
	)
	</pre> </dd>
	<dt>depvar </dt>
	<dd>TEXT. The name of the column containing the dependent variable. </dd>
	<dt>indepvar </dt>
	<dd>TEXT. Expression list to evaluate for the independent variables. An intercept variable is not assumed. The number of independent variables cannot exceed 65535. </dd>
	<dt>max_num_iterations (optional) </dt>
	<dd>INTEGER, default: 20. The maximum number of iterations that are allowed. </dd>
	<dt>optimizer (optional) </dt>
	<dd>TEXT, default: 'irls'. The name of the optimizer to use, 'newton', or 'irls' (iteratively reweighted least squares). </dd>
	<dt>precision (optional) </dt>
	<dd>FLOAT8, default: 0.0001. The difference between log-likelihood values in successive iterations that should indicate convergence. Note that a non-positive value here disables the convergence criterion, and execution will only stop after <code>max_num_iterations</code> iterations. </dd>
	<dt>ref_category (optional) </dt>
	<dd>INTEGER, default: 0. The reference category ranges from [0, <em>numCategories</em> – 1]. </dd>
	</dl>
	<p><a class="anchor" id="notes"></a></p>
	<dl class="section note"><dt>Note</dt><dd>All table names can be optionally schema qualified and all table and column names should follow the same case-sensitivity and quoting rules as in the database.</dd></dl>
	<p><a class="anchor" id="output"></a></p>
	<dl class="section user"><dt>Output Format</dt><dd>The output of the <a class="el" href="multilogistic_8sql__in.html#a7be20ccb465d47808e18149140fc666f">mlogregr()</a> function is a composite type with the following fields: <dl class="arglist">
	<dt>ref_category </dt>
	<dd>INTEGER. The reference category. Categories are encoded as integers with values from {0, 1, 2,..., <em>numCategories</em> – 1} </dd>
	<dt>coef </dt>
	<dd>FLOAT8[]. An array of coefficients, \( \boldsymbol c \). </dd>
	<dt>log_likelihood </dt>
	<dd>FLOAT8. The log-likelihood, \( l(\boldsymbol c) \). </dd>
	<dt>std_err </dt>
	<dd>FLOAT8[]. An array of the standard errors. </dd>
	<dt>z_stats </dt>
	<dd>FLOAT8[]. An array of the Wald z-statistics. </dd>
	<dt>p_values </dt>
	<dd>FLOAT8[]. An array of the Wald p-values. </dd>
	<dt>odds_ratios </dt>
	<dd>FLOAT8[]. An array of the odds ratios. </dd>
	<dt>condition_no </dt>
	<dd>FLOAT8. The condition number of the matrix, computed using the coefficients of the iteration immediately preceding convergence. </dd>
	<dt>num_iterations </dt>
	<dd>INTEGER. The number of iterations executed before the algorithm completed. </dd>
	</dl>
	</dd></dl>
	<p><a class="anchor" id="examples"></a></p>
	<dl class="section user"><dt>Examples</dt><dd><ol type="1">
	<li>Create the training data table. <pre class="fragment">sql> DROP TABLE IF EXISTS patients;
	sql> CREATE TABLE patients (id INTEGER NOT NULL, second_attack INTEGER,
	treatment INTEGER, trait_anxiety INTEGER);
	sql> COPY patients FROM STDIN WITH DELIMITER '\|';
	1 \| 1 \| 1 \| 70
	3 \| 1 \| 1 \| 50
	5 \| 1 \| 0 \| 40
	7 \| 1 \| 0 \| 75
	9 \| 1 \| 0 \| 70
	11 \| 0 \| 1 \| 65
	13 \| 0 \| 1 \| 45
	15 \| 0 \| 1 \| 40
	17 \| 0 \| 0 \| 55
	19 \| 0 \| 0 \| 50
	2 \| 1 \| 1 \| 80
	4 \| 1 \| 0 \| 60
	6 \| 1 \| 0 \| 65
	8 \| 1 \| 0 \| 80
	10 \| 1 \| 0 \| 60
	12 \| 0 \| 1 \| 50
	14 \| 0 \| 1 \| 35
	16 \| 0 \| 1 \| 50
	18 \| 0 \| 0 \| 45
	20 \| 0 \| 0 \| 60
	\.
	</pre></li>
	<li>Run the multilogistic regression function. <pre class="fragment">sql> \x on
	Expanded display is on.
	sql> SELECT * FROM madlib.mlogregr('patients', 'second_attack',
	'ARRAY[1, treatment, trait_anxiety]', 20,
	'irls', 0.0001, 1);
	-[ RECORD 1 ]--+----------------------------------------------------------
	ref_category \| 1
	coef \| {6.3634699417818,1.02410605239327,-0.119044916668605}
	log_likelihood \| -9.41018298388876
	std_err \| {3.21389766375092,1.17107844860319,0.0549790458269306}
	z_stats \| {1.97998524145757,0.874498248699546,-2.16527796868917}
	p_values \| {0.0477051870698143,0.381846973530451,0.0303664045046178}
	odds_ratios \| {580.256322802414,2.78460505656035,0.887767924288744}
	condition_no \| 106329.420371447
	num_iterations \| 5
	</pre></li>
	</ol>
	</dd></dl>
	<p><a class="anchor" id="seealso"></a></p>
	<dl class="section see"><dt>See Also</dt><dd>File <a class="el" href="multilogistic_8sql__in.html" title="SQL functions for multinomial logistic regression. ">multilogistic.sql_in</a> documenting the multinomial <a class="el" href="logistic_8sql__in.html#a4ded9be5c8b111dbb3109efaad83d69e" title="Evaluate the usual logistic function in an under-/overflow-safe way. ">logistic</a> regression functions </dd>
	<dd>
	grp_logregr</dd></dl>
	<p><a class="anchor" id="background"></a></p>
	<dl class="section user"><dt>Technical Background</dt><dd>Multinomial logistic regression models the outcomes of categorical dependent random variables (denoted \( Y \in \{ 0,1,2 \ldots k \} \)). The model assumes that the conditional mean of the dependent categorical variables is the logistic function of an affine combination of independent variables (usually denoted \( \boldsymbol x \)). That is, <p class="formulaDsp">
	\[ E[Y \mid \boldsymbol x] = \sigma(\boldsymbol c^T \boldsymbol x) \]
	</p>
	for some unknown vector of coefficients \( \boldsymbol c \) and where \( \sigma(x) = \frac{1}{1 + \exp(-x)} \) is the logistic function. Multinomial logistic regression finds the vector of coefficients \( \boldsymbol c \) that maximizes the likelihood of the observations.</dd></dl>
	<p>Let</p>
	<ul>
	<li>\( \boldsymbol y \in \{ 0,1 \}^{n \times k} \) denote the vector of observed dependent variables, with \( n \) rows and \( k \) columns, containing the observed values of the dependent variable,</li>
	<li>\( X \in \mathbf R^{n \times k} \) denote the design matrix with \( k \) columns and \( n \) rows, containing all observed vectors of independent variables \( \boldsymbol x_i \) as rows.</li>
	</ul>
	<p>By definition, </p>
	<p class="formulaDsp">
	\[ P[Y = y_i \| \boldsymbol x_i] = \sigma((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i) \,. \]
	</p>
	<p> Maximizing the likelihood \( \prod_{i=1}^n \Pr(Y = y_i \mid \boldsymbol x_i) \) is equivalent to maximizing the log-likelihood \( \sum_{i=1}^n \log \Pr(Y = y_i \mid \boldsymbol x_i) \), which simplifies to </p>
	<p class="formulaDsp">
	\[ l(\boldsymbol c) = -\sum_{i=1}^n \log(1 + \exp((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i)) \,. \]
	</p>
	<p> The Hessian of this objective is \( H = -X^T A X \) where \( A = \text{diag}(a_1, \dots, a_n) \) is the diagonal matrix with \( a_i = \sigma(\boldsymbol c^T \boldsymbol x) \cdot \sigma(-\boldsymbol c^T \boldsymbol x) \,. \) Since \( H \) is non-positive definite, \( l(\boldsymbol c) \) is convex. There are many techniques for solving convex optimization problems. Currently, logistic regression in MADlib can use:</p>
	<ul>
	<li>Iteratively Reweighted Least Squares</li>
	</ul>
	<p>We estimate the standard error for coefficient \( i \) as </p>
	<p class="formulaDsp">
	\[ \mathit{se}(c_i) = \left( (X^T A X)^{-1} \right)_{ii} \,. \]
	</p>
	<p> The Wald z-statistic is </p>
	<p class="formulaDsp">
	\[ z_i = \frac{c_i}{\mathit{se}(c_i)} \,. \]
	</p>
	<p>The Wald \( p \)-value for coefficient \( i \) gives the probability (under the assumptions inherent in the Wald test) of seeing a value at least as extreme as the one observed, provided that the null hypothesis ( \( c_i = 0 \)) is true. Letting \( F \) denote the cumulative density function of a standard normal distribution, the Wald \( p \)-value for coefficient \( i \) is therefore </p>
	<p class="formulaDsp">
	\[ p_i = \Pr(\|Z\| \geq \|z_i\|) = 2 \cdot (1 - F( \|z_i\| )) \]
	</p>
	<p> where \( Z \) is a standard normally distributed random variable.</p>
	<p>The odds ratio for coefficient \( i \) is estimated as \( \exp(c_i) \).</p>
	<p>The condition number is computed as \( \kappa(X^T A X) \) during the iteration immediately <em>preceding</em> convergence (i.e., \( A \) is computed using the coefficients of the previous iteration). A large condition number (say, more than 1000) indicates the presence of significant multicollinearity.</p>
	<p>The multinomial logistic regression uses a default reference category of zero, and the regression coefficients in the output are in the order described below. For a problem with \( K \) dependent variables \( (1, ..., K) \) and \( J \) categories \( (0, ..., J-1) \), let \( {m_{k,j}} \) denote the coefficient for dependent variable \( k \) and category \( j \). The output is \( {m_{k_1, j_0}, m_{k_1, j_1} \ldots m_{k_1, j_{J-1}}, m_{k_2, j_0}, m_{k_2, j_1}, \ldots m_{k_2, j_{J-1}} \ldots m_{k_K, j_{J-1}}} \). The order is NOT CONSISTENT with the multinomial regression marginal effect calculation with function <em>marginal_mlogregr</em>. This is deliberate because the interfaces of all multinomial regressions (robust, clustered, ...) will be moved to match that used in marginal.</p>
	<p><a class="anchor" id="literature"></a></p>
	<dl class="section user"><dt>Literature</dt><dd></dd></dl>
	<p>A collection of nice write-ups, with valuable pointers into further literature:</p>
	<p>[1] Annette J. Dobson: An Introduction to Generalized Linear Models, Second Edition. Nov 2001</p>
	<p>[2] Cosma Shalizi: Statistics 36-350: Data Mining, Lecture Notes, 18 November 2009, <a href="http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf">http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf</a></p>
	<p>[3] Scott A. Czepiel: Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation, Retrieved Jul 12 2012, <a href="http://czep.net/stat/mlelr.pdf">http://czep.net/stat/mlelr.pdf</a> </p>
	</div><!-- contents -->
	</div><!-- doc-content -->
	<!-- start footer part -->
	<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
	<ul>
	<li class="footer">Generated on Thu Jan 9 2014 20:35:40 for MADlib by
	<a href="http://www.doxygen.org/index.html">
	<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.4 </li>
	</ul>
	</div>
	</body>
	</html>