| <!DOCTYPE html > |
| <html> |
| <head> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" /> |
| <title>Spark 3.2.2 ScalaDoc - org.apache.spark.mllib.util</title> |
| <meta name="description" content="Spark 3.2.2 ScalaDoc - org.apache.spark.mllib.util" /> |
| <meta name="keywords" content="Spark 3.2.2 ScalaDoc org.apache.spark.mllib.util" /> |
| <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> |
| |
| |
| <link href="../../../../../lib/index.css" media="screen" type="text/css" rel="stylesheet" /> |
| <link href="../../../../../lib/template.css" media="screen" type="text/css" rel="stylesheet" /> |
| <link href="../../../../../lib/diagrams.css" media="screen" type="text/css" rel="stylesheet" id="diagrams-css" /> |
| <script type="text/javascript" src="../../../../../lib/jquery.min.js"></script> |
| <script type="text/javascript" src="../../../../../lib/jquery.panzoom.min.js"></script> |
| <script type="text/javascript" src="../../../../../lib/jquery.mousewheel.min.js"></script> |
| <script type="text/javascript" src="../../../../../lib/index.js"></script> |
| <script type="text/javascript" src="../../../../../index.js"></script> |
| <script type="text/javascript" src="../../../../../lib/scheduler.js"></script> |
| <script type="text/javascript" src="../../../../../lib/template.js"></script> |
| |
| <script type="text/javascript"> |
| /* this variable can be used by the JS to determine the path to the root document */ |
| var toRoot = '../../../../../'; |
| </script> |
| |
| </head> |
| <body> |
| <div id="search"> |
| <span id="doc-title">Spark 3.2.2 ScalaDoc<span id="doc-version"></span></span> |
| <span class="close-results"><span class="left"><</span> Back</span> |
| <div id="textfilter"> |
| <span class="input"> |
| <input autocapitalize="none" placeholder="Search" id="index-input" type="text" accesskey="/" /> |
| <i class="clear material-icons"></i> |
| <i id="search-icon" class="material-icons"></i> |
| </span> |
| </div> |
| </div> |
| <div id="search-results"> |
| <div id="search-progress"> |
| <div id="progress-fill"></div> |
| </div> |
| <div id="results-content"> |
| <div id="entity-results"></div> |
| <div id="member-results"></div> |
| </div> |
| </div> |
| <div id="content-scroll-container" style="-webkit-overflow-scrolling: touch;"> |
| <div id="content-container" style="-webkit-overflow-scrolling: touch;"> |
| <div id="subpackage-spacer"> |
| <div id="packages"> |
| <h1>Packages</h1> |
| <ul> |
| <li name="_root_.root" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="_root_"></a><a id="root:_root_"></a> |
| <span class="permalink"> |
| <a href="../../../../../index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../../../../../index.html"><span class="name">root</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../../../../../index.html" class="extype" name="_root_">root</a></dd></dl></div> |
| </li><li name="_root_.org" visbl="pub" class="indented1 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="org"></a><a id="org:org"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../../../../index.html"><span class="name">org</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../../../../../index.html" class="extype" name="_root_">root</a></dd></dl></div> |
| </li><li name="org.apache" visbl="pub" class="indented2 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="apache"></a><a id="apache:apache"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../../../index.html"><span class="name">apache</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../../../../index.html" class="extype" name="org">org</a></dd></dl></div> |
| </li><li name="org.apache.spark" visbl="pub" class="indented3 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="spark"></a><a id="spark:spark"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="Core Spark functionality." href="../../index.html"><span class="name">spark</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Core Spark functionality.</p><div class="fullcomment"><div class="comment cmt"><p>Core Spark functionality. <a href="../../SparkContext.html" class="extype" name="org.apache.spark.SparkContext">org.apache.spark.SparkContext</a> serves as the main entry point to |
| Spark, while <a href="../../rdd/RDD.html" class="extype" name="org.apache.spark.rdd.RDD">org.apache.spark.rdd.RDD</a> is the data type representing a distributed collection, |
| and provides most parallel operations.</p><p>In addition, <a href="../../rdd/PairRDDFunctions.html" class="extype" name="org.apache.spark.rdd.PairRDDFunctions">org.apache.spark.rdd.PairRDDFunctions</a> contains operations available only on RDDs |
| of key-value pairs, such as <code>groupByKey</code> and <code>join</code>; <a href="../../rdd/DoubleRDDFunctions.html" class="extype" name="org.apache.spark.rdd.DoubleRDDFunctions">org.apache.spark.rdd.DoubleRDDFunctions</a> |
| contains operations available only on RDDs of Doubles; and |
| <a href="../../rdd/SequenceFileRDDFunctions.html" class="extype" name="org.apache.spark.rdd.SequenceFileRDDFunctions">org.apache.spark.rdd.SequenceFileRDDFunctions</a> contains operations available on RDDs that can |
| be saved as SequenceFiles. These operations are automatically available on any RDD of the right |
| type (e.g. RDD[(Int, Int)] through implicit conversions.</p><p>Java programmers should reference the <a href="../../api/java/index.html" class="extype" name="org.apache.spark.api.java">org.apache.spark.api.java</a> package |
| for Spark programming APIs in Java.</p><p>Classes and methods marked with <span class="experimental badge" style="float: none;"> |
| Experimental</span> are user-facing features which have not been officially adopted by the |
| Spark project. These are subject to change or removal in minor releases.</p><p>Classes and methods marked with <span class="developer badge" style="float: none;"> |
| Developer API</span> are intended for advanced users want to extend Spark through lower |
| level interfaces. These are subject to changes or removal in minor releases. |
| </p></div><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../../../index.html" class="extype" name="org.apache">apache</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib" visbl="pub" class="indented4 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="mllib"></a><a id="mllib:mllib"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="RDD-based machine learning APIs (in maintenance mode)." href="../index.html"><span class="name">mllib</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">RDD-based machine learning APIs (in maintenance mode).</p><div class="fullcomment"><div class="comment cmt"><p>RDD-based machine learning APIs (in maintenance mode).</p><p>The <code>spark.mllib</code> package is in maintenance mode as of the Spark 2.0.0 release to encourage |
| migration to the DataFrame-based APIs under the <a href="../../ml/index.html" class="extype" name="org.apache.spark.ml">org.apache.spark.ml</a> package. |
| While in maintenance mode,</p><ul><li>no new features in the RDD-based <code>spark.mllib</code> package will be accepted, unless they block |
| implementing new features in the DataFrame-based <code>spark.ml</code> package;</li><li>bug fixes in the RDD-based APIs will still be accepted.</li></ul><p>The developers will continue adding more features to the DataFrame-based APIs in the 2.x series |
| to reach feature parity with the RDD-based APIs. |
| And once we reach feature parity, this package will be deprecated. |
| </p></div><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../../index.html" class="extype" name="org.apache.spark">spark</a></dd><dt>See also</dt><dd><span class="cmt"><p><a href="https://issues.apache.org/jira/browse/SPARK-4591">SPARK-4591</a> to track |
| the progress of feature parity</p></span></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.classification" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="classification"></a><a id="classification:classification"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/classification/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../classification/index.html"><span class="name">classification</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.clustering" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="clustering"></a><a id="clustering:clustering"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/clustering/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../clustering/index.html"><span class="name">clustering</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.evaluation" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="evaluation"></a><a id="evaluation:evaluation"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/evaluation/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../evaluation/index.html"><span class="name">evaluation</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.feature" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="feature"></a><a id="feature:feature"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/feature/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../feature/index.html"><span class="name">feature</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.fpm" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="fpm"></a><a id="fpm:fpm"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/fpm/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../fpm/index.html"><span class="name">fpm</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.linalg" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="linalg"></a><a id="linalg:linalg"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/linalg/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../linalg/index.html"><span class="name">linalg</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.optimization" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="optimization"></a><a id="optimization:optimization"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/optimization/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../optimization/index.html"><span class="name">optimization</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.pmml" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="pmml"></a><a id="pmml:pmml"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/pmml/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../pmml/index.html"><span class="name">pmml</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.random" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="random"></a><a id="random:random"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/random/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../random/index.html"><span class="name">random</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.rdd" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="rdd"></a><a id="rdd:rdd"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/rdd/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../rdd/index.html"><span class="name">rdd</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.recommendation" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="recommendation"></a><a id="recommendation:recommendation"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/recommendation/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../recommendation/index.html"><span class="name">recommendation</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.regression" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="regression"></a><a id="regression:regression"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/regression/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../regression/index.html"><span class="name">regression</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.stat" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="stat"></a><a id="stat:stat"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/stat/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="" href="../stat/index.html"><span class="name">stat</span></a> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.tree" visbl="pub" class="indented5 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="tree"></a><a id="tree:tree"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/tree/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <a title="This package contains the default implementation of the decision tree algorithm, which supports:" href="../tree/index.html"><span class="name">tree</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">This package contains the default implementation of the decision tree algorithm, which supports:</p><div class="fullcomment"><div class="comment cmt"><p>This package contains the default implementation of the decision tree algorithm, which supports:</p><ul><li>binary classification,</li><li>regression,</li><li>information loss calculation with entropy and Gini for classification and |
| variance for regression,</li><li>both continuous and categorical features. |
| </li></ul></div><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util" visbl="pub" class="indented5 current" data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="util"></a><a id="util:util"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <span class="name">util</span> |
| </span> |
| |
| <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd><a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></dd></dl></div> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="DataValidators$.html" title="A collection of methods used to validate data before applying ML algorithms."></a> |
| <a href="DataValidators$.html" title="A collection of methods used to validate data before applying ML algorithms.">DataValidators</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="KMeansDataGenerator$.html" title="Generate test data for KMeans."></a> |
| <a href="KMeansDataGenerator$.html" title="Generate test data for KMeans.">KMeansDataGenerator</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="LinearDataGenerator$.html" title="Generate sample data used for Linear Data."></a> |
| <a href="LinearDataGenerator$.html" title="Generate sample data used for Linear Data.">LinearDataGenerator</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="trait" href="Loader.html" title="Trait for classes which can load models and transformers from files."></a> |
| <a href="Loader.html" title="Trait for classes which can load models and transformers from files.">Loader</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="LogisticRegressionDataGenerator$.html" title="Generate test data for LogisticRegression."></a> |
| <a href="LogisticRegressionDataGenerator$.html" title="Generate test data for LogisticRegression.">LogisticRegressionDataGenerator</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="MFDataGenerator$.html" title="Generate RDD(s) containing data for Matrix Factorization."></a> |
| <a href="MFDataGenerator$.html" title="Generate RDD(s) containing data for Matrix Factorization.">MFDataGenerator</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="MLUtils$.html" title="Helper methods to load, save and pre-process data used in MLLib."></a> |
| <a href="MLUtils$.html" title="Helper methods to load, save and pre-process data used in MLLib.">MLUtils</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="object" href="SVMDataGenerator$.html" title="Generate sample data used for SVM."></a> |
| <a href="SVMDataGenerator$.html" title="Generate sample data used for SVM.">SVMDataGenerator</a> |
| </li><li class="current-entities indented5"> |
| <span class="separator"></span> |
| <a class="trait" href="Saveable.html" title="Trait for models and transformers which may be saved as files."></a> |
| <a href="Saveable.html" title="Trait for models and transformers which may be saved as files.">Saveable</a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div id="content"> |
| <body class="package value"> |
| <div id="definition"> |
| <div class="big-circle package">p</div> |
| <p id="owner"><a href="../../../../index.html" class="extype" name="org">org</a>.<a href="../../../index.html" class="extype" name="org.apache">apache</a>.<a href="../../index.html" class="extype" name="org.apache.spark">spark</a>.<a href="../index.html" class="extype" name="org.apache.spark.mllib">mllib</a></p> |
| <h1>util<span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/index.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span></h1> |
| |
| </div> |
| |
| <h4 id="signature" class="signature"> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">package</span> |
| </span> |
| <span class="symbol"> |
| <span class="name">util</span> |
| </span> |
| </h4> |
| |
| |
| <div id="comment" class="fullcommenttop"></div> |
| |
| |
| <div id="mbrsel"> |
| <div class="toggle"></div> |
| <div id="memberfilter"> |
| <i class="material-icons arrow"></i> |
| <span class="input"> |
| <input id="mbrsel-input" placeholder="Filter all members" type="text" accesskey="/" /> |
| </span> |
| <i class="clear material-icons"></i> |
| </div> |
| <div id="filterby"> |
| <div id="order"> |
| <span class="filtertype">Ordering</span> |
| <ol> |
| |
| <li class="alpha in"><span>Alphabetic</span></li> |
| |
| </ol> |
| </div> |
| |
| <div id="visbl"> |
| <span class="filtertype">Visibility</span> |
| <ol><li class="public in"><span>Public</span></li><li class="all out"><span>All</span></li></ol> |
| </div> |
| </div> |
| </div> |
| |
| <div id="template"> |
| <div id="allMembers"> |
| |
| |
| <div id="types" class="types members"> |
| <h3>Type Members</h3> |
| <ol><li name="org.apache.spark.mllib.util.Loader" visbl="pub" class="indented0 " data-isabs="true" fullComment="yes" group="Ungrouped"> |
| <a id="Loader[M<:org.apache.spark.mllib.util.Saveable]extendsAnyRef"></a><a id="Loader[M<:Saveable]:Loader[M]"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/Loader.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">trait</span> |
| </span> |
| <span class="symbol"> |
| <a title="Trait for classes which can load models and transformers from files." href="Loader.html"><span class="name">Loader</span></a><span class="tparams">[<span name="M">M <: <a href="Saveable.html" class="extype" name="org.apache.spark.mllib.util.Saveable">Saveable</a></span>]</span><span class="result"> extends <span class="extype" name="scala.AnyRef">AnyRef</span></span> |
| </span> |
| |
| <p class="shortcomment cmt">Trait for classes which can load models and transformers from files.</p><div class="fullcomment"><div class="comment cmt"><p>Trait for classes which can load models and transformers from files. |
| This should be inherited by an object paired with the model class. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"1.3.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.Saveable" visbl="pub" class="indented0 " data-isabs="true" fullComment="yes" group="Ungrouped"> |
| <a id="SaveableextendsAnyRef"></a><a id="Saveable:Saveable"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/Saveable.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">trait</span> |
| </span> |
| <span class="symbol"> |
| <a title="Trait for models and transformers which may be saved as files." href="Saveable.html"><span class="name">Saveable</span></a><span class="result"> extends <span class="extype" name="scala.AnyRef">AnyRef</span></span> |
| </span> |
| |
| <p class="shortcomment cmt">Trait for models and transformers which may be saved as files.</p><div class="fullcomment"><div class="comment cmt"><p>Trait for models and transformers which may be saved as files. |
| This should be inherited by the class which implements model instances. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"1.3.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li></ol> |
| </div> |
| |
| |
| |
| <div class="values members"> |
| <h3>Value Members</h3> |
| <ol> |
| <li name="org.apache.spark.mllib.util.DataValidators" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="DataValidators"></a><a id="DataValidators:DataValidators"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/DataValidators$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="A collection of methods used to validate data before applying ML algorithms." href="DataValidators$.html"><span class="name">DataValidators</span></a><span class="result"> extends <span class="extype" name="org.apache.spark.internal.Logging">Logging</span></span> |
| </span> |
| |
| <p class="shortcomment cmt">A collection of methods used to validate data before applying ML algorithms.</p><div class="fullcomment"><div class="comment cmt"><p>A collection of methods used to validate data before applying ML algorithms. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.KMeansDataGenerator" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="KMeansDataGenerator"></a><a id="KMeansDataGenerator:KMeansDataGenerator"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/KMeansDataGenerator$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Generate test data for KMeans." href="KMeansDataGenerator$.html"><span class="name">KMeansDataGenerator</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Generate test data for KMeans.</p><div class="fullcomment"><div class="comment cmt"><p>Generate test data for KMeans. This class first chooses k cluster centers |
| from a d-dimensional Gaussian distribution scaled by factor r and then creates a Gaussian |
| cluster with scale 1 around each center. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.LinearDataGenerator" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="LinearDataGenerator"></a><a id="LinearDataGenerator:LinearDataGenerator"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/LinearDataGenerator$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Generate sample data used for Linear Data." href="LinearDataGenerator$.html"><span class="name">LinearDataGenerator</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Generate sample data used for Linear Data.</p><div class="fullcomment"><div class="comment cmt"><p>Generate sample data used for Linear Data. This class generates |
| uniformly random values for every feature and adds Gaussian noise with mean <code>eps</code> to the |
| response variable <code>Y</code>. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.LogisticRegressionDataGenerator" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="LogisticRegressionDataGenerator"></a><a id="LogisticRegressionDataGenerator:LogisticRegressionDataGenerator"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/LogisticRegressionDataGenerator$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Generate test data for LogisticRegression." href="LogisticRegressionDataGenerator$.html"><span class="name">LogisticRegressionDataGenerator</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Generate test data for LogisticRegression.</p><div class="fullcomment"><div class="comment cmt"><p>Generate test data for LogisticRegression. This class chooses positive labels |
| with probability <code>probOne</code> and scales features for positive examples by <code>eps</code>. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.MFDataGenerator" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="MFDataGenerator"></a><a id="MFDataGenerator:MFDataGenerator"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/MFDataGenerator$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Generate RDD(s) containing data for Matrix Factorization." href="MFDataGenerator$.html"><span class="name">MFDataGenerator</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Generate RDD(s) containing data for Matrix Factorization.</p><div class="fullcomment"><div class="comment cmt"><p>Generate RDD(s) containing data for Matrix Factorization.</p><p>This method samples training entries according to the oversampling factor |
| 'trainSampFact', which is a multiplicative factor of the number of |
| degrees of freedom of the matrix: rank*(m+n-rank).</p><p>It optionally samples entries for a testing matrix using |
| 'testSampFact', the percentage of the number of training entries |
| to use for testing.</p><p>This method takes the following inputs: |
| sparkMaster (String) The master URL. |
| outputPath (String) Directory to save output. |
| m (Int) Number of rows in data matrix. |
| n (Int) Number of columns in data matrix. |
| rank (Int) Underlying rank of data matrix. |
| trainSampFact (Double) Oversampling factor. |
| noise (Boolean) Whether to add gaussian noise to training data. |
| sigma (Double) Standard deviation of added gaussian noise. |
| test (Boolean) Whether to create testing RDD. |
| testSampFact (Double) Percentage of training data to use as test data. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.MLUtils" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="MLUtils"></a><a id="MLUtils:MLUtils"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/MLUtils$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Helper methods to load, save and pre-process data used in MLLib." href="MLUtils$.html"><span class="name">MLUtils</span></a><span class="result"> extends <span class="extype" name="org.apache.spark.internal.Logging">Logging</span></span> |
| </span> |
| |
| <p class="shortcomment cmt">Helper methods to load, save and pre-process data used in MLLib.</p><div class="fullcomment"><div class="comment cmt"><p>Helper methods to load, save and pre-process data used in MLLib. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li><li name="org.apache.spark.mllib.util.SVMDataGenerator" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped"> |
| <a id="SVMDataGenerator"></a><a id="SVMDataGenerator:SVMDataGenerator"></a> |
| <span class="permalink"> |
| <a href="../../../../../org/apache/spark/mllib/util/SVMDataGenerator$.html" title="Permalink"> |
| <i class="material-icons"></i> |
| </a> |
| </span> |
| <span class="modifier_kind"> |
| <span class="modifier"></span> |
| <span class="kind">object</span> |
| </span> |
| <span class="symbol"> |
| <a title="Generate sample data used for SVM." href="SVMDataGenerator$.html"><span class="name">SVMDataGenerator</span></a> |
| </span> |
| |
| <p class="shortcomment cmt">Generate sample data used for SVM.</p><div class="fullcomment"><div class="comment cmt"><p>Generate sample data used for SVM. This class generates uniform random values |
| for the features and adds Gaussian noise with weight 0.1 to generate labels. |
| </p></div><dl class="attributes block"> <dt>Annotations</dt><dd> |
| <span class="name">@Since</span><span class="args">(<span> |
| |
| <span class="symbol">"0.8.0"</span> |
| </span>)</span> |
| |
| </dd></dl></div> |
| </li> |
| </ol> |
| </div> |
| |
| |
| |
| |
| </div> |
| |
| <div id="inheritedMembers"> |
| |
| |
| </div> |
| |
| <div id="groupedMembers"> |
| <div class="group" name="Ungrouped"> |
| <h3>Ungrouped</h3> |
| |
| </div> |
| </div> |
| |
| </div> |
| |
| <div id="tooltip"></div> |
| |
| <div id="footer"> </div> |
| </body> |
| </div> |
| </div> |
| </div> |
| </body> |
| </html> |
