layout: global title: Dimensionality Reduction - RDD-based API displayTitle: Dimensionality Reduction - RDD-based API license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Dimensionality reduction is the process of reducing the number of variables under consideration. It can be used to extract latent features from raw and noisy features or compress data while maintaining the structure. spark.mllib
provides support for dimensionality reduction on the RowMatrix class.
Singular value decomposition (SVD) factorizes a matrix into three matrices: $U$, $\Sigma$, and $V$ such that
\[ A = U \Sigma V^T, \]
where
For large matrices, usually we don't need the complete factorization but only the top singular values and its associated singular vectors. This can save storage, de-noise and recover the low-rank structure of the matrix.
If we keep the top $k$ singular values, then the dimensions of the resulting low-rank matrix will be:
$U$
: $m \times k$
,$\Sigma$
: $k \times k$
,$V$
: $n \times k$
.We assume $n$ is smaller than $m$. The singular values and the right singular vectors are derived from the eigenvalues and the eigenvectors of the Gramian matrix $A^T A$. The matrix storing the left singular vectors $U$, is computed via matrix multiplication as $U = A (V S^{-1})$, if requested by the user via the computeU parameter. The actual method to use is determined automatically based on the computational cost:
spark.mllib
provides SVD functionality to row-oriented matrices, provided in the RowMatrix class.
{% include_example python/mllib/svd_example.py %}
The same code applies to IndexedRowMatrix
if U
is defined as an IndexedRowMatrix
.
{% include_example scala/org/apache/spark/examples/mllib/SVDExample.scala %}
The same code applies to IndexedRowMatrix
if U
is defined as an IndexedRowMatrix
.
{% include_example java/org/apache/spark/examples/mllib/JavaSVDExample.java %}
The same code applies to IndexedRowMatrix
if U
is defined as an IndexedRowMatrix
.
Principal component analysis (PCA) is a statistical method to find a rotation such that the first coordinate has the largest variance possible, and each succeeding coordinate, in turn, has the largest variance possible. The columns of the rotation matrix are called principal components. PCA is used widely in dimensionality reduction.
spark.mllib
supports PCA for tall-and-skinny matrices stored in row-oriented format and any Vectors.
The following code demonstrates how to compute principal components on a RowMatrix
and use them to project the vectors into a low-dimensional space.
Refer to the RowMatrix
Python docs for details on the API.
{% include_example python/mllib/pca_rowmatrix_example.py %}
The following code demonstrates how to compute principal components on a RowMatrix
and use them to project the vectors into a low-dimensional space.
Refer to the RowMatrix
Scala docs for details on the API.
{% include_example scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala %}
The following code demonstrates how to compute principal components on source vectors and use them to project the vectors into a low-dimensional space while keeping associated labels:
Refer to the PCA
Scala docs for details on the API.
{% include_example scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala %}
The following code demonstrates how to compute principal components on a RowMatrix
and use them to project the vectors into a low-dimensional space.
Refer to the RowMatrix
Java docs for details on the API.
{% include_example java/org/apache/spark/examples/mllib/JavaPCAExample.java %}