| = Machine Learning |
| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| |
| This section of the math expressions user guide covers machine learning |
| functions. |
| |
| == Feature Scaling |
| |
| Before performing machine learning operations its often necessary to |
| scale the feature vectors so they can be compared at the same scale. |
| |
| All the scaling function operate on vectors and matrices. |
| When operating on a matrix the rows of the matrix are scaled. |
| |
| === Min/Max Scaling |
| |
| The `minMaxScale` function scales a vector or matrix between a minimum and maximum value. |
| By default it will scale between 0 and 1 if min/max values are not provided. |
| |
| Below is a simple example of min/max scaling between 0 and 1. |
| Notice that once brought into the same scale the vectors are the same. |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40, 50), |
| b=array(200, 300, 400, 500), |
| c=matrix(a, b), |
| d=minMaxScale(c)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "d": [ |
| [ |
| 0, |
| 0.3333333333333333, |
| 0.6666666666666666, |
| 1 |
| ], |
| [ |
| 0, |
| 0.3333333333333333, |
| 0.6666666666666666, |
| 1 |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 0 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Standardization |
| |
| The `standardize` function scales a vector so that it has a mean of 0 and a standard deviation of 1. |
| Standardization can be used with machine learning algorithms, such as |
| https://en.wikipedia.org/wiki/Support_vector_machine[Support Vector Machine (SVM)], that perform better |
| when the data has a normal distribution. |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40, 50), |
| b=array(200, 300, 400, 500), |
| c=matrix(a, b), |
| d=standardize(c)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "d": [ |
| [ |
| -1.161895003862225, |
| -0.3872983346207417, |
| 0.3872983346207417, |
| 1.161895003862225 |
| ], |
| [ |
| -1.1618950038622249, |
| -0.38729833462074165, |
| 0.38729833462074165, |
| 1.1618950038622249 |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 17 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Unit Vectors |
| |
| The `unitize` function scales vectors to a magnitude of 1. A vector with a |
| magnitude of 1 is known as a unit vector. Unit vectors are preferred when the vector math deals |
| with vector direction rather than magnitude. |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40, 50), |
| b=array(200, 300, 400, 500), |
| c=matrix(a, b), |
| d=unitize(c)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "d": [ |
| [ |
| 0.2721655269759087, |
| 0.40824829046386296, |
| 0.5443310539518174, |
| 0.6804138174397716 |
| ], |
| [ |
| 0.2721655269759087, |
| 0.4082482904638631, |
| 0.5443310539518174, |
| 0.6804138174397717 |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 6 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Distance and Distance Measures |
| |
| The `distance` function computes the distance for two numeric arrays or a distance matrix for the columns of a matrix. |
| |
| There are five distance measure functions that return a function that performs the actual distance calculation: |
| |
| * `euclidean` (default) |
| * `manhattan` |
| * `canberra` |
| * `earthMovers` |
| * `haversineMeters` (Geospatial distance measure) |
| |
| The distance measure functions can be used with all machine learning functions |
| that support distance measures. |
| |
| Below is an example for computing Euclidean distance for two numeric arrays: |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40, 50), |
| b=array(21, 29, 41, 49), |
| c=distance(a, b)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": 2 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 0 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| Below the distance is calculated using *Manahattan* distance. |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40, 50), |
| b=array(21, 29, 41, 49), |
| c=distance(a, b, manhattan())) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": 4 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 1 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| |
| Below is an example for computing a distance matrix for columns |
| of a matrix: |
| |
| [source,text] |
| ---- |
| let(a=array(20, 30, 40), |
| b=array(21, 29, 41), |
| c=array(31, 40, 50), |
| d=matrix(a, b, c), |
| c=distance(d)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "e": [ |
| [ |
| 0, |
| 15.652475842498529, |
| 34.07345007480164 |
| ], |
| [ |
| 15.652475842498529, |
| 0, |
| 18.547236990991408 |
| ], |
| [ |
| 34.07345007480164, |
| 18.547236990991408, |
| 0 |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 24 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == K-Means Clustering |
| |
| The `kmeans` functions performs k-means clustering of the rows of a matrix. |
| Once the clustering has been completed there are a number of useful functions available |
| for examining the clusters and centroids. |
| |
| The examples below cluster _term vectors_. |
| The section <<term-vectors.adoc#term-vectors,Text Analysis and Term Vectors>> offers |
| a full explanation of these features. |
| |
| === Centroid Features |
| |
| In the example below the `kmeans` function is used to cluster a result set from the Enron email data-set |
| and then the top features are extracted from the cluster centroids. |
| |
| [source,text] |
| ---- |
| let(a=select(random(enron, q="body:oil", rows="500", fl="id, body"), <1> |
| id, |
| analyze(body, body_bigram) as terms), |
| b=termVectors(a, maxDocFreq=.10, minDocFreq=.05, minTermLength=14, exclude="_,copyright"),<2> |
| c=kmeans(b, 5), <3> |
| d=getCentroids(c), <4> |
| e=topFeatures(d, 5)) <5> |
| ---- |
| |
| Let's look at what data is assigned to each variable: |
| |
| <1> *`a`*: The `random` function returns a sample of 500 documents from the "enron" |
| collection that match the query "body:oil". The `select` function selects the `id` and |
| and annotates each tuple with the analyzed bigram terms from the `body` field. |
| <2> *`b`*: The `termVectors` function creates a TF-IDF term vector matrix from the |
| tuples stored in variable *`a`*. Each row in the matrix represents a document. The columns of the matrix |
| are the bigram terms that were attached to each tuple. |
| <3> *`c`*: The `kmeans` function clusters the rows of the matrix into 5 clusters. The k-means clustering is performed using the Euclidean distance measure. |
| <4> *`d`*: The `getCentroids` function returns a matrix of cluster centroids. Each row in the matrix is a centroid |
| from one of the 5 clusters. The columns of the matrix are the same bigrams terms of the term vector matrix. |
| <5> *`e`*: The `topFeatures` function returns the column labels for the top 5 features of each centroid in the matrix. |
| This returns the top 5 bigram terms for each centroid. |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "e": [ |
| [ |
| "enron enronxgate", |
| "north american", |
| "energy services", |
| "conference call", |
| "power generation" |
| ], |
| [ |
| "financial times", |
| "chief financial", |
| "financial officer", |
| "exchange commission", |
| "houston chronicle" |
| ], |
| [ |
| "southern california", |
| "california edison", |
| "public utilities", |
| "utilities commission", |
| "rate increases" |
| ], |
| [ |
| "rolling blackouts", |
| "public utilities", |
| "electricity prices", |
| "federal energy", |
| "price controls" |
| ], |
| [ |
| "california edison", |
| "regulatory commission", |
| "southern california", |
| "federal energy", |
| "power generators" |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 982 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Cluster Features |
| |
| The example below examines the top features of a specific cluster. This example uses the same techniques |
| as the centroids example but the top features are extracted from a cluster rather than the centroids. |
| |
| [source,text] |
| ---- |
| let(a=select(random(collection3, q="body:oil", rows="500", fl="id, body"), |
| id, |
| analyze(body, body_bigram) as terms), |
| b=termVectors(a, maxDocFreq=.09, minDocFreq=.03, minTermLength=14, exclude="_,copyright"), |
| c=kmeans(b, 25), |
| d=getCluster(c, 0), <1> |
| e=topFeatures(d, 4)) <2> |
| ---- |
| |
| <1> The `getCluster` function returns a cluster by its index. Each cluster is a matrix containing term vectors |
| that have been clustered together based on their features. |
| <2> The `topFeatures` function is used to extract the top 4 features from each term vector |
| in the cluster. |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "e": [ |
| [ |
| "electricity board", |
| "maharashtra state", |
| "power purchase", |
| "state electricity", |
| "reserved enron" |
| ], |
| [ |
| "electricity board", |
| "maharashtra state", |
| "state electricity", |
| "purchase agreement", |
| "independent power" |
| ], |
| [ |
| "maharashtra state", |
| "reserved enron", |
| "federal government", |
| "state government", |
| "dabhol project" |
| ], |
| [ |
| "purchase agreement", |
| "power purchase", |
| "electricity board", |
| "maharashtra state", |
| "state government" |
| ], |
| [ |
| "investment grade", |
| "portland general", |
| "general electric", |
| "holding company", |
| "transmission lines" |
| ], |
| [ |
| "state government", |
| "state electricity", |
| "purchase agreement", |
| "electricity board", |
| "maharashtra state" |
| ], |
| [ |
| "electricity board", |
| "state electricity", |
| "energy management", |
| "maharashtra state", |
| "energy markets" |
| ], |
| [ |
| "electricity board", |
| "maharashtra state", |
| "state electricity", |
| "state government", |
| "second quarter" |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 978 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Multi K-Means Clustering |
| |
| K-means clustering will produce different results depending on |
| the initial placement of the centroids. K-means is fast enough |
| that multiple trials can be performed and the best outcome selected. |
| |
| The `multiKmeans` function runs the k-means clustering algorithm for a given number of trials and selects the |
| best result based on which trial produces the lowest intra-cluster variance. |
| |
| The example below is identical to centroids example except that it uses `multiKmeans` with 100 trials, |
| rather than a single trial of the `kmeans` function. |
| |
| [source,text] |
| ---- |
| let(a=select(random(collection3, q="body:oil", rows="500", fl="id, body"), |
| id, |
| analyze(body, body_bigram) as terms), |
| b=termVectors(a, maxDocFreq=.09, minDocFreq=.03, minTermLength=14, exclude="_,copyright"), |
| c=multiKmeans(b, 5, 100), |
| d=getCentroids(c), |
| e=topFeatures(d, 5)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "e": [ |
| [ |
| "enron enronxgate", |
| "energy trading", |
| "energy markets", |
| "energy services", |
| "unleaded gasoline" |
| ], |
| [ |
| "maharashtra state", |
| "electricity board", |
| "state electricity", |
| "energy trading", |
| "chief financial" |
| ], |
| [ |
| "price controls", |
| "electricity prices", |
| "francisco chronicle", |
| "wholesale electricity", |
| "power generators" |
| ], |
| [ |
| "southern california", |
| "california edison", |
| "public utilities", |
| "francisco chronicle", |
| "utilities commission" |
| ], |
| [ |
| "california edison", |
| "power purchases", |
| "system operator", |
| "term contracts", |
| "independent system" |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 1182 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Fuzzy K-Means Clustering |
| |
| The `fuzzyKmeans` function is a soft clustering algorithm which |
| allows vectors to be assigned to more then one cluster. The `fuzziness` parameter |
| is a value between 1 and 2 that determines how fuzzy to make the cluster assignment. |
| |
| After the clustering has been performed the `getMembershipMatrix` function can be called |
| on the clustering result to return a matrix describing which clusters each vector belongs to. |
| There is a row in the matrix for each vector that was clustered. There is a column in the matrix |
| for each cluster. The values in the columns are the probability that the vector belonged to the specific |
| cluster. |
| |
| A simple example will make this more clear. In the example below 300 documents are analyzed and |
| then turned into a term vector matrix. Then the `fuzzyKmeans` function clusters the |
| term vectors into 12 clusters with a fuzziness factor of 1.25. |
| |
| [source,text] |
| ---- |
| let(a=select(random(collection3, q="body:oil", rows="300", fl="id, body"), |
| id, |
| analyze(body, body_bigram) as terms), |
| b=termVectors(a, maxDocFreq=.09, minDocFreq=.03, minTermLength=14, exclude="_,copyright"), |
| c=fuzzyKmeans(b, 12, fuzziness=1.25), |
| d=getMembershipMatrix(c), <1> |
| e=rowAt(d, 0), <2> |
| f=precision(e, 5)) <3> |
| ---- |
| |
| <1> The `getMembershipMatrix` function is used to return the membership matrix; |
| <2> and the first row of membership matrix is retrieved with the `rowAt` function. |
| <3> The `precision` function is then applied to the first row |
| of the matrix to make it easier to read. |
| |
| This expression returns a single vector representing the cluster membership probabilities for the first |
| term vector. Notice that the term vector has the highest association with the 12^th^ cluster, |
| but also has significant associations with the 3^rd^, 5^th^, 6^th^ and 7^th^ clusters: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "f": [ |
| 0, |
| 0, |
| 0.178, |
| 0, |
| 0.17707, |
| 0.17775, |
| 0.16214, |
| 0, |
| 0, |
| 0, |
| 0, |
| 0.30504 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 2157 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == K-Nearest Neighbor (KNN) |
| |
| The `knn` function searches the rows of a matrix for the |
| k-nearest neighbors of a search vector. The `knn` function |
| returns a matrix of the k-nearest neighbors. |
| |
| The `knn` function supports changing of the distance measure by providing one of these |
| distance measure functions as the fourth parameter: |
| |
| * `euclidean` (Default) |
| * `manhattan` |
| * `canberra` |
| * `earthMovers` |
| |
| The example below builds on the clustering examples to demonstrate the `knn` function. |
| |
| [source,text] |
| ---- |
| let(a=select(random(collection3, q="body:oil", rows="500", fl="id, body"), |
| id, |
| analyze(body, body_bigram) as terms), |
| b=termVectors(a, maxDocFreq=.09, minDocFreq=.03, minTermLength=14, exclude="_,copyright"), |
| c=multiKmeans(b, 5, 100), |
| d=getCentroids(c), <1> |
| e=rowAt(d, 0), <2> |
| g=knn(b, e, 3), <3> |
| h=topFeatures(g, 4)) <4> |
| ---- |
| |
| <1> In the example, the centroids matrix is set to variable *`d`*. |
| <2> The first centroid vector is selected from the matrix with the `rowAt` function. |
| <3> Then the `knn` function is used to find the 3 nearest neighbors |
| to the centroid vector in the term vector matrix (variable *`b`*). |
| <4> The `topFeatures` function is used to request the top 4 featurs of the term vectors in the knn matrix. |
| |
| The `knn` function returns a matrix with the 3 nearest neighbors based on the |
| default distance measure which is euclidean. Finally, the top 4 features |
| of the term vectors in the nearest neighbor matrix are returned: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "h": [ |
| [ |
| "california power", |
| "electricity supply", |
| "concerned about", |
| "companies like" |
| ], |
| [ |
| "maharashtra state", |
| "california power", |
| "electricity board", |
| "alternative energy" |
| ], |
| [ |
| "electricity board", |
| "maharashtra state", |
| "state electricity", |
| "houston chronicle" |
| ] |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 1243 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == K-Nearest Neighbor Regression |
| |
| K-nearest neighbor regression is a non-linear, multi-variate regression method. Knn regression is a lazy learning |
| technique which means it does not fit a model to the training set in advance. Instead the |
| entire training set of observations and outcomes are held in memory and predictions are made |
| by averaging the outcomes of the k-nearest neighbors. |
| |
| The `knnRegress` function prepares the training set for use with the `predict` function. |
| |
| Below is an example of the `knnRegress` function. In this example 10,000 random samples |
| are taken, each containing the variables `filesize_d`, `service_d` and `response_d`. The pairs of |
| `filesize_d` and `service_d` will be used to predict the value of `response_d`. |
| |
| [source,text] |
| ---- |
| let(samples=random(collection1, q="*:*", rows="10000", fl="filesize_d, service_d, response_d"), |
| filesizes=col(samples, filesize_d), |
| serviceLevels=col(samples, service_d), |
| outcomes=col(samples, response_d), |
| observations=transpose(matrix(filesizes, serviceLevels)), |
| lazyModel=knnRegress(observations, outcomes , 5)) |
| ---- |
| |
| This expression returns the following response. Notice that `knnRegress` returns a tuple describing the regression inputs: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "lazyModel": { |
| "features": 2, |
| "robust": false, |
| "distance": "EuclideanDistance", |
| "observations": 10000, |
| "scale": false, |
| "k": 5 |
| } |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 170 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Prediction and Residuals |
| |
| The output of `knnRegress` can be used with the `predict` function like other regression models. |
| |
| In the example below the `predict` function is used to predict results for the original training |
| data. The sumSq of the residuals is then calculated. |
| |
| [source,text] |
| ---- |
| let(samples=random(collection1, q="*:*", rows="10000", fl="filesize_d, service_d, response_d"), |
| filesizes=col(samples, filesize_d), |
| serviceLevels=col(samples, service_d), |
| outcomes=col(samples, response_d), |
| observations=transpose(matrix(filesizes, serviceLevels)), |
| lazyModel=knnRegress(observations, outcomes , 5), |
| predictions=predict(lazyModel, observations), |
| residuals=ebeSubtract(outcomes, predictions), |
| sumSqErr=sumSq(residuals)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "sumSqErr": 1920290.1204126712 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 3796 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Setting Feature Scaling |
| |
| If the features in the observation matrix are not in the same scale then the larger features |
| will carry more weight in the distance calculation then the smaller features. This can greatly |
| impact the accuracy of the prediction. The `knnRegress` function has a `scale` parameter which |
| can be set to `true` to automatically scale the features in the same range. |
| |
| The example below shows `knnRegress` with feature scaling turned on. |
| |
| Notice that when feature scaling is turned on the `sumSqErr` in the output is much lower. |
| This shows how much more accurate the predictions are when feature scaling is turned on in |
| this particular example. This is because the `filesize_d` feature is significantly larger then |
| the `service_d` feature. |
| |
| [source,text] |
| ---- |
| let(samples=random(collection1, q="*:*", rows="10000", fl="filesize_d, service_d, response_d"), |
| filesizes=col(samples, filesize_d), |
| serviceLevels=col(samples, service_d), |
| outcomes=col(samples, response_d), |
| observations=transpose(matrix(filesizes, serviceLevels)), |
| lazyModel=knnRegress(observations, outcomes , 5, scale=true), |
| predictions=predict(lazyModel, observations), |
| residuals=ebeSubtract(outcomes, predictions), |
| sumSqErr=sumSq(residuals)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "sumSqErr": 4076.794951120683 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 3790 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| |
| === Setting Robust Regression |
| |
| The default prediction approach is to take the mean of the outcomes of the k-nearest |
| neighbors. If the outcomes contain outliers the mean value can be skewed. Setting |
| the `robust` parameter to `true` will take the median outcome of the k-nearest neighbors. |
| This provides a regression prediction that is robust to outliers. |
| |
| === Setting the Distance Measure |
| |
| The distance measure can be changed for the k-nearest neighbor search by adding a distance measure |
| function to the `knnRegress` parameters. Below is an example using `manhattan` distance. |
| |
| [source,text] |
| ---- |
| let(samples=random(collection1, q="*:*", rows="10000", fl="filesize_d, service_d, response_d"), |
| filesizes=col(samples, filesize_d), |
| serviceLevels=col(samples, service_d), |
| outcomes=col(samples, response_d), |
| observations=transpose(matrix(filesizes, serviceLevels)), |
| lazyModel=knnRegress(observations, outcomes, 5, manhattan(), scale=true), |
| predictions=predict(lazyModel, observations), |
| residuals=ebeSubtract(outcomes, predictions), |
| sumSqErr=sumSq(residuals)) |
| ---- |
| |
| This expression returns the following response: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "sumSqErr": 4761.221942288098 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 3571 |
| } |
| ] |
| } |
| } |
| ---- |
