| = Time Series |
| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| This section of the user guide provides an overview of time series *aggregation*, |
| *smoothing* and *differencing*. |
| |
| == Time Series Aggregation |
| |
| The `timeseries` function performs fast, distributed time |
| series aggregation leveraging Solr's builtin faceting and date math capabilities. |
| |
| The example below performs a monthly time series aggregation: |
| |
| [source,text] |
| ---- |
| timeseries(collection1, |
| q=*:*, |
| field="recdate_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "recdate_dt": "2012-01", |
| "count(*)": 8703 |
| }, |
| { |
| "recdate_dt": "2012-02", |
| "count(*)": 8648 |
| }, |
| { |
| "recdate_dt": "2012-03", |
| "count(*)": 8621 |
| }, |
| { |
| "recdate_dt": "2012-04", |
| "count(*)": 8533 |
| }, |
| { |
| "recdate_dt": "2012-05", |
| "count(*)": 8792 |
| }, |
| { |
| "recdate_dt": "2012-06", |
| "count(*)": 8598 |
| }, |
| { |
| "recdate_dt": "2012-07", |
| "count(*)": 8679 |
| }, |
| { |
| "recdate_dt": "2012-08", |
| "count(*)": 8469 |
| }, |
| { |
| "recdate_dt": "2012-09", |
| "count(*)": 8637 |
| }, |
| { |
| "recdate_dt": "2012-10", |
| "count(*)": 8536 |
| }, |
| { |
| "recdate_dt": "2012-11", |
| "count(*)": 8785 |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 16 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Vectorizing the Time Series |
| |
| Before a time series result can be operated on by math expressions |
| the data will need to be vectorized. Specifically |
| in the example above, the aggregation field count(*) will need to by moved into an array. |
| As described in the Streams and Vectorization section of the user guide, the `col` function can be used |
| to copy a numeric column from a list of tuples into an array. |
| |
| The expression below demonstrates the vectorization of the count(*) field. |
| |
| [source,text] |
| ---- |
| let(a=timeseries(collection1, |
| q=*:*, |
| field="test_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)), |
| b=col(a, count(*))) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "b": [ |
| 8703, |
| 8648, |
| 8621, |
| 8533, |
| 8792, |
| 8598, |
| 8679, |
| 8469, |
| 8637, |
| 8536, |
| 8785 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 5 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Smoothing |
| |
| Time series smoothing is often used to remove the noise from a time series and help |
| spot the underlying trends. |
| The math expressions library has three *sliding window* approaches |
| for time series smoothing. The *sliding window* approaches use a summary value |
| from a sliding window of the data to calculate a new set of smoothed data points. |
| |
| The three *sliding window* functions are lagging indicators, which means |
| they don't start to move in the direction of the trend until the trend effects |
| the summary value of the sliding window. Because of this lagging quality these smoothing |
| functions are often used to confirm the direction of the trend. |
| |
| === Moving Average |
| |
| The `movingAvg` function computes a simple moving average over a sliding window of data. |
| The example below generates a time series, vectorizes the count(*) field and computes the |
| moving average with a window size of 3. |
| |
| The moving average function returns an array that is of shorter length |
| then the original data set. This is because results are generated only when a full window of data |
| is available for computing the average. With a window size of three the moving average will |
| begin generating results at the 3rd value. The prior values are not included in the result. |
| |
| This is true for all the sliding window functions. |
| |
| [source,text] |
| ---- |
| let(a=timeseries(collection1, |
| q=*:*, |
| field="test_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)), |
| b=col(a, count(*)), |
| c=movingAvg(b, 3)) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": [ |
| 8657.333333333334, |
| 8600.666666666666, |
| 8648.666666666666, |
| 8641, |
| 8689.666666666666, |
| 8582, |
| 8595, |
| 8547.333333333334, |
| 8652.666666666666 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 7 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Exponential Moving Average |
| |
| The `expMovingAvg` function uses a different formula for computing the moving average that |
| responds faster to changes in the underlying data. This means that it is |
| less of a lagging indicator then the simple moving average. |
| |
| Below is an example that computes an exponential moving average: |
| |
| [source,text] |
| ---- |
| let(a=timeseries(collection1, q=*:*, |
| field="test_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)), |
| b=col(a, count(*)), |
| c=expMovingAvg(b, 3)) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": [ |
| 8657.333333333334, |
| 8595.166666666668, |
| 8693.583333333334, |
| 8645.791666666668, |
| 8662.395833333334, |
| 8565.697916666668, |
| 8601.348958333334, |
| 8568.674479166668, |
| 8676.837239583334 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 5 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Moving Median |
| |
| The `movingMedian` function uses the median of the sliding window rather than the average. |
| In many cases the moving median will be more *robust* to outliers then moving averages. |
| |
| Below is an example computing the moving median: |
| |
| [source,text] |
| ---- |
| let(a=timeseries(collection1, |
| q=*:*, |
| field="test_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)), |
| b=col(a, count(*)), |
| c=movingMedian(b, 3)) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": [ |
| 8648, |
| 8621, |
| 8621, |
| 8598, |
| 8679, |
| 8598, |
| 8637, |
| 8536, |
| 8637 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 7 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| == Differencing |
| |
| Differencing is often used to remove the |
| trend or seasonality from a time series. This is known as making a time series |
| *stationary*. |
| |
| === First Difference |
| |
| The actual technique of differencing is to use the difference between values rather than the |
| original values. The *first difference* takes the difference between a value and the value |
| that came directly before it. The first difference is often used to remove the trend |
| from a time series. |
| |
| In the example below, the `diff` function computes the first difference of a time series. |
| The result array length is one value smaller then the original array. |
| This is because the `diff` function only returns a result for values |
| where the prior value has been subtracted. |
| |
| [source,text] |
| ---- |
| let(a=timeseries(collection1, |
| q=*:*, |
| field="test_dt", |
| start="2012-01-20T17:33:18Z", |
| end="2012-12-20T17:33:18Z", |
| gap="+1MONTH", |
| format="YYYY-MM", |
| count(*)), |
| b=col(a, count(*)), |
| c=diff(b)) |
| ---- |
| |
| When this expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "c": [ |
| -55, |
| -27, |
| -88, |
| 259, |
| -194, |
| 81, |
| -210, |
| 168, |
| -101, |
| 249 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 11 |
| } |
| ] |
| } |
| } |
| ---- |
| |
| === Lagged Differences |
| |
| The `diff` function has an optional second parameter to specify a lag in the difference. |
| If a lag is specified the difference is taken between a value and the value at a specified |
| lag in the past. Lagged differences are often used to remove seasonality from a time series. |
| |
| The simple example below demonstrates how lagged differencing works. |
| Notice that the array in the example follows a simple repeated pattern. This type of pattern |
| is often displayed with seasonality. In this example we can remove this pattern using |
| the `diff` function with a lag of 4. This will subtract the value lagging four indexes |
| behind the current index. Notice that result set size is the original array size minus the lag. |
| This is because the `diff` function only returns results for values where the lag of 4 |
| is possible to compute. |
| |
| [source,text] |
| ---- |
| let(a=array(1,2,5,2,1,2,5,2,1,2,5), |
| b=diff(a, 4)) |
| ---- |
| |
| Expression is sent to the `/stream` handler it responds with: |
| |
| [source,json] |
| ---- |
| { |
| "result-set": { |
| "docs": [ |
| { |
| "b": [ |
| 0, |
| 0, |
| 0, |
| 0, |
| 0, |
| 0, |
| 0 |
| ] |
| }, |
| { |
| "EOF": true, |
| "RESPONSE_TIME": 0 |
| } |
| ] |
| } |
| } |
| ---- |
