| = Log Analytics |
| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| This section of the user guide provides an introduction to Solr log analytics. |
| |
| NOTE: This is an appendix of the <<math-expressions.adoc#,Visual Guide to Streaming Expressions and Math Expressions>>. All the functions described below are covered in detail in the guide. |
| See the <<math-start.adoc#,Getting Started>> chapter to learn how to get started with visualizations and Apache Zeppelin. |
| |
| == Loading |
| |
| The out-of-the-box Solr log format can be loaded into a Solr index using the `bin/postlogs` command line tool |
| located in the `bin/` directory of the Solr distribution. |
| |
| NOTE: If working from the source distribution the |
| distribution must first be built before `postlogs` can be run. |
| |
| The `postlogs` script is designed to be run from the root directory of the Solr distribution. |
| |
| The `postlogs` script takes two parameters: |
| |
| * Solr base URL (with collection): `http://localhost:8983/solr/logs` |
| * File path to root of the logs directory: All files found under this directory (including sub-directories) will be indexed. |
| If the path points to a single log file only that log file will be loaded. |
| |
| Below is a sample execution of the `postlogs` tool: |
| |
| [source,text] |
| ---- |
| ./bin/postlogs http://localhost:8983/solr/logs /var/logs/solrlogs |
| ---- |
| |
| The example above will index all the log files under `/var/logs/solrlogs` to the `logs` collection found at the base url `http://localhost:8983/solr`. |
| |
| == Exploring |
| |
| Log exploration is often the first step in log analytics and visualization. |
| |
| When working with unfamiliar installations exploration can be used to understand which collections are |
| covered in the logs, what shards and cores are in those collections and the types of operations being |
| performed on those collections. |
| |
| Even with familiar Solr installations exploration is still extremely |
| important while troubleshooting because it will often turn up surprises such as unknown errors or |
| unexpected admin or indexing operations. |
| |
| === Sampling |
| |
| The first step in exploration is to take a random sample from the `logs` collection |
| with the `random` function. |
| |
| In the example below the `random` function is run with one |
| parameter which is the name of the collection to sample. |
| |
| image::images/math-expressions/logs-sample.png[] |
| |
| The sample contains 500 random records with their full field list. By looking |
| at this sample we can quickly learn about the *fields* available in the `logs` collection. |
| |
| === Time Period |
| |
| Each log record contains a time stamp in the `date_dt` field. |
| Its often useful to understand what time period the logs cover and how many log records have been |
| indexed. |
| |
| The `stats` function can be run to display this information. |
| |
| image::images/math-expressions/logs-dates.png[] |
| |
| |
| === Record Types |
| |
| One of the key fields in the index is the `type_s` field which is the type of log |
| record. |
| |
| The `facet` expression can be used to visualize the different types of log records and how many |
| records of each type are in the index. |
| |
| image::images/math-expressions/logs-type.png[] |
| |
| |
| === Collections |
| |
| Another important field is the `collection_s` field which is the collection that the |
| log record was generated from. |
| |
| The `facet` expression can be used to visualize the different collections and how many log records |
| they generate. |
| |
| image::images/math-expressions/logs-collection.png[] |
| |
| |
| === Record Type by Collection |
| |
| A two dimensional `facet` can be run to visualize the record types by collection. |
| |
| image::images/math-expressions/logs-type-collection.png[] |
| |
| |
| === Time Series |
| |
| The `timeseries` function can be used to visualize a time series for a specific time range |
| of the logs. |
| |
| In the example below a time series is used to visualize the log record counts |
| at 15 second intervals. |
| |
| image::images/math-expressions/logs-time-series.png[] |
| |
| Notice that there is a very low level of log activity up until hour 21 minute 27. |
| Then a burst of log activity occurs from minute 27 to minute 52. |
| |
| This is then followed by a large spike of log activity. |
| |
| The example below breaks this down further by adding a query on the `type_s` field to only |
| visualize *query* activity in the log. |
| |
| |
| image::images/math-expressions/logs-time-series2.png[] |
| |
| Notice the query activity accounts for more then half of the burst of log records between |
| 21:27 and 21:52. But the query activity does not account for the large spike in |
| log activity that follows. |
| |
| We can account for that spike by changing the search to include only *update*, *commit*, |
| and *deleteByQuery* records in the logs. We can also narrow by collection |
| so we know where these activities are taking place. |
| |
| |
| image::images/math-expressions/logs-time-series3.png[] |
| |
| Through the various exploratory queries and visualizations we now have a much |
| better understanding of what's contained in the logs. |
| |
| |
| == Query Counting |
| |
| Distributed searches produce more than one log record for each query. There will be one *top level* log |
| record for |
| the top level distributed query and a *shard level* log record on one replica from each shard. There may also |
| be a set of *ids* queries to retrieve fields by id from the shards to complete the page of results. |
| |
| There are fields in the log index that can be used to differentiate between the three types of query records. |
| |
| The examples below use the `stats` function to count the different types of query records in the logs. |
| The same queries can be used with `search`, `random` and `timeseries` functions to return results |
| for specific types of query records. |
| |
| === Top Level Queries |
| |
| To find all the top level queries in the logs, add a query to limit results to log records with `distrib_s:true` as follows: |
| |
| image::images/math-expressions/query-top-level.png[] |
| |
| |
| === Shard Level Queries |
| |
| To find all the shard level queries that are not IDs queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:false` |
| as follows: |
| |
| image::images/math-expressions/query-shard-level.png[] |
| |
| |
| === ID Queries |
| |
| To find all the *ids* queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:true` |
| as follows: |
| |
| image::images/math-expressions/query-ids.png[] |
| |
| |
| == Query Performance |
| |
| One of the important tasks of Solr log analytics is understanding how well a Solr cluster is performing. |
| |
| The `qtime_i` field contains the query time (QTime) in milliseconds |
| from the log records. |
| There are number of powerful visualizations and statistical approaches for analyzing query performance. |
| |
| |
| === QTime Scatter Plot |
| |
| Scatter plots can be used to visualize random samples of the `qtime_i` |
| field. |
| The example below demonstrates a scatter plot of 500 random samples |
| from the `ptest1` collection of log records. |
| |
| In this example, `qtime_i` is plotted on the y-axis and the x-axis is simply a sequence to spread the query times out across the plot. |
| |
| NOTE: The `x` field is included in the field list. |
| The `random` function automatically generates a sequence for the x-axis when `x` is included in the field list. |
| |
| image::images/math-expressions/qtime-scatter.png[] |
| |
| From this scatter plot we can tell a number of important things about the query times: |
| |
| * The sample query times range from a low of 122 to a high of 643. |
| * The mean appears to be just above 400 ms. |
| * The query times tend to cluster closer to the mean and become less frequent as they move away |
| from the mean. |
| |
| |
| === Highest QTime Scatter Plot |
| |
| It's often useful to be able to visualize the highest query times recorded in the log data. |
| This can be done by using the `search` function and sorting on `qtime_i desc`. |
| |
| In the example below the `search` function returns the highest 500 query times from the `ptest1` collection and sets the results to the variable `a`. |
| Then the `col` function is used to extract the `qtime_i` column from the result set into a vector, which is set to variable `y`. |
| |
| Then the `zplot` function is used plot the query times on the y-axis of the scatter plot. |
| |
| NOTE: The `rev` function is used to reverse the query times vector so the visualization displays from lowest to highest query times. |
| |
| image::images/math-expressions/qtime-highest-scatter.png[] |
| |
| From this plot we can see that the 500 highest query times start at 510ms and slowly move higher, until the last 10 spike upwards, culminating at the highest query time of 2529ms. |
| |
| |
| === QTime Distribution |
| |
| In this example a visualization is created which shows the |
| distribution of query times rounded to the nearest second. |
| |
| The example below starts by taking a random sample of 10000 log records with a `type_s`* of `query`. |
| The results of the `random` function are assigned to the variable `a`. |
| |
| The `col` function is then used extract the `qtime_i` field from the results. |
| The vector of query times is set to variable `b`. |
| |
| The `scalarDivide` function is then used to divide all elements of the query time vector by 1000. |
| This converts the query times from milli-seconds to seconds. |
| The result is set to variable `c`. |
| |
| The `round` function then rounds all elements of the query times vector to the nearest second. |
| This means all query times less than 500ms will round to 0. |
| |
| The `freqTable` function is then applied to the vector of query times rounded to |
| the nearest second. |
| |
| The resulting frequency table is shown in the visualization below. |
| The x-axis is the number of seconds. |
| The y-axis is the number of query times that rounded to each second. |
| |
| image::images/math-expressions/qtime-dist.png[] |
| |
| Notice that roughly 93 percent of the query times rounded to 0, meaning they were under 500ms. |
| About 6 percent round to 1 and the rest rounded to either 2 or 3 seconds. |
| |
| |
| === QTime Percentiles Plot |
| |
| A percentile plot is another powerful tool for understanding the distribution of query times in the logs. |
| The example below demonstrates how to create and interpret percentile plots. |
| |
| In this example an `array` of percentiles is created and set to variable `p`. |
| |
| Then a random sample of 10000 log records is drawn and set to variable `a`. |
| The `col` function is then used to extract the `qtime_i` field from the sample results and this vector is set to variable `b`. |
| |
| The `percentile` function is then used to calculate the value at each percentile for the vector of query times. |
| The array of percentiles set to variable `p` tells the `percentile` function |
| which percentiles to calculate. |
| |
| Then the `zplot` function is used to plot the *percentiles* on the x-axis and |
| the *query time* at each percentile on the y-axis. |
| |
| image::images/math-expressions/query-qq.png[] |
| |
| From the plot we can see that the 80th percentile has a query time of 464ms. |
| This means that 80% percent of queries are below 464ms. |
| |
| === QTime Time Series |
| |
| A time series aggregation can also be run to visualization how QTime changes over time. |
| |
| The example below shows a time series, area chart that visualizes *average query time* at 15 second intervals for a 3 minute section of a log. |
| |
| image::images/math-expressions/qtime-series.png[] |
| |
| |
| == Performance Troubleshooting |
| |
| If query analysis determines that queries are not performing as expected then log analysis can also be used to troubleshoot the cause of the slowness. |
| The section below demonstrates several approaches for locating the source of query slowness. |
| |
| === Slow Nodes |
| |
| In a distributed search the final search performance is only as fast as the slowest responding shard in the cluster. |
| Therefore one slow node can be responsible for slow overall search time. |
| |
| The fields `core_s`, `replica_s` and `shard_s` are available in the log records. |
| These fields allow average query time to be calculated by *core*, *replica* or *shard*. |
| |
| The `core_s` field is particularly useful as its the most granular element and |
| the naming convention often includes the collection, shard and replica information. |
| |
| The example below uses the `facet` function to calculate `avg(qtime_i)` by core. |
| |
| image::images/math-expressions/slow-nodes.png[] |
| |
| Notice in the results that the `core_s` field contains information about the |
| *collection*, *shard*, and *replica*. |
| The example also shows that qtime seems to be significantly higher for certain cores in the same collection. |
| This should trigger a deeper investigation as to why those cores might be performing slower. |
| |
| === Slow Queries |
| |
| If query analysis shows that most queries are performing well but there are outlier queries that are slow, one reason for this may be that specific queries are slow. |
| |
| The `q_s` and `q_t` fields both hold the value of the *q* parameter from Solr requests. |
| The `q_s` field is a string field and the `q_t` field has been tokenized. |
| |
| The `search` function can be used to return the top N slowest queries in the logs by sorting the results by `qtime_i desc`. the example |
| below demonstrates this: |
| |
| image::images/math-expressions/slow-queries.png[] |
| |
| Once the queries have been retrieved they can be inspected and tried individually to determine if the query is consistently slow. |
| If the query is shown to be slow a plan to improve the query performance |
| can be devised. |
| |
| === Commits |
| |
| Commits and activities that cause commits, such as full index replications, can result in slower query performance. |
| Time series visualization can help to determine if commits are |
| related to degraded performance. |
| |
| The first step is to visualize the query performance issue. |
| The time series below limits the log results to records that are type `query` and computes the `max(qtime_i)` at ten minute intervals. |
| The plot shows the day, hour and minute on the x-axis and `max(qtime_i)` in milliseconds on the y-axis. |
| Notice there are some extreme spikes in max `qtime_i` that need to be understood. |
| |
| image::images/math-expressions/query-spike.png[] |
| |
| |
| The next step is to generate a time series that counts commits across the same time intervals. |
| The time series below uses the same `start`, `end` and `gap` as the initial time series. |
| But this time series is computed for records that have a type of `commit`. |
| The count for the commits is calculated and plotted on y-axis. |
| |
| Notice that there are spikes in commit activity that appear near the spikes in max `qtime_i`. |
| |
| image::images/math-expressions/commit-series.png[] |
| |
| The final step is to overlay the two time series in the same plot. |
| |
| This is done by performing both time series and setting the results to variables, in this case |
| `a` and `b`. |
| |
| Then the `date_dt` and `max(qtime_i)` fields are extracted as vectors from the first time series and set to variables using the `col` function. |
| And the `count(*)` field is extracted from the second time series. |
| |
| The `zplot` function is then used to plot the time stamp vector on the x-axis and the max qtimes and commit count vectors on y-axis. |
| |
| NOTE: The `minMaxScale` function is used to scale both vectors |
| between 0 and 1 so they can be visually compared on the same plot. |
| |
| image::images/math-expressions/overlay-series.png[] |
| |
| Notice in this plot that the commit count seems to be closely related to spikes |
| in max `qtime_i`. |
| |
| == Errors |
| |
| The log index will contain any error records found in the logs. Error records will have a `type_s` field value of `error`. |
| |
| The example below searches for error records: |
| |
| image::images/math-expressions/search-error.png[] |
| |
| |
| If the error is followed by a stack trace the stack trace will be present in the searchable field `stack_t`. |
| The example below shows a search on the `stack_t` field and the stack trace presented in the result. |
| |
| image::images/math-expressions/stack.png[] |