Annotating Change Points in Grafana

Change points found by analyze can be exported as Grafana annotations using the --update-grafana flag:

$ otava analyze <test or group> --update-grafana

The annotations generated by Otava get the following tags:

  • otava
  • change-point
  • test:<test name>
  • metric:<metric name>
  • tags configured in the tags property of the test
  • tags configured in the annotate property of the test
  • tags configured in the annotate property of the metric

Additionally, the annotate property supports variable tags:

  • %{TEST_NAME} - name of the test
  • %{METRIC_NAME} - name of the metric
  • %{GRAPHITE_PATH} - resolves to the path to the data in Graphite
  • %{GRAPHITE_PATH_COMPONENTS} - splits the path of the data in Graphite into separate components and each path component is exported as a separate tag
  • %{GRAPHITE_PREFIX} - resolves to the prefix of the path to the data in Graphite (the part of the path up to the metric suffix)
  • %{GRAPHITE_PREFIX_COMPONENTS} - similar as %{GRAPHITE_PATH_COMPONENTS} but splits the prefix of the path instead of the path

Example

[!TIP] See otava.yaml for the full Graphite & Grafana example.

Start docker-compose with Graphite in one tab:

docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes --build

Run otava in another tab:

docker-compose -f examples/graphite/docker-compose.yaml run otava otava analyze my-product.test --since=-10m --update-grafana

Expected output:

time                       run    branch    version    commit      throughput    response_time    cpu_usage
-------------------------  -----  --------  ---------  --------  ------------  ---------------  -----------
2024-12-14 22:45:10 +0000                                               61160               87          0.2
2024-12-14 22:46:10 +0000                                               60160               85          0.3
2024-12-14 22:47:10 +0000                                               60960               89          0.1
                                                                 ············                   ···········
                                                                        -5.6%                       +300.0%
                                                                 ············                   ···········
2024-12-14 22:48:10 +0000                                               57123               88          0.8
2024-12-14 22:49:10 +0000                                               57980               87          0.9
2024-12-14 22:50:10 +0000                                               56950               85          0.7

Open local Grafana in your browser (use default admin/admin credentials), open Benchmarks dashboard, and see your data points and annotations on both charts.: