sdks/python/apache_beam/testing/analyzers/analyze_metric_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "# Licensed to the Apache Software Foundation (ASF) under one\n",
        "# or more contributor license agreements.  See the NOTICE file\n",
        "# distributed with this work for additional information\n",
        "# regarding copyright ownership.  The ASF licenses this file\n",
        "# to you under the Apache License, Version 2.0 (the\n",
        "# \"License\"); you may not use this file except in compliance\n",
        "# with the License.  You may obtain a copy of the License at\n",
        "#\n",
        "#   http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing,\n",
        "# software distributed under the License is distributed on an\n",
        "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
        "# KIND, either express or implied.  See the License for the\n",
        "# specific language governing permissions and limitations\n",
        "# under the License."
      ],
      "metadata": {
        "id": "fCjymAKWJiTh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# this notebook intended for internal testing purpose."
      ],
      "metadata": {
        "id": "CCAvj4mQFR5x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Apache Beam can be installed directly from the main branch of https://github.com/apache/beam or can be installed via `pip install apache_beam>=2.45.0`"
      ],
      "metadata": {
        "id": "IL7coD4DJqzG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/apache/beam.git\n",
        "!pip install -r beam/sdks/python/build-requirements.txt\n",
        "!pip install -e beam/sdks/python[gcp]\n",
        "!pip install matplotlib\n",
        "!pip install pandas"
      ],
      "metadata": {
        "id": "yW4okqmpECqY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import necessary dependencies"
      ],
      "metadata": {
        "id": "ZPt3DbZcL-Ki"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from apache_beam.testing.load_tests import load_test_metrics_utils\n",
        "from apache_beam.testing.load_tests.load_test_metrics_utils import BigQueryMetricsFetcher"
      ],
      "metadata": {
        "id": "xYGgc-tpE9qY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bq_project = 'apache-beam-testing'\n",
        "bq_dataset = '<bq-dataset-name>' # sample value: beam_run_inference\n",
        "bq_table = '<bq-table>' # sample value: torch_inference_imagenet_results_resnet152\n",
        "metric_name = '<perf-alerted-metric-name>' # sample value: mean_load_model_latency_milli_secs\n",
        "\n",
        "query = f\"\"\"\n",
        "      SELECT *\n",
        "      FROM {bq_project}.{bq_dataset}.{bq_table}\n",
        "      WHERE CONTAINS_SUBSTR(({load_test_metrics_utils.METRICS_TYPE_LABEL}), '{metric_name}')\n",
        "      ORDER BY {load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL} DESC\n",
        "      LIMIT 30\n",
        "    \"\"\"\n"
      ],
      "metadata": {
        "id": "nyMmUpRrD_zV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the performance/load test store the results in BigQuery using this [schema](https://github.com/apache/beam/blob/83679216cce2d52dbeb7e837f06ca1d57b31d509/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py#L66),\n",
        "then fetch the metric_values for a `metric_name` for the last `30` runs and display a plot using matplotlib.\n"
      ],
      "metadata": {
        "id": "RwlsXCLbVs_2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "big_query_metrics_fetcher = BigQueryMetricsFetcher()\n",
        "metric_data: pd.DataFrame = big_query_metrics_fetcher.fetch(query=query)"
      ],
      "metadata": {
        "id": "rmOE_odNEBFK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# sort the data to view it in chronological order.\n",
        "metric_data.sort_values(\n",
        "      by=[load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL], inplace=True)"
      ],
      "metadata": {
        "id": "q-i3qLpGV5Ly"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "metric_data.plot(x=load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL,\n",
        "                 y=load_test_metrics_utils.VALUE_LABEL)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "vbFoxdxHVvtQ"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}