| |
| <!DOCTYPE html> |
| <html lang="en" dir=ZgotmplZ> |
| |
| <head> |
| |
| |
| |
| <link rel="stylesheet" href="/bootstrap/css/bootstrap.min.css"> |
| <script src="/bootstrap/js/bootstrap.bundle.min.js"></script> |
| <link rel="stylesheet" type="text/css" href="/font-awesome/css/font-awesome.min.css"> |
| <script src="/js/anchor.min.js"></script> |
| <script src="/js/flink.js"></script> |
| <link rel="canonical" href="https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/"> |
| |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="description" content="Backpressure monitoring in the web UI |
| The backpressure topic was tackled from different angles over the last couple of years. However, when it comes to identifying and analyzing sources of backpressure, things have changed quite a bit in the recent Flink releases (especially with new additions to metrics and the web UI in Flink 1.13). This post will try to clarify some of these changes and go into more detail about how to track down the source of backpressure, but first…"> |
| <meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="How to identify the source of backpressure?" /> |
| <meta property="og:description" content="Backpressure monitoring in the web UI |
| The backpressure topic was tackled from different angles over the last couple of years. However, when it comes to identifying and analyzing sources of backpressure, things have changed quite a bit in the recent Flink releases (especially with new additions to metrics and the web UI in Flink 1.13). This post will try to clarify some of these changes and go into more detail about how to track down the source of backpressure, but first…" /> |
| <meta property="og:type" content="article" /> |
| <meta property="og:url" content="https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/" /><meta property="article:section" content="posts" /> |
| <meta property="article:published_time" content="2021-07-07T00:00:00+00:00" /> |
| <meta property="article:modified_time" content="2021-07-07T00:00:00+00:00" /> |
| <title>How to identify the source of backpressure? | Apache Flink</title> |
| <link rel="manifest" href="/manifest.json"> |
| <link rel="icon" href="/favicon.png" type="image/x-icon"> |
| <link rel="stylesheet" href="/book.min.22eceb4d17baa9cdc0f57345edd6f215a40474022dfee39b63befb5fb3c596b5.css" integrity="sha256-IuzrTRe6qc3A9XNF7dbyFaQEdAIt/uObY777X7PFlrU="> |
| <script defer src="/en.search.min.2698f0d1b683dae4d6cb071668b310a55ebcf1c48d11410a015a51d90105b53e.js" integrity="sha256-Jpjw0baD2uTWywcWaLMQpV688cSNEUEKAVpR2QEFtT4="></script> |
| <!-- |
| Made with Book Theme |
| https://github.com/alex-shpak/hugo-book |
| --> |
| |
| <meta name="generator" content="Hugo 0.124.1"> |
| |
| |
| <script> |
| var _paq = window._paq = window._paq || []; |
| |
| |
| _paq.push(['disableCookies']); |
| |
| _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="//analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '1']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| |
| </head> |
| |
| <body dir=ZgotmplZ> |
| |
| |
| |
| <header> |
| <nav class="navbar navbar-expand-xl"> |
| <div class="container-fluid"> |
| <a class="navbar-brand" href="/"> |
| <img src="/img/logo/png/100/flink_squirrel_100_color.png" alt="Apache Flink" height="47" width="47" class="d-inline-block align-text-middle"> |
| <span>Apache Flink</span> |
| </a> |
| <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <i class="fa fa-bars navbar-toggler-icon"></i> |
| </button> |
| <div class="collapse navbar-collapse" id="navbarSupportedContent"> |
| <ul class="navbar-nav"> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">About</a> |
| <ul class="dropdown-menu"> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/flink-architecture/">Architecture</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/flink-applications/">Applications</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/flink-operations/">Operations</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/use-cases/">Use Cases</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/powered-by/">Powered By</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/roadmap/">Roadmap</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/community/">Community & Project Info</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/security/">Security</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/what-is-flink/special-thanks/">Special Thanks</a> |
| |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| |
| |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Getting Started</a> |
| <ul class="dropdown-menu"> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/">With Flink<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/docs/try-flink-kubernetes-operator/quick-start/">With Flink Kubernetes Operator<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/">With Flink CDC<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/try-flink-ml/quick-start/">With Flink ML<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/getting-started/project-setup.html">With Flink Stateful Functions<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/overview/">Training Course<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| |
| |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a> |
| <ul class="dropdown-menu"> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/">Flink 1.19 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-master/">Flink Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">Kubernetes Operator 1.8 (latest)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main">Kubernetes Operator Main (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable">CDC 3.0 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-master">CDC Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">ML 2.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-master">ML Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">Stateful Functions 3.3 (stable)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-master">Stateful Functions Master (snapshot)<i class="link fa fa-external-link title" aria-hidden="true"></i> |
| </a> |
| |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| |
| |
| <li class="nav-item dropdown"> |
| <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">How to Contribute</a> |
| <ul class="dropdown-menu"> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/overview/">Overview</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/contribute-code/">Contribute Code</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/reviewing-prs/">Review Pull Requests</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/code-style-and-quality-preamble/">Code Style and Quality Guide</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/contribute-documentation/">Contribute Documentation</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/documentation-style-guide/">Documentation Style Guide</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/improve-website/">Contribute to the Website</a> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| <a class="dropdown-item" href="/how-to-contribute/getting-help/">Getting Help</a> |
| |
| |
| </li> |
| |
| </ul> |
| </li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <li class="nav-item"> |
| |
| |
| <a class="nav-link" href="/posts/">Flink Blog</a> |
| |
| |
| </li> |
| |
| |
| |
| |
| |
| <li class="nav-item"> |
| |
| |
| <a class="nav-link" href="/downloads/">Downloads</a> |
| |
| |
| </li> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| </ul> |
| <div class="book-search"> |
| <div class="book-search-spinner hidden"> |
| <i class="fa fa-refresh fa-spin"></i> |
| </div> |
| <form class="search-bar d-flex" onsubmit="return false;"su> |
| <input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/"> |
| <i class="fa fa-search search"></i> |
| <i class="fa fa-circle-o-notch fa-spin spinner"></i> |
| </form> |
| <div class="book-search-spinner hidden"></div> |
| <ul id="book-search-results"></ul> |
| </div> |
| </div> |
| </div> |
| </nav> |
| <div class="navbar-clearfix"></div> |
| </header> |
| |
| |
| <main class="flex"> |
| <section class="container book-page"> |
| |
| <article class="markdown"> |
| <h1> |
| <a href="/2021/07/07/how-to-identify-the-source-of-backpressure/">How to identify the source of backpressure?</a> |
| </h1> |
| |
| |
| |
| July 7, 2021 - |
| |
| |
| |
| Piotr Nowojski |
| |
| <a href="https://twitter.com/PiotrNowojski">(@PiotrNowojski)</a> |
| |
| |
| |
| |
| <p><div> |
| <img src="/img/blog/2021-07-07-backpressure/animated.png" alt="Backpressure monitoring in the web UI"/> |
| <p class="align-center">Backpressure monitoring in the web UI</p> |
| </div> |
| <p>The backpressure topic was tackled from different angles over the last couple of years. However, when it comes |
| to identifying and analyzing sources of backpressure, things have changed quite a bit in the recent Flink releases |
| (especially with new additions to metrics and the web UI in Flink 1.13). This post will try to clarify some of |
| these changes and go into more detail about how to track down the source of backpressure, but first…</p> |
| <h2 id="what-is-backpressure"> |
| What is backpressure? |
| <a class="anchor" href="#what-is-backpressure">#</a> |
| </h2> |
| <p>This has been explained very well in an old, but still accurate, <a href="https://www.ververica.com/blog/how-flink-handles-backpressure">post by Ufuk Celebi</a>. |
| I highly recommend reading it if you are not familiar with this concept. For a much deeper and low-level understanding of |
| the topic and how Flink’s network stack works, there is a more <a href="https://alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632">advanced explanation available here</a>.</p> |
| <p>At a high level, backpressure happens if some operator(s) in the Job Graph cannot process records at the |
| same rate as they are received. This fills up the input buffers of the subtask that is running this slow operator. |
| Once the input buffers are full, backpressure propagates to the output buffers of the upstream subtasks. |
| Once those are filled up, the upstream subtasks are also forced to slow down their records’ processing |
| rate to match the processing rate of the operator causing this bottleneck down the stream. Backpressure |
| further propagates up the stream until it reaches the source operators.</p> |
| <p>As long as the load and available resources are static and none of the operators produce short bursts of |
| data (like windowing operators), those input/output buffers should only be in one of two states: almost empty |
| or almost full. If the downstream operator or subtask is able to keep up with the influx of data, the |
| buffers will be empty. If not, then the buffers will be full [<sup>1</sup>]. In fact, checking the buffers’ usage metrics |
| was the basis of the previously recommended way on how to detect and analyze backpressure described <a href="https://flink.apache.org/2019/07/23/flink-network-stack-2.html#backpressure">a couple |
| of years back by Nico Kruber</a>. |
| As I mentioned in the beginning, Flink now offers much better tools to do the same job, but before we get to that, |
| there are two questions worth asking.</p> |
| <h3 id="why-should-i-care-about-backpressure"> |
| Why should I care about backpressure? |
| <a class="anchor" href="#why-should-i-care-about-backpressure">#</a> |
| </h3> |
| <p>Backpressure is an indicator that your machines or operators are overloaded. The buildup of backpressure |
| directly affects the end-to-end latency of the system, as records are waiting longer in the queues before |
| being processed. Secondly, aligned checkpointing takes longer with backpressure, while unaligned checkpoints |
| will be larger (you can read more about aligned and unaligned checkpoints <a href="//nightlies.apache.org/flink/flink-docs-release-1.13/docs/concepts/stateful-stream-processing/#checkpointing">in the documentation</a>. |
| If you are struggling with checkpoint barriers propagation times, taking care of backpressure would most |
| likely help to solve the problem. Lastly, you might just want to optimize your job in order to reduce |
| the costs of running the job.</p> |
| <p>In order to address the problem for all cases, one needs to be aware of it, then locate and analyze it.</p> |
| <h3 id="why-shouldnt-i-care-about-backpressure"> |
| Why shouldn’t I care about backpressure? |
| <a class="anchor" href="#why-shouldnt-i-care-about-backpressure">#</a> |
| </h3> |
| <p>Frankly, you do not always have to care about the presence of backpressure. Almost by definition, lack |
| of backpressure means that your cluster is at least ever so slightly underutilized and over-provisioned. |
| If you want to minimize idling resources, you probably can not avoid incurring some backpressure. This |
| is especially true for batch processing.</p> |
| <h2 id="how-to-detect-and-track-down-the-source-of-backpressure"> |
| How to detect and track down the source of backpressure? |
| <a class="anchor" href="#how-to-detect-and-track-down-the-source-of-backpressure">#</a> |
| </h2> |
| <p>One way to detect backpressure is to use <a href="//nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/metrics/#system-metrics">metrics</a>, |
| however, in Flink 1.13 it’s no longer necessary to dig so deep. In most cases, it should be enough to just |
| look at the job graph in the Web UI.</p> |
| <div> |
| <img src="/img/blog/2021-07-07-backpressure/simple-example.png"/> |
| </div> |
| <p>The first thing to note in the example above is that different tasks have different colors. Those colors |
| represent a combination of two factors: under how much backpressure this task is and how busy it is. Idling |
| tasks will be blue, fully busy tasks will be red hot, and fully backpressured tasks will be black. Anything |
| in between will be a combination/shade of those three colors. With this knowledge, one can easily spot the |
| backpressured tasks (black). The busiest (red) task downstream of the backpressured tasks will most likely |
| be the source of the backpressure (the bottleneck).</p> |
| <p>If you click on one particular task and go into the “BackPressure” tab you will be able to further dissect |
| the problem and check what is the busy/backpressured/idle status of every subtask in that task. For example, |
| this is especially handy if there is a data skew and not all subtasks are equally utilized.</p> |
| <div> |
| <img src="/img/blog/2021-07-07-backpressure/subtasks.png" alt="Backpressure among subtasks"/> |
| <p class="align-center">Backpressure among subtasks</p> |
| </div> |
| <p>In the above example, we can clearly see which subtasks are idling, which are backpressured, and that |
| none of them are busy. And frankly, in a nutshell, that should be enough to quickly understand what is |
| happening with your Job :) However, there are a couple of more details worth explaining.</p> |
| <h3 id="what-are-those-numbers"> |
| What are those numbers? |
| <a class="anchor" href="#what-are-those-numbers">#</a> |
| </h3> |
| <p>If you are curious how it works underneath, we can go a little deeper. At the base of this new mechanism |
| we have three <a href="//nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/metrics/#io">new metrics</a> |
| that are exposed and calculated by each subtask:</p> |
| <ul> |
| <li><code>idleTimeMsPerSecond</code></li> |
| <li><code>busyTimeMsPerSecond</code></li> |
| <li><code>backPressuredTimeMsPerSecond</code></li> |
| </ul> |
| <p>Each of them measures the average time in milliseconds per second that the subtask spent being idle, |
| busy, or backpressured respectively. Apart from some rounding errors they should complement each other and |
| add up to 1000ms/s. In essence, they are quite similar to, for example, CPU usage metrics.</p> |
| <p>Another important detail is that they are being averaged over a short period of time (a couple of seconds) |
| and they take into account everything that is happening inside the subtask’s thread: operators, functions, |
| timers, checkpointing, records serialization/deserialization, network stack, and other Flink internal |
| overheads. A <code>WindowOperator</code> that is busy firing timers and producing results will be reported as busy or backpressured. |
| A function doing some expensive computation in <code>CheckpointedFunction#snapshotState</code> call, for instance flushing |
| internal buffers, will also be reported as busy.</p> |
| <p>One limitation, however, is that <code>busyTimeMsPerSecond</code> and <code>idleTimeMsPerSecond</code> metrics are oblivious |
| to anything that is happening in separate threads, outside of the main subtask’s execution loop. |
| Fortunately, this is only relevant for two cases:</p> |
| <ul> |
| <li>Custom threads that you manually spawn in your operators (a discouraged practice).</li> |
| <li>Old-style sources that implement the deprecated <code>SourceFunction</code> interface. Such sources will report <code>NaN</code>/<code>N/A</code> |
| as the value for busyTimeMsPerSecond. For more information on the topic of Data Sources please |
| <a href="//nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/datastream/sources/">take a look here</a>.</li> |
| </ul> |
| <div> |
| <img src="/img/blog/2021-07-07-backpressure/source-task-busy.png" alt="Old-style sources do not report busy time"/> |
| <p class="align-center">Old-style sources do not report busy time</p> |
| </div> |
| <p>In order to present those raw numbers in the web UI, those metrics need to be aggregated from all subtasks |
| (on the job graph we are showing only tasks). This is why the web UI presents the maximal value from all |
| subtasks of a given task and why the aggregated maximal values of busy and backpressured may not add up to 100%. |
| One subtask can be backpressured at 60%, while another can be busy at 60%. This can result in a task that |
| is both backpressured and busy at 60%.</p> |
| <h3 id="varying-load"> |
| Varying load |
| <a class="anchor" href="#varying-load">#</a> |
| </h3> |
| <p>There is one more thing. Do you remember that those metrics are measured and averaged over a couple of seconds? |
| Keep this in mind when analyzing jobs or tasks with varying load, such as (sub)tasks containing a <code>WindowOperator</code> |
| that is firing periodically. Both the subtask with a constant load of 50% and the subtask that alternates every |
| second between being fully busy and fully idle will be reporting the same value of <code>busyTimeMsPerSecond</code> |
| of 500ms/s.</p> |
| <p>Furthermore, varying load and especially firing windows can move the bottleneck to a different place in |
| the job graph:</p> |
| <div> |
| <img src="/img/blog/2021-07-07-backpressure/bottleneck-zoom.png" alt="Bottleneck alternating between two tasks"/> |
| <p class="align-center">Bottleneck alternating between two tasks</p> |
| </div> |
| <div> |
| <img src="/img/blog/2021-07-07-backpressure/sliding-window.png" alt="SlidingWindowOperator"/> |
| <p class="align-center">SlidingWindowOperator</p> |
| </div> |
| <p>In this particular example, <code>SlidingWindowOperator</code> was the bottleneck as long as it was accumulating records. |
| However, as soon as it starts to fire its windows (once every 10 seconds), the downstream task |
| <code>SlidingWindowCheckMapper -> Sink: SlidingWindowCheckPrintSink</code> becomes the bottleneck and <code>SlidingWindowOperator</code> |
| gets backpressured. As those busy/backpressured/idle metrics are averaging time over a couple of seconds, |
| this subtlety is not immediately visible and has to be read between the lines. On top of that, the web UI |
| is updating its state only once every 10 seconds, which makes spotting more frequent changes a bit more difficult.</p> |
| <h2 id="what-can-i-do-with-backpressure"> |
| What can I do with backpressure? |
| <a class="anchor" href="#what-can-i-do-with-backpressure">#</a> |
| </h2> |
| <p>In general this is a complex topic that is worthy of a dedicated blog post. It was, to a certain extent, |
| addressed in <a href="https://flink.apache.org/2019/07/23/flink-network-stack-2.html#:~:text=this%20is%20unnecessary.-,What%20to%20do%20with%20Backpressure%3F,-Assuming%20that%20you">previous blog posts</a>. |
| In short, there are two high-level ways of dealing with backpressure. Either add more resources (more machines, |
| faster CPU, more RAM, better network, using SSDs…) or optimize usage of the resources you already have |
| (optimize the code, tune the configuration, avoid data skew). In either case, you first need to analyze |
| what is causing backpressure by:</p> |
| <ol> |
| <li>Identifying the presence of backpressure.</li> |
| <li>Locating which subtask(s) or machines are causing it.</li> |
| <li>Digging deeper into what part of the code is causing it and which resource is scarce.</li> |
| </ol> |
| <p>Backpressure monitoring improvements and metrics can help you with the first two points. To tackle the |
| last one, profiling the code can be the way to go. To help with profiling, also starting from Flink 1.13, |
| <a href="http://www.brendangregg.com/flamegraphs.html">Flame Graphs</a> are <a href="//nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/flame_graphs/">integrated into Flink’s web UI</a>. |
| Flame Graphs is a well known profiling tool and visualization technique and I encourage you to give it a try.</p> |
| <p>But keep in mind that after locating where the bottleneck is, you can analyze it the same way you would |
| any other non-distributed application (by checking resource utilization, attaching a profiler, etc). |
| Usually there is no silver bullet for problems like this. You can try to scale up but sometimes it might |
| not be easy or practical to do.</p> |
| <p>Anyway… The aforementioned improvements to backpressure monitoring allow us to easily detect the source of backpressure, |
| and Flame Graphs can help us to analyze why a particular subtask is causing problems. Together those two |
| features should make the previously quite tedious process of debugging and performance analysis of Flink |
| jobs that much easier! Please upgrade to Flink 1.13.x and try them out!</p> |
| <p>[<sup>1</sup>] There is a third possibility. In a rare case when network exchange is actually the bottleneck in your job, |
| the downstream task will have empty input buffers, while upstream output buffers will be full. <a class="anchor" id="1"></a></p> |
| </p> |
| </article> |
| |
| |
| |
| |
| |
| |
| |
| <div class="edit-this-page"> |
| <p> |
| <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications">Want to contribute translation?</a> |
| </p> |
| <p> |
| <a href="//github.com/apache/flink-web/edit/asf-site/docs/content/posts/2021-07-07-backpressure.md"> |
| Edit This Page<i class="fa fa-edit fa-fw"></i> |
| </a> |
| </p> |
| </div> |
| |
| </section> |
| |
| <aside class="book-toc"> |
| |
| |
| |
| <nav id="TableOfContents"><h3>On This Page <a href="javascript:void(0)" class="toc" onclick="collapseToc()"><i class="fa fa-times" aria-hidden="true"></i></a></h3> |
| <ul> |
| <li> |
| <ul> |
| <li><a href="#what-is-backpressure">What is backpressure?</a> |
| <ul> |
| <li><a href="#why-should-i-care-about-backpressure">Why should I care about backpressure?</a></li> |
| <li><a href="#why-shouldnt-i-care-about-backpressure">Why shouldn’t I care about backpressure?</a></li> |
| </ul> |
| </li> |
| <li><a href="#how-to-detect-and-track-down-the-source-of-backpressure">How to detect and track down the source of backpressure?</a> |
| <ul> |
| <li><a href="#what-are-those-numbers">What are those numbers?</a></li> |
| <li><a href="#varying-load">Varying load</a></li> |
| </ul> |
| </li> |
| <li><a href="#what-can-i-do-with-backpressure">What can I do with backpressure?</a></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| |
| |
| </aside> |
| <aside class="expand-toc hidden"> |
| <a class="toc" onclick="expandToc()" href="javascript:void(0)"> |
| <i class="fa fa-bars" aria-hidden="true"></i> |
| </a> |
| </aside> |
| |
| </main> |
| |
| <footer> |
| |
| |
| |
| <div class="separator"></div> |
| <div class="panels"> |
| <div class="wrapper"> |
| <div class="panel"> |
| <ul> |
| <li> |
| <a href="https://flink-packages.org/">flink-packages.org</a> |
| </li> |
| <li> |
| <a href="https://www.apache.org/">Apache Software Foundation</a> |
| </li> |
| <li> |
| <a href="https://www.apache.org/licenses/">License</a> |
| </li> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <li> |
| <a href="/zh/"> |
| <i class="fa fa-globe" aria-hidden="true"></i> 中文版 |
| </a> |
| </li> |
| |
| |
| |
| </ul> |
| </div> |
| <div class="panel"> |
| <ul> |
| <li> |
| <a href="/what-is-flink/security">Security</a--> |
| </li> |
| <li> |
| <a href="https://www.apache.org/foundation/sponsorship.html">Donate</a> |
| </li> |
| <li> |
| <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
| </li> |
| </ul> |
| </div> |
| <div class="panel icons"> |
| <div> |
| <a href="/posts"> |
| <div class="icon flink-blog-icon"></div> |
| <span>Flink blog</span> |
| </a> |
| </div> |
| <div> |
| <a href="https://github.com/apache/flink"> |
| <div class="icon flink-github-icon"></div> |
| <span>Github</span> |
| </a> |
| </div> |
| <div> |
| <a href="https://twitter.com/apacheflink"> |
| <div class="icon flink-twitter-icon"></div> |
| <span>Twitter</span> |
| </a> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| <hr/> |
| |
| <div class="container disclaimer"> |
| <p>The contents of this website are © 2024 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p> |
| </div> |
| |
| |
| |
| </footer> |
| |
| </body> |
| </html> |
| |
| |
| |
| |
| |
| |
