| |
| <!DOCTYPE html> |
| <html lang="en" dir=> |
| |
| <head> |
| <meta name="generator" content="Hugo 0.113.0"> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="description" content="What is Apache Flink? — Operations # Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Since many streaming applications are designed to run continuously with minimal downtime, a stream processor must provide excellent failure recovery, as well as tooling to monitor and maintain applications while they are running. |
| Apache Flink puts a strong focus on the operational aspects of stream processing. Here, we explain Flink’s failure recovery mechanism and present its features to manage and supervise running applications."> |
| <meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Operations" /> |
| <meta property="og:description" content="What is Apache Flink? — Operations # Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Since many streaming applications are designed to run continuously with minimal downtime, a stream processor must provide excellent failure recovery, as well as tooling to monitor and maintain applications while they are running. |
| Apache Flink puts a strong focus on the operational aspects of stream processing. Here, we explain Flink’s failure recovery mechanism and present its features to manage and supervise running applications." /> |
| <meta property="og:type" content="article" /> |
| <meta property="og:url" content="https://flink.apache.org/what-is-flink/flink-operations/" /><meta property="article:section" content="what-is-flink" /> |
| |
| |
| <title>Operations | Apache Flink</title> |
| <link rel="manifest" href="/manifest.json"> |
| <link rel="icon" href="/favicon.png" type="image/x-icon"> |
| <link rel="alternate" hreflang="zh" href="https://flink.apache.org/zh/what-is-flink/flink-operations/" title="运维"> |
| |
| <link rel="stylesheet" href="/book.min.e3b33391dbc1f4b2cc47778e2f4b86c744ded3ccc82fdfb6f08caf91d8607f9a.css" integrity="sha256-47MzkdvB9LLMR3eOL0uGx0Te08zIL9+28Iyvkdhgf5o="> |
| <script defer src="/en.search.min.8592fd2e43835d2ef6fab8eb9b8969ee6ad1bdb888a636e37e28032f8bd9887d.js" integrity="sha256-hZL9LkODXS72+rjrm4lp7mrRvbiIpjbjfigDL4vZiH0="></script> |
| <!-- |
| Made with Book Theme |
| https://github.com/alex-shpak/hugo-book |
| --> |
| |
| |
| |
| <link rel="stylesheet" type="text/css" href="/font-awesome/css/font-awesome.min.css"> |
| <script src="/js/anchor.min.js"></script> |
| <script src="/js/flink.js"></script> |
| <link rel="canonical" href="https://flink.apache.org/what-is-flink/flink-operations/"> |
| |
| |
| <script> |
| var _paq = window._paq = window._paq || []; |
| |
| |
| _paq.push(['disableCookies']); |
| |
| _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]); |
| _paq.push(['trackPageView']); |
| _paq.push(['enableLinkTracking']); |
| (function() { |
| var u="//analytics.apache.org/"; |
| _paq.push(['setTrackerUrl', u+'matomo.php']); |
| _paq.push(['setSiteId', '1']); |
| var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; |
| g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); |
| })(); |
| </script> |
| |
| </head> |
| |
| <body dir=> |
| <input type="checkbox" class="hidden toggle" id="menu-control" /> |
| <input type="checkbox" class="hidden toggle" id="toc-control" /> |
| <main class="container flex"> |
| <aside class="book-menu"> |
| |
| |
| |
| <nav> |
| |
| |
| <a id="logo" href="/"> |
| <img width="70%" src="/flink-header-logo.svg"> |
| </a> |
| |
| <div class="book-search"> |
| <input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" /> |
| <div class="book-search-spinner hidden"></div> |
| <ul id="book-search-results"></ul> |
| </div> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <input type="checkbox" id="section-4117fb24454a2c30ee86e524839e77ec" class="toggle" checked /> |
| <label for="section-4117fb24454a2c30ee86e524839e77ec" class="flex justify-between flink-menu-item">What is Apache Flink?<span>▾</span> |
| </label> |
| |
| <ul> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-ffd5922da551e96e0481423fab94c463" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/what-is-flink/flink-architecture/" class="">Architecture</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-fc28f08b67476edb77e00e03b6c7c2e0" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/what-is-flink/flink-applications/" class="">Applications</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-612df33a02d5d4ee78d718abaab5b5b4" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/what-is-flink/flink-operations/" class="active">Operations</a> |
| </label> |
| |
| |
| </li> |
| |
| </ul> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-f1ecec07350bd6810050d40158878749" class="flex justify-between flink-menu-item"> |
| <a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/" style="color:black" class="">What is Stateful Functions? <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-4113a4c3072cb35f6fd7a0d4e098ee70" class="flex justify-between flink-menu-item"> |
| <a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/" style="color:black" class="">What is Flink ML? <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-b39c70259d0abbe2bf1d8d645425f84d" class="flex justify-between flink-menu-item"> |
| <a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/" style="color:black" class="">What is the Flink Kubernetes Operator? <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-53e0b1afcb9ccaf779dc285aa272a014" class="flex justify-between flink-menu-item"> |
| <a href="https://nightlies.apache.org/flink/flink-table-store-docs-stable/" style="color:black" class="">What is Flink Table Store? <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-f4973f06a66f063045b4ebdacaf3127d" class="flex justify-between flink-menu-item"> |
| <a href="/use-cases/" class="">Use Cases</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-0f1863835376e859ac438ae9529daff2" class="flex justify-between flink-menu-item"> |
| <a href="/powered-by/" class="">Powered By</a> |
| </label> |
| |
| |
| |
| |
| |
| <br/> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-f383f23a96a43d8d0cc66aeb0237e26a" class="flex justify-between flink-menu-item"> |
| <a href="/downloads/" class="">Downloads</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <input type="checkbox" id="section-c727fab97b4d77e5b28ce8c448fb9000" class="toggle" /> |
| <label for="section-c727fab97b4d77e5b28ce8c448fb9000" class="flex justify-between flink-menu-item">Getting Started<span>▾</span> |
| </label> |
| |
| <ul> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-f45abaa99ab076108b9a5b94edbc6647" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/" style="color:black" class="">With Flink <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-efe2166e9dce6f72e126dcc2396b4402" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/getting-started/project-setup.html" style="color:black" class="">With Flink Stateful Functions <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-7e268d0a469b1093bb33d71d093eb7b9" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/try-flink-ml/quick-start/" style="color:black" class="">With Flink ML <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-cc7147cd0441503127bfaf6f219d4fbb" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/docs/try-flink-kubernetes-operator/quick-start/" style="color:black" class="">With Flink Kubernetes Operator <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-660ca694e416d8ca9176dda52a60d637" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-table-store-docs-stable/docs/try-table-store/quick-start/" style="color:black" class="">With Flink Table Store <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-75db0b47bf4ae9c247aadbba5fbd720d" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/overview/" style="color:black" class="">Training Course <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| </ul> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <input type="checkbox" id="section-6318075fef29529089951a49d413d083" class="toggle" /> |
| <label for="section-6318075fef29529089951a49d413d083" class="flex justify-between flink-menu-item">Documentation<span>▾</span> |
| </label> |
| |
| <ul> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-9a8122d8912450484d1c25394ad40229" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-docs-stable/" style="color:black" class="">Flink 1.17 (stable) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-8b2fd3efb702be3783ba98d650707e3c" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-docs-master/" style="color:black" class="">Flink Master (snapshot) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-5317a079cddb964c59763c27607f43d9" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/" style="color:black" class="">Stateful Functions 3.2 (stable) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-25b72f108b7156e94d91b04853d8813a" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-statefun-docs-master" style="color:black" class="">Stateful Functions Master (snapshot) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-27ca02ef0fc9328fd4cb17a730bb50ab" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-ml-docs-stable/" style="color:black" class="">ML 2.3 (stable) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-6d895ec5ad127a29a6a9ce101328ccdf" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-ml-docs-master" style="color:black" class="">ML Master (snapshot) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-0def4384f0d0330c940232737a100605" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/" style="color:black" class="">Kubernetes Operator 1.5 (latest) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-a2c75d90005425982ba8f26ae0e160a3" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main" style="color:black" class="">Kubernetes Operator Main (snapshot) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-07b85e4b2f61b1526bf202c64460abcd" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-table-store-docs-stable/" style="color:black" class="">Table Store 0.3 (stable) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-9b9a0032b1e858a34c125d828d1a0718" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="https://nightlies.apache.org/flink/flink-table-store-docs-master/" style="color:black" class="">Table Store Master (snapshot) <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| </li> |
| |
| </ul> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-63d6a565d79aa2895f70806a46021c07" class="flex justify-between flink-menu-item"> |
| <a href="/getting-help/" class="">Getting Help</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-1d5066022b83f4732dc80f4e9eaa069a" class="flex justify-between flink-menu-item"> |
| <a href="https://flink-packages.org/" style="color:black" class="">flink-packages.org <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| |
| |
| <br/> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-7821b78a97db9e919426e86121a7be9c" class="flex justify-between flink-menu-item"> |
| <a href="/community/" class="">Community & Project Info</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-8c042831df4e371c4ef9375f1df06f35" class="flex justify-between flink-menu-item"> |
| <a href="/roadmap/" class="">Roadmap</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <input type="checkbox" id="section-73117efde5302fddcb193307d582b588" class="toggle" /> |
| <label for="section-73117efde5302fddcb193307d582b588" class="flex justify-between flink-menu-item">How to Contribute<span>▾</span> |
| </label> |
| |
| <ul> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-6646b26b23a3e79b8de9c552ee76f6dd" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/overview/" class="">Overview</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-e6ab9538b82cd5f94103b971adb7c1a9" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/contribute-code/" class="">Contribute Code</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-1c09e1358485e82d9b3f5f689d4ced65" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/reviewing-prs/" class="">Review Pull Requests</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-ed01e0defd235498fa3c9a2a0b3302fb" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/code-style-and-quality-preamble/" class="">Code Style and Quality Guide</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-4e8d5e9924cf15f397711b0d82e15650" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/contribute-documentation/" class="">Contribute Documentation</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-ddaa8307917e5ba7f60ba3316711e492" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/documentation-style-guide/" class="">Documentation Style Guide</a> |
| </label> |
| |
| |
| </li> |
| |
| <li> |
| |
| |
| |
| |
| |
| <label for="section-390a72c171cc82f180a308b95fc3aa72" class="flex justify-between flink-menu-item flink-menu-child"> |
| <a href="/how-to-contribute/improve-website/" class="">Contribute to the Website</a> |
| </label> |
| |
| |
| </li> |
| |
| </ul> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-9d3ddfd487223d5a199ba301f25c88c6" class="flex justify-between flink-menu-item"> |
| <a href="/security/" class="">Security</a> |
| </label> |
| |
| |
| |
| |
| |
| <br/> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <label for="section-a07783f405300745807d39eacf150420" class="flex justify-between flink-menu-item"> |
| <a href="/posts/" class="">Flink Blog</a> |
| </label> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <br/> |
| <hr class="menu-break"> |
| |
| |
| <label for="section-f71a7070dbb7b669824a6441408ded70" class="flex justify-between flink-menu-item"> |
| <a href="https://github.com/apache/flink" style="color:black" class="">Flink on GitHub <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| <label for="section-2ccaaab8c67f3105bbf7df75faca8027" class="flex justify-between flink-menu-item"> |
| <a href="https://twitter.com/apacheflink" style="color:black" class="">@ApacheFlink <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label> |
| |
| |
| |
| <hr class="menu-break"> |
| <table> |
| <tr> |
| <th colspan="2"> |
| <label for="section-78c2028200542d78f8c1a8f6b4cbb36b" class="flex justify-between flink-menu-item"> |
| <a href="https://www.apache.org/" style="color:black" class="">Apache Software Foundation <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label></th> |
| </tr> |
| <tr> |
| <td> |
| <label for="section-794df3791a8c800841516007427a2aa3" class="flex justify-between flink-menu-item"> |
| <a href="https://www.apache.org/licenses/" style="color:black" class="">License <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label></td> |
| <td> |
| <label for="section-2fae32629d4ef4fc6341f1751b405e45" class="flex justify-between flink-menu-item"> |
| <a href="https://www.apache.org/security/" style="color:black" class="">Security <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label></td> |
| </tr> |
| <tr> |
| <td> |
| <label for="section-0584e445d656b83b431227bb80ff0c30" class="flex justify-between flink-menu-item"> |
| <a href="https://www.apache.org/foundation/sponsorship.html" style="color:black" class="">Donate <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label></td> |
| <td> |
| <label for="section-00d06796e489999226fb5bb27fe1b3b2" class="flex justify-between flink-menu-item"> |
| <a href="https://www.apache.org/foundation/thanks.html" style="color:black" class="">Thanks <i class="link fa fa-external-link title" aria-hidden="true"></i></a> |
| </label></td> |
| </tr> |
| </table> |
| |
| <hr class="menu-break"> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <a href="/zh/what-is-flink/flink-operations/" class="flex align-center"> |
| <i class="fa fa-globe" aria-hidden="true"></i> |
| 中文版 |
| </a> |
| |
| <script src="/js/track-search-terms.js"></script> |
| |
| |
| </nav> |
| |
| |
| |
| |
| <script>(function(){var e=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(){localStorage.setItem("menu.scrollTop",e.scrollTop)}),e.scrollTop=localStorage.getItem("menu.scrollTop")})()</script> |
| |
| |
| |
| </aside> |
| |
| <div class="book-page"> |
| <header class="book-header"> |
| |
| <div class="flex align-center justify-between"> |
| <label for="menu-control"> |
| <img src="/svg/menu.svg" class="book-icon" alt="Menu" /> |
| </label> |
| |
| <strong>Operations</strong> |
| |
| <label for="toc-control"> |
| |
| <img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" /> |
| |
| </label> |
| </div> |
| |
| |
| |
| <aside class="hidden clearfix"> |
| |
| |
| |
| <nav id="TableOfContents"><h3>On This Page <button class="toc" onclick="collapseToc()"><i class="fa fa-compress" aria-hidden="true"></i></button></h3> |
| <ul> |
| <li><a href="#what-is-apache-flink--operations">What is Apache Flink? — Operations</a> |
| <ul> |
| <li><a href="#run-your-applications-non-stop-247">Run Your Applications Non-Stop 24/7</a></li> |
| <li><a href="#update-migrate-suspend-and-resume-your-applications">Update, Migrate, Suspend, and Resume Your Applications</a></li> |
| <li><a href="#monitor-and-control-your-applications">Monitor and Control Your Applications</a></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| |
| |
| </aside> |
| |
| |
| </header> |
| |
| |
| |
| |
| |
| |
| |
| <article class="markdown"><h1 id="what-is-apache-flink--operations"> |
| What is Apache Flink? — Operations |
| <a class="anchor" href="#what-is-apache-flink--operations">#</a> |
| </h1> |
| <p>Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Since many streaming applications are designed to run continuously with minimal downtime, a stream processor must provide excellent failure recovery, as well as tooling to monitor and maintain applications while they are running.</p> |
| <p>Apache Flink puts a strong focus on the operational aspects of stream processing. Here, we explain Flink’s failure recovery mechanism and present its features to manage and supervise running applications.</p> |
| <h2 id="run-your-applications-non-stop-247"> |
| Run Your Applications Non-Stop 24/7 |
| <a class="anchor" href="#run-your-applications-non-stop-247">#</a> |
| </h2> |
| <p>Machine and process failures are ubiquitous in distributed systems. A distributed stream processor like Flink must recover from failures in order to be able to run streaming applications 24/7. Obviously, this does not only mean to restart an application after a failure but also to ensure that its internal state remains consistent, such that the application can continue processing as if the failure had never happened.</p> |
| <p>Flink provides several features to ensure that applications keep running and remain consistent:</p> |
| <ul> |
| <li><strong>Consistent Checkpoints</strong>: Flink’s recovery mechanism is based on consistent checkpoints of an application’s state. In case of a failure, the application is restarted and its state is loaded from the latest checkpoint. In combination with resettable stream sources, this feature can guarantee <em>exactly-once state consistency</em>.</li> |
| <li><strong>Efficient Checkpoints</strong>: Checkpointing the state of an application can be quite expensive if the application maintains terabytes of state. Flink can perform asynchronous and incremental checkpoints, in order to keep the impact of checkpoints on the application’s latency SLAs very small.</li> |
| <li><strong>End-to-End Exactly-Once</strong>: Flink features transactional sinks for specific storage systems that guarantee that data is only written out exactly once, even in case of failures.</li> |
| <li><strong>Integration with Cluster Managers</strong>: Flink is tightly integrated with cluster managers, such as <a href="https://hadoop.apache.org">Hadoop YARN</a> or <a href="https://kubernetes.io">Kubernetes</a>. When a process fails, a new process is automatically started to take over its work.</li> |
| <li><strong>High-Availability Setup</strong>: Flink features a high-availability mode that eliminates all single-points-of-failure. The HA-mode is based on <a href="https://zookeeper.apache.org">Apache ZooKeeper</a>, a battle-proven service for reliable distributed coordination.</li> |
| </ul> |
| <h2 id="update-migrate-suspend-and-resume-your-applications"> |
| Update, Migrate, Suspend, and Resume Your Applications |
| <a class="anchor" href="#update-migrate-suspend-and-resume-your-applications">#</a> |
| </h2> |
| <p>Streaming applications that power business-critical services need to be maintained. Bugs need to be fixed and improvements or new features need to be implemented. However, updating a stateful streaming application is not trivial. Often one cannot simply stop the applications and restart a fixed or improved version because one cannot afford to lose the state of the application.</p> |
| <p>Flink’s <em>Savepoints</em> are a unique and powerful feature that solves the issue of updating stateful applications and many other related challenges. A savepoint is a consistent snapshot of an application’s state and therefore very similar to a checkpoint. However in contrast to checkpoints, savepoints need to be manually triggered and are not automatically removed when an application is stopped. A savepoint can be used to start a state-compatible application and initialize its state. Savepoints enable the following features:</p> |
| <ul> |
| <li><strong>Application Evolution</strong>: Savepoints can be used to evolve applications. A fixed or improved version of an application can be restarted from a savepoint that was taken from a previous version of the application. It is also possible to start the application from an earlier point in time (given such a savepoint exists) to repair incorrect results produced by the flawed version.</li> |
| <li><strong>Cluster Migration</strong>: Using savepoints, applications can be migrated (or cloned) to different clusters.</li> |
| <li><strong>Flink Version Updates</strong>: An application can be migrated to run on a new Flink version using a savepoint.</li> |
| <li><strong>Application Scaling</strong>: Savepoints can be used to increase or decrease the parallelism of an application.</li> |
| <li><strong>A/B Tests and What-If Scenarios</strong>: The performance or quality of two (or more) different versions of an application can be compared by starting all versions from the same savepoint.</li> |
| <li><strong>Pause and Resume</strong>: An application can be paused by taking a savepoint and stopping it. At any later point in time, the application can be resumed from the savepoint.</li> |
| <li><strong>Archiving</strong>: Savepoints can be archived to be able to reset the state of an application to an earlier point in time.</li> |
| </ul> |
| <h2 id="monitor-and-control-your-applications"> |
| Monitor and Control Your Applications |
| <a class="anchor" href="#monitor-and-control-your-applications">#</a> |
| </h2> |
| <p>Just like any other service, continuously running streaming applications need to be supervised and integrated into the operations infrastructure, i.e., monitoring and logging services, of an organization. Monitoring helps to anticipate problems and react ahead of time. Logging enables root-cause analysis to investigate failures. Finally, easily accessible interfaces to control running applications are an important feature.</p> |
| <p>Flink integrates nicely with many common logging and monitoring services and provides a REST API to control applications and query information.</p> |
| <ul> |
| <li><strong>Web UI</strong>: Flink features a web UI to inspect, monitor, and debug running applications. It can also be used to submit executions for execution or cancel them.</li> |
| <li><strong>Logging</strong>: Flink implements the popular slf4j logging interface and integrates with the logging frameworks <a href="https://logging.apache.org/log4j/2.x/">log4j</a> or <a href="https://logback.qos.ch/">logback</a>.</li> |
| <li><strong>Metrics</strong>: Flink features a sophisticated metrics system to collect and report system and user-defined metrics. Metrics can be exported to several reporters, including <a href="https://en.wikipedia.org/wiki/Java_Management_Extensions">JMX</a>, Ganglia, <a href="https://graphiteapp.org/">Graphite</a>, <a href="https://prometheus.io/">Prometheus</a>, <a href="https://github.com/etsy/statsd">StatsD</a>, <a href="https://www.datadoghq.com/">Datadog</a>, and <a href="https://www.slf4j.org/">Slf4j</a>.</li> |
| <li><strong>REST API</strong>: Flink exposes a REST API to submit a new application, take a savepoint of a running application, or cancel an application. The REST API also exposes metadata and collected metrics of running or completed applications.</li> |
| </ul> |
| </article> |
| |
| |
| |
| <footer class="book-footer"> |
| |
| |
| |
| |
| |
| |
| |
| <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications">Want to contribute translation?</a> |
| <br><br> |
| <a href="//github.com/apache/flink-web/edit/asf-site/docs/content/what-is-flink/flink-operations.md" style="color:black"><i class="fa fa-edit fa-fw"></i>Edit This Page</a> |
| |
| |
| |
| |
| </footer> |
| |
| |
| |
| <div class="book-comments"> |
| |
| </div> |
| |
| |
| |
| <label for="menu-control" class="hidden book-menu-overlay"></label> |
| </div> |
| |
| |
| <aside class="book-toc"> |
| |
| |
| |
| <nav id="TableOfContents"><h3>On This Page <button class="toc" onclick="collapseToc()"><i class="fa fa-compress" aria-hidden="true"></i></button></h3> |
| <ul> |
| <li><a href="#what-is-apache-flink--operations">What is Apache Flink? — Operations</a> |
| <ul> |
| <li><a href="#run-your-applications-non-stop-247">Run Your Applications Non-Stop 24/7</a></li> |
| <li><a href="#update-migrate-suspend-and-resume-your-applications">Update, Migrate, Suspend, and Resume Your Applications</a></li> |
| <li><a href="#monitor-and-control-your-applications">Monitor and Control Your Applications</a></li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| |
| |
| </aside> |
| <aside class="expand-toc"> |
| <button class="toc" onclick="expandToc()"> |
| <i class="fa fa-expand" aria-hidden="true"></i> |
| </button> |
| </aside> |
| |
| </main> |
| |
| |
| </body> |
| |
| </html> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
