| <!doctype html> |
| <html class="docs-version-current" lang="en" dir="ltr"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="generator" content="Docusaurus v2.0.0-beta.14"> |
| <link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache Linkis Blog RSS Feed"> |
| <link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="Apache Linkis Blog Atom Feed"> |
| <link rel="search" type="application/opensearchdescription+xml" title="Apache Linkis" href="/opensearch.xml"><title data-react-helmet="true">Involve Prometheus into Linkis | Apache Linkis</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://linkis.incubator.apache.org/docs/1.1.3/deployment/involve_prometheus_into_linkis"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:version" content="current"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="docs-default-current"><meta data-react-helmet="true" property="og:title" content="Involve Prometheus into Linkis | Apache Linkis"><meta data-react-helmet="true" name="description" content="This article describes how to enable Prometheus to monitor all running Linkis services."><meta data-react-helmet="true" property="og:description" content="This article describes how to enable Prometheus to monitor all running Linkis services."><link data-react-helmet="true" rel="shortcut icon" href="/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://linkis.incubator.apache.org/docs/1.1.3/deployment/involve_prometheus_into_linkis"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/1.1.3/deployment/involve_prometheus_into_linkis" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/zh-CN/docs/1.1.3/deployment/involve_prometheus_into_linkis" hreflang="zh-CN"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/1.1.3/deployment/involve_prometheus_into_linkis" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://AE29KQB3IA-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.92d87943.css"> |
| <link rel="preload" href="/assets/js/runtime~main.a4c60f50.js" as="script"> |
| <link rel="preload" href="/assets/js/main.42471b13.js" as="script"> |
| </head> |
| <body> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--light_4Vu1 navbar__logo"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--dark_uzRr navbar__logo"><b class="navbar__title">Apache Linkis(Incubating)</b></a><a class="navbar__item navbar__link" href="/">Home</a><a class="navbar__item navbar__link" href="/faq/main">FAQ</a><a class="navbar__item navbar__link" href="/download/main">Download</a><a class="navbar__item navbar__link" href="/community/how-to-subscribe">Community</a><a class="navbar__item navbar__link" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/user">Users</a><div class="navbar__item dropdown dropdown--hoverable"><a class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link">Doc</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/1.1.3/introduction">Next(1.1.3)</a></li><li><a class="dropdown__link" href="/docs/latest/introduction">1.1.2</a></li><li><a class="dropdown__link" href="/docs/1.1.1/introduction">1.1.1</a></li><li><a class="dropdown__link" href="/docs/1.1.0/introduction">1.1.0</a></li><li><a class="dropdown__link" href="/docs/1.0.3/introduction">1.0.3</a></li><li><a class="dropdown__link" href="/docs/1.0.2/introduction">1.0.2</a></li><li><a class="dropdown__link" href="/docs/0.11.0/introduction">0.11.0</a></li><li><a class="dropdown__link" href="/versions">All Version</a></li></ul></div><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link"><span><svg t="1631348384596" class="iconLanguage_EbrZ" viewBox="0 0 1024 1024" version="1.1" p-id="557" width="20" height="20"><path d="M547.797333 638.208l-104.405333-103.168 1.237333-1.28a720.170667 720.170667 0 0 0 152.490667-268.373333h120.448V183.082667h-287.744V100.906667H347.605333v82.218666H59.818667V265.386667h459.178666a648.234667 648.234667 0 0 1-130.304 219.946666 643.242667 643.242667 0 0 1-94.976-137.728H211.541333a722.048 722.048 0 0 0 122.453334 187.434667l-209.194667 206.378667 58.368 58.368 205.525333-205.525334 127.872 127.829334 31.232-83.84m231.424-208.426667h-82.218666l-184.96 493.312h82.218666l46.037334-123.306667h195.242666l46.464 123.306667h82.218667l-185.002667-493.312m-107.690666 287.744l66.56-178.005333 66.602666 178.005333z" fill="currentColor" p-id="558"></path></svg><span>English</span></span></a><ul class="dropdown__menu"><li><a href="/docs/1.1.3/deployment/involve_prometheus_into_linkis" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-transform:capitalize">English</a></li><li><a href="/zh-CN/docs/1.1.3/deployment/involve_prometheus_into_linkis" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-transform:capitalize">简体中文</a></li></ul></div><div class="searchBox_Bc3W"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper docs-wrapper docs-doc-page"><div class="docPage_lDyR"><button class="clean-btn backToTopButton_i9tI" type="button"><svg viewBox="0 0 24 24" width="28"><path d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z" fill="currentColor"></path></svg></button><aside class="docSidebarContainer_0YBq"><div class="sidebar_a3j0"><nav class="menu thin-scrollbar menu_cyFh menuWithAnnouncementBar_+O1J"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/1.1.3/introduction">Introduction</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/1.1.3/release">Version overview</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/1.1.3/release-notes-1.1.3">Release Notes 1.1.3-RC1</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#">Deployment</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/quick_deploy">Quick Deployment</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/cluster_deployment">Cluster Deployment</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/engine_conn_plugin_installation">EngineConnPlugin Installation</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/installation_hierarchical_structure">Installation Directory Structure</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/unpack_hierarchical_structure">installation package directory structure</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/sourcecode_hierarchical_structure">Source Code Directory Structure</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/web_install">Linkis Console Deployment</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/involve_skywalking_into_linkis">Involve SkyWaling into Linkis</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/start_metadatasource">DataSource</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/deploy_linkis_without_hdfs">Deploy Linkis without HDFS</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/linkis_scriptis_install">Installation and deployment of the tool scriptis</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/1.1.3/deployment/involve_prometheus_into_linkis">Involve Prometheus into Linkis</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.1.3/deployment/involve_knife4j_into_linkis">Involve Knife4j into Linkis</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">User Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Engine Usage</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">API Docs</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Table Structure</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Architecture</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Development Doc</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Upgrade Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Tuning And Troubleshooting</a></li></ul></nav></div></aside><main class="docMainContainer_r8cw"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_zHA2"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is unreleased documentation for <!-- -->Apache Linkis<!-- --> <b>Next(1.1.3)</b> version.</div><div class="margin-top--md">For up-to-date documentation, see the <b><a href="/docs/latest/introduction">latest version</a></b> (<!-- -->1.1.2<!-- -->).</div></div><div class="docItemContainer_oiyr"><article><span class="theme-doc-version-badge badge badge--secondary">Version: <!-- -->Next(1.1.3)</span><div class="tocCollapsible_aw-L theme-doc-toc-mobile tocMobile_Tx6Y"><button type="button" class="clean-btn tocCollapsibleButton_zr6a">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Involve Prometheus into Linkis</h1></header><p>This article describes how to enable Prometheus to monitor all running Linkis services.</p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="1-introduction-to-prometheus"></a>1. Introduction to Prometheus<a class="hash-link" href="#1-introduction-to-prometheus" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="11-what-is-prometheus"></a>1.1 What is Prometheus<a class="hash-link" href="#11-what-is-prometheus" title="Direct link to heading">#</a></h3><p>Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.</p><p>In the context of microservice, it provides the service discovery feature, enabling to find targets dynamically from service register center, like Eureka, Consul, etc, and pull the metrics from API endpoint over http protocol.</p><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="12-prometheus-architecture"></a>1.2 Prometheus Architecture<a class="hash-link" href="#12-prometheus-architecture" title="Direct link to heading">#</a></h3><p>This diagram illustrates the architecture of Prometheus and some of its ecosystem components:</p><p><img src="https://prometheus.io/assets/architecture.png"></p><p>Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.</p><p><img src="/assets/images/prometheus_architecture-b8e59c4fd7626cf1ec1fb8dfc261d7e6.jpg"></p><p>In the context of Linkis, we will use Eureka (Service Discover)SD in Prometheus to retrieve scrape targets using the Eureka REST API. And Prometheus will periodically check the REST endpoint and create a target for every app instance.</p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="2-how-to-enable-prometheus"></a>2. How to Enable Prometheus<a class="hash-link" href="#2-how-to-enable-prometheus" title="Direct link to heading">#</a></h2><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="21-enable-prometheus-when-installing-linkis"></a>2.1 Enable Prometheus when installing Linkis<a class="hash-link" href="#21-enable-prometheus-when-installing-linkis" title="Direct link to heading">#</a></h3><p>Modify the configuration item <code>PROMETHEUS_ENABLE</code> in linkis-env.sh of Linkis.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI bash"><pre tabindex="0" class="prism-code language-bash codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(189, 147, 249);font-style:italic">PROMETHEUS_ENABLE</span><span class="token operator">=</span><span class="token plain">true</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>After running the <code>install.sh</code>, it's expected to see the configuration related to <code>prometheus</code> is appended inside the following files:</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## application-linkis.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">eureka</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">metadata-map</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">prometheus.path</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> $</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">prometheus.path</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">$</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">prometheus.endpoint</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">management</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">endpoints</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">web</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">exposure</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">include</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> refresh</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">info</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">health</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">metrics</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">prometheus</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## application-eureka.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">eureka</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">metadata-map</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">prometheus.path</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> $</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">prometheus.path</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/actuator/prometheus</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">management</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">endpoints</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">web</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">exposure</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">include</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> refresh</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">info</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">health</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">metrics</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">prometheus</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## linkis.properties ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.prometheus.enable=true</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.server.user.restful.uri.pass.auth=/api/rest_j/v1/actuator/prometheus</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Then inside each computation engine, like spark, flink or hive, it's needed to add the same configuration <strong>manually</strong>.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## linkis-engineconn.properties ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.prometheus.enable=true</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.server.user.restful.uri.pass.auth=/api/rest_j/v1/actuator/prometheus</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="22-enable-prometheus-after-installation"></a>2.2 Enable Prometheus after installation<a class="hash-link" href="#22-enable-prometheus-after-installation" title="Direct link to heading">#</a></h3><p>Modify<code>${LINKIS_HOME}/conf/application-linkis.yml</code>, add <code>prometheus</code> as exposed endpoints.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## application-linkis.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">management</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">endpoints</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">web</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">exposure</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#Add prometheus</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">include</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> refresh</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">info</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">health</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">metrics</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">prometheus</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Modify<code>${LINKIS_HOME}/conf/application-eureka.yml</code>, add <code>prometheus</code> as exposed endpoints.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## application-eureka.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">management</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">endpoints</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">web</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">exposure</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)">#Add prometheus</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">include</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> refresh</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">info</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">health</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">metrics</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain">prometheus</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Modify<code>${LINKIS_HOME}/conf/linkis.properties</code>, remove the comment <code>#</code> before <code>prometheus.enable</code></p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## linkis.properties ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">wds.linkis.prometheus.enable=true</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">...</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><h3><a aria-hidden="true" tabindex="-1" class="anchor anchor__h3 anchorWithStickyNavbar_y2LR" id="23-start-linkis"></a>2.3 Start Linkis<a class="hash-link" href="#23-start-linkis" title="Direct link to heading">#</a></h3><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI bash"><pre tabindex="0" class="prism-code language-bash codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token plain">$ </span><span class="token function" style="color:rgb(80, 250, 123)">bash</span><span class="token plain"> linkis-start-all.sh</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>After start the services, it's expected to access the prometheus endpoint of each microservice in the Linkis, for example, http://linkishost:9103/api/rest_j/v1/actuator/prometheus.</p><div class="admonition admonition-caution alert alert--warning"><div class="admonition-heading"><h5><span class="admonition-icon"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>注意</h5></div><div class="admonition-content"><p>The prometheus endpoint of gateway/eureka don't include the prefix <code>api/rest_j/v1</code>, and the complete endpoint will be http://linkishost:9001/actuator/prometheus</p></div></div><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="3-demo-for-deploying-the-prometheus-alertmanager-and-grafana"></a>3. Demo for Deploying the Prometheus, Alertmanager and Grafana<a class="hash-link" href="#3-demo-for-deploying-the-prometheus-alertmanager-and-grafana" title="Direct link to heading">#</a></h2><p>Usually the monitoring setup for a cloud native application will be deployed on kubernetes with service discovery and high availability (e.g. using a kubernetes operator like Prometheus Operator). To quickly prototype dashboards and experiment with different metric type options (e.g. histogram vs gauge) you may need a similar setup locally. This sector explains how to setup locally a Prometheus/Alert Manager and Grafana monitoring stack with Docker Compose.</p><p>First, lets define a general component of the stack as follows:</p><ul><li><p>An Alert Manager container that exposes its UI at 9093 and read its configuration from alertmanager.conf</p></li><li><p>A Prometheus container that exposes its UI at 9090 and read its configuration from prometheus.yml and its list of alert rules from alert_rules.yml</p></li><li><p>A Grafana container that exposes its UI at 3000, with list of metrics sources defined in grafana_datasources.yml and configuration in grafana_config.ini</p></li><li><p>The following docker-compose.yml file summaries the configuration of all those components:</p></li></ul><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## docker-compose.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">version</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"3"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">networks</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">default</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">external</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token boolean important">true</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> my</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">network</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">services</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">prometheus</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> prom/prometheus</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">latest</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">container_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> prometheus</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./config/prometheus.yml</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/etc/prometheus/prometheus.yml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./config/alertrule.yml</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/etc/prometheus/alertrule.yml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./prometheus/prometheus_data</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/prometheus</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'--config.file=/etc/prometheus/prometheus.yml'</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">ports</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"9090:9090"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">alertmanager</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> prom/alertmanager</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">latest</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">container_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> alertmanager</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./config/alertmanager.yml</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/etc/alertmanager/alertmanager.yml</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">ports</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"9093:9093"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">grafana</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> grafana/grafana</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">latest</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">container_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> grafana</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">environment</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> GF_SECURITY_ADMIN_PASSWORD=123456</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> GF_USERS_ALLOW_SIGN_UP=false</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./grafana/provisioning/dashboards</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/etc/grafana/provisioning/dashboards</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./grafana/provisioning/datasources</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/etc/grafana/provisioning/datasources</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> ./grafana/grafana_data</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">/var/lib/grafana</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">ports</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"3000:3000"</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Second, to define some alerts based on metrics in Prometheus, you can group then into an alert_rules.yml, so you could validate those alerts are properly triggered in your local setup before configuring them in the production instance. |
| As an example, the following configration convers the usual metrics used to monitor Linkis services.</p><ul><li>a. Down instance</li><li>b. High Cpu for each JVM instance (>80%)</li><li>c. High Heap memory for each JVM instance (>80%)</li><li>d. High NonHeap memory for each JVM instance (>80%)</li><li>e. High Waiting thread for each JVM instance (100)</li></ul><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## alertrule.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">groups</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisAlert</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">rules</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">alert</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisNodeDown</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">expr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> last_over_time(up</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKISI.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application</span><span class="token tag" style="color:rgb(255, 121, 198)">!=</span><span class="token plain">"LINKIS</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">CG</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">ENGINECONN"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">1m</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain">)== 0</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">for</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 15s</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">severity</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> critical</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> Linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $labels.instance }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">summary</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"instance: {{ $labels.instance }} down"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">description</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"Linkis instance(s) is/are down in last 1m"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $value }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">alert</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisNodeCpuHigh</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">expr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> system_cpu_usage</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">= 0.8</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">for</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 1m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">severity</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> warning</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> Linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $labels.instance }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">summary</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"instance: {{ $labels.instance }} cpu overload"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">description</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"CPU usage is over 80% for over 1min"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $value }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">alert</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisNodeHeapMemoryHigh</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">expr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> sum(jvm_memory_used_bytes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> area="heap"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">) by(instance) </span><span class="token important">*100/sum(jvm_memory_max_bytes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> area="heap"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">) by(instance) </span><span class="token punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">= 50</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">for</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 1m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">severity</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> warning</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> Linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $labels.instance }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">summary</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"instance: {{ $labels.instance }} memory(heap) overload"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">description</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"Memory usage(heap) is over 80% for over 1min"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $value }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">alert</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisNodeNonHeapMemoryHigh</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">expr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> sum(jvm_memory_used_bytes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> area="nonheap"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">) by(instance) </span><span class="token important">*100/sum(jvm_memory_max_bytes</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> area="nonheap"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain">) by(instance) </span><span class="token punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">= 60</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">for</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 1m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">severity</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> warning</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> Linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $labels.instance }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">summary</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"instance: {{ $labels.instance }} memory(nonheap) overload"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">description</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"Memory usage(nonheap) is over 80% for over 1min"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $value }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">alert</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> LinkisWaitingThreadHigh</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">expr</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> jvm_threads_states_threads</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">job="linkis"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> application=~"LINKIS.</span><span class="token important">*"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">,</span><span class="token plain"> state="waiting"</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">= 100</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">for</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 1m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">severity</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> warning</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">service</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> Linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">instance</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $labels.instance }}"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">summary</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"instance: {{ $labels.instance }} waiting threads is high"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">description</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"waiting threads is over 100 for over 1min"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"{{ $value }}"</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p><strong>Note</strong>: Since once the service instance is shutdown, it will not be one of the target of Prometheus Eureka SD, and <code>up</code> metrics will not return any data after a short time. Thus we will collect if the <code>up=0</code> in the last one minute to determine whether the service is alive or not. </p><p>Third, and most importantly define Prometheus configuration in prometheus.yml file. This will defines:</p><ul><li>the global settings like scrapping interval and rules evaluation interval</li><li>the connection information to reach AlertManager and the rules to be evaluated</li><li>the connection information to application metrics endpoint. |
| This is an example configration file for Linkis:</li></ul><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## prometheus.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># my global config</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">global</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">scrape_interval</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 30s </span><span class="token comment" style="color:rgb(98, 114, 164)"># By default, scrape targets every 15 seconds.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">evaluation_interval</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 30s </span><span class="token comment" style="color:rgb(98, 114, 164)"># By default, scrape targets every 15 seconds.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">alerting</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">alertmanagers</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">static_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">targets</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token string" style="color:rgb(255, 121, 198)">'alertmanager:9093'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Load and evaluate rules in this file every 'evaluation_interval' seconds.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">rule_files</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">"alertrule.yml"</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># A scrape configuration containing exactly one endpoint to scrape:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Here it's Prometheus itself.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">scrape_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">job_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'prometheus'</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">static_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">targets</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token string" style="color:rgb(255, 121, 198)">'localhost:9090'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">job_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> linkis</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">eureka_sd_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)"># the endpoint of your eureka instance</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">server</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">linkis</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">host</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain">20303/eureka</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">relabel_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">source_labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">__meta_eureka_app_name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">target_label</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> application</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">source_labels</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token plain">__meta_eureka_app_instance_metadata_prometheus_path</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">action</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> replace</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">target_label</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> __metrics_path__</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">regex</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> (.+)</span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Forth, the following configuration defines how alerts will be sent to external webhook.</p><div class="codeBlockContainer_J+bg"><div class="codeBlockContent_csEI yaml"><pre tabindex="0" class="prism-code language-yaml codeBlock_rtdJ thin-scrollbar" style="color:#F8F8F2;background-color:#282A36"><code class="codeBlockLines_1zSZ"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">## alertmanager.yml ##</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">global</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">resolve_timeout</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 5m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">route</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">receiver</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'webhook'</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">group_by</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token string" style="color:rgb(255, 121, 198)">'alertname'</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)"># How long to wait to buffer alerts of the same group before sending a notification initially.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">group_wait</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 1m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)"># How long to wait before sending an alert that has been added to a group for which there has already been a notification.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">group_interval</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 5m</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token comment" style="color:rgb(98, 114, 164)"># How long to wait before re-sending a given alert that has already been sent in a notification.</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">repeat_interval</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> 12h</span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key atrule">receivers</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'webhook'</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">webhook_configs</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain"> </span><span class="token key atrule">send_resolved</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token boolean important">true</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token key atrule">url</span><span class="token punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token punctuation" style="color:rgb(248, 248, 242)">{</span><span class="token plain">your</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">webhook</span><span class="token punctuation" style="color:rgb(248, 248, 242)">-</span><span class="token plain">url</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token punctuation" style="color:rgb(248, 248, 242)">}</span><span class="token plain"></span></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"> |
| </span></span></code></pre><button type="button" aria-label="Copy code to clipboard" class="copyButton_M3SB clean-btn">Copy</button></div></div><p>Finally, after defining all the configuration file as well as the docker compose file we can start the monitoring stack with <code>docker-compose up</code></p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="4-result-display"></a>4. Result display<a class="hash-link" href="#4-result-display" title="Direct link to heading">#</a></h2><p>On Prometheus page, it's expected to see all the Linkis service instances as shown below: |
| <img src="/assets/images/prometheus_screenshot-fbd3f3d69fff7a50160471e7e893c1ae.jpg"></p><p>When the Grafana is accessible, you need to import the prometheus as datasource in Grafana, and import the dashboard template with id 11378, which is normally used for springboot service(2.1+). |
| Then you can view one living dashboard of Linkis there.</p><p><img src="/assets/images/grafana_screenshot-dbea39db333224610c66dcd0582ac709.jpg"></p><p>You can also try to integrate the Prometheus alter manager with your own webhook, where you can see if the alter message is fired.</p></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/incubator-linkis-website/edit/dev/docs/deployment/involve_prometheus_into_linkis.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/1.1.3/deployment/linkis_scriptis_install"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">« <!-- -->Installation and deployment of the tool scriptis</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/1.1.3/deployment/involve_knife4j_into_linkis"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Involve Knife4j into Linkis<!-- --> »</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-introduction-to-prometheus" class="table-of-contents__link">1. Introduction to Prometheus</a><ul><li><a href="#11-what-is-prometheus" class="table-of-contents__link">1.1 What is Prometheus</a></li><li><a href="#12-prometheus-architecture" class="table-of-contents__link">1.2 Prometheus Architecture</a></li></ul></li><li><a href="#2-how-to-enable-prometheus" class="table-of-contents__link">2. How to Enable Prometheus</a><ul><li><a href="#21-enable-prometheus-when-installing-linkis" class="table-of-contents__link">2.1 Enable Prometheus when installing Linkis</a></li><li><a href="#22-enable-prometheus-after-installation" class="table-of-contents__link">2.2 Enable Prometheus after installation</a></li><li><a href="#23-start-linkis" class="table-of-contents__link">2.3 Start Linkis</a></li></ul></li><li><a href="#3-demo-for-deploying-the-prometheus-alertmanager-and-grafana" class="table-of-contents__link">3. Demo for Deploying the Prometheus, Alertmanager and Grafana</a></li><li><a href="#4-result-display" class="table-of-contents__link">4. Result display</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/latest/introduction">Document</a></li><li class="footer__item"><a class="footer__link-item" href="/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Releases<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Issue Tracker<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Pull Requests<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Subscribe Mailing List</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/how-to-subscribe">How to Subscribe</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Subscribe Mail<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Mail Archive<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div><img style="height:50px" alt="Apache Software Foundation" src="/img/incubator-logo.svg"><p style="color: #999999; padding: 0 20px 30px;font-weight:400;text-align:left">Apache Linkis is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p><p></p> |
| <p style="padding: 0 20px 30px;color: #999999;font-weight: 400;"> Copyright © 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0. Apache Linkis, Apache Incubator, Apache, the Apache feather logo, the Apache Linkis logo and the Apache Incubator project logo are trademarks of The Apache Software Foundation.</p> |
| <div></div></div></div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.a4c60f50.js"></script> |
| <script src="/assets/js/main.42471b13.js"></script> |
| </body> |
| </html> |
