| <!doctype html> |
| <html class="docs-version-1.0.2" lang="en" dir="ltr"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="generator" content="Docusaurus v2.0.0-beta.14"> |
| <link rel="alternate" type="application/rss+xml" href="/blog/rss.xml" title="Apache Linkis Blog RSS Feed"> |
| <link rel="alternate" type="application/atom+xml" href="/blog/atom.xml" title="Apache Linkis Blog Atom Feed"> |
| <link rel="search" type="application/opensearchdescription+xml" title="Apache Linkis" href="/opensearch.xml"><title data-react-helmet="true">Resource Manager | Apache Linkis</title><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" property="og:url" content="https://linkis.incubator.apache.org/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager"><meta data-react-helmet="true" name="docsearch:language" content="en"><meta data-react-helmet="true" name="docsearch:version" content="1.0.2"><meta data-react-helmet="true" name="docsearch:docusaurus_tag" content="docs-default-1.0.2"><meta data-react-helmet="true" property="og:title" content="Resource Manager | Apache Linkis"><meta data-react-helmet="true" name="description" content="1. Background"><meta data-react-helmet="true" property="og:description" content="1. Background"><link data-react-helmet="true" rel="shortcut icon" href="/img/favicon.ico"><link data-react-helmet="true" rel="canonical" href="https://linkis.incubator.apache.org/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager" hreflang="en"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/zh-CN/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager" hreflang="zh-CN"><link data-react-helmet="true" rel="alternate" href="https://linkis.incubator.apache.org/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager" hreflang="x-default"><link data-react-helmet="true" rel="preconnect" href="https://AE29KQB3IA-dsn.algolia.net" crossorigin="anonymous"><link rel="stylesheet" href="/assets/css/styles.92d87943.css"> |
| <link rel="preload" href="/assets/js/runtime~main.a4c60f50.js" as="script"> |
| <link rel="preload" href="/assets/js/main.42471b13.js" as="script"> |
| </head> |
| <body> |
| <script>!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}()</script><div id="__docusaurus"> |
| <div><a href="#" class="skipToContent_OuoZ">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--light_4Vu1 navbar__logo"><img src="/img/logo.png" alt="Apache Linkis Logo" class="themedImage_TMUO themedImage--dark_uzRr navbar__logo"><b class="navbar__title">Apache Linkis(Incubating)</b></a><a class="navbar__item navbar__link" href="/">Home</a><a class="navbar__item navbar__link" href="/faq/main">FAQ</a><a class="navbar__item navbar__link" href="/download/main">Download</a><a class="navbar__item navbar__link" href="/community/how-to-subscribe">Community</a><a class="navbar__item navbar__link" href="/blog">Blog</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/user">Users</a><div class="navbar__item dropdown dropdown--hoverable"><a class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a class="navbar__link">Doc</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/1.1.3/introduction">Next(1.1.3)</a></li><li><a class="dropdown__link" href="/docs/latest/introduction">1.1.2</a></li><li><a class="dropdown__link" href="/docs/1.1.1/introduction">1.1.1</a></li><li><a class="dropdown__link" href="/docs/1.1.0/introduction">1.1.0</a></li><li><a class="dropdown__link" href="/docs/1.0.3/introduction">1.0.3</a></li><li><a class="dropdown__link" href="/docs/1.0.2/introduction">1.0.2</a></li><li><a class="dropdown__link" href="/docs/0.11.0/introduction">0.11.0</a></li><li><a class="dropdown__link" href="/versions">All Version</a></li></ul></div><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" class="navbar__link"><span><svg t="1631348384596" class="iconLanguage_EbrZ" viewBox="0 0 1024 1024" version="1.1" p-id="557" width="20" height="20"><path d="M547.797333 638.208l-104.405333-103.168 1.237333-1.28a720.170667 720.170667 0 0 0 152.490667-268.373333h120.448V183.082667h-287.744V100.906667H347.605333v82.218666H59.818667V265.386667h459.178666a648.234667 648.234667 0 0 1-130.304 219.946666 643.242667 643.242667 0 0 1-94.976-137.728H211.541333a722.048 722.048 0 0 0 122.453334 187.434667l-209.194667 206.378667 58.368 58.368 205.525333-205.525334 127.872 127.829334 31.232-83.84m231.424-208.426667h-82.218666l-184.96 493.312h82.218666l46.037334-123.306667h195.242666l46.464 123.306667h82.218667l-185.002667-493.312m-107.690666 287.744l66.56-178.005333 66.602666 178.005333z" fill="currentColor" p-id="558"></path></svg><span>English</span></span></a><ul class="dropdown__menu"><li><a href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" style="text-transform:capitalize">English</a></li><li><a href="/zh-CN/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager" target="_self" rel="noopener noreferrer" class="dropdown__link" style="text-transform:capitalize">简体中文</a></li></ul></div><div class="searchBox_Bc3W"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper docs-wrapper docs-doc-page"><div class="docPage_lDyR"><button class="clean-btn backToTopButton_i9tI" type="button"><svg viewBox="0 0 24 24" width="28"><path d="M7.41 15.41L12 10.83l4.59 4.58L18 14l-6-6-6 6z" fill="currentColor"></path></svg></button><aside class="docSidebarContainer_0YBq"><div class="sidebar_a3j0"><nav class="menu thin-scrollbar menu_cyFh menuWithAnnouncementBar_+O1J"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/1.0.2/introduction">Introduction</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Deployment</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">User Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Engine Usage</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">API Docs</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#">Architecture</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/overview">Overview</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/difference_between_1.0_and_0.x">Difference Between 1.0 And 0.x</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/job_submission_preparation_and_execution_process">Job Submission</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/add_an_engine_conn">Add an EngineConn</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#" tabindex="0">Commons</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#" tabindex="0">Computation Governance Services</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/overview">Overview</a></li><li class="theme-doc-sidebar-item-category menu__list-item"><a class="menu__link menu__link--sublist menu__link--active" href="#" tabindex="0">Linkis Manager</a><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/overview">Overview</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/app_manager">App Manager</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/label_manager">Label Manager</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager">Resource Manager</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#" tabindex="0">Engine</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/entrance">Entrance Architecture Design</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" tabindex="0" href="/docs/1.0.2/architecture/computation_governance_services/linkis-cli">Linkis-Client Architecture Design</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#" tabindex="0">Public Enhancement Services</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#" tabindex="0">Microservice Governance Services</a></li></ul></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Development Doc</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Upgrade Guide</a></li><li class="theme-doc-sidebar-item-category menu__list-item menu__list-item--collapsed"><a class="menu__link menu__link--sublist" href="#">Tuning And Troubleshooting</a></li><li class="theme-doc-sidebar-item-link menu__list-item"><a class="menu__link" href="/docs/1.0.2/contact">Contact Us</a></li></ul></nav></div></aside><main class="docMainContainer_r8cw"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_zHA2"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is documentation for <!-- -->Apache Linkis<!-- --> <b>1.0.2</b>, which is no longer actively maintained.</div><div class="margin-top--md">For up-to-date documentation, see the <b><a href="/docs/latest/architecture/computation_governance_services/linkis_manager/resource_manager">latest version</a></b> (<!-- -->1.1.2<!-- -->).</div></div><div class="docItemContainer_oiyr"><article><span class="theme-doc-version-badge badge badge--secondary">Version: <!-- -->1.0.2</span><div class="tocCollapsible_aw-L theme-doc-toc-mobile tocMobile_Tx6Y"><button type="button" class="clean-btn tocCollapsibleButton_zr6a">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Resource Manager</h1></header><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="1-background"></a>1. Background<a class="hash-link" href="#1-background" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->ResourceManager (RM for short) is the computing resource management module of Linkis. All EngineConn (EC for short), EngineConnManager (ECM for short), and even external resources including Yarn are managed by RM. RM can manage resources based on users, ECM, or other granularities defined by complex tags. </p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="2-the-role-of-rm-in-linkis"></a>2. The role of RM in Linkis<a class="hash-link" href="#2-the-role-of-rm-in-linkis" title="Direct link to heading">#</a></h2><p><img alt="01" src="/assets/images/rm-01-3eb30eebca5162efc4a8b52af116dca9.png"> |
| <img alt="02" src="/assets/images/rm-02-16e4f450fde11be7226ea8029cac1b55.png"><br> |
| <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->As a part of Linkis Manager, RM mainly functions as follows: maintain the available resource information reported by ECM, process the resource application submitted by ECM, record the actual resource usage information reported by EC in real time during the life cycle after successful application, and provide query current resource usage The relevant interface of the situation.<br> |
| <!-- -->In Linkis, other services that interact with RM mainly include: </p><ol><li>Engine Manager, ECM for short: Processes the microservices that start the engine connector request. As a resource provider, ECM is responsible for registering and unregistering resources with RM. At the same time, as the manager of the engine, ECM is responsible for applying for resources from RM instead of the new engine connector that is about to start. For each ECM instance, there is a corresponding resource record in the RM, which contains information such as the total resources and protection resources it provides, and dynamically updates the used resources.<br><img alt="03" src="/assets/images/rm-03-88c5d2f15f1c504a47b5f255124ad493.png"> </li><li>The engine connector, referred to as EC, is the actual execution unit of user operations. At the same time, as the actual user of the resource, the EC is responsible for reporting the actual use of the resource to the RM. Each EC has a corresponding resource record in the RM: during the startup process, it is reflected as a locked resource; during the running process, it is reflected as a used resource; after being terminated, the resource record is subsequently deleted.<br><img alt="04" src="/assets/images/rm-04-f3788f8ef5c9d98c9fe3562794fff7ea.png"> </li></ol><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="3-resource-type-and-format"></a>3. Resource type and format<a class="hash-link" href="#3-resource-type-and-format" title="Direct link to heading">#</a></h2><p><img alt="05" src="/assets/images/rm-05-fcca9073b4eba3afa8e7bf22c7b096b5.png"><br> |
| <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->As shown in the figure above, all resource classes implement a top-level Resource interface, which defines the calculation and comparison methods that all resource classes need to support, and overloads the corresponding mathematical operators to enable resources to be Directly calculated and compared like numbers. </p><table><thead><tr><th>Operator</th><th>Correspondence Method</th><th>Operator</th><th>Correspondence Method</th></tr></thead><tbody><tr><td>+</td><td>add</td><td>></td><td>moreThan</td></tr><tr><td>-</td><td>minus</td><td>\<</td><td>lessThan</td></tr><tr><td>*</td><td>multiply</td><td>=</td><td>equals</td></tr><tr><td>/</td><td>divide</td><td>><!-- -->=</td><td>notLessThan</td></tr><tr><td>\<=</td><td>notMoreThan</td><td></td><td></td></tr></tbody></table><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The currently supported resource types are shown in the following table. All resources have corresponding json serialization and deserialization methods, which can be stored in json format and transmitted across the network: </p><table><thead><tr><th>Resource Type</th><th>Description</th></tr></thead><tbody><tr><td>MemoryResource</td><td>Memory Resource</td></tr><tr><td>CPUResource</td><td>CPU Resource</td></tr><tr><td>LoadResource</td><td>Both memory and CPU resources</td></tr><tr><td>YarnResource</td><td>Yarn queue resources (queue, queue memory, queue CPU, number of queue instances)</td></tr><tr><td>LoadInstanceResource</td><td>Server resources (memory, CPU, number of instances)</td></tr><tr><td>DriverAndYarnResource</td><td>Driver and executor resources (with server resources and Yarn queue resources at the same time)</td></tr><tr><td>SpecialResource</td><td>Other custom resources</td></tr></tbody></table><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="4-available-resource-management"></a>4. Available resource management<a class="hash-link" href="#4-available-resource-management" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The available resources in the RM mainly come from two sources: the available resources reported by the ECM, and the resource limits configured according to tags in the Configuration module.<br> |
| <strong>ECM resource report</strong>: </p><ol><li><p>When the ECM is started, it will broadcast the ECM registration message. After receiving the message, the RM will register the resource according to the content contained in the message. The resource-related content includes:</p><ol><li><p>Total resources: the total number of resources that the ECM can provide.</p></li><li><p>Protect resources: When the remaining resources are less than this resource, no further resources are allowed to be allocated.</p></li><li><p>Resource type: such as LoadResource, DriverAndYarnResource and other type names.</p></li><li><p>Instance information: machine name plus port name.</p></li></ol></li><li><p>After RM receives the resource registration request, it adds a record in the resource table, the content is consistent with the parameter information of the interface, and finds the label representing the ECM through the instance information, and adds an association in the resource and label association table recording.</p></li><li><p>When the ECM is closed, it will broadcast a message that the ECM is closed. After receiving the message, the RM will go offline according to the ECM instance information in the message, that is, delete the resource and associated records corresponding to the ECM instance tag. </p></li></ol><p><strong>Configuration module tag resource configuration</strong> |
| <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->In the Configuration module, users can configure the number of resources based on different tag combinations, such as limiting the maximum available resources of the User/Creator/EngineType combination.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The RM queries the Configuration module for resource information through the RPC message, using the combined tag as the query condition, and converts it into a Resource object to participate in subsequent comparison and recording. </p><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="5-resource-usage-management"></a>5. Resource Usage Management<a class="hash-link" href="#5-resource-usage-management" title="Direct link to heading">#</a></h2><p><strong>Receive user's resource application:</strong> </p><ol><li><p>When LinkisManager receives a request to start EngineConn, it will call RM's resource application interface to apply for resources. The resource application interface accepts an optional time parameter. When the waiting time for applying for a resource exceeds the limit of the time parameter, the resource application will be automatically processed as a failure.<br> |
| <strong>Judging whether there are enough resources:</strong><br> |
| <!-- -->That is, to determine whether the remaining available resources are greater than the requested resources, if greater than or equal to, the resources are sufficient; otherwise, the resources are insufficient.</p></li><li><p>RM preprocesses the label information attached to the resource application, and filters, combines and converts the original labels according to the rules (such as combining the User/Creator label and EngineType label), which makes the subsequent resource judgment more granular flexible.</p></li><li><p>Lock each converted label one by one, so that their corresponding resource records remain unchanged during the processing of resource applications.</p></li><li><p>According to each label:</p><ol><li><p>Query the corresponding resource record from the database through the Persistence module. If the record contains the remaining available resources, it is directly used for comparison.</p></li><li><p>If there is no direct remaining available resource record, it will be calculated by the formula of <!-- -->[Remaining Available Resource=Maximum Available Resource-Used Resource-Locked Resource-Protected Resource]<!-- -->.</p></li><li><p>If there is no maximum available resource record, request the Configuration module to see if there is configured resource information, if so, use the formula for calculation, if not, skip the resource judgment for this tag.</p></li><li><p>If there is no resource record, skip the resource judgment for this tag.</p></li></ol></li><li><p>As long as one tag is judged to be insufficient in resources, the resource application will fail, and each tag will be unlocked one by one.</p></li><li><p>Only when all tags are judged to be sufficient resources, can the resource application be successfully passed and proceed to the next step. </p></li></ol><p><strong>lock by application of resources:</strong></p><ol><li><p>The number of resource request by generating a new record in the resource table, and associated with each tag.</p></li><li><p>If there is a tag corresponding to the remaining available resource record, the corresponding number of the abatement.</p></li><li><p>Generate a timed task, the lock checks whether these resources are actually used after a certain time, if the timeout is not used, it is mandatory recycling.</p></li><li><p>unlock each tag.</p></li></ol><p><strong>report the actual use of resources:</strong></p><ol><li><p>EngineConn after the start, broadcast a resource utilization message. RM after receiving the message, check whether the label corresponding to the EngineConn lock resource record, and if not, an error.</p></li><li><p>If you have locked resource, the EngineConn all labels associated lock.</p></li><li><p>For each tag, the resource record corresponding lock record for the conversion of used resources.</p></li><li><p>Unlock all labels.</p></li></ol><p><strong>Release actual used resources:</strong></p><ol><li><p>EngineConn after the end of the life cycle, recycling broadcast messages. RM after receiving the message, check whether the EngineConn corresponding label resources have been recorded.</p></li><li><p>If so, all the labels associated EngineConn be locked.</p></li></ol><p>3, minus the amount used in the corresponding resource record for each label.</p><ol start="4"><li><p>If there is a tag corresponding to the remaining available resource record, the corresponding increase in number.</p></li><li><p>The unlocking each tag</p></li></ol><h2><a aria-hidden="true" tabindex="-1" class="anchor anchor__h2 anchorWithStickyNavbar_y2LR" id="6-external-resource-management"></a>6. External resource management<a class="hash-link" href="#6-external-resource-management" title="Direct link to heading">#</a></h2><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->In RM, in order to classify resources more flexibly and expansively, support multi-cluster resource management and control, and at the same time make it easier to access new external resources, the following considerations have been made in the design:</p><ol><li><p>Unified management of resources through tags. After the resource is registered, it is associated with the tag, so that the attributes of the resource can be expanded infinitely. At the same time, resource applications are also tagged to achieve flexible matching.</p></li><li><p>Abstract the cluster into one or more tags, and maintain the environmental information corresponding to each cluster tag in the external resource management module to achieve dynamic docking.</p></li><li><p>Abstract a general external resource management module. If you need to access new external resource types, you can convert different types of resource information into Resource entities in the RM as long as you implement a fixed interface to achieve unified management.<br> |
| <img alt="06" src="/assets/images/rm-06-20c8ef614561c8c23052d6e59ec4360a.png"> </p></li></ol><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->Other modules of RM obtain external resource information through the interface provided by ExternalResourceService.</p><p> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- --> <!-- -->The ExternalResourceService obtains information about external resources through resource types and tags:</p><ol><li><p>The type, label, configuration and other attributes of all external resources (such as cluster name, Yarn web |
| url, Hadoop version and other information) are maintained in the linkis<!-- -->_<!-- -->external<!-- -->_<!-- -->resource<!-- -->_<!-- -->provider table.</p></li><li><p>For each resource type, there is an implementation of the ExternalResourceProviderParser interface, which parses the attributes of external resources, converts the information that can be matched to the Label into the corresponding Label, and converts the information that can be used as a parameter to request the resource interface into params . Finally, an ExternalResourceProvider instance that can be used as a basis for querying external resource information is constructed.</p></li><li><p>According to the resource type and label information in the parameters of the ExternalResourceService method, find the matching ExternalResourceProvider, generate an ExternalResourceRequest based on the information in it, and formally call the API provided by the external resource to initiate a resource information request.</p></li></ol></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/incubator-linkis-website/edit/dev/versioned_docs/version-1.0.2/architecture/computation_governance_services/linkis_manager/resource_manager.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_mS5F" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_mt2f"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><div class="pagination-nav__item"><a class="pagination-nav__link" href="/docs/1.0.2/architecture/computation_governance_services/linkis_manager/label_manager"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">« <!-- -->Label Manager</div></a></div><div class="pagination-nav__item pagination-nav__item--next"><a class="pagination-nav__link" href="/docs/1.0.2/architecture/computation_governance_services/engine/engine_conn"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">EngineConn Design<!-- --> »</div></a></div></nav></div></div><div class="col col--3"><div class="tableOfContents_vrFS thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-background" class="table-of-contents__link">1. Background</a></li><li><a href="#2-the-role-of-rm-in-linkis" class="table-of-contents__link">2. The role of RM in Linkis</a></li><li><a href="#3-resource-type-and-format" class="table-of-contents__link">3. Resource type and format</a></li><li><a href="#4-available-resource-management" class="table-of-contents__link">4. Available resource management</a></li><li><a href="#5-resource-usage-management" class="table-of-contents__link">5. Resource Usage Management</a></li><li><a href="#6-external-resource-management" class="table-of-contents__link">6. External resource management</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/docs/latest/introduction">Document</a></li><li class="footer__item"><a class="footer__link-item" href="/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Releases<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Community</div><ul class="footer__items"><li class="footer__item"><a href="https://github.com/apache/incubator-linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>GitHub<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Issue Tracker<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://github.com/apache/incubator-linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Pull Requests<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div><div class="col footer__col"><div class="footer__title">Subscribe Mailing List</div><ul class="footer__items"><li class="footer__item"><a class="footer__link-item" href="/community/how-to-subscribe">How to Subscribe</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Subscribe Mail<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item"><span>Mail Archive<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_wgqa"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></span></a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div><img style="height:50px" alt="Apache Software Foundation" src="/img/incubator-logo.svg"><p style="color: #999999; padding: 0 20px 30px;font-weight:400;text-align:left">Apache Linkis is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p><p></p> |
| <p style="padding: 0 20px 30px;color: #999999;font-weight: 400;"> Copyright © 2022 The Apache Software Foundation. Licensed under the Apache License, Version 2.0. Apache Linkis, Apache Incubator, Apache, the Apache feather logo, the Apache Linkis logo and the Apache Incubator project logo are trademarks of The Apache Software Foundation.</p> |
| <div></div></div></div></div></div></footer></div> |
| <script src="/assets/js/runtime~main.a4c60f50.js"></script> |
| <script src="/assets/js/main.42471b13.js"></script> |
| </body> |
| </html> |
