| <div role="region"><a href="#" class="skipToContent_fXgn">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Navigation bar toggle" class="navbar__toggle clean-btn" type="button" tabindex="0"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo.svg" alt="apache devlake" class="themedImage_ToTc themedImage--light_HNdA"><img src="/img/logo.svg" alt="apache devlake" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate">Apache DevLake</b></a></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Docs</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/Overview/Introduction">Next</a></li><li><a class="dropdown__link" href="/docs/v1.0/Overview/Introduction">v1.0 (Stable)</a></li><li><a class="dropdown__link" href="/docs/v0.21/Overview/Introduction">v0.21</a></li><li><a class="dropdown__link" href="/docs/v0.20/Overview/Introduction">v0.20</a></li><li><a class="dropdown__link" href="/docs/v0.19/Overview/Introduction">v0.19</a></li><li><a class="dropdown__link" href="/docs/v0.18/Overview/Introduction">v0.18</a></li><li><a class="dropdown__link" href="/docs/v0.17/Overview/Introduction">v0.17</a></li><li><a class="dropdown__link" href="/docs/v0.16/Overview/Introduction">v0.16</a></li><li><a class="dropdown__link" href="/docs/v0.15/Overview/Introduction">v0.15</a></li></ul></div><a class="navbar__item navbar__link" href="/livedemo/EngineeringLeads/DORA">Use Cases</a><a class="navbar__item navbar__link" href="/community/">Community</a><a class="navbar__item navbar__link" href="/team">Team</a><a class="navbar__item navbar__link" href="/blogOverview">Blog</a><a href="https://github.com/apache/devlake" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">GitHub</a><a class="navbar__item navbar__link" href="/download">Download</a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">ASF</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Foundation</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">License</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">Events</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">Security</a></li><li><a href="https://privacy.apache.org/policies/privacy-policy-public.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Sponsorship</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Thanks</a></li></ul></div><div class="toggle_vylO colorModeToggle_DEke"><button class="clean-btn toggleButton_gllP toggleButtonDisabled_aARS" type="button" disabled="" title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)"><svg viewBox="0 0 24 24" width="24" height="24" class="lightToggleIcon_pyhR"><path fill="currentColor" d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"></path></svg><svg viewBox="0 0 24 24" width="24" height="24" class="darkToggleIcon_wfgR"><path fill="currentColor" d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"></path></svg></button></div><div class="searchBox_ZlJk"><div class="navbar__search searchBarContainer_NW3z"><input placeholder="Search" aria-label="Search" class="navbar__search-input"><div class="loadingRing_RJI3 searchBarLoadingRing_YnHq"><div></div><div></div><div></div><div></div></div></div></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebar_njMd"><nav class="menu thin-scrollbar menu_SIkG"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--active" aria-expanded="true" href="/docs/Overview">Overview</a><button aria-label="Toggle the collapsible sidebar category 'Overview'" type="button" class="clean-btn menu__caret"></button></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/Overview/Introduction">Introduction</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/Overview/Architecture">Architecture</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/Overview/KeyConcepts">Key Concepts</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/Overview/SupportedDataSources">Supported Data Sources</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/Overview/References">API References</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/Overview/Roadmap">Roadmap</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/GettingStarted">Getting Started</a><button aria-label="Toggle the collapsible sidebar category 'Getting Started'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/Config UI">Config UI</a><button aria-label="Toggle the collapsible sidebar category 'Config UI'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/DORA">DORA</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/Metrics">Metrics</a><button aria-label="Toggle the collapsible sidebar category 'Metrics'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/DataModels">Data Models</a><button aria-label="Toggle the collapsible sidebar category 'Data Models'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/DeveloperManuals">Developer Manuals</a><button aria-label="Toggle the collapsible sidebar category 'Developer Manuals'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/Plugins">Plugins</a><button aria-label="Toggle the collapsible sidebar category 'Plugins'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/Troubleshooting">Troubleshooting</a><button aria-label="Toggle the collapsible sidebar category 'Troubleshooting'" type="button" class="clean-btn menu__caret"></button></div></li></ul></nav></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_OVgt"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item"><a class="breadcrumbs__link" itemprop="item" href="/docs/Overview"><span itemprop="name">Overview</span></a><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Key Concepts</span><meta itemprop="position" content="2"></li></ul></nav><span class="theme-doc-version-badge badge badge--secondary">Version: Next</span><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Key Concepts</h1></header><p><em>Last updated: Nov 9, 2023</em></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="in-config-ui">In Config UI<a class="hash-link" href="#in-config-ui" title="Direct link to heading"></a></h2><p>The following terms are arranged in the order of their appearance in the actual user workflow in the config UI.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="data-source">Data Source<a class="hash-link" href="#data-source" title="Direct link to heading"></a></h3><p><strong>A data source is a specific DevOps tool from which you wish to sync your data, such as GitHub, GitLab, Jira and Jenkins.</strong></p><p>Typically, DevLake uses one <a href="#data-plugins">data plugin</a> to pull data for a single data source. For example, the <a href="/docs/Plugins/jira">jira</a> plugin is used to fetch data from Jira.</p><p>However, there are cases where DevLake may use multiple data plugins for a single data source. This approach is employed to enhance the synchronization speed and provide other advantages. For instance, when retrieving data from GitHub or GitLab, aside from the <a href="/docs/Plugins/github">github</a> and <a href="/docs/Plugins/gitlab">gitlab</a> plugins, the <a href="/docs/Plugins/gitextractor">gitextractor</a> is also used to fetch data. In these cases, DevLake still recognizes GitHub or GitLab as a single data source.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="data-connection">Data Connection<a class="hash-link" href="#data-connection" title="Direct link to heading"></a></h3><p><strong>A data connection is a specific instance of a <a href="#data-source">data source</a>.</strong> It stores the necessary access information, such as the endpoint URL and authentication token, to establish a connection to that data source.</p><p>A single data source can have one or more data connections associated with it. This allows you to connect to and retrieve data from different instances or installations of the same data source.</p><p>To set up a new data connection, it is recommended to use the 'Data Connections' page in DevLake. This page provides a convenient interface for adding and configuring data connections. Once a data connection is set up, you can later associate it with a DevLake project.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="data-scope">Data Scope<a class="hash-link" href="#data-scope" title="Direct link to heading"></a></h3><p><strong>A data scope is the top-level 'container' in a data source</strong>. For example, a data scope for Jira is a Jira board, for TAPD is a TAPD workspace, for GitHub/GitLab/BitBucket is a repo, for Jenkins is a Jenkins job, etc.</p><p>You can add multiple data scopes to a data connection to determine which data to collect. Data scopes vary for different data sources.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="scope-config">Scope Config<a class="hash-link" href="#scope-config" title="Direct link to heading"></a></h3><p><strong>A scope config refers to the configuration of a data scope.</strong> It defines the specific data entities to be collected and the transformations to be applied to that data.</p><p>Each data scope can have at most one scope config associated with it; while a scope config can be shared among multiple data scopes under the same data connection.</p><p>A scope config consists of two parts: <a href="#data-entities">Data Entities</a> and <a href="#transformations">Transformations</a>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="data-entities">Data Entities<a class="hash-link" href="#data-entities" title="Direct link to heading"></a></h4><p>Data entities refer to the specific data fields that are collected from different data domains. Check the <a href="/docs/Overview/SupportedDataSources#data-collection-scope-by-each-plugin">supported data entities</a> of each data source.</p><p>Data entities are categorized into <a href="/docs/DataModels/DevLakeDomainLayerSchema#data-models">six data domains</a> in DevLake: Issue Tracking, Source Code Management, Code Review, CI/CD, Code Quality, and Cross-Domain.</p><p>When setting up the scope config of a GitHub data connection, you have the flexibility to choose which specific data entities you want to collect. if you only want to collect 'repos', 'commits', and 'pull requests' while excluding 'issues' and 'workflow runs', you need to check the 'Source Code Management' and 'Code Review' domains, and uncheck 'Issue Tracking' or 'CI/CD' domains.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="transformations">Transformations<a class="hash-link" href="#transformations" title="Direct link to heading"></a></h4><p>Transformations are configurations for users to customize how DevLake transforms raw API responses to the domain layer data.</p><p>Although configuring transformation rules is not mandatory, certain pre-built dashboards, such as <a href="/livedemo/EngineeringLeads/DORA">DORA</a> and <a href="/livedemo/EngineeringLeads/WeeklyBugRetro">Weekly Bug Retro</a> require them to display the metrics accurately. If you leave the rules blank or have not configured them correctly, only a few <a href="/livedemo/DataSources/GitHub">data source dashboards</a> will be displayed as expected.</p><p>You can find the required transformations in the 'Dashboard Introduction' panel in each pre-built dashboard.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="project">Project<a class="hash-link" href="#project" title="Direct link to heading"></a></h3><p><strong>On a high level, a DevLake project can be viewed as a real-world project or product line.</strong> It represents a specific initiative or endeavor within the software development domain.</p><p><strong>On a lower level, a DevLake project is a way of organizing and grouping data from different domains.</strong> DevLake uses various <a href="#data-scope">data scopes</a>, such as repos, boards, cicd_scopes, and cq_projects as the 'container' to associate different types of data to a specific project.</p><ul><li>A project has a <a href="#Bluepirnts">blueprint</a> for data collection and metric computation.</li><li>DevLake measures DORA metrics at the project level. Each project has a set of DORA metrics. For example, if a user associates 'Jenkins Job A' and 'Jira board B' with project M, only the 'deployments' from 'Jenkins Job A' and the 'incidents' from 'Jira board B' will be considered when calculating the Change Failure Rate metric for project M. |
| <img loading="lazy" src="/assets/images/project_pipeline-06b394bb2f3453ffd888f20affbe2d5d.png" width="2594" height="1810" class="img_ev3q"></li></ul><h3 class="anchor anchorWithStickyNavbar_LWe7" id="blueprint">Blueprint<a class="hash-link" href="#blueprint" title="Direct link to heading"></a></h3><p><strong>A blueprint serves as the plan to synchronize data from data sources into the DevLake platform.</strong> Creating a blueprint consists of four steps:</p><ol><li>Adding <a href="#data-connections">data connections</a>: You can add one or more data connections to a blueprint, depending on the data sources you want to sync with DevLake. Each data connection represents a specific data source, such as GitHub or Jira.</li><li>Setting up the <a href="#data-scope">data scope</a>: When adding a data connection, you can choose to collect all or part of the configured data scopes of the data connection.</li><li>Setting up the sync policy: You can specify the sync frequency and the time range for data collection.</li></ol><p>The relationship between 'Blueprint', 'Project' and 'Data Connection' is explained as follows:</p><p><img loading="lazy" alt="Blueprint ERD" src="/assets/images/blueprint-erd-762692dfe07100fd9c045dd22ec90954.svg" width="1352" height="667" class="img_ev3q"></p><ul><li>A blueprint will be automatically created by along the creation with a DevLake project.</li><li>Each blueprint can have multiple data connections.</li><li>Each data connection can have multiple data scopes.</li><li>Each set of data scope only consists of one GitHub/GitLab project or Jira board, along with their corresponding data entities.</li><li>Each set of data scope can only have one set of scope config.</li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="apis-and-config-ui-advanced-mode">APIs and Config UI Advanced Mode<a class="hash-link" href="#apis-and-config-ui-advanced-mode" title="Direct link to heading"></a></h2><p>Typically, the following terms do not appear in the regular mode of the Config UI, but can be very useful if you use <a href="/docs/Overview/References">DevLake's APIs</a> or the advanced mode of Config UI.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="data-plugins">Data Plugins<a class="hash-link" href="#data-plugins" title="Direct link to heading"></a></h3><p><strong>A data plugin is a specific module that syncs or transforms data.</strong> There are two types of data plugins: Data Collection Plugins and Data Transformation Plugins.</p><p>Data Collection Plugins pull data from one or more data sources. DevLake supports 8 data plugins in this category: <code>ae</code>, <code>feishu</code>, <code>gitextractor</code>, <code>github</code>, <code>gitlab</code>, <code>jenkins</code>, <code>jira</code> and <code>tapd</code>.</p><p>Data Transformation Plugins transform the data pulled by other Data Collection Plugins. <code>refdiff</code> is currently the only plugin in this category.</p><p>Although the names of the data plugins are not displayed in the regular mode of DevLake Configuration UI, they can be used directly in JSON in the Advanced Mode.</p><p>For detailed information about the relationship between data sources and data plugins, please refer to <a href="/docs/Overview/SupportedDataSources">Supported Data Sources</a>.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="pipelines">Pipelines<a class="hash-link" href="#pipelines" title="Direct link to heading"></a></h3><p><strong>A pipeline is an orchestration of <a href="#tasks">tasks</a> of data <code>collection</code>, <code>extraction</code>, <code>conversion</code> and <code>enrichment</code>, defined in the DevLake API.</strong> A pipeline is composed of one or multiple <a href="#stages">stages</a> that are executed in a sequential order. Any error occurring during the execution of any stage, task or subtask will cause the immediate fail of the pipeline.</p><p>The composition of a pipeline is explained as follows: |
