---
title: "Supported Data Sources"
description: >
  Data sources that DevLake supports
sidebar_position: 5
---

## Data Sources and Data Plugins

Apache DevLake(incubating) supports the following data sources. The data from each data source is collected with one or more plugins.

| Data Source                    | Domain(s)                                                  | Supported Versions                   | Config UI Availability | Triggered Plugins           | Collection Mode                                                |
| ------------------------------ | ---------------------------------------------------------- | ------------------------------------ | ---------------------- | --------------------------- | -------------------------------------------------------------- |
| GitHub (include GitHub Action) | Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud                                | Available              | `github`, `gitextractor`    | Full Refresh                                                   |
| GitLab (include GitLabCI)      | Source Code Management, Code Review, Issue Tracking, CI/CD | Cloud, Community Edition 13.x+       | Available              | `gitlab`, `gitextractor`    | Full Refresh, Incremental Sync(for `issues`,`MRs`)             |
| Gitee                          | Source Code Management, Code Review, Issue Tracking        | Cloud                                | Not Available          | `gitee`, `gitextractor`     | Full Refresh, Incremental Sync(for `issues`,`MRs`)             |
| BitBucket                      | Source Code Management, Code Review                        | Cloud                                | Not Available          | `bitbucket`, `gitextractor` | Full Refresh                                                   |
| Jira                           | Issue Tracking                                             | Cloud, Server 7.x+, Data Center 7.x+ | Available              | `jira`                      | Full Refresh, Incremental Sync(for `issues` and related)       |
| TAPD                           | Issue Tracking                                             | Cloud                                | Not Available          | `tapd`                      | Full Refresh, Incremental Sync(for `stories`, `bugs`, `tasks`) |
| Jenkins                        | CI/CD                                                      | 2.263.x+                             | Available              | `jenkins`                   | Full Refresh                                                   |
| Feishu                         | Calendar                                                   | Cloud                                | Not Available          | `feishu`                    | Full Refresh                                                   |
| AE                             | Source Code Management                                     |                                      | Not Available          | `ae`                        | Full Refresh                                                   |

## Data Collection Scope By Each Plugin

This table shows the entities collected by each plugin. Domain layer entities in this table are consistent with the entities [here](/DataModels/DevLakeDomainLayerSchema.md).
✅ : Collect by default.
💪 : Collect not by default. You need to add the corresponding subtasks to collect these entities in the [advanced mode](../UserManuals/ConfigUI/AdvancedMode.md).

| Domain Layer Entities                                                                       | ae  | dora | gitextractor | incoming webhook | github | gitlab | jenkins | jira | refdiff | tapd |
| ------------------------------------------------------------------------------------------- | --- | ---- | ------------ | ---------------- | ------ | ------ | ------- | ---- | ------- | ---- |
| [accounts](../DataModels/DevLakeDomainLayerSchema.md/#accounts)                             |     |      |              |                  | ✅     | ✅     |         | ✅   |         | ✅   |
| [board_issues](../DataModels/DevLakeDomainLayerSchema.md/#board_issues)                     |     |      |              |                  | ✅     | ✅     |         | ✅   |         | ✅   |
| [board_repos](../DataModels/DevLakeDomainLayerSchema.md/#board_repos)                       |     |      |              |                  | ✅     | ✅     |         | ✅   |         |      |
| [board_sprints](../DataModels/DevLakeDomainLayerSchema.md/#board_sprints)                   |     |      |              |                  | ✅     |        |         | ✅   |         | ✅   |
| [boards](../DataModels/DevLakeDomainLayerSchema.md/#boards)                                 |     |      |              |                  | ✅     | ✅     |         | ✅   |         | ✅   |
| [cicd_pipeline_commits](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipeline_commits)   |     | ✅   |              |                  | ✅     | ✅     | ✅      |      |         |      |
| [cicd_pipelines](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipelines)                 |     | ✅   |              |                  | ✅     | ✅     | ✅      |      |         |      |
| [cicd_scopes](../DataModels/DevLakeDomainLayerSchema.md/#cicd_scopes)                       |     | ✅   |              |                  | ✅     | ✅     | ✅      |      |         |      |
| [cicd_tasks](../DataModels/DevLakeDomainLayerSchema.md/#cicd_tasks)                         |     | ✅   |              | 💪               | ✅     | ✅     | ✅      |      |         |      |
| [commit_file_components](../DataModels/DevLakeDomainLayerSchema.md/#commit_file_components) |     |      | ✅           |                  |        |        |         |      |         |      |
| [commit_files](../DataModels/DevLakeDomainLayerSchema.md/#commit_files)                     |     |      | ✅           |                  |        |        |         |      |         |      |
| [commit_line_change](../DataModels/DevLakeDomainLayerSchema.md/#commit_line_change)         |     |      | ✅           |                  |        |        |         |      |         |      |
| [commit_parents](../DataModels/DevLakeDomainLayerSchema.md/#commit_parents)                 |     |      | ✅           |                  |        |        |         |      |         |      |
| [commits](../DataModels/DevLakeDomainLayerSchema.md/#commits)                               | ✅  |      | ✅           |                  | 💪     | 💪     |         |      |         |      |
| [commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#commits_diffs)                   |     |      |              |                  |        |        |         |      | ✅      |      |
| [components](../DataModels/DevLakeDomainLayerSchema.md/#components)                         |     |      |              |                  |        |        |         |      |         |      |
| [finished_commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#finished_commits_diffs) |     |      |              |                  |        |        |         |      |         |      |
| [issue_changelogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_changelogs)             |     |      |              |                  |        |        |         | ✅   |         | ✅   |
| [issue_comments](../DataModels/DevLakeDomainLayerSchema.md/#issue_commentswip)              |     |      |              |                  | ✅     |        |         |      |         | ✅   |
| [issue_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_commits)                   |     |      |              |                  |        |        |         | ✅   |         | ✅   |
| [issue_labels](../DataModels/DevLakeDomainLayerSchema.md/#issue_labels)                     |     |      |              |                  | ✅     | ✅     |         |      |         | ✅   |
| [issue_repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_repo_commits)         |     |      |              |                  |        |        |         | ✅   |         |      |
| [issue_worklogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_worklogs)                 |     |      |              |                  |        |        |         | ✅   |         | ✅   |
| [issues](../DataModels/DevLakeDomainLayerSchema.md/#issues)                                 |     |      |              |                  | ✅     |        |         | ✅   |         | ✅   |
| [project_issue_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_issue_metrics)   |     | ✅   |              |                  | ✅     | ✅     |         | ✅   |         | ✅   |
| [project_mapping](../DataModels/DevLakeDomainLayerSchema.md/#project_mapping)               |     | ✅   |              |                  | ✅     | ✅     | ✅      | ✅   |         | ✅   |
| [project_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_metrics)               |     | ✅   |              |                  | ✅     | ✅     | ✅      | ✅   |         | ✅   |
| [project_pr_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_pr_metrics)         |     | ✅   |              |                  | ✅     | ✅     |         |      |         | ✅   |
| [projects](../DataModels/DevLakeDomainLayerSchema.md/#project)                              |     | ✅   |              |                  | ✅     | ✅     | ✅      | ✅   |         | ✅   |
| [pull_request_comments](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_comments)   |     |      |              |                  | ✅     | ✅     |         |      |         |      |
| [pull_request_commits](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_commits)     |     |      |              |                  | ✅     | ✅     |         |      |         |      |
| [pull_request_issues](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_issues)       |     |      |              |                  | ✅     |        |         |      |         |      |
| [pull_request_labels](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_labels)       |     |      |              |                  | ✅     | ✅     |         |      |         |      |
| [pull_requests](../DataModels/DevLakeDomainLayerSchema.md/#pull_requests)                   |     |      |              |                  | ✅     | ✅     |         |      |         |      |
| [ref_commits](../DataModels/DevLakeDomainLayerSchema.md/#ref_commits)                       |     |      |              |                  |        |        |         |      | ✅      |      |
| [refs](../DataModels/DevLakeDomainLayerSchema.md/#refs)                                     |     |      | ✅           |                  |        |        |         |      | ✅      |      |
| [refs_issues_diffs](../DataModels/DevLakeDomainLayerSchema.md/#refs_issues_diffs)           |     |      |              |                  |        |        |         |      | ✅      |      |
| [ref_pr_cherry_picks](../DataModels/DevLakeDomainLayerSchema.md/#ref_pr_cherry_picks)       |     |      |              |                  |        |        |         |      | ✅      |      |
| [repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#repo_commits)                     |     |      | ✅           |                  | 💪     | 💪     |         |      |         |      |
| [repo_snapshot](../DataModels/DevLakeDomainLayerSchema.md/#repo_snapshot)                   |     |      | ✅           |                  |        |        |         |      |         |      |
| [repos](../DataModels/DevLakeDomainLayerSchema.md/#repos)                                   |     |      |              |                  | ✅     | ✅     |         |      |         |      |
| [sprint_issues](../DataModels/DevLakeDomainLayerSchema.md/#sprint_issues)                   |     |      |              |                  | ✅     |        |         | ✅   |         | ✅   |
| [sprints](../DataModels/DevLakeDomainLayerSchema.md/#sprints)                               |     |      |              |                  | ✅     |        |         | ✅   |         | ✅   |
| [team_users](../DataModels/DevLakeDomainLayerSchema.md/#team_users)                         |     |      |              |                  |        |        |         |      |         |      |
| [teams](../DataModels/DevLakeDomainLayerSchema.md/#teams)                                   |     |      |              |                  |        |        |         |      |         |      |
| [user_account](../DataModels/DevLakeDomainLayerSchema.md/#user_accounts)                    |     |      |              |                  |        |        |         |      |         |      |
| [users](../DataModels/DevLakeDomainLayerSchema.md/#users)                                   |     |      |              |                  |        |        |         | ✅   |         | ✅   |

## Data Sync Policy

**bold:** means it may collect slowly.

**\*bold\*:** means it may collect very slowly.

### Jira

| Subtask Name               | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| -------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
| CollectStatusMeta          | 1                               | -                                       | -                            |
| CollectProjectsMeta        | <10                             | ❌                                      | -                            |
| CollectIssueTypesMeta      | <10                             | ❌                                      | -                            |
| CollectIssuesMeta          | <10^4                           | ✅                                      | ✅                           |
| CollectIssueChangelogsMeta | 1000~10^5                       | ✅                                      | ✅                           |
| CollectAccountsMeta        | <10^3                           | ❌                                      | ❌                           |
| CollectWorklogsMeta        | 1000~10^5                       | ✅                                      | ✅                           |
| CollectRemotelinksMeta     | 1000~10^5                       | ✅                                      | ✅                           |
| CollectSprintsMeta         | <100                            | ❌                                      | ❌                           |
| CollectEpicsMeta           | <100                            | ❌                                      | ✅                           |

### Jenkins

| Subtask Name         | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| -------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
| CollectApiBuildsMeta | ≈100                            | ❌                                      | ❌                           |
| CollectApiStagesMeta | ≈10^4                           | ❌                                      | ✅                           |

### Gitlab

| Subtask Name                | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| --------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
| CollectApiIssuesMeta        | <10^4                           | ✅                                      | ✅                           |
| CollectApiMergeRequestsMeta | <10^3                           | ✅                                      | ✅                           |
| CollectApiMrNotesMeta       | <10^5                           | ❌                                      | ✅                           |
| CollectApiMrCommitsMeta     | <10^5                           | ❌                                      | ✅                           |
| **CollectApiPipelinesMeta** | <10^4                           | ✅                                      | ❌                           |
| CollectApiJobsMeta          | <10^5                           | ❌                                      | ✅                           |

### Github

| Subtask Name                       | Estimated Max Number of Request     | Does It support Incremental Collection? | Does It Support Time Filter? |
| ---------------------------------- | ----------------------------------- | --------------------------------------- | ---------------------------- |
| ---------------------------------  | Common                              | -----------------------                 |                              |
| CollectMilestonesMeta              | ≈10                                 | ✅                                       | ❌                            |
| CollectRunsMeta                    | <10^4                               | ✅                                       | ✅                            |
| CollectApiCommentsMeta             | 400 (max page that GitHub supports) | ✅                                       | ✅                            |
| **CollectApiEventsMeta**           | 400 (max page that GitHub supports) | ❌                                       | ❌                            |
| CollectApiPullRequestReviewsMeta   | <10^5                               | ✅                                       | ✅                            |
| ---------------------------------  | Graphql Only (Default)              | -----------------------                 |                              |
| CollectIssueMeta                   | ≈10^4                               | ❌                                       | ✅                            |
| CollectPrMeta                      | ≈10^3                               | ❌                                       | ✅                            |
| CollectCheckRunMeta                | <10^4                               | ❌                                       | ✅                            |
| CollectAccountMeta                 | ≈10^2                               | ❌                                       | -                            |
| ---------------------------------  | Restful Only (Not by Default)       | -----------------------                 |                              |
| CollectApiIssuesMeta               | ≈10^4                               | ✅                                       | ❌                            |
| CollectApiPullRequestsMeta         | ≈10^2                               | ❌                                       | ❌                            |
| CollectApiPullRequestCommitsMeta   | ≈10^4                               | ✅                                       | ✅                            |
| **CollectApiPrReviewCommentsMeta** | ≈10^4                               | ✅                                       | ✅                            |
| **CollectAccountsMeta**            | ≈10^4                               | ❌                                       | ❌                            |
| **CollectAccountOrgMeta**          | ≈10^4                               | ❌                                       | ❌                            |
| CollectJobsMeta                    | <10^6                               | ❌                                       | ✅                            |
| CollectApiCommitsMeta              | Not enabled                         | -                                       | -                            |
| CollectApiCommitStatsMeta          | Not enabled                         | -                                       | -                            |

### Feishu

| Subtask Name                  | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| ----------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
| CollectMeetingTopUserItemMeta | ≈10^3                           | ❌                                      | ✅                           |

### Bitbucket

| Subtask Name                        | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| ----------------------------------- | ------------------------------- | --------------------------------------- | ---------------------------- |
| ~~CollectApiRepoMeta~~              | 1                               | ❌                                      | ❌                           |
| CollectApiPullRequestsMeta          | ≈10^3                           | ❌                                      | ❌                           |
| **CollectApiIssuesMeta**            | ≈10^4                           | ❌                                      | ❌                           |
| **CollectApiPrCommentsMeta**        | ≈10^5                           | ❌                                      | ❌                           |
| **\*CollectApiIssueCommentsMeta\*** | ≈10^6                           | ❌                                      | ❌                           |
| **CollectApiPipelinesMeta**         | <10^4                           | ❌                                      | ❌                           |
| CollectApiDeploymentsMeta           | <10^2                           | ❌                                      | ❌                           |

### Gitee

| Subtask Name                         | Estimated Max Number of Request | Does It support Incremental Collection? | Does It Support Time Filter? |
| ------------------------------------ | ------------------------------- | --------------------------------------- | ---------------------------- |
| ~~CollectApiRepoMeta~~               | 1                               | ❌                                      | ❌                           |
| CollectApiPullRequestsMeta           | ≈10^3                           | ✅                                      | ❌                           |
| **CollectApiIssuesMeta**             | ≈10^4                           | ✅                                      | ❌                           |
| **CollectCommitsMeta?**              | ≈10^4                           | ✅                                      | ❌                           |
| **CollectApiPrCommentsMeta**         | ≈10^5                           | ❌                                      | ❌                           |
| **\*CollectApiIssueCommentsMeta\***  | ≈10^6                           | ✅                                      | ❌                           |
| **CollectApiPullRequestCommitsMeta** | ≈10^5                           | ❌                                      | ❌                           |
| **CollectApiPullRequestReviewsMeta** | ≈10^5                           | ❌                                      | ❌                           |
| **\*CollectApiCommitStatsMeta\***    | ≈10^6 (Not enable)              | ❌                                      | ❌                           |