feat(github): collect GitHub issue fields and map them onto domain issues (#9038)
* feat(github): collect GitHub issue fields and map them onto domain issues
GitHub issue fields are organization-level structured issue metadata that
went generally available on 2026-07-02. They are typed, mutually exclusive
within a field, and shared across every repository in the organization —
which is what teams currently approximate with `type:`-style labels.
Collects issue field values and lets a scope config map a field onto an
issue column, where it takes precedence over the existing label regexes.
- New table `_tool_github_issue_field_values`, one row per issue per field,
carrying a queryable text form of the value alongside the original JSON.
- New subtasks Collect/Extract Issue Field Values, both disabled by default
so existing pipelines are unaffected until a mapping is configured.
- New scope config keys issueFieldPriority, issueFieldSeverity,
issueFieldComponent, issueFieldStoryPoint and issueFieldDueDate, each
holding a field *name*.
The mapping is applied in the issue convertor rather than written back into
`_tool_github_issues`: the collector iterates that table to build its request
URLs, so a subtask that both read and wrote it was a cycle in the subtask
graph. Converting instead also keeps the tool layer as raw GitHub truth and
avoids adding columns there.
A 404 from the field-values endpoint is treated as "no field values" so an
organization that has never configured issue fields, or a token that cannot
see them, does not fail the whole task.
* test(github): add an e2e dataflow test for issue field values
Covers extraction and the scope config mapping end to end against a real
database, which the unit tests could not reach.
Extraction asserts the value normalisation per data type: a single_select
resolving to its option name and colour, an integral number rendering as
"5" rather than "5.0", a fractional number keeping its precision, a
multi_select joining option names while keeping the raw JSON array, and a
null value producing an empty value.
Conversion asserts the mapping reaches the domain issue -- priority,
component, story point and due date -- and, for the case that matters,
that an unparseable value is skipped with a warning rather than failing
the task or writing a wrong value: issue #7 carries "soon" in a date
field and a null priority, and comes out with neither set while the other
issues are untouched.
* fix(github): register the issue field values table in GetTablesInfo
Test_GetPluginTablesInfo compares the plugin's declared tables against the
ones its migrations create, and the new table was missing from the list:
table_info_test.go:121: The following tables are not returned by the
TablesInfo method
_tool_github_issue_field_values
Adds GithubIssueFieldValue to Github.GetTablesInfo(). Verified inside the
mericodev/lake-builder image the unit-test job uses, since the plugins
package needs libgit2 to build.Apache DevLake is an open-source dev data platform that ingests, analyzes, and visualizes the fragmented data from DevOps tools to extract insights for engineering excellence, developer experience, and community growth.
Apache DevLake is used by Engineering Leads, Open Source Software Maintainers and development teams looking to make better sense of their development process and to bring a more data-driven approach to their own practices. You can ask Apache DevLake many questions regarding your development process. Just connect and query.
The main way you interact with DevLake is through the integrated dashboards powered by Grafana.
Dashboards for Engineering Leads
Dashboards for OSS Maintainers
DevLake supports connections to many popular development tools, including GitHub, GitLab, Jenkins, Jira, Sonarqube and more. Here you can find all data sources supported by DevLake, their scopes, supported versions and more!
You can set up Apache DevLake by following our step-by-step instructions for either Docker Compose or Helm. Feel free to ask the community if you get stuck at any point.
Please see detailed usage instructions. Here's an overview on how to get started using DevLake.
Install using either Docker Compose or Helm.
The DevLake Configuration UI will guide you through the process (a Blueprint) to define the data connections, data scope, transformation and sync frequency of the data you wish to collect.
You can track the progress of the Blueprint you have just set up.
Once the first run of the Blueprint is completed, you can view the corresponding dashboards.
If the pre-built dashboards are limited for your use cases, you can always customize or create your own metrics or dashboards with SQL.
Please read the contribution guidelines before you make contribution. The following docs list the resources you might need to know after you decided to make contribution.
If you plan to contribute code to Apache DevLake, we have instructions on how to get started with setting up your Development environemtn.
One of the best ways to get started contributing is by improving DevLake's documentation.
Message us on Slack
This project is licensed under Apache License 2.0 - see the LICENSE file for details.