fix(jira): stop the changelog convertor hiding unconverted items forever (#9042)
The incremental filter on the changelog convertor used created_at:
_tool_jira_issue_changelog_items.created_at >= ?
created_at is stamped when the row is first inserted and never moves again.
So a changelog item that was collected during one window but not converted
in that window can never be selected by any later incremental run -- the
timestamp it is filtered on is permanently in the past. Nothing errors; the
rows simply stay in the tool layer.
That matches the report in #8834: a partial, silent shortfall in
issue_changelogs from the same sync run, persisting across runs, varying by
project. It also explains why the reporter's check looked clean -- they
inspected _devlake_collector_latest_state, which is the collector's state,
not the convertor's.
Switched to updated_at, which the extractor's upsert refreshes
(OnConflict{UpdateAll: true}), so a re-collected item is reconsidered. Of
the 36 convertors in the code base, this was the only one filtering on
created_at; the other 35 already use updated_at.
This does not widen the board filter, so it does not carry the cost klesh
raised against moving board_id into the join: no board task converts
anything outside its own board.
Separately, the board filter is now reported rather than silent. After a
successful conversion the subtask counts collected changelog items whose
issue is not on this board and logs the number with the reason. That is one
aggregate query per board task, and it turns an unexplained shortfall into
a logged figure. Diagnostic failures are logged, never propagated.
Tests: an e2e dataflow test driving the convertor with a changelog for an
issue on no board, asserting it is excluded, that nothing is attached to
issue id 0, and that in-scope changelogs still convert; plus a regression
guard on the filter column, since reverting it is a one-token change that
silently restores permanent data loss.Apache DevLake is an open-source dev data platform that ingests, analyzes, and visualizes the fragmented data from DevOps tools to extract insights for engineering excellence, developer experience, and community growth.
Apache DevLake is used by Engineering Leads, Open Source Software Maintainers and development teams looking to make better sense of their development process and to bring a more data-driven approach to their own practices. You can ask Apache DevLake many questions regarding your development process. Just connect and query.
The main way you interact with DevLake is through the integrated dashboards powered by Grafana.
Dashboards for Engineering Leads
Dashboards for OSS Maintainers
DevLake supports connections to many popular development tools, including GitHub, GitLab, Jenkins, Jira, Sonarqube and more. Here you can find all data sources supported by DevLake, their scopes, supported versions and more!
You can set up Apache DevLake by following our step-by-step instructions for either Docker Compose or Helm. Feel free to ask the community if you get stuck at any point.
Please see detailed usage instructions. Here's an overview on how to get started using DevLake.
Install using either Docker Compose or Helm.
The DevLake Configuration UI will guide you through the process (a Blueprint) to define the data connections, data scope, transformation and sync frequency of the data you wish to collect.
You can track the progress of the Blueprint you have just set up.
Once the first run of the Blueprint is completed, you can view the corresponding dashboards.
If the pre-built dashboards are limited for your use cases, you can always customize or create your own metrics or dashboards with SQL.
Please read the contribution guidelines before you make contribution. The following docs list the resources you might need to know after you decided to make contribution.
If you plan to contribute code to Apache DevLake, we have instructions on how to get started with setting up your Development environemtn.
One of the best ways to get started contributing is by improving DevLake's documentation.
Message us on Slack
This project is licensed under Apache License 2.0 - see the LICENSE file for details.