feat(q-dev): add logging data ingestion and enrich Kiro dashboards (#8767)

* feat(q-dev): add logging data ingestion and enrich Kiro dashboards

Add support for ingesting S3 logging data (GenerateAssistantResponse and
GenerateCompletions events) into new database tables, and enrich all three
Kiro Grafana dashboards with additional metrics.

Changes:
- New models: QDevChatLog and QDevCompletionLog for logging event data
- New extractor: s3_logging_extractor.go parses JSON.gz logging files
- Updated S3 collector to also handle .json.gz files
- Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions)
- New dashboard: "Kiro AI Activity Insights" with 10 panels including
  model usage distribution, active hours, conversation depth, feature
  adoption (Steering/Spec), file type usage, and prompt/response trends
- Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration,
  and Dev (Agentic) metric panels
- Fixed "Kiro Usage Dashboard" per-user table to sort by user_id
- Migration script for new tables

* fix(q-dev): use separate base path for logging S3 prefixes

Logging data lives under a different S3 prefix ("logging/") than user
report data ("user-report/"). Add LoggingBasePath option (defaults to
"logging") so logging prefixes are constructed correctly.

* fix(q-dev): auto-scan logging path without extra config

Kiro exports to two well-known S3 prefixes in the same bucket:
- user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports)
- logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs)

When AccountId is set, automatically scan both paths. The "logging"
prefix is hardcoded since it's a standard Kiro export convention.
No additional configuration needed.

* fix(q-dev): update scope tooltip to mention logging data scanning

* fix(q-dev): fix scope ID routing and CSV/JSON file separation

Three fixes:
1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope
   IDs containing "/" (e.g. "034362076319/2026") work in URL paths
2. CSV extractor now filters for .csv files only, preventing it from
   trying to parse .json.gz logging files as CSV
3. Frontend scope API calls now encodeURIComponent(scopeId) for safe
   URL encoding

* fix(q-dev): resolve *scopeId route conflict with dispatcher pattern

The catch-all *scopeId route conflicts with *scopeId/latest-sync-state.
Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a
GetScopeDispatcher that checks for /latest-sync-state suffix and
dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId.

* fix(q-dev): use URL-safe scope ID format (underscore separator)

Scope IDs like "034362076319/2026" break URL routing because "/" is a
path separator. Change ID format to "034362076319_2026" (underscore)
when AccountId is set. The Prefix field still uses "/" for S3 path
matching. Revert to standard :scopeId routes since IDs are now safe.

Note: existing scopes need to be recreated after this change.

* fix(q-dev): use NoPKModel instead of Model in archived logging models

archived.Model only has ID+timestamps, missing RawDataOrigin fields
(_raw_data_params etc.) that common.NoPKModel includes. This caused
"Unknown column '_raw_data_params'" errors at runtime.

* fix(q-dev): fix GROUP BY in per-user table to merge display_name variants

Remove display_name from GROUP BY so same user_id with different
display_name values gets merged. Use MAX(display_name) in SELECT.

* fix(q-dev): normalize logging user IDs to match CSV short UUID format

Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report
uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to
one user_id across both data sources.

* fix(q-dev): normalize user IDs in CSV extractors and sort table DESC

Apply normalizeUserId to both createUserReportData and
createUserDataWithDisplayName so user_report CSV data also strips
the "d-{directoryId}." prefix. Change per-user table sort to
ORDER BY user_id DESC.

* style(q-dev): fix gofmt formatting in chat_log models

* perf(q-dev): parallelize logging S3 downloads and batch DB writes

Optimize logging extractor performance:
- 10 goroutine workers for parallel S3 file downloads
- Batch 50 files per DB transaction instead of 1-per-file
- sync.Map cache for display name resolution (avoid repeated IAM calls)
- Parse records in memory during download, write all at once

This should improve throughput from ~1.5 files/sec to ~15+ files/sec
for typical logging file sizes.

* fix(q-dev): check tx.Rollback error return to satisfy errcheck lint

* feat(q-dev): add per-user model usage table and models column

Add "Per-User Model Usage" table (panel 11) showing each user's
request count and avg prompt/response length per model_id. Also add
"Models Used" column to the Per-User Activity table.

* fix(q-dev): remove per-user model usage table, keep models column only

* feat(q-dev): add Kiro Executive Dashboard with cross-source analytics

New dashboard "Kiro Executive Dashboard" with 12 panels covering:
- KPIs: WAU, credits efficiency, acceptance rate, steering adoption
- Trends: weekly active users, new vs returning users
- Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec
- Cost: credits pace vs projected monthly, idle power users
- Quality: acceptance rate trends, code review findings, test generation
- Efficiency: per-user productivity table with credits/line ratio

Correlates data across user_report (credits), user_data (code metrics),
and chat_log (interaction patterns) for holistic Kiro usage insights.

* fix(q-dev): fix pie charts to show per-row slices instead of single total

Set reduceOptions.values=true so Grafana treats each SQL result row as
a separate pie slice. Fixes Model Usage Distribution, File Type Usage,
Kiro Feature Adoption, and Active File Types pie charts.

* fix(q-dev): cast Hour to string for Active Hours bar chart x-axis

* fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues

1. qdev_user_report Panel 4 (Subscription Tier Distribution): set
   reduceOptions.values=true to show per-tier slices
2. qdev_user_data Panel 6 (User Interactions): remove display_name
   from GROUP BY, use MAX(display_name) to merge same user

* fix(q-dev): prevent data inflation in user_report JOIN user_data

user_report has multiple rows per (user_id, date) due to client_type
(KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date).
A direct JOIN causes user_data metrics to be counted multiple times.

Fix: pre-aggregate user_report by (user_id, date) in a subquery before
joining, so the JOIN is always 1:1.

Affects: Credits Efficiency stat and User Productivity table.
18 files changed
tree: 615aab3b475b24108a33570d679940ef5e628a59
  1. .devcontainer/
  2. .github/
  3. backend/
  4. config-ui/
  5. devops/
  6. grafana/
  7. resources/
  8. .asf.yaml
  9. .dockerignore
  10. .editorconfig
  11. .gitattributes
  12. .gitignore
  13. .licenserc.yaml
  14. AGENTS.md
  15. DISCLAIMER
  16. docker-compose-dev.yml
  17. docker-compose.datasources.yml
  18. env.example
  19. LICENSE
  20. Makefile
  21. NOTICE
  22. README.md
README.md

Apache DevLake(Incubating)

PRs Welcome Dockerhub pulls unit-test Go Report Card Slack Twitter

🤔 What is Apache DevLake?

Apache DevLake is an open-source dev data platform that ingests, analyzes, and visualizes the fragmented data from DevOps tools to extract insights for engineering excellence, developer experience, and community growth.

Apache DevLake is used by Engineering Leads, Open Source Software Maintainers and development teams looking to make better sense of their development process and to bring a more data-driven approach to their own practices. You can ask Apache DevLake many questions regarding your development process. Just connect and query.

🎯 What can be accomplished with Apache DevLake?

  1. Your Dev Data lives in many silos and tools. DevLake brings them all together to give you a complete view of your Software Development Life Cycle (SDLC).
  2. From DORA to scrum retros, DevLake implements metrics effortlessly with prebuilt dashboards supporting common frameworks and goals.
  3. DevLake fits teams of all shapes and sizes, and can be readily extended to support new data sources, metrics, and dashboards, with a flexible framework for data collection and transformation.

👉 Live Demos

The main way you interact with DevLake is through the integrated dashboards powered by Grafana.

Live DORA Dashboard

Dashboards for Engineering Leads

Dashboards for OSS Maintainers

💪 Supported Data Sources

DevLake supports connections to many popular development tools, including GitHub, GitLab, Jenkins, Jira, Sonarqube and more. Here you can find all data sources supported by DevLake, their scopes, supported versions and more!

🚀 Getting Started

Installation

You can set up Apache DevLake by following our step-by-step instructions for either Docker Compose or Helm. Feel free to ask the community if you get stuck at any point.

🤓 Usage

Please see detailed usage instructions. Here's an overview on how to get started using DevLake.

1. Set up DevLake

Install using either Docker Compose or Helm.

2. Create a Blueprint

The DevLake Configuration UI will guide you through the process (a Blueprint) to define the data connections, data scope, transformation and sync frequency of the data you wish to collect.

3. Track the Blueprint's progress

You can track the progress of the Blueprint you have just set up.

4. View the pre-built dashboards

Once the first run of the Blueprint is completed, you can view the corresponding dashboards.

5. Customize the dashboards with SQL

If the pre-built dashboards are limited for your use cases, you can always customize or create your own metrics or dashboards with SQL.

Contributing

Please read the contribution guidelines before you make contribution. The following docs list the resources you might need to know after you decided to make contribution.

👩🏾‍💻 Contributing Code

If you plan to contribute code to Apache DevLake, we have instructions on how to get started with setting up your Development environemtn.

📄 Contributing Documentation

One of the best ways to get started contributing is by improving DevLake's documentation.

⌚ Roadmap

  • Roadmap: Detailed roadmaps for DevLake.

💙 Community

Message us on Slack

📄 License

This project is licensed under Apache License 2.0 - see the LICENSE file for details.