Apache DevLake - AI Coding Agent Instructions

Project Overview

Apache DevLake is a dev data platform that ingests data from DevOps tools (GitHub, GitLab, Jira, Jenkins, etc.), transforms it into standardized domain models, and enables metrics/dashboards via Grafana.

Architecture

Three-Layer Data Model

  1. Raw Layer (_raw_* tables): JSON data collected from APIs, stored for replay/debugging
  2. Tool Layer (_tool_* tables): Plugin-specific models extracted from raw data
  3. Domain Layer (standardized tables): Normalized models in backend/core/models/domainlayer/ - CODE, TICKET, CICD, CODEREVIEW, CODEQUALITY, CROSS

Key Components

  • backend/: Go server + plugins (main codebase)
  • backend/python/: Python plugin framework via RPC
  • config-ui/: React frontend (TypeScript, Vite, Ant Design)
  • grafana/: Dashboard definitions

Plugin Development (Go)

Plugin Structure

Each plugin in backend/plugins/<name>/ follows this layout:

api/         # REST endpoints (connections, scopes, scope-configs)
impl/        # Plugin implementation (implements core interfaces)
models/      # Tool layer models + migrationscripts/
tasks/       # Collectors, Extractors, Converters
e2e/         # Integration tests with CSV fixtures

Required Interfaces

See backend/plugins/gitlab/impl/impl.go for reference:

  • PluginMeta: Name, Description, RootPkgPath
  • PluginTask: SubTaskMetas(), PrepareTaskData()
  • PluginModel: GetTablesInfo() - must list all models or CI fails
  • PluginMigration: MigrationScripts() for DB schema evolution
  • PluginSource: Connection(), Scope(), ScopeConfig()

Subtask Pattern (Collector → Extractor → Converter)

// 1. Register subtask in tasks/register.go via init()
func init() {
    RegisterSubtaskMeta(&CollectIssuesMeta)
}

// 2. Define dependencies for execution order
var CollectIssuesMeta = plugin.SubTaskMeta{
    Name:         "Collect Issues",
    Dependencies: []*plugin.SubTaskMeta{}, // or reference other metas
}

API Collectors

Migration Scripts

  • Located in models/migrationscripts/
  • Register all scripts in register.go's All() function
  • Version format: YYYYMMDD_description.go

Build & Development Commands

# From repo root
make dep              # Install Go + Python dependencies
make build            # Build plugins + server
make dev              # Build + run server
make godev            # Go-only dev (no Python plugins)
make unit-test        # Run all unit tests
make e2e-test         # Run E2E tests

# From backend/
make swag             # Regenerate Swagger docs (required after API changes)
make lint             # Run golangci-lint

Running Locally

docker-compose -f docker-compose-dev.yml up mysql grafana  # Start deps
make dev                                                     # Run server on :8080
cd config-ui && yarn && yarn start                          # UI on :4000

Testing

Unit Tests

Place *_test.go files alongside source. Use mocks from backend/mocks/.

E2E Tests for Plugins

Use CSV fixtures in e2e/ directory. See backend/test/helper/ for the Go test client that can spin up an in-memory DevLake instance.

Integration Testing

helper.ConnectLocalServer(t, &helper.LocalClientConfig{
    ServerPort:   8080,
    DbURL:        "mysql://merico:merico@127.0.0.1:3306/lake",
    CreateServer: true,
    Plugins:      []plugin.PluginMeta{gitlab.Gitlab{}},
})

Python Plugins

Located in backend/python/plugins/. Use Poetry for dependencies. See backend/python/README.md.

Code Conventions

  • Tool model table names: _tool_<plugin>_<entity> (e.g., _tool_gitlab_issues)
  • Domain model IDs: Use didgen.NewDomainIdGenerator for consistent cross-plugin IDs
  • All plugins must be independent - no cross-plugin imports
  • Apache 2.0 license header required on all source files

Common Pitfalls

  • Forgetting to add models to GetTablesInfo() fails plugins/table_info_test.go
  • Migration scripts must be added to All() in register.go
  • API changes require running make swag to update Swagger docs
  • Python plugins require libgit2 for gitextractor functionality