AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

Repository Overview

This is the Apache HugeGraph-Computer repository containing two distinct graph computing systems:

computer (Java/Maven): A distributed BSP/Pregel-style graph processing framework that runs on Kubernetes or YARN
vermeer (Go): A high-performance in-memory graph computing platform with master-worker architecture

Both integrate with HugeGraph for graph data input/output.

Build & Test Commands

Computer (Java)

Prerequisites:

JDK 11 for building/running
JDK 8 for HDFS dependencies
Maven 3.5+
For K8s module: run mvn clean install first to generate CRD classes under computer-k8s

Build:

cd computer
mvn clean compile -Dmaven.javadoc.skip=true

Tests:

# Unit tests
mvn test -P unit-test

# Integration tests
mvn test -P integrate-test

Run single test:

# Run specific test class
mvn test -P unit-test -Dtest=ClassName

# Run specific test method
mvn test -P unit-test -Dtest=ClassName#methodName

License check:

mvn apache-rat:check

Package:

mvn clean package -DskipTests

Vermeer (Go)

Prerequisites:

Go 1.23+
curl and unzip (for downloading binary dependencies)

First-time setup:

cd vermeer
make init  # Downloads supervisord and protoc binaries, installs Go deps

Build:

make          # Build for current platform
make build-linux-amd64
make build-linux-arm64

Development build with hot-reload UI:

go build -tags=dev

Clean:

make clean      # Remove built binaries and generated assets
make clean-all  # Also remove downloaded tools

Run:

# Using binary directly
./vermeer --env=master
./vermeer --env=worker

# Using script (configure in vermeer.sh)
./vermeer.sh start master
./vermeer.sh start worker

Regenerate protobuf (if proto files changed):

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0

# Generate (adjust protoc path for your platform)
vermeer/tools/protoc/linux64/protoc vermeer/apps/protos/*.proto --go-grpc_out=vermeer/apps/protos/. --go_out=vermeer/apps/protos/. # please note remove license header if any

Architecture

Computer (Java) - BSP/Pregel Framework

Module Structure:

computer-api: Public interfaces for graph processing (Computation, Vertex, Edge, Aggregator, Combiner, GraphFactory)
computer-core: Runtime implementation (WorkerService, MasterService, messaging, BSP coordination, managers)
computer-algorithm: Built-in algorithms (PageRank, LPA, WCC, SSSP, TriangleCount, etc.)
computer-driver: Job submission and driver-side coordination
computer-k8s: Kubernetes deployment integration
computer-yarn: YARN deployment integration
computer-k8s-operator: Kubernetes operator for job management
computer-dist: Distribution packaging
computer-test: Integration and unit tests

Key Design Patterns:

API/Implementation Separation: Algorithms depend only on computer-api interfaces; computer-core provides runtime implementation. Algorithms are dynamically loaded via config.
Manager Pattern: WorkerService composes multiple managers (MessageSendManager, MessageRecvManager, WorkerAggrManager, DataServerManager, SortManagers, SnapshotManager, etc.) with lifecycle hooks: initAll(), beforeSuperstep(), afterSuperstep(), closeAll().
BSP Coordination: Explicit barrier synchronization via etcd (EtcdBspClient). Each superstep follows:
- workerStepPrepareDone → waitMasterStepPrepareDone
- Local compute (vertices process messages)
- workerStepComputeDone → waitMasterStepComputeDone
- Aggregators/snapshots
- workerStepDone → waitMasterStepDone (master returns SuperstepStat)
Computation Contract: Algorithms implement Computation<M extends Value>:
- compute0(context, vertex): Initialize at superstep 0
- compute(context, vertex, messages): Process messages in subsequent supersteps
- Access to aggregators, combiners, and message sending via ComputationContext

Important Files:

Algorithm contract: computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/worker/Computation.java
Runtime orchestration: computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java
BSP coordination: computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/bsp/Bsp4Worker.java
Example algorithm: computer/computer-algorithm/src/main/java/org/apache/hugegraph/computer/algorithm/centrality/pagerank/PageRank.java

Vermeer (Go) - In-Memory Computing Engine

Directory Structure:

algorithms/: Go algorithm implementations (pagerank.go, sssp.go, louvain.go, etc.)
apps/:
- bsp/: BSP coordination helpers
- graphio/: HugeGraph I/O adapters (reads via gRPC to store/pd, writes via HTTP REST)
- master/: Master scheduling, HTTP endpoints, worker management
- compute/: Worker-side compute logic
- protos/: Generated protobuf/gRPC definitions
- common/: Utilities, logging, metrics
client/: Client libraries
tools/: Binary dependencies (supervisord, protoc)
ui/: Web UI assets

Key Patterns:

Maker/Registry Pattern: Graph loaders/writers register themselves via init() (e.g., LoadMakers[LoadTypeHugegraph] = &HugegraphMaker{}). Master selects loader by type.
HugeGraph Integration:
- hugegraph.go implements HugegraphMaker, HugegraphLoader, HugegraphWriter
- Queries PD via gRPC for partition metadata
- Streams vertex/edge data via gRPC from store (ScanPartition)
- Writes results back via HugeGraph HTTP REST API
Master-Worker: Master schedules LoadPartition tasks to workers, manages worker lifecycle via WorkerManager/WorkerClient, exposes HTTP admin endpoints.

Important Files:

HugeGraph integration: vermeer/apps/graphio/hugegraph.go
Master scheduling: vermeer/apps/master/tasks/tasks.go
Worker management: vermeer/apps/master/workers/workers.go
HTTP endpoints: vermeer/apps/master/services/http_master.go
Scheduler: vermeer/apps/master/bl/scheduler_bl.go

Integration with HugeGraph

Computer (Java):

WorkerInputManager reads vertices/edges from HugeGraph via GraphFactory abstraction
Graph data is partitioned and distributed to workers via input splits

Vermeer (Go):

Directly queries HugeGraph PD (metadata service) for partition information
Uses gRPC to stream graph data from HugeGraph store
Writes computed results back via HugeGraph HTTP REST API (adds properties to vertices)

Development Workflow

Adding a New Algorithm (Computer):

Create class in computer-algorithm implementing Computation<MessageType>
Implement compute0() for initialization and compute() for message processing
Use context.sendMessage() or context.sendMessageToAllEdges() for message passing
Register aggregators in beforeSuperstep(), read/write in compute()
Configure algorithm class name in job config

K8s-Operator Development:

CRD classes are auto-generated; run mvn clean install in computer-k8s-operator first
Generated classes appear in computer-k8s/target/generated-sources/
CRD generation script: computer-k8s-operator/crd-generate/Makefile

Vermeer Asset Updates:

Web UI assets must be regenerated after changes: cd asset && go generate
Or use make generate-assets from vermeer root
For dev mode with hot-reload: go build -tags=dev

Testing Notes

Computer:

Integration tests require etcd, HDFS, HugeGraph, and Kubernetes (see .github/workflows/computer-ci.yml)
Test environment setup scripts in computer-dist/src/assembly/travis/
Unit tests run in isolation without external dependencies

Vermeer:

Test scripts in vermeer/test/,with vermeer_test.go and vermeer_test.sh
Configuration files in vermeer/config/ (master.ini, worker.ini templates)

CI/CD

CI pipeline (.github/workflows/computer-ci.yml) runs:

License check (Apache RAT)
Setup HDFS (Hadoop 3.3.2)
Setup Minikube/Kubernetes
Load test data into HugeGraph
Compile with Java 11
Run integration tests (-P integrate-test)
Run unit tests (-P unit-test)
Upload coverage to Codecov

Important Notes

Computer K8s module: Must run mvn clean install before editing to generate CRD classes
Java version: Build requires JDK 11; HDFS dependencies require JDK 8
Vermeer binary deps: First-time builds need make init to download supervisord/protoc
BSP coordination: Computer uses etcd for barrier synchronization (configure via BSP_ETCD_URL)
Memory management: Both systems auto-manage memory by spilling to disk when needed