Apache Iggy is a persistent message streaming platform in Rust. Thread-per-core shared-nothing, io_uring + compio. Transports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs: Rust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem ingests from / egresses to external systems via dlopened plugins.
Skills live under
.claude/skills/, with per-skill links under.agents/skills/for Codex. Load connectors-overview first for any change undercore/connectors/. Other subsystems follow the repo-wide principles in this file.
iggy_connector_sdk MAJOR version or changing any FFI signature in core/connectors/sdk/src/{sink,source}.rs (breaks every pre-built plugin .so).core/binary_protocol/ wire-format types tagged #[repr(C)] (breaks every running client).core/connectors/runtime/src/state.rs) in a way that changes the atomic-rename protocol (corruption risk).master, amending shipped commits, or skipping hooks with --no-verify / --no-gpg-sign.Most PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.
prek run before the PR is opened. If prek isn‘t installed, ask the user to install it (cargo install prek, then prek install). Don’t substitute the individual checks.# Verification order is enforced by CI (.github/actions/rust/pre-merge). cargo fmt --all # step 1 cargo sort --no-format --workspace # step 2 (--no-format mandatory) cargo clippy --all-features --all-targets -- -D warnings # step 3 cargo test -p <crate> # step 4 # Other CI checks (mirror in pre-commit; pass --fix where supported) ./scripts/ci/taplo.sh # TOML ./scripts/ci/markdownlint.sh # Markdown ./scripts/ci/shellcheck.sh # Shell ./scripts/ci/license-headers.sh # Apache headers ./scripts/ci/trailing-whitespace.sh ./scripts/ci/trailing-newline.sh ./scripts/ci/sync-rustc-version.sh ./scripts/ci/python-sdk-version-sync.sh ./scripts/ci/uv-lock-check.sh ./scripts/ci/binary-artifacts.sh --check ./scripts/ci/third-party-licenses.sh --validate --manifest <path> # Run things cargo run --bin iggy-server cargo run --bin iggy -- <subcommand> IGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors ./scripts/run-bdd-tests.sh ./scripts/run-benches.sh
Pre-commit hooks wire the same scripts. Install with pre-commit install (Python) or prek install (Rust drop-in) - both read .pre-commit-config.yaml.
iggy/ ├── core/ │ ├── server/ Iggy server binary (Viewstamped Replication) │ ├── sdk/ Rust client SDK │ ├── cli/ iggy CLI │ ├── connectors/ Connectors runtime + SDK + sinks/sources │ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...) │ ├── binary_protocol/ Wire format (stable) │ ├── configs/ Config plumbing + ConfigEnv derive │ ├── ai/mcp/ MCP server │ ├── bench/ Benchmark suite │ ├── integration/ Cross-crate integration tests │ ├── simulator/ Deterministic simulator │ ├── consensus/ Raft (Miri-checked unsafe iobuf) │ ├── journal/ WAL / journal │ ├── message_bus/ In-process bus │ ├── shard/ Shard runtime │ ├── harness_derive/ #[iggy_harness] proc macro │ └── metadata, partitions, server_common, configs_derive, clock, tools ├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs ├── bdd/ Cross-language BDD (Gherkin) ├── examples/ Per-SDK runnable examples ├── helm/ Helm charts ├── web/ Web UI └── scripts/ CI + release + dev scripts
| Task | Location |
|---|---|
| Wire protocol | core/binary_protocol/ |
| Server | core/server/src/ |
| Rust client SDK | core/sdk/src/ |
| Connectors | core/connectors/ -> connector-* skills |
| Integration tests | core/integration/tests/ |
| Cross-SDK BDD | bdd/ |
| Benchmark suite | core/bench/ |
Each SKILL.md has YAML frontmatter (name, description). For connectors work, load connectors-overview first as router. Claude Code uses /skill-name; Codex uses $skill-name or /skills. Keep one source under .claude/skills/ and link its directory from .agents/skills/. Keep templates and references inside the source directory.
Review skills are user-invoked only. Keep disable-model-invocation: true for Claude Code and policy.allow_implicit_invocation: false in agents/openai.yaml for Codex. Do not replicate their steps without an explicit invocation. When a non-trivial change passes verification, suggest /team-review <target> in Claude Code or $team-review <target> in Codex.
Review skills share execution instructions for target preparation and client-specific agent tools. Run ./scripts/ci/skills.sh to check skill metadata, links, and invocation policies.
licenserc.toml; common examples: .rs uses // ..., Cargo.toml and shell use # ....fmt --all -> sort --no-format --workspace -> clippy --all-features --all-targets -- -D warnings -> test. Plus taplo.sh --check. CI enforces all.--no-format to cargo sort. Without it multi-line feature arrays flatten and taplo re-expands them, churning the workspace.FromStr not parse_foo, Display not format_foo, From/TryFrom not to_foo_from_bar. Structured Err enums, not String.use inside functions or blocks. Group: std, external crates, then crate::.b, p, t, m. Match literal API field names as log labels.cargo install without authorization. Toolchain pinned in rust-toolchain.toml. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.unwrap() / expect() on Results from external I/O outside tests.tokio::sync::Mutex (not std::sync::Mutex) wherever a lock is held across .await. Hold locks briefly: lock, clone, drop guard, then do I/O.&self, &str, &[T]. Vec::with_capacity(n) when size is known.#[serde(default)] or Option<T>.| Tool | Where |
|---|---|
cargo fmt, cargo sort, cargo clippy | pre-commit + CI |
cargo machete | CI machete task |
cargo nextest + cargo-rail | CI test-* DAG-scoped |
cargo llvm-cov | CI coverage |
cargo http-registry | CI verify-publish |
| Miri | CI miri (binary_protocol + consensus) |
taplo, markdownlint-cli, shellcheck | pre-commit + CI |
typos | pre-commit |
cargo about + license-checker-rseidelsohn | scripts/ci/third-party-licenses.sh |
HawkEye / license-headers.sh | pre-commit + CI |
uv, ruff | pre-commit (Python) |
golangci-lint | pre-push (Go) |
#[cfg(test)] mod tests at EOF of src/lib.rs (or per-module).<crate>/tests/.core/integration/tests/. Harness in core/integration/src/harness/ spawns real iggy-server (ServerHandle), connectors runtime (ConnectorsRuntimeHandle), MCP server (McpHandle), client (ClientHandle). #[iggy_harness] proc macro wires fixtures. Suites: cli/, cluster/, config_provider/, connectors/, data_integrity/, mcp/, sdk/, server/, state/, storage/. External backends inside connectors/ use testcontainers-modules. Per-test logs land in test_logs/.bdd/. Gherkin features against every SDK. Driver: scripts/run-bdd-tests.sh.BDD test naming: given_X_when_Y_should_Z (3-part). Promote to 4-part given_X_when_Y_then_Z_should_W only with a distinct “then” intermediate. Be consistent inside a file.
Filter a single integration test by path. Build the binaries the harness launches first: it runs whatever is already in target/, so a stale one fails as though the change under test broke it.
cargo build --bin iggy-server --bin iggy-connectors cargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json
testcontainers-modules spawns containers per test.cargo sort without --no-format will churn the workspace. Use the pre-commit hook.rust-toolchain.toml. Older system cargo fails on transitive deps. prefer rustup so the pin auto-resolves.test_logs/ grows quickly. Wipe between major refactors.binary_protocol + consensus. Cannot emulate io_uring syscalls. do not try to expand Miri to crates that pull compio.--test target. Filter by test path inside the single mod.rs binary.Command::cargo_bin only resolves a path, so a stale iggy-server, iggy-connectors or iggy-mcp is used silently. CI builds first (cargo build --locked --bin iggy-server --bin iggy); a local run does not, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.Created by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:
local_data/ - server datalocal_state/ - connectors runtime state filestest_logs/ - per-test logs + local_data snapshotsperformance_results/ - benchmark outputtarget/ - cargo buildfeat(connectors): .... Subject <= 72 chars. Allowed types/scopes enforced by .github/workflows/pr-title.yml.scripts/extract-version.sh and scripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh.scripts/verify-crates-publish.sh against cargo-http-registry.scripts/ci/third-party-licenses.sh (ASF policy).