Status note: this is the historical design document for the architecture rewrite (a design record, not a living guide). The current implementation status and maturity of each capability are governed by the capability status table in the main README. For the user-facing views see Architecture overview and the feature guides.
Branch:
refactor/unified-runtime-pipeline— based on the direction defined in this document.Core change: drop the MQ's Producer Group / Consumer Group semantics entirely — the MQ is used purely as a storage layer, and EventMesh maintains its own subscription and dispatch semantics. Internally everything is
EventMeshFrame(CloudEvent is demoted to one of the external ingress formats); externally the multi-protocol surface (CloudEvents / MeshMessage / A2A) converts directly to Frame via theFrameAdaptorSPI, never through each other. The SDK is simplified to HTTP-only.v2.0 architecture deepening (2026-08-13, absorbed from
eventmesh-architecture-refinement.md):
EventMeshFrameend-to-end internally (storage SPI + dispatch pipeline
- carriers all Frame-typed).
- External protocol conversion moved into the
FrameAdaptorSPI (CloudEvents / MeshMessage / A2A each an independent plugin, zero CE-intermediary conversions).- Offset path-② (purely local, no meta reporting, no group.id; takeover via MQ replay).
- Full-stickiness load balancing (session-assignment-layer
recommend, self-reported instance load, no forwarding).
The pre-rewrite architecture inherited Kafka/RocketMQ's Producer Group / Consumer Group concepts, which caused:
Redesign direction: EventMesh becomes a pure “CloudEvents over MQ” event bus. The MQ is EventMesh's storage backend; EventMesh manages its own subscription and dispatch logic; clients only need HTTP + CloudEvents.
| Rule | Meaning |
|---|---|
| MQ without semantics | Kafka/RocketMQ are used only as “a durable FIFO queue”; no Producer Group / Consumer Group / Tag concept is ever exposed to clients |
| EventMesh-owned subscriptions | All dispatch logic (who receives which message, under which policy) is maintained by EventMesh itself, never delegated to the MQ |
| EventMesh-owned offsets | EventMesh manages the consumption position of every subscription (topic#clientId#partition → offset), modeled on the RocketMQ client OffsetStore, persisted in RocksDB — independent of MQ Consumer Group offsets |
| Minimal SDK | One SDK (HTTP), two objects (CloudEvent + Subscription), three APIs (publish / subscribe / unsubscribe) |
| EventMeshFrame internally | Everything internal is EventMeshFrame (fixed header + KV attributes + raw data); CloudEvent is demoted to one external ingress format; external protocols (CloudEvents/MeshMessage/A2A) each convert directly to Frame via the FrameAdaptor SPI, never through each other |
| Independently deployed connectors | The Connector Runtime is a separate process talking to the EventMesh Runtime over HTTP + CloudEvents — no shared internals, independent lifecycles and OffsetStores |
Implementation status snapshot (v2.0 / 2026-08-13 audit): of the iron rules, “EventMeshFrame internally / independent connectors / minimal HTTP-only SDK” have landed; “EventMesh-owned subscriptions / offsets” hold for single-instance (offset path-② purely local + MQ-replay takeover, meta reporting off by default). Multi-instance coordination moved to full stickiness (session-assignment-layer recommend, forwarding layer disabled). External protocols go through the
FrameAdaptorSPI (CloudEvents as an independent plugin + MeshMessage + A2A each converting directly to Frame, zero CE-intermediary). See §19.
previous design redesigned EventMesh SDK: ├─ TCP SDK (Package protocol) → deleted entirely ├─ HTTP SDK (EventMeshMessage) → simplified to CloudEvents-only └─ gRPC SDK (proto CloudEvent) → deleted entirely MQ role: ├─ Group semantics exposed to clients ───→ pure storage, no semantics ├─ MQ-side load balancing ───→ EventMesh SubscriptionManager └─ Tag filtering (subExpression) ───→ EventMesh MULTICAST CloudEvent filters
┌─────────────────────────────────────────────────────────────────┐
│ EventMesh Client │
│ (HTTP SDK, CloudEvents-only) │
│ cloudEventsClient.publish(CloudEvent) │
│ cloudEventsClient.subscribe(topic, handler) │
│ cloudEventsClient.unsubscribe(topic) │
└────────────────────────────┬────────────────────────────────────┘
│ HTTP POST / GET (application/cloudevents+json)
▼
┌─────────────────────────────────────────────────────────────────┐
│ EventMesh Runtime │
│ (single process, unified) │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ HTTP endpoint (the only transport layer) │ │
│ │ POST /events/publish → Ingress pipeline │ │
│ │ POST /events/subscribe → SubscriptionManager │ │
│ │ POST /events/unsubscribe │ │
│ │ GET /events/poll → PushService delivery │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Ingress pipeline │ │
│ │ AuthFilter → RateLimitFilter → AclFilter │ │
│ │ → ProtocolFilter → TransformerEngine → RouterEngine │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ EventMeshFrame │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Storage plugin (the MQ as a storage layer) │ │
│ │ · single Producer, EventMesh-managed │ │
│ │ · writes by topic partition │ │
│ │ · no Producer Group / clientId / producerGroup config │ │
│ │ · single Consumer — pulls everything, dispatch per │ │
│ │ EventMesh subscription rules, no Consumer Group │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ SubscriptionManager (the core) │ │
│ │ manages every client subscription; dispatches by policy: │ │
│ │ · LOAD_BALANCE (round-robin / least-connections) │ │
│ │ · BROADCAST (every subscriber) │ │
│ │ · MULTICAST (subject/type/header match) │ │
│ │ · own OffsetStore (RocksDB) │ │
│ └──────────────────────────────┬─────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ PushService (long-poll / SSE / WebSocket transports) │ │
│ └────────────────────────────────────────────────────────────┘ │
└────────────────────────────┬────────────────────────────────────┘
│ Kafka / RocketMQ (pure storage)
▼
┌──────────────────────┐
│ Kafka / RocketMQ │
│ (a distributed WAL) │
│ · no Producer Group │
│ · no Consumer Group │
│ · no Tag filtering │
│ · EventMesh is the │
│ only producer + │
│ consumer │
└──────────────────────┘
The problem being removed. The old storage plugin exposed MQ semantics:
// eventmesh-storage-rocketmq (old) MeshMQProducer.java: producerGroup: String // RocketMQ ProducerGroup — clients had to understand it createTransactionProducer() // transactional messages EventMesh had to inherit MeshMQConsumer.java: consumerGroup: String // RocketMQ ConsumerGroup — the core problem subscribe(topic, subExpression) // subExpression = RocketMQ Tag filtering push() / pull() // two consumption modes mixed
Configuring consumerGroup through the EventMesh SDK means the client is still using RocketMQ's Consumer Group semantics — EventMesh was not really providing its own subscription/dispatch model.
The target interface (landed as MeshStoragePlugin): only two core concepts — send(frame) and offset-scoped poll, plus assignPartitions / commitOffset — with no producerGroup, consumerGroup, or tag parameters. (Status note: the Kafka implementation initially kept an internal group.id (gap G7); assignPartitions/commitOffset landed for Kafka first and were partially stubbed for RocketMQ.)
The old SDK family (TCP / HTTP / gRPC) had three object models and API styles in parallel. The redesign collapsed them into one HTTP SDK with CloudEventsClient exposing publish / subscribe / unsubscribe (plus the later additions: request-reply, streaming, lite topics — see the client guide).
All entry points (HTTP publish, A2A, legacy bridge) converge on one frame-typed ingress path — today's UniIngressService — so filters, quota, and rate limiting have exactly one place to live.
The old consumer model (TCP Consumer with session maps, HTTP PushConsumer with LRU caches, gRPC push) was replaced by a single PushService with pluggable transports (long-poll / SSE / WebSocket) fed by the SubscriptionManager — all frame-typed, all sharing ACK / retry / quota.
The Connector Runtime and EventMesh Runtime are two completely independent processes communicating over HTTP + CloudEvents, sharing no internal components:
┌─────────────────────────────┐ HTTP / CloudEvents ┌──────────────────────────────┐ │ Connector Runtime │ ←────────────────────→ │ EventMesh Runtime │ │ Source/Sink manager │ POST /events/publish │ ingress → storage (MQ) │ │ · connector lifecycle │ │ SubscriptionManager │ │ · external poll/write │ GET /events/poll │ · own offsets (RocksDB) │ │ · independent offsets │ ←───────────────────── │ · dispatch (LB/Bcast/Mcast)│ └─────────────────────────────┘ └──────────────────────────────┘
Source connectors poll() external systems into CloudEvents; sink connectors put(CloudEvent) out; each maintains its own offsets.
A single launcher (EventMeshApplication) starts traffic HTTP + admin (+ opt-in WebSocket). The deletion list removed the TCP/gRPC server paths, eventmesh-registry, and dead modules (~59% of the tree). The phased plan tracked the migration; the evidence trail lives in architecture review evidence.
Foundational decisions, confirmed 2026-07-02. Later design and implementation follow this section; changes must be appended as new decisions noting what they supersede.
Strict adherence to the “MQ without semantics” rule — EventMesh implements multi-instance consumption coordination entirely itself: a self-written partition-assignment protocol + leases + centralized offset storage + cluster-wide subscription sync. No compromise that reuses MQ Consumer Group rebalancing internally. Highest development cost; strongest autonomy.
(v2.0 note: multi-instance coordination was later simplified to the full-stickiness model — see Load balancing — with partition ownership retained behind PARTITION_OWNED_PULL.)
Client-facing delivery is at-least-once, with clients deduplicating on the CloudEvents id. Exactly-once is not implemented for the delivery side. ACK + retry + DLQ + STICKY ordering + client dedup all hold. Exactly-once remains confined to connector offset management (see §16).
Keep a single HTTP-only CloudEvents SDK; delete the TCP and gRPC SDKs. This is an irreversible action — all existing TCP/gRPC clients must migrate to the HTTP SDK. Low-latency needs are covered by the optional WebSocket/SSE transports. (The shipped runtime keeps the legacy TCP path as a compatibility bridge; see Protocols & SDKs.)
The code baseline is the then-current workspace (which had already absorbed meta / openconnect / A2A / CloudEvents / retry modules), not a clean rewrite on Apache develop. Customizations (the masa submodules: connector-wemq, trace-, registry-, security-acl) are plugin-shaped and were slated for rewrite against the new SPIs.
The global control plane belongs to the Meta registry (instance heartbeats, partition assignments, clientId routing, cluster subscription view, offset replicas, dynamic rules); the Admin Server only manages (Jobs, DLQ browse/replay, ops commands, metric aggregation) and never performs strongly-consistent coordination. Degradation: Meta down → instances self-assign; Admin down → ops impaired, Runtime fine. The dual registry abstraction (eventmesh-registry) was dropped in favor of the single Meta path.
A synchronous request() API over HTTP request-response (suspended request, correlationId matching, Meta routing for cross-instance replies). Timeout = failure; late replies are dropped by default (clear semantics, no DLQ). request-reply is RPC semantics — no re-delivery, no DLQ — deliberately isolated from the at-least-once pub/sub path.
The implementation targets Java 21 (virtual threads for suspended push channels — near-zero cost per parked connection). Storage keeps multiple backends with S3Stream added alongside Kafka/RocketMQ (integration posture A: a minimal S3StreamStoragePlugin; EventMesh still fully self-coordinates partitions; S3Stream is a data WAL only). RocksDB's clarified position: the local full copy + Meta write-offload + degraded fallback for offsets — not redundant with the Meta tier.
Two different layers, often confused:
Layer A: connector-internal progress (offset persistence) Scope: Source pull offset / Sink write-confirm offset Where: inside the Connector Runtime; no external clients involved Semantics: exactly-once (local RocksDB + remote Admin double-write) Purpose: a connector restart neither loses nor double-processes external data Layer B: client-facing delivery (subscribers receiving messages) Scope: EventMesh → SDK subscriber (post-publish delivery) Where: SubscriptionManager → PushService → client poll/ACK Semantics: at-least-once + client idempotency (decision 15.2) Purpose: no message loss; duplicates possible, deduped by id at the client
Exactly-once is feasible for layer A (self-contained offset state) and prohibitively expensive for layer B (transactional delivery + a global dedup store) — hence the split.
requester: POST /events/request
body: CloudEvent { ..., x-em-reply-to: "reply.<reqId>", x-em-correlation-id: reqId }
(the HTTP request suspends, blocking for a reply or timeout)
↓ EventMesh routes like a publish (ingress → storage.send)
responder: subscribed to the request topic, processes, then
POST /events/reply
body: CloudEvent { x-em-correlation-id: reqId, data: reply }
↓ EventMesh matches correlationId to the suspended request
requester: the suspended request returns the reply ← unblocked
timeout → the request fails (same semantics as the old TCP sync call)
Key points: a correlationId→AsyncContext matching table with timeout sweep; cross-instance replies route via the Meta routing table; late replies are dropped; never re-delivered and never dead-lettered (RPC side effects must not repeat). (Gap G10: self-addressed cross-instance reply routing was later covered by the sticky-instance model.)
The end-to-end test-case design (§18) mapped every guarantee above to a named test; the current test inventory and per-PR evidence live in review evidence.
The streaming design (§20–21, integrated from the streaming-session, sdk-streaming-call, and lite-streaming-call designs) defines the two orthogonal streaming modes — Mode 1 (client → agent streaming call; the runtime mediates, channels multiplex with client affinity; sessionId format <agentId>:<uuid> routes with zero lookups on the : prefix) and Mode 2 (session pub/sub with deterministic lite naming, no agent, no matchmaking) — both carried by EventMeshFrame STREAM_REQ / STREAM_CHUNK messages. The SDK surface is client.streaming(): openSession → call → forEach and subscribeSession / openSessionPublisher. See Streaming & push transports for the current user-facing guide.
Appendices A–F of the original document carried the configuration key table, the full CloudEvents extension field set, deployment topology examples, the develop-code migration mapping, the customization (masa) migration inventory, and the v1.11 implementation-gap analysis. The still-current content from those appendices lives in the feature and reference pages of this documentation tree (configuration, protocols, offset management, load balancing, frame protocol); the point-in-time gap/migration tables are superseded by the architecture review evidence and are intentionally not reproduced.
Document version: v2.0 (English rewrite of the Chinese original) · integrated 2026-08-13 · related: offset management, load balancing, frame protocol.