[ISSUE #5231#5232#5233] Harden ConnectorRuntime against data loss; split connector-api SPI module; add plugin unit tests (#4642) (#5328)
* [ISSUE #5231#5232#5233] Harden ConnectorRuntime against data loss, split connector-api SPI module, add plugin unit tests
P0 — data-loss hardening (follow-up to #5231/#5232/#5233):
- runSourceOnce(): per-event isolation — one bad event (publish throws) no
longer kills the source loop or strands the rest of the batch; null events
in a poll() batch are skipped and counted
- runSinkOnce(): per-delivery ACK isolation — one ACK failure no longer loses
the offsets/ACKs of the remaining deliveries; EventMesh re-delivers
- offsetStore.put failures no longer abort the publish/commit path
- new getSourcePublishFailures() counter surfaces skipped events to the admin API
- 4 new unit tests covering all three hardening scenarios
P1 — connector-api module split (plan: docs/contrib/connector-api-split-plan.md):
- new :eventmesh-connector-api module with the 7 SPI interfaces
(SourceConnector, SinkConnector, EventMeshEndpoint, HttpCaller,
ConnectorOffsetStore, CloudEventSerializer, PollEntry) — package name kept
as org.apache.eventmesh.connector so the 23 plugins need zero code changes
- all 23 plugin build.gradles now depend on :eventmesh-connector-api instead
of :eventmesh-connector-runtime; compile-time enforcement in addition to
the ArchUnit guard
- new ArchUnit rule ruleConnectorPluginsDependOnlyOnSpi (FAIL mode) + canary
negative test proving the rule actually fires
- dist-connector packaging ships the api jar in apps/ next to the runtime jar
P1 — configuration:
- eventmesh.properties gains a commented Connector Runtime section
P0 — plugin unit tests (issue #4642 follow-up):
- file: FileSourceConnectorTest (5) + FileSinkConnectorTest (5)
- kafka: KafkaSinkConnectorTest (1, MockProducer)
- pulsar: PulsarSinkConnectorTest (2, smoke)
- rocketmq: RocketmqSinkConnectorTest (3, smoke)
- file connectors gain close()Quietly() to release file handles (Windows)
Verified: :eventmesh-connector-api:build, :eventmesh-connector-runtime:build,
:connector-{file,kafka,pulsar,rocketmq}:build, :eventmesh-runtime:compileJava,
:eventmesh-architecture-guard:test (11/11) all green locally.
* Add missing Apache license header to FakePluginCanary (fix CI checkstyle + skywalking-eyes)
📦 Documentation | 📔 Examples | ⚙️ Roadmap | 🌐 简体中文
Apache EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
EventMesh adopts a unified CloudEvents-over-MQ architecture. The message queue (MQ) acts as a pure write-ahead log (WAL) for durable storage only — there are no consumer groups, no tags, and no broker-side subscription semantics. Instead, the stateless EventMesh Runtime owns all delivery logic: its SubscriptionManager maintains the subscription registry and offset tracking, and dispatches events with load-balance, broadcast, and multicast semantics. Applications interact through a lightweight HTTP + CloudEvents 1.0 SDK (publish / subscribe / unsubscribe), while integration with external systems runs in a standalone Connector Runtime via the connector SPI.
Apache EventMesh is packed with features that help users build event-driven applications with ease. Here are the highlights that set EventMesh apart:
Core architecture
publish, subscribe, unsubscribe); no heavyweight client, no vendor lock-in.Extensibility & ecosystem
Each EventMesh surface carries an explicit maturity status. The table below is the single source of truth — module-level docs link here instead of restating their status. See docs for the per-capability guides.
| Capability | Status | Recommendation | Migration target |
|---|---|---|---|
| HTTP + CloudEvents | GA target | Recommended — the primary user path (CloudEventsClient + /events/*) | Primary path |
| Kafka / RocketMQ storage (4.x, 5.x) | GA target | Recommended — pluggable WAL backends, TCK-covered (MeshStoragePluginTCK) | Primary path |
| SSE / WebSocket push | Beta | Usable — integration-tested; unified ACK/redelivery semantics still landing | Unified push transports |
| Connector Runtime | Beta | Usable — 24 connector plugins on the standalone runtime | New connector SPI migration |
| A2A / Agent Gateway | Experimental | Evaluate — task store + runtime bridge landed (#5302/#5304); reaper & Meta-backed agent cards pending | Unified Runtime A2A |
| TCP / gRPC / OpenMessaging SDKs | Legacy-compatible | Existing users only — kept so old clients run unmodified; not extended | HTTP + CloudEvents |
Status meanings:
Migrating off TCP / gRPC SDKs? The legacy clients keep working against the current runtime; see the client guide for the HTTP + CloudEvents replacement (
CloudEventsClient).
CloudEventsClient walkthrough (pub/sub, request-reply, streaming, lite topics)A full step-by-step walkthrough — prerequisites, backend choice, run via Docker or from source, first publish, three receive transports, unsubscribe, and the SDK path — lives in Getting started. The first-event examples in that guide work against the standard ports (8080 HTTP, 8081 admin, 8082 WebSocket, 8083 connector admin).
Each contributor has played an important role in promoting the robust development of Apache EventMesh. We sincerely appreciate all contributors who have contributed code and documents.
Apache EventMesh enriches the CNCF Cloud Native Landscape.
Apache EventMesh is licensed under the Apache License, Version 2.0.
| WeChat Assistant | WeChat Public Account | Slack |
|---|---|---|
| Join Slack Chat(Please open an issue if this link is expired) |
Bi-weekly meeting : #Tencent meeting : 346-6926-0133
Bi-weekly meeting record : bilibili
| Name | Description | Subscribe | Unsubscribe | Archive |
|---|---|---|---|---|
| Users | User discussion | Subscribe | Unsubscribe | Mail Archives |
| Development | Development discussion (Design Documents, Issues, etc.) | Subscribe | Unsubscribe | Mail Archives |
| Commits | Commits to related repositories | Subscribe | Unsubscribe | Mail Archives |
| Issues | Issues or PRs comments and reviews | Subscribe | Unsubscribe | Mail Archives |