中文版见
docs/proxy-admin-zh.md。
RocketMQ 5.0 moved client access behind the stateless Proxy, but operations still observe clients through broker-side structures (ConsumerManager on the broker, Remoting-era admin commands). gRPC clients attached to a Proxy are invisible to those tools: the control plane cannot answer “which SDK clients are online, what do they subscribe to, are they healthy” without indirect metrics heuristics.
A control-plane dashboard needs a standard server-side interface to read complete gRPC client data. This change implements that interface on the Proxy itself.
Admin serviceThe gRPC contract is the upstream Admin service defined in apache/rocketmq/v2/admin.proto of the apache/rocketmq-apis repository (main branch, consumed as a git submodule — see §6). This implementation deliberately does NOT introduce a bespoke ProxyAdminService in the protocol layer: the control-plane RPC surface lives in the same versioned, community-reviewed proto that the dashboard and multi-language SDKs already consume, and stays additive-compatible with upstream evolution.
The proxy implements AdminGrpc.AdminImplBase (all 16 RPCs) and binds it on a dedicated admin gRPC server.
MessagingService.Admin service, apache/rocketmq/v2/admin.proto).proxy.admin.* ACL 2.0 resources with read-only / high-privilege action separation.The upstream Admin service is bound on its own gRPC server/port (grpcAdminServerPort, default 8088) with its own interceptor chain, separate from the data plane (8081). The whole surface is gated by a global kill switch grpcAdminServerEnable, which defaults to false: the admin server is opt-in and is not started unless explicitly enabled.
Rationale: control-plane traffic must never contend with the data plane, and the admin port can be firewalled to the operations network only. The admin server intentionally does NOT expose channelz or proto reflection. It reuses the data plane's GrpcChannelManager / GrpcClientSettingsManager so online clients are visible to admin queries. Runtime executors introduced by this feature are instance-owned and registered in the Proxy lifecycle: the admin timeout scheduler stops with ProxyAdminGrpcService, and the blocking gateway executor stops with AdminService.
proxy.admin.* resourcesCredentials arrive in the standard gRPC Authorization metadata (same scheme as the data plane, ACL 2.0 signature). Authentication runs as an interceptor, ordered after the standard pipeline‘s HeaderInterceptor (registered before it, since gRPC invokes interceptors in reverse registration order). That ordering is load-bearing: HeaderInterceptor replaces the inbound x-mq-channel-id with the channel id derived from the transport, and authentication results are cached per channel id, so authenticating on the client-supplied value would let a request on one connection reuse another connection’s cached success and skip signature verification.
Every RPC maps to one resource + one action:
| Resource | RPCs | Actions |
|---|---|---|
proxy.admin.client | ListSubscription, ListConsumerConnection (List); DescribeSubscription, DescribeGroupAccumulation, GetConsumerRunningInfo, QueryTimeSpan (Get) | List / Get |
proxy.admin.config | ChangeLogLevel | Update |
proxy.admin.connection | PrintThreadStackTrace, VerifyMessage | Update (high privilege) |
proxy.admin.route | GetTopicRoute | Get |
proxy.admin.ops | GetProxyRuntimeStats, DescribeTopicStatus, QueryMessage (Get); ResetGroupOffset (Update); DeleteSubscription (Delete); AdminSendMessage (Pub) | Get / Update / Delete / Pub |
Modeling note: ACL 2.0 resource types are cluster/namespace/topic/group. The admin resources are modeled as CLUSTER-typed literals with reserved names (resource keys cluster:proxy.admin.<module>), which yields exact least-privilege matching without colliding with real cluster names and without changing the auth core.
Modes:
grpcAdminServerAuthEnable=false → open surface (same semantics as the data plane);proxy.admin.* resource;grpcAdminServerAuthEnable=true → fail-closed. The admin surface must be able to both authenticate the caller and enforce the per-method ACL, so this mode requires the cluster-wide authentication and authorization switches to be on. If either is off every request is refused (authentication off → UNAUTHENTICATED; authorization off → FAILED_PRECONDITION) rather than served without a real permission check. This is deliberate: the ACL 2.0 evaluator is itself gated by the cluster authorization switch, so serving requests with authorization off would silently pass every call, including the destructive ones.Audit: every served RPC writes [PROXY-ADMIN-AUDIT] subject/method/resource/ action/sourceIp to the auth audit logger (satisfies the Console-user + AK + resource + operation audit tuple requirement together with the ACL 2.0 engine's own audit log).
A gRPC client is attached to exactly one proxy at a time, but a dashboard must still see the whole cluster from any single admin endpoint. Two mechanisms make that work without adding scope/peering fields to the protocol:
ListConsumerConnection, ListSubscription, DescribeSubscription) merge the broker-side ConsumerConnection view (which covers remoting clients, and gRPC clients replicated between proxies by HeartbeatSyncer) with this proxy's own ConsumerManager (which covers the gRPC v2 clients attached locally). The result is deduplicated by group + topic, so the answer does not scale with the number of online consumers and does not depend on which proxy is queried.PrintThreadStackTrace, VerifyMessage and GetConsumerRunningInfo need the live telemetry stream that only the owning proxy holds. ProxyAdminForwarder detects a client owned by a peer (its ConsumerManager entry is a RemoteChannel whose getRemoteProxyIp() is the peer‘s localServeAddr) and forwards the whole RPC — original metadata included — to the peer’s admin port, guarding against loops with the x-mq-admin-forwarded header. The peer admin port is assumed uniform across the cluster (the heartbeat sync payload does not carry it); a cluster running heterogeneous admin ports would need that added to the heartbeat record first.GetProxyRuntimeStats intentionally reports the local process (its own connection/producer/consumer counts), so a dashboard that wants per-node stats queries each proxy directly.
| Source | RPCs | Mechanism |
|---|---|---|
| Cluster-visible connections | ListConsumerConnection, ListSubscription, DescribeSubscription | broker-side ConsumerConnection (remoting + peer-synced clients) merged with this proxy's ConsumerManager (local gRPC v2 clients) |
| Client-directed telemetry relay | PrintThreadStackTrace, VerifyMessage, GetConsumerRunningInfo | telemetry command relayed to the owning GrpcClientChannel; forwarded to the peer proxy when the client is remote (D3) |
| Broker gateway (async, multi-broker fan-out) | DescribeTopicStatus, DescribeGroupAccumulation, ResetGroupOffset, QueryMessage, QueryTimeSpan, GetTopicRoute, DeleteSubscription | the proxy's own async AdminService remoting gateway; each broker hop is bounded by its own deadline so a slow/unresponsive broker cannot hang the RPC |
| Proxy producer path | AdminSendMessage | MessagingProcessor.sendMessage (system properties honoured for timer / FIFO messages) |
| Proxy runtime | ChangeLogLevel | relocated logback API on the root logger |
Every broker-facing call goes through the asynchronous AdminService gateway and is fanned out to all relevant brokers concurrently, so the gRPC executor thread is never blocked.
The connection/subscription listings are cluster-visible and include both remoting clients (via the broker-side ConsumerConnection) and gRPC v2 clients (via the proxy's ConsumerManager). The client-directed telemetry RPCs (PrintThreadStackTrace, VerifyMessage, GetConsumerRunningInfo) operate on the gRPC v2 client that holds a proxy telemetry stream, with cross-proxy forwarding; their full running-info payload is bounded by what the telemetry protocol can carry (see §8, honest boundaries).
| Requirement | Delivered as | Notes |
|---|---|---|
| ListClients (filter by group/topic/clientId prefix, paged) | ListConsumerConnection(group[, topic]) | clientId-prefix filter and pagination are not expressible in the upstream contract; documented boundary of this partial delivery |
| DescribeClient (SDK version, subscriptions, heartbeat, auth, Pop progress) | DescribeSubscription (per-client subscriptions) + GetConsumerRunningInfo (subscriptions + running info) | heartbeat/auth-status/Pop-progress fields do not exist in the upstream contract |
| ListClientsByGroup / ListClientsByTopic | ListConsumerConnection(group[, topic]) | fully covered |
| Multi-proxy cluster aggregation | cluster-visible connection data + client-directed RPC forwarding | see D3 |
M2+ items (config hot update, quotas, connection control, Pop/batch diagnostics, route observation streaming) are not part of the upstream Admin contract and are out of scope for this delivery.
The apache/rocketmq-apis repository (main branch, containing apache/rocketmq/v2/admin.proto) is consumed as a git submodule (rocketmq-apis/, see .gitmodules); CI workflows check it out with submodules: true.
rocketmq-proto module generates the Java + gRPC stubs for apache/rocketmq/v2/{definition,service,admin}.proto at build time via protobuf-maven-plugin; its version follows the reactor (${revision}), so no published proto artifact is required to build this repository.//rocketmq-proto:rocketmq-proto builds the same classes through a genrule over the submodule (surfaced as the @rocketmq_apis external repository), using pinned protoc / protoc-gen-grpc-java binaries and the well-known-type protos.| Key | Default | Meaning |
|---|---|---|
grpcAdminServerEnable | false | kill switch; the admin surface is opt-in. Set true to start the admin gRPC server; false (default) = not started |
grpcAdminServerPort | 8088 | dedicated admin gRPC port (<=0 disables) |
grpcAdminServerAuthEnable | false | fail-closed mode; requires cluster authentication and authorization to be enabled (see D2) |
grpcAdminServerRequestTimeoutMillis | 3000 | per-broker-hop deadline for the broker calls an admin RPC fans out to |
The admin surface never fills a field it cannot truthfully supply; the following are answered with an explicit status or left unset rather than faked:
SubscriptionGroupConfig.subscriptionDataSet (the broker only reads it), so there is no persisted per-topic subscription to remove. The RPC returns NOT_FOUND with an explanation and leaves the consumer group and its offsets untouched. Distributions that need this RPC back it with an external subscription store that populates the same field.message_id (unique-key index) and message_key. The subscription / lite_topic / pure time-range scan variants need a paging cursor the open-source broker does not provide and return BAD_REQUEST.in_tps / out_tps unset: the open-source proxy keeps no per-process throughput counter, and reporting 0 would be indistinguishable from an idle proxy.subscriptions: the v2 telemetry protocol has no reply message carrying properties, message_queue_table or consume_status_table. The RPC returns OK with an explanatory status message rather than an empty shell.create_timestamp / tags unset because the broker does not record a topic creation time.Admin service — ListConsumerConnection / DescribeSubscription / GetConsumerRunningInfo / DescribeGroupAccumulation, served cluster-visibly with cross-proxy forwarding, plus the full 16-RPC Admin surface for dashboard integration.| Criterion | Status |
|---|---|
| Design document + stable backward-compatible proto contract | this document + upstream admin.proto (rocketmq-apis main) |
| Client query RPCs merged into the server repo | ProxyAdminGrpcService implements all 16 Admin RPCs on the proxy |
| Independent ACL control, read-only/high-risk separation, least-privilege doc | D2 resources/actions + §11 (Least-Privilege Configuration Guide) below |
| E2E with the dashboard | contract = upstream Admin service, frozen for dashboard integration (cross-repo) |
The admin surface authorizes every RPC against dedicated proxy.admin.* ACL 2.0 resources (see §4 decision D2 above). This section shows the minimum-permission policy for each operational role.
Resources (ACL 2.0 keys; modeled as cluster-typed literals with reserved names):
| Resource key | Protects |
|---|---|
cluster:proxy.admin.client | online client query & client diagnostics |
cluster:proxy.admin.config | proxy runtime log-level change |
cluster:proxy.admin.connection | client-directed telemetry commands: thread stack, verify message (HIGH) |
cluster:proxy.admin.route | topic route view |
cluster:proxy.admin.ops | broker-facing ops: stats/topic status/message query (read) and reset offset / delete subscription / admin send (HIGH) |
Action classes:
Get, ListUpdate, Delete, PubThe server maps every RPC to exactly one (resource, action) pair; granting a read-only action can never authorize a high-privilege RPC.
All commands run against any broker/namesrv of the cluster (ACL 2.0 storage). Create users first:
sh mqadmin createUser -n <namesrv-addr> -u <username> -p <password>
Online clients, subscriptions, accumulation, diagnostics, config/route views.
sh mqadmin updateAcl -n <namesrv-addr> \ -s user:rip2-ro \ -r cluster:proxy.admin.client,cluster:proxy.admin.config,cluster:proxy.admin.route,cluster:proxy.admin.ops \ -a Get,List \ -d Allow
Note: Get,List on proxy.admin.ops covers the read-only broker-facing RPCs; the mutating ops RPCs require Update/Delete/Pub and stay denied.
Role A plus the ability to direct telemetry commands (thread stack / verify message) at a connected client.
sh mqadmin updateAcl -n <namesrv-addr> \ -s user:rip2-oncall \ -r cluster:proxy.admin.connection \ -a Update \ -d Allow # plus the Role A grant above
Offset reset, subscription deletion, admin send, client diagnostics commands.
sh mqadmin updateAcl -n <namesrv-addr> \ -s user:rip2-admin \ -r cluster:proxy.admin.client,cluster:proxy.admin.config,cluster:proxy.admin.connection,cluster:proxy.admin.route,cluster:proxy.admin.ops \ -a Get,List,Update,Delete,Pub \ -d Allow
(Keep Role C accounts to a minimum; every use is recorded in the auth audit log with the [PROXY-ADMIN-AUDIT] prefix.)
Restrict admin access to the operations network via the -i sourceIp option:
sh mqadmin updateAcl -n <namesrv-addr> \ -s user:rip2-ro \ -r cluster:proxy.admin.client \ -a Get,List \ -d Allow \ -i 10.0.0.0/8
By default the admin server follows the cluster-wide authentication/authorization switches (same behavior as the data plane). Fail-closed mode makes the admin surface refuse to serve anything it cannot fully authorize:
# proxy.json / -D grpcAdminServerAuthEnable=true grpcAdminServerAuthEnable: true
With grpcAdminServerAuthEnable=true, the admin surface requires the cluster authentication and authorization switches to be enabled. If authentication is off, requests are rejected with UNAUTHENTICATED; if authorization is off, they are rejected with FAILED_PRECONDITION. This guarantees the proxy.admin.* ACL is genuinely enforced — use it when the admin port cannot be network-isolated.
The surface is off by default (grpcAdminServerEnable=false). To enable it:
grpcAdminServerEnable: true # start the admin gRPC server on grpcAdminServerPort
To keep it disabled (the default), leave grpcAdminServerEnable: false, or set grpcAdminServerPort to 0 / negative — the admin gRPC server is then not started at all.
Every served admin RPC logs subject (Console login user / AK), method, resource, action and source IP to the auth audit logger; denied requests are logged by the ACL 2.0 engine itself. This satisfies the four-tuple audit requirement (Console user + AK + resource + operation).