Multi-tenancy guide

The Spark Connect Gateway can serve many tenants through a single deployment without their workloads interfering with each other. This guide is the operator entry point for multi-tenant setups: pick a deployment shape, find the sample config, follow the links to the deep references.

If you're after a specific topic, the dedicated guides go deeper:

deployment.md — full chart reference, every knob, install/upgrade/uninstall.
observability.md — metrics, audit log schema, log/PromQL examples.
runbook.md — symptom → diagnosis → fix.

What “multi-tenant” means here

A tenant is whatever string the gateway resolves for an inbound RPC. The session-affinity routing key is (tenant, user_id, session_id), so (team-a, alice, sess-1) is a different binding from (team-b, alice, sess-1) — two tenants can reuse session ids without colliding.

What the gateway gives you at the data plane:

Capability	How
Per-tenant identity from auth	`tenantResolver` reads the JWT/static-token tenant claim, a gRPC metadata header, or a fixed string.
Per-tenant backend pools	`tenantPools.overrides` pins a tenant to a dedicated Spark Connect cluster; everything else falls through to a shared default.
Per-tenant quotas	`rateLimit.overrides` sets a token-bucket RPS / burst per tenant; opt-in per-user dimension inside a tenant; per-replica or Redis-shared via `rateLimit.store`.
Per-pool backend credentials	`backendToken` presents a `spark.connect.authenticate.token` bearer to each pool's backends (per-tenant overrides supported), so backends themselves refuse clients that bypass the gateway — see Enforcing the trust boundary.
Tenant-aware audit trail	Every audit event (`session.create`, `auth.failure`, `rpc.error`, …) carries the resolved tenant in a structured field.
Strict-isolation mode	Pair `onMissing=reject` (resolver) with `onUnknownTenant=reject` (pools) so RPCs from unmapped tenants never reach a backend.

What's deliberately not here — see What's not here yet at the end.

The four building blocks

Each knob is documented at length in deployment.md. The summary below is just enough to make the decision guide make sense.

1. `tenantResolver` — where does the tenant string come from?

source is one of from_claim (auth identity), from_metadata (x-tenant header), or always_default (single-tenant). onMissing decides what happens when the source yields nothing — use_default falls back to defaultName, reject returns Unauthenticated. See Multi-tenant: picking a tenant resolver for the full decision table.

2. `tenantPools` — which backends does each tenant use?

backendDiscovery at the top of values.yaml configures the default pool; tenantPools.overrides.<tenant> adds a dedicated pool (static or k8s discovery). tenantPools.onUnknownTenant mirrors the resolver's onMissing policy at the routing layer. See Per-tenant pools.

3. `rateLimit` — protect tenants from each other's bursts

Off by default. Set rateLimit.enabled: true and supply rateLimit.default plus optional rateLimit.overrides.<tenant>. Quota violations surface as RESOURCE_EXHAUSTED and increment scg_rate_limit_rejected_total{tenant, scope}. See Per-tenant rate limiting.

4. `audit` — who did what, when

Emits four event types by default (session.create, session.release, auth.failure, rpc.error) tagged with target=scg::audit so you can split them out in Loki/Splunk. Every event carries the tenant. See Audit logging for chart config and the event schema for fields and sample queries.

Decision guide

Three deployment shapes cover almost everything we've seen.

Shape A: Permissive multi-tenant (shared backends, per-tenant metrics)

You want one Spark Connect cluster shared across many tenants, but you want metrics, audit, and (optionally) quotas labeled per tenant. No tenant-specific pools. New tenants don't need a config change.

Pick this when:

All tenants tolerate sharing the same Spark cluster.
Onboarding a new tenant should be a no-op for the gateway.
You still want per-tenant observability and quotas.

See Sample 1.

Shape B: Strict multi-tenant (pinned pools per tenant)

Each tenant gets its own Spark Connect cluster. The gateway rejects RPCs from any tenant that isn't explicitly mapped — onboarding is a config change. Tokens without a tenant claim are also rejected.

Pick this when:

Tenants must be isolated at the data plane (different Spark clusters per team, regulated tenants, noisy-neighbor risk too high to share).
Operators are okay rolling a chart upgrade per new tenant.

See Sample 2.

Shape C: Single-tenant

You want the gateway's full feature surface (auth, observability, audit) but only have one tenant. Every RPC routes to tenant="default".

Pick this when:

You don't need multi-tenant routing yet.
You want the audit stream + per-tenant metric labels even though there's only one tenant.

See Sample 3 — this is the chart's default, no per-tenant config needed.

Sample configs

Each block below is a complete excerpt for the multi-tenant-relevant sections of values.yaml. Drop them into your existing values file alongside auth / discovery / TLS / etc. — only the multi-tenant knobs are shown here.

Sample 1: Permissive multi-tenant

# JWT-based auth with a tenant claim; clients without one land in
# the "default" bucket.
tenantResolver:
  source: from_claim
  onMissing: use_default
  defaultName: default

# No per-tenant pool overrides — every tenant uses the default
# backend pool. New tenants need zero config.
tenantPools:
  onUnknownTenant: use_default
  overrides: {}

# Per-tenant quotas keep one tenant from monopolizing the shared
# backends. The default applies to every tenant unless overridden.
rateLimit:
  enabled: true
  default:
    rpcsPerSecond: 100
    burst: 200
  overrides:
    team-a:
      rpcsPerSecond: 500
      burst: 1000

audit:
  enabled: true
  logSuccessfulRpcs: false

What you get:

scg_rpcs_total, scg_rate_limit_rejected_total{tenant}, and every audit event carry the resolved tenant.
team-a gets 5× the quota of the default tier; everyone else shares the default.
A token without a tenant claim still works — it routes as tenant="default".

Sample 2: Strict multi-tenant

# JWT must carry a tenant claim. Unauthenticated otherwise.
tenantResolver:
  source: from_claim
  onMissing: reject
  defaultName: default

# Each tenant has its own Spark Connect cluster. Anything not in
# `overrides` is PermissionDenied — operators must register new
# tenants before they can connect.
tenantPools:
  onUnknownTenant: reject
  overrides:
    team-a:
      type: static
      addresses:
        - "spark-team-a-1.svc.cluster.local:15002"
        - "spark-team-a-2.svc.cluster.local:15002"
    team-b:
      type: k8s
      namespace: spark-team-b
      serviceName: spark-connect
      port: 15002

# Per-tenant quotas, optional per-user dimension on the enterprise
# tier to stop one user inside team-c from using its entire quota.
rateLimit:
  enabled: true
  default:
    rpcsPerSecond: 100
    burst: 200
  overrides:
    team-a:
      rpcsPerSecond: 500
      burst: 1000
    team-c:
      rpcsPerSecond: 2000
      burst: 5000
      perUserRpcsPerSecond: 200
      perUserBurst: 400

audit:
  enabled: true
  logSuccessfulRpcs: false

What you get:

A misconfigured client (no tenant claim, or a typo tenant string) fails at the gateway — the request never reaches a backend.
Each tenant‘s pool is health-checked independently when healthCheck.enabled: true; an unhealthy team-a backend doesn’t affect team-b routing.
Adding a new tenant is helm upgrade with a new entry under tenantPools.overrides.

Sample 3: Single-tenant

# Defaults — included only to make the shape explicit. You can omit
# this entire block; the chart defaults already produce this
# behaviour.
tenantResolver:
  source: from_claim
  onMissing: use_default
  defaultName: default

tenantPools:
  onUnknownTenant: use_default
  overrides: {}

rateLimit:
  enabled: false

audit:
  enabled: true
  logSuccessfulRpcs: false

This is the no-multi-tenant-config shape. Every RPC ends up in tenant="default"; pool routing is unchanged; audit captures the four default events; metrics gain the tenant label but it's the same value everywhere.

Migration paths

Adopting the gateway without multi-tenancy

Helm upgrade. No values.yaml changes needed — the chart's defaults preserve single-pool single-tenant behaviour. Watch scg_rpcs_total{code="OK"} stay flat across the rollout.

Permissive → Strict

You're starting from Sample 1 and want to lock down to Sample 2. Order matters because flipping onMissing to reject is client-affecting:

Pool first: add tenantPools.overrides entries for every tenant currently in production. Keep onUnknownTenant: use_default for now — new entries just override the pool, no client gets rejected.
Verify the new pools are healthy: scg_backend_pool_size > 0 for each tenant; kubectl logs shows requests routing to the new pools.
Flip pool policy: tenantPools.onUnknownTenant: reject. Any client whose tenant isn't in your overrides list starts seeing PermissionDenied — make sure the list is complete first. See Tenant getting PermissionDenied with “no configured pool” for the recovery path if you miss one.
Update auth-side: ensure your IdP issues tenant claims for every legitimate client. Roll this change before step 5.
Flip resolver policy: tenantResolver.onMissing: reject. Any client whose token has no tenant claim now gets Unauthenticated — see Clients suddenly getting Unauthenticated after a config change.

Adding a new tenant under Strict mode

Add the tenant under tenantPools.overrides.<name> (and rateLimit.overrides.<name> if it needs a different quota).
helm upgrade — the chart rolls the pods with the new ConfigMap.
Issue tokens with the matching tenant claim.

Without step 1, the new tenant's first RPC hits PermissionDenied: tenant "<name>" has no configured pool and the audit log records rpc.error events with the resolved tenant — exactly the signal you want for “I forgot to register this tenant”.

Verifying isolation

If you want confidence the multi-tenant stack is doing what you expect in your environment, the crates/proxy/tests/multitenant_e2e.rs integration test wires every multi-tenant feature together and asserts isolation along four axes:

Pool + affinity — two tenants using the same session_id bind to different backends and stay there on repeats.
Rate limit — one tenant exhausting its quota leaves the other unaffected; metrics increment only on the offending tenant.
Audit labeling — every audit event carries the resolved tenant from the verified identity, not anything client-forgeable.
Auth-level Reject — onMissing=reject + a tenantless token produces Unauthenticated before the request reaches a backend.

The same four invariants are the right things to assert in your own staging environment. The Go/Python/JVM Spark Connect clients all accept a Bearer token via metadata, so you can drive equivalent assertions from any of them.

Distributed rate limiting

The rate limiter has two backends. Pick by your replica count:

`rateLimit.store`	Where state lives	Effective cluster-wide quota
`memory` (default)	Per gateway replica	`N × default.rpcsPerSecond` for N replicas
`redis`	One Redis instance, shared	Exactly `default.rpcsPerSecond`

The memory backend is fine for single-replica deployments or back-pressure-style limiting where being inside 2× of the configured quota is acceptable. Switch to redis when you need a strict cluster-wide cap — typical SaaS billing-tier enforcement.

Sample redis config:

rateLimit:
  enabled: true
  store: redis
  redis:
    url: "redis://redis.spark-connect.svc:6379"
    keyPrefix: "scg-rl"
    keyTtlSecs: 3600
    onFailure: open      # open | closed
  default:
    rpcsPerSecond: 100
    burst: 200

Fail mode is the question worth thinking about up front:

onFailure: open (default) — when Redis is unreachable, admit the RPC and bump scg_rate_limit_redis_errors_total{tenant, reason}. Matches the Redis affinity-store's fail-soft behaviour: availability over strict quotas. The error metric makes the outage visible so you can alert on a sustained nonzero rate.
onFailure: closed — when Redis is unreachable, reject the RPC with ResourceExhausted. Pick this if a Redis outage must not become a quota-bypass vector (regulated SaaS). Note that this makes Redis a hard request-path dependency — Redis down means every RPC throttled.

Operational notes:

The limiter uses a Lua script via plain EVAL/EVALSHA, not the redis-cell module. It works on any Redis 6+, including managed offerings (ElastiCache, MemoryStore, Upstash) that don't allow loadmodule.
Reuse the same Redis instance as the affinity store — the limiter uses a different keyPrefix (default scg-rl vs. affinity‘s scg) so a FLUSH of one won’t disturb the other. Helm chart defaults assume a shared redis Service.
Bucket keys self-expire after keyTtlSecs so abandoned tenants don't leak Redis memory.

Verify by running two gateway replicas with the same Redis: their combined RPS for one tenant must stay under default.rpcsPerSecond + burst, not double it. The integration test crates/ratelimit/tests/redis_integration.rs covers this case under two_replicas_share_the_bucket.

What's not here yet

The gateway deliberately scopes itself to the data plane. Tenant-related roadmap items, summarised (the project-wide list with planned shapes lives in ROADMAP.md):

Weighted backend selection per tenant — every tenant pool currently uses round-robin / pool-internal load distribution. A weighted scheme for tiered backends inside one tenant is on the roadmap.
Cold-start provisioning — the gateway routes to existing backends; it doesn't ask K8s to create a Spark Connect server on demand for a tenant. Stand up pools out-of-band.
Warm pool per tenant — no pre-provisioning logic.

Multi-tenancy guide

What “multi-tenant” means here

The four building blocks

1. tenantResolver — where does the tenant string come from?

2. tenantPools — which backends does each tenant use?

3. rateLimit — protect tenants from each other's bursts

4. audit — who did what, when

Decision guide

Shape A: Permissive multi-tenant (shared backends, per-tenant metrics)

Shape B: Strict multi-tenant (pinned pools per tenant)

Shape C: Single-tenant

Sample configs

Sample 1: Permissive multi-tenant

Sample 2: Strict multi-tenant

Sample 3: Single-tenant

Migration paths

Adopting the gateway without multi-tenancy

Permissive → Strict

Adding a new tenant under Strict mode

Verifying isolation

Distributed rate limiting

What's not here yet

1. `tenantResolver` — where does the tenant string come from?

2. `tenantPools` — which backends does each tenant use?

3. `rateLimit` — protect tenants from each other's bursts

4. `audit` — who did what, when