iggy-gateway-kafka)Foundation layer for apache/iggy#3421: a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata, CreateTopics, DeleteTopics, DescribeConfigs and AlterConfigs use Iggy. InitProducerId and consumer group coordination work with or without one.
Stub warning: with
IGGY_KAFKA_BRIDGE_ENABLED=true, Produce, Fetch, ListOffsets, Metadata, CreateTopics, DeleteTopics, DescribeConfigs and AlterConfigs use Iggy. With the bridge off (the default), those eight are stubs: Produce, Fetch and ListOffsets answer retriableNOT_LEADER_OR_FOLLOWER(6), CreateTopics, DeleteTopics, DescribeConfigs and AlterConfigs answerNOT_CONTROLLER(41), and Metadata reports every requested topic unknown. CreateTopics, DeleteTopics and AlterConfigs run as the bridge's own Iggy user: with the bridge on andIGGY_KAFKA_SASL_ENABLEDoff (the default), any client that can reach this port can create topics (up to 1000 partitions each), delete topics and their data, or shorten a topic‘sretention.mslow enough that the cleaner drops sealed segments, deleting data. With SASL on,DeleteTopicsandAlterConfigsadditionally require the authenticated principal to holdmanage_topics; without it every resource/name in the request failsTOPIC_AUTHORIZATION_FAILED(29) and nothing is written or deleted.CreateTopicshas no such gate yet - SASL on does not protect it. Every client reads and writes as the bridge’s Iggy user. See docs/SCOPE.md.InitProducerId does real work too, with or without the bridge: it allocates a producer id, so a stock idempotent producer starts instead of failing at startup.
Consumer group coordination is not a stub either:
FindCoordinator,JoinGroup,Heartbeat,LeaveGroup,SyncGroup,DescribeGroupsandListGroupsare real, with real membership, rebalances, graceful leave, session expiry and admin views of that membership (docs/CONSUMER_GROUPS.md).kafka-consumer-groups.sh --list,--describe --membersand--describe --stateuse those views. The default--describecalls OffsetFetch (API key 9), which is not implemented, so that mode does not work. With the bridge off, Metadata reports every topic unknown, so a consumer joins a group and is assigned 0 partitions. With the bridge on, OffsetCommit and OffsetFetch keep each group‘s offsets in Iggy, so a group resumes from its last commit, also after a gateway restart (docs/OFFSET_STORAGE.md). Every Iggy server must run a build with the external group consumer kind (#4392) before the first commit. In a partly upgraded cluster, one commit can make a partition stop committing. In a cluster, a node that has not applied a group’s first commit answers “no offset”, so a consumer onauto.offset.reset=latestcan skip records until #4409.
cargo run -p iggy-gateway-kafka
Default bind: 127.0.0.1:9093. Environment variables:
| Variable | Default | Description |
|---|---|---|
IGGY_KAFKA_BIND_ADDR | 127.0.0.1:9093 | TCP address to listen on |
IGGY_KAFKA_ADVERTISED_HOST | bind IP | Hostname/IP clients use to reach this broker (required when binding to 0.0.0.0/::) |
IGGY_KAFKA_ADVERTISED_PORT | bind port | Port advertised in Metadata responses |
IGGY_KAFKA_MAX_CONNECTIONS | 1024 | Maximum concurrent connections before new ones are rejected |
IGGY_KAFKA_MAX_FRAME_SIZE | 8388608 | Maximum accepted request frame size in bytes |
IGGY_KAFKA_IDLE_TIMEOUT_SECS | 600 | Seconds a connection may sit idle before the next frame's length prefix arrives |
IGGY_KAFKA_READ_TIMEOUT_SECS | 15 | Seconds allowed to read a frame body once its length prefix arrives |
IGGY_KAFKA_WRITE_TIMEOUT_SECS | 10 | Seconds allowed to write a response frame |
IGGY_KAFKA_SHUTDOWN_DRAIN_TIMEOUT_SECS | 25 | Seconds graceful shutdown waits for in-flight connections before abandoning them. The bridge then signs its Iggy clients out, 5 s at most |
IGGY_KAFKA_INSTANCE_ID | 0 | This gateway's number among the gateways fronting one Iggy cluster. It is the high half of every producer id InitProducerId hands out, and Kafka requires those to be cluster-unique, so give every gateway its own value. A single gateway can leave it at 0. |
IGGY_KAFKA_BRIDGE_ENABLED | false | Connect the Iggy bridge at startup. While false every API answers with its stub, and the IGGY_KAFKA_IGGY_* variables below are read by nothing. A failed connection is fatal, not a downgrade to stubs. |
IGGY_KAFKA_SASL_ENABLED | false | Require SASL/PLAIN authentication before serving any other API (true or false, nothing else) |
IGGY_KAFKA_PRE_AUTH_TIMEOUT_SECS | 15 | Seconds an unauthenticated connection may sit between frames. Waiting for an authentication slot and the verification itself each get this budget, the verification's starting once it holds a slot. Separate from the 10-minute idle timeout that applies once authenticated |
IGGY_KAFKA_MAX_CONCURRENT_AUTHENTICATIONS | 4 | Credential verifications the gateway runs at once, across all connections. Each costs a password hash on an Iggy shard thread. This bounds the gateway‘s side only: a check that times out frees its slot while its hash keeps running inside Iggy. Size it below the Iggy node’s shard count |
The fastest way to see a Kafka client talk to Iggy: two containers, no local Rust toolchain to build them. (Reading the record back below uses docker compose exec, not a local build either.)
Kafka wire (TCP 9093) Iggy wire (TCP 8090) kcat / librdkafka ─────────────────────▶ gateway ─────────────────────▶ iggy-server (your Kafka client) iggy-gateway-kafka (real broker)
cd gateways/kafka docker compose up --build
iggy-server pulls the prebuilt apache/iggy:edge image (refreshed on a master push only when server or iggy-cli is affected, or web or core/server/Dockerfile changes), so only iggy-gateway-kafka compiles from source. Its Dockerfile mounts a cargo registry/target cache, so a first run costs one real compile and every run after reuses it, even across a source change elsewhere in the workspace. Testing a local iggy-server change: comment out image: and uncomment build: in docker-compose.yml first. That build mounts cargo registry, git, and target caches, so the first run is still a full release build of several minutes and later runs reuse those mounts.
This starts iggy-server (the real Iggy broker) and iggy-gateway-kafka (bridge enabled, pointed at that server) on a shared Docker network, with the gateway waiting for the server's health check before it starts. From another terminal:
# Broker discovery works with just kcat - it has no create-topic mode of its own, though. kcat -b 127.0.0.1:9093 -L
A full create-topic-then-produce flow needs a client that can send CreateTopics (there is no auto-create on Produce). With Python‘s kafka-python (illustrative - not exercised by an automated test, so adjust if the library’s API has moved since this was written):
from kafka.admin import KafkaAdminClient, NewTopic from kafka import KafkaProducer admin = KafkaAdminClient(bootstrap_servers="127.0.0.1:9093") admin.create_topics([NewTopic(name="orders", num_partitions=1, replication_factor=1)]) producer = KafkaProducer(bootstrap_servers="127.0.0.1:9093") producer.send("orders", b"hello from kafka-python").get(timeout=5)
Read the record back with a real Kafka consumer - Fetch is wired to Iggy with the bridge on (see the stub warning above), so kcat reads it the same way it would against a real broker:
kcat -b 127.0.0.1:9093 -C -t orders -o beginning -e
-e exits once kcat catches up rather than waiting for more. No consumer group needed for a one-shot read; see docs/CONSUMER_GROUPS.md for group-based consuming. Polling Iggy directly still works too, if you want to see the record from the other side:
docker compose exec iggy-server /usr/local/bin/iggy --username iggy \ --password iggy-gateway-quickstart message poll kafka orders 0 --offset 0 --message-count 1
Ports: gateway 9093 (Kafka wire, the one a client dials), Iggy 8090 (TCP, iggy CLI/SDK), 3000/8080/8092 (Iggy HTTP/QUIC/WebSocket, unused by the gateway itself but published for direct Iggy inspection).
Config: the compose file sets fixed dev-only root credentials (IGGY_ROOT_PASSWORD) so the gateway's IGGY_KAFKA_IGGY_PASSWORD is known ahead of time - never reuse them outside a local stack. See gateways/kafka/docker-compose.yml and the environment variable tables below for every other knob.
Limitations (see the stub warning above for the full list): single gateway, single Iggy node, SASL off by default. This is a development quick start, not a production deployment shape. If your local apache/iggy:edge predates #4392, run docker compose pull, because that server refuses every offset commit.
This quick start‘s topics use Iggy’s default Durability::Replicated - a Produce ack is a quorum commit, not an fsync. A docker kill or host power loss can lose acked records despite the named Docker volume; the volume protects a clean docker compose down/up, not a crash. Neither this quick start nor phase1_e2e_tests.rs exercises crash durability - the automated test and the manual read-back above both only prove a produced record is readable from the still-live process that wrote it. docs/MANUAL_TESTING.md's Category I covers a restart (a graceful docker compose restart, which flushes on the way down) as a smoke test for “does the volume actually persist across a container restart” - it is not a crash test and does not contradict this limitation.
Success criteria (tracked for #3539): a new contributor should reach a successful produce (and an Iggy-side read-back confirming it landed) within 15 minutes of git clone, and every Phase 1 API key (CreateTopics, Metadata, Produce, ListOffsets) has an automated test exercising the real compiled binaries end to end - tests/phase1_e2e_tests.rs is that test, spawning real iggy-server and iggy-gateway-kafka processes (not KafkaGateway::run in-process, as every other suite in this crate does) and driving CreateTopics → Metadata → Produce → ListOffsets over real TCP, then reading the produced records back through the Iggy SDK.
cargo test -p iggy-gateway-kafka
See docs/TEST_SUITE.md for the full suite catalog (cargo test -p iggy-gateway-kafka -- --list for the exact current test names - the count has drifted out of sync with the actual suites before, so it isn't pinned here).
Some api_handler_tests, server_e2e_tests, and version_firewall_tests cases require wire fixtures under tools/kafka-tool/kafka_messages/ (gitignored locally; CI generates them via scripts/ci-wire-fixtures.sh):
./gateways/kafka/scripts/ci-wire-fixtures.sh generate cargo test -p iggy-gateway-kafka ./gateways/kafka/scripts/ci-wire-fixtures.sh cleanup # optional
Or generate only the keys the tests need:
for key in 0 1 2 10 11 12 13 14 19 22; do cargo run -p kafka-message-gen -- generate \ --output gateways/kafka/tools/kafka-tool/kafka_messages \ --api-key "$key" done
Before check-in, run the procedure in docs/MANUAL_TESTING.md (smoke, version firewall, kcat, adversarial cases).
See docs/SCOPE.md for #3421 deliverables, supported API key/version table, and post-foundation TODO backlog.
retention.msDelivery through this gateway is at-least-once, and stays at-least-once across a gateway restart. Transactions are not supported, and will not be. An idempotent Kafka producer is given a producer id so that it starts, but its retries are not deduplicated: Produce uses the producer id only to pick the producer's Iggy session, and ignores the epoch and sequence, so a retry after a network timeout writes the record twice, and both copies reach the stream with their own offsets.
Iggy deduplicates writes on its own partition plane, and that does not close this gap, because it guards the hop from the gateway to Iggy rather than the hop from the producer to the gateway. docs/IDEMPOTENCE.md has the detail and what closing it needs.
Transactions are not supported, and will not be. AddPartitionsToTxn (24), AddOffsetsToTxn (25), EndTxn (26) and TxnOffsetCommit (28) are never advertised, so a conforming client never sends one; InitProducerId with a transactional_id answers UNSUPPORTED_VERSION (35); and a Produce request carrying a transactional_id gets UNSUPPORTED_VERSION (35) on every partition rather than having its records stored as if they were ordinary ones. None of those closes the connection. docs/SCOPE.md's Transactions section has the ordering and the reasoning.
A stock consumer with a group.id joins, takes the partitions its own assignor hands it, commits, and resumes from its commit. With the bridge on, commits live in Iggy, so a group also resumes after a gateway restart. tests/kafka_client_e2e_tests.rs checks both with the Java consumer, and a member that leaves hands its partition to the other at its commit.
Limits:
group.protocol=consumer (KIP-848) fails.group.instance.id is accepted but not honoured. A static member that restarts causes two rebalances.kafka-consumer-groups.sh cannot list or describe a group.Off by default. With IGGY_KAFKA_SASL_ENABLED=true the gateway requires SASL/PLAIN before it serves any other API, and it verifies the credentials by logging into Iggy with them.
The username and password a Kafka client sends are an Iggy username and password. There is no mapping table and no credential store in the gateway: create an Iggy user for each Kafka principal and point the client at it. Credentials are verified against IGGY_KAFKA_IGGY_ADDR.
IGGY_KAFKA_SASL_ENABLED=true IGGY_KAFKA_IGGY_ADDR=127.0.0.1:8090 cargo run -p iggy-gateway-kafka
Transport security to Iggy is configured separately from the Kafka side, because the two protect different hops. These variables cover only the connection the credential check makes, and are refused while SASL is off. The bridge's own client connects without TLS at any setting, and that connection carries IGGY_KAFKA_IGGY_PASSWORD in the clear:
| Variable | Default | Description |
|---|---|---|
IGGY_KAFKA_IGGY_TLS_ENABLED | false | Encrypt the credential check's connection to Iggy (true or false, nothing else). The bridge connection is not covered. Required if the Iggy server only accepts TLS, otherwise every verification fails as unreachable |
IGGY_KAFKA_IGGY_TLS_DOMAIN | derived from the address | Name checked against the Iggy server certificate |
IGGY_KAFKA_IGGY_TLS_CA_FILE | SDK bundled roots | PEM roots to trust. Note the SDK does not use the system trust store |
Client side, for example with kcat:
kcat -b 127.0.0.1:9093 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=PLAIN \ -X sasl.username=alice -X sasl.password=s3cret -L
Four things to know before switching it on:
ApiVersions advertisement only while it is on, and unauthenticated clients are refused.IGGY_KAFKA_MAX_CONCURRENT_AUTHENTICATIONS bounds the checks the gateway runs at once, and a peer whose login was rejected is refused for a delay that doubles per rejection, from 0.5s up to 30s.DescribeAcls renders the authenticated principal's Iggy permissions as Kafka ACL bindings, so kafka-acls.sh --list works against the gateway. It is read only: CreateAcls and DeleteAcls are not implemented and not advertised.
--list here is a snapshot of the caller‘s own grants, not the broker-wide dump it is against a Kafka cluster. A principal sees its own permissions and nobody else’s, because the gateway holds no administrative credentials, so filtering on another User: returns an empty listing whatever that user holds, and root‘s listing is root’s grants rather than a catalog of every binding. Only global permissions are rendered, as wildcard bindings, and the view is a snapshot taken when the connection authenticated, so a permission changed afterwards is invisible until the client reconnects. docs/ACL_MAPPING.md has the mapping table and what is deliberately left out.
Full reasoning, including what was rejected and why, is in docs/AUTHENTICATION.md.
src/bridge/ is the SDK integration layer: connects to Iggy, maps Kafka topics to Iggy streams/topics, provisions them on demand, looks up high watermarks (one or many partitions of a topic per call) for ListOffsets, probes and polls partitions for Fetch, and stores and reads group offsets for OffsetCommit and OffsetFetch. Produce (#3535), Fetch (#3536), ListOffsets (#3537), Metadata (#3534), CreateTopics (#3538), DescribeConfigs and AlterConfigs , OffsetCommit and OffsetFetchcall it. Topic configuration behavior is in docs/CONFIGS.md. (#3542) call it. Tested by bridge's own unit tests, tests/bridge_iggy_integration_tests.rs and the tests/*_real_bridge_tests.rs suites. All but the unit tests start a real iggy-server.
One partition, one Iggy send. Each partition answers for itself.
| Field | Gateway |
|---|---|
| Partition | Index as sent. Both count from 0. |
| Base offset | From the send confirmation. -1 if none. |
log_start_offset | Always -1. |
acks | 0, 1, -1 write the same. Other values: 21, before any per-partition check. |
acks=0 | Writes, answers nothing. Any failed partition closes the connection. |
| Topics | Creates none. Missing: 3. Bad name: 17. |
timeout_ms | Honored, max 20 s. Past it: 7. |
| Compression | gzip, snappy, lz4. zstd from v7, else 76. |
| Producer id | Picks the producer's Iggy session. See Concurrency ceiling. |
| Producer epoch, sequence | Ignored, so a retry writes twice. |
| Timestamps over about 71 min apart | Clamped into one send. kafka.ts keeps the real one. |
| Several batches in one partition | 87, as Kafka. |
| Bytes after the batch | 87. |
| More records than the batch declares | 87, as Kafka. |
| Same partition twice in one request | Written twice. Kafka keeps the last. |
| Repeated header name in a record | 87. |
Produce v0-2, acks=0 | Closes the connection. |
| Undecodable request | Closes the connection. |
| Code | When | Client |
|---|---|---|
| 10 | Record, send or partition too large, even alone | Java splits multi-record batches. Else fails. |
| 87 | Record the gateway cannot map. Reason in error_message from v8. | Fails. |
| 35 | Transactional or control batch | Fails. |
| 6 | Request budget ran out. Nothing written. | Retries. |
| 7 | Deadline passed: in the send, waiting for decode memory, or behind the producer's own earlier send. Or connection lost mid-send. May be written. | Retries. Can duplicate. |
Partition cap: max_frame_size decompressed bytes, max_frame_size / 64 record slots, 3 headers per slot. Past it: 10.
Request budget: 8 partition caps, refused partitions included. Past it: 6 for the rest, not decoded.
Decode memory: 5 partition caps across all requests, counted in bytes. A request holds what its largest partition may decode: its bytes or 64 per record slot, whichever is more, or a whole cap when compressed. One holding 64 KiB or more first takes one of 4 large slots, and holds no memory while it waits for one. So large requests hold 4 caps at most, and leave 1 to the small ones.
A request waits for its producer's turn first, and holds no slot and no memory while it does. Past its deadline waiting for any of the three: 7. A send that outlives its request keeps its slot and its memory until it ends, 45 s at most.
docs/BRIDGE_MAPPING.md describes what a record becomes once it is stored.
Per request: one probe per topic, shared with other Fetches, then polls of each partition with records, in request order. One uncompressed batch per partition.
| Field | Gateway |
|---|---|
| Partition, offset | As sent. Both systems count from 0. |
| Iggy replica | The one on the node each bridge client is on. Every client signs in at the metadata leader, and a view change or a refused call can move it. So in a cluster, a probe and a poll can read nodes that are not at the same offset. |
| Offset below the oldest kept one | Reads from the oldest kept one. If none is kept, no records until the next write, because a cluster replica that lost records after a crash reads the same. Kafka answers 1 here, so auto.offset.reset=none gets no error. |
| Recreated topic | Iggy starts each partition at offset 0 again, and Fetch cannot see it. Seek consumers to 0 after you recreate a topic. Otherwise writes past a consumer's old offset make it skip the records below. |
high_watermark | One past the last committed offset. |
last_stable_offset | Same as high_watermark. No transactions. |
log_start_offset | Always -1. |
| Sessions | None opened. session_id is always 0, so every request is a full one. |
max_wait_ms | Honored, max 18 s. |
min_bytes | Honored, as Kafka does: the request waits until the records read reach it. New bytes are estimated from Iggy's average message size. |
max_bytes | Honored, max IGGY_KAFKA_MAX_FRAME_SIZE. |
partition_max_bytes | Honored. |
| First partition with records | At least one record, even past every byte limit (KIP-74). |
| Records per partition | At most 1000 per request, polled up to 16 at a time. |
isolation_level, replica id, epochs, rack, forgotten topics | Ignored. |
| Code | When | Java consumer |
|---|---|---|
OFFSET_OUT_OF_RANGE (1) | Offset below 0. Or an offset still past high_watermark after 30 s of 6 on the same connection, counted from its first 6 there. A read there in range starts the 30 s over. A node that lags clears within them. A partition with no offset past 0 never answers 1, because Iggy reports that while it loads, fences or rebuilds one. | Resets its offset (auto.offset.reset). |
UNKNOWN_TOPIC_OR_PARTITION (3) | Topic or partition not in Iggy, or a name Kafka refuses | Refreshes metadata, retries. |
NOT_LEADER_OR_FOLLOWER (6) | Iggy unreachable, too slow, not signed in, loading the partition, returning no records where it reports some, or an offset past the end (see 1). Alone, it goes out after max_wait_ms. | Refreshes metadata, retries. |
TOPIC_AUTHORIZATION_FAILED (29) | The bridge's Iggy user lacks permission. Fetch reads as that user, not as the client. | Fails. |
UNKNOWN_SERVER_ERROR (-1) | Anything else, such as a stored message the gateway cannot map. Alone, it goes out after max_wait_ms. Each connection reads the partition again at most once per max_wait_ms. | Retries the same offset. |
FETCH_SESSION_ID_NOT_FOUND (70) | Top level. The request continues a session. | Sends a full request. |
The Java consumer throws on any other partition code, so Fetch sends none.
Limits:
0 with no records.max_bytes each, or one larger record.warn.max_bytes gets no poll, unless the request still owes its first record.ListOffsets answers 6 for it all along, and warns too.| Variable | Default | Description |
|---|---|---|
IGGY_KAFKA_IGGY_ADDR | 127.0.0.1:8090 | Address of the Iggy server to bridge to |
IGGY_KAFKA_IGGY_USERNAME | iggy | Iggy username |
IGGY_KAFKA_IGGY_PASSWORD | none - required | Iggy password. No default: iggy-server only uses the well-known iggy/iggy root credentials when started with --with-default-root-credentials (dev-only); otherwise it generates a random password, so a hardcoded default here could never be right and would invite running as root unnoticed |
IGGY_KAFKA_IGGY_STREAM | kafka | Default Iggy stream for a Kafka topic with no explicit mapping override |
IGGY_KAFKA_TOPIC_MAP_PATH | unset | Path to a topic-mapping TOML file (see below); omit to use only the default rule |
IGGY_KAFKA_IGGY_MAX_MESSAGE_SIZE | 64MiB | Iggy's message_bus.max_message_size. Set both together. Larger partitions answer 10 |
IGGY_KAFKA_PRODUCE_SESSIONS | 256 | Producers that get an Iggy client of their own. The rest share one. 0: all share one |
IGGY_KAFKA_PRODUCE_SESSION_IDLE_SECS | 600 | An unused Produce client closes after this long |
The initial connect retries a fixed, bounded number of times (RECONNECTION_RETRIES = 3, not the Iggy SDK client's own default of unlimited retries, one dial per second, forever), and the whole attempt - retries included - is capped at REQUEST_TIMEOUT (15s) wall-clock, so IggyBridge::connect fails in bounded time whether the address refuses the connection or silently drops it, instead of blocking the calling task indefinitely. Every other bridge call (ensure_stream_and_topic, high_watermark(s), probe, close) carries the same REQUEST_TIMEOUT for the same reason: the SDK reconnects internally, mid-call, on a transport error, through the same undead-lined dial path - a bridge call made well after the initial connect can still hit this if Iggy becomes unreachable later.
send_records uses the Produce deadline instead:
Default rule, no config file needed: a Kafka topic orders maps to Iggy stream IGGY_KAFKA_IGGY_STREAM (default kafka), topic orders - the Kafka topic name carries over unchanged. Override specific topics with a TOML file:
default_stream = "kafka" [topics.orders] stream = "billing" topic = "orders_v2" # A Kafka topic name containing dots needs the key quoted, or TOML parses it as nested # tables ([topics.org] containing [apache] containing [kafka]) instead of one topic named # "org.apache.kafka.events". [topics."org.apache.kafka.events"] stream = "billing" topic = "kafka_events"
Point IGGY_KAFKA_TOPIC_MAP_PATH at the file to load it; topics not listed under [topics.*] still fall back to the default rule.
default_stream is required in a map file - it has no #[serde(default)], unlike topics - so an override-only file with no default_stream key fails to load rather than falling back to kafka. When both IGGY_KAFKA_TOPIC_MAP_PATH and IGGY_KAFKA_IGGY_STREAM are set, the file's own default_stream always wins and the env var is ignored entirely: a TOML file is a complete mapping document, not an overlay on top of the env var.
ensure_stream_and_topic(kafka_topic, partition_count) creates the mapped Iggy stream and topic if either is missing. Idempotent when repeated with the same partition_count: a no-op if both already exist with that count, and a NameAlreadyExists race against a concurrent caller creating the same stream/topic is treated as success, not an error - the goal is “it exists,” not “this call created it.” A different partition_count against an already-existing topic returns BridgeError::PartitionCountMismatch rather than silently keeping the old count or growing it - two concurrent callers requesting different counts for the same topic must not both see success.
Topics created this way have no message expiry - Iggy‘s own server default, not Kafka’s 7-day default. AlterConfigs can set that expiry afterwards through retention.ms (docs/CONFIGS.md). Until a client does, messages never expire, so a Kafka app that assumes the 7-day default will accumulate data on this bridge.
They also use Iggy‘s default durability, Durability::Replicated - quorum commit without an additional stable-storage barrier, with the disk write itself threshold-gated (flushed at 1024 messages or 1 MiB of unflushed data, whichever comes first). Kafka’s own defaults take the same posture, so this isn't a wrong choice, but on a single node both can lose an acked write to a power cut before that threshold is reached - worth knowing rather than discovering later.
IggyClient of its own, one request at a time, so its batches land in order. Different producers send at once.IGGY_KAFKA_PRODUCE_SESSIONS (256) of these clients are open. A full pool closes its least recently used free client for a new producer, if that client sat unused for 30 s. Otherwise the producer shares 1 more client, one request at a time, and moves to its own at a later request, once there is room and none of its requests runs.IGGY_KAFKA_MAX_CONNECTIONS (1024) allows more producers than sessions, so raise the session limit to give each its own.metadata.clients_table_max (8192), and a slot of partition.dedup_clients_max (4096) in each partition it writes.IGGY_KAFKA_PRODUCE_SESSION_IDLE_SECS (600) unused, checked every 30 s.IGGY_KAFKA_PRODUCE_SESSIONS=0: all producers share one client, one request at a time.max.in.flight.requests.per.connection=1, or retries=0. Otherwise a retried batch can land after later ones.IGGY_KAFKA_PRODUCE_SESSIONS=0.Fetches share topic probes: one get_topic per topic at a time, for all consumers together, on the probe client. A caught-up consumer of a busy topic can start one per Fetch. A waiting Fetch costs about 10 get_topic calls per second per topic. ListOffsets probes on that client too.
BridgeError::to_kafka_error_code() maps Iggy failures to Kafka wire error codes:
UNKNOWN_TOPIC_OR_PARTITION (3). This includes the generic ResourceNotFound that a partition request returns when the server cannot resolve itUnauthorized) → TOPIC_AUTHORIZATION_FAILED (29) - a real, fixable-by-the-Kafka-operator ACL problemIGGY_KAFKA_IGGY_USERNAME/_PASSWORD are wrong) → UNKNOWN_SERVER_ERROR (-1), deliberately not 29 - the Kafka client can’t fix a bridge-side credential misconfiguration, and blaming its own ACLs for one is worse than an unexplained fatal errorNOT_LEADER_OR_FOLLOWER (6). On a send, Disconnected, EmptyResponse, TcpError and StaleClient → 7 instead: the write may have landedTransientNotCommitted) → REQUEST_TIMED_OUT (7) - retriable in real Kafka too, chosen because it's what a real broker sends for the same shape of failure, not to make a client stop retryingBridgeError::Timeout) → REQUEST_TIMED_OUT (7), the same code and the same reasoning as TransientNotCommitted above - the SDK‘s write/read run on a task this timeout cannot abort, so the outcome is unknown, not known-safe-to-retry (see Connection config’s timeout caveat above)INVALID_TOPIC_EXCEPTION (17), checked before any Iggy call is madePartitionCountMismatch → TOPIC_ALREADY_EXISTS (36, not INVALID_PARTITIONS - that code's own text is “below 1”, a different condition)TooManyPartitions) → INVALID_PARTITIONS (37), reachable through ensure_topic‘s partition_count argument once it exceeds the server’s capPartitionsLimitReached, 2022) → POLICY_VIOLATION (44). See metadata.partitions_max belowUNKNOWN_SERVER_ERROR (-1)Fetch folds these into the codes a consumer handles: 7 → 6, 17 → 3, and any other code → -1. OffsetCommit and OffsetFetch fold 6, 7 and RequestTooOld into COORDINATOR_LOAD_IN_PROGRESS (14), which the Java client retries in both (to_offset_error_code).
These are Iggy server limits, not gateway settings. A Kafka client cannot act on any of them, so an operator has to.
| Limit | Default | Where |
|---|---|---|
| Consumer offset keys per partition, per consumer kind | 4096, ceiling 262144 | partition.consumer_offsets_max |
| Partitions of one node, all streams and topics | 0, no cap | metadata.partitions_max |
| One user header name, and one header value | 255 bytes | fixed, user_headers.rs |
| All user headers of one message | 100 KB | fixed, MAX_USER_HEADERS_SIZE |
| Message payload | 64 MB | fixed, MAX_PAYLOAD_SIZE |
Only the first two are configurable. A Kafka consumer group commits one offset key per partition it holds, so partition.consumer_offsets_max is what bounds the number of groups that can commit against one partition. Passing it returns TooManyConsumerOffsets (3024), which reaches the client as UNKNOWN_SERVER_ERROR because Kafka has no code for the condition. The gateway logs the real Iggy error, so the server log is where an operator diagnoses it.
A CreateTopics that would take the node past metadata.partitions_max returns PartitionsLimitReached (2022), which reaches the client as POLICY_VIOLATION (44). The cap counts committed partitions only, so creates that run at the same time can go past it.
The other three decide when a Kafka record goes into the envelope instead of being stored natively. See docs/OFFSET_STORAGE.md and docs/BRIDGE_MAPPING.md.