Kafka Docker Image Examples

Introduction

This document contains examples for the Docker image. Docker Compose files are provided in this directory for the example use cases.

For the usage guide of the Docker image (default configs, file input, environment variables, SASL and SSL modes), see the Getting Started > Docker page of the Kafka documentation on the website (source: docs/getting-started/docker.md).

Examples

docker-compose-files directory contains Docker Compose files for some example configs to run apache/kafka OR apache/kafka-native Docker image.
Pass the IMAGE variable with the Docker Compose file to specify which Docker image to use for bringing up the containers.

# to bring up containers using apache/kafka docker image
IMAGE=apache/kafka:latest <docker compose command>

# to bring up containers using apache/kafka-native docker image
IMAGE=apache/kafka-native:latest <docker compose command>

Run the commands from root of the repository.
Checkout single-node examples for quick small examples to play around with.
cluster contains multi node examples, for combined mode as well as isolated mode.
Kafka server running on Docker container can be accessed using cli scripts or your own client code.
Make sure jars are built, if you decide to use cli scripts of this repo.

Single Node

These examples are for understanding various ways inputs can be provided and kafka can be configured in Docker container.
Examples are present inside docker-compose-files/single-node directory.
Plaintext:
- This is the simplest compose file.
- We are using environment variables purely for providing configs.
- KAFKA_LISTENERS is getting supplied. But if it was not provided, defaulting would have kicked in and we would have used KAFKA_ADVERTISED_LISTENERS to generate KAFKA_LISTENERS, by replacing the host with 0.0.0.0.
- Note that we have provided a CLUSTER_ID, but it's not mandatory as there is a default cluster id present in container.
- We had to provide KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR and set it explicitly to 1, because if we don't provide it default value provided by kafka will be taken which is 3.
- We have also set hostname of the container. It can be kept same as the container name for clarity.
- To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up
```
- To produce messages using client scripts:
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
```

SSL:

Note that here we are using environment variables to pass configs.
Notice how secrets folder is mounted to docker container.
In case of environment variable it is mandatory to keep the files in /etc/kafka/secrets folder in docker container, given that the path of the files will be derived from that, as we are just providing file names in other SSL configs.
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up

To produce messages using client scripts (Ensure that java version >= 17):

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties

File Input:

Here ssl configs are provided via file input.
Notice that now full file path is needed for the configs.
Note that there is extra volume mount now.
Configs provided via environment variable will override the file input configs.
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up

To produce messages using client scripts (Ensure that java version >= 17):

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties

SASL_PLAINTEXT:

Here we use SASL/PLAIN mechanism for authentication.
A JAAS configuration file is mounted into the container via a volume mount.
KAFKA_OPTS is set to point to the JAAS config file.
Similar to the plaintext example, two listeners are configured: one for inter-broker communication and one for client-to-broker communication. Both use the SASL_PLAINTEXT security protocol.
Two users are configured in the JAAS file: admin (for inter-broker) and alice (for clients).
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml up

To produce messages using client scripts (Ensure that java version >= 17):

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties

Multi Node Cluster

These examples are for real world use cases where multiple nodes of kafka are required.
Combined:
- Examples are present in docker-compose-files/cluster/combined directory.
- Plaintext:
  - Each broker must expose a unique port to host machine.
    - For example broker-1, broker2 and broker3 are listening on port 9092, they're exposing it to the host via ports 29092, 39092 and 49092 respectively.
  - Here important thing to note is that to ensure that kafka brokers are accessible both to clients as well as to each other we have introduced an additional listener.
  - PLAINTEXT is supposed to be listener accessible to other brokers.
    - The inter broker listener advertised by the brokers is exposed on container's hostname. This is done so that brokers can find each other in Docker network.
  - PLAINTEXT_HOST is supposed to be listener accessible to the clients.
    - The port advertised for host machine is done on localhost, as this is the hostname (in this example) that client will use to connect with kafka running inside Docker container.
  - Here we take advantage of hostname set for each broker and set the listener accordingly.
  - To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up
```
  - To access using client script:
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092
```
- SSL:
  - Similar to Plaintext example, for inter broker communication in SSL mode, SSL-INTERNAL is required and for client to broker communication, SSL is required.
  - Note that KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM is set to empty as hostname was not set in credentials. This won't be needed in production use cases.
  - Also note that for example we have used the same credentials for all brokers. Make sure each broker has it's own secrets.
  - To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up
```
  - To produce messages using client scripts (Ensure that java version >= 17):
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties
```
- SASL_PLAINTEXT:
  - Similar to Plaintext example, with SASL/PLAIN authentication added.
  - Similar to Plaintext example, two listeners are configured for inter-broker and client-to-broker communication, both using the SASL_PLAINTEXT security protocol. Controllers use PLAINTEXT.
  - Each broker mounts the same JAAS config file. In production, each broker should have its own credentials.
  - To run the example:
```
# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml up
```
  - To produce messages using client scripts (Ensure that java version >= 17):
```
# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties
```

Isolated:

Examples are present in docker-compose-files/cluster/isolated directory.

Plaintext:

Here controllers and brokers are configured separately.
It's a good practice to define that brokers depend on controllers.
In this case also we have same listeners setup as mentioned in combined case.
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up

To access using client script:

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092

SSL:

Pretty much same as combined example, with controllers and brokers separated now.
Note that SSL-INTERNAL is only for inter broker communication and controllers are using PLAINTEXT.
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up

To produce messages using client scripts (Ensure that java version >= 17):

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties

SASL_PLAINTEXT:

Same as combined SASL_PLAINTEXT example, with controllers and brokers separated.
SASL_PLAINTEXT is only for inter-broker and client communication. Controllers use PLAINTEXT.
To run the example:

# Run from root of the repo

# JVM based Apache Kafka Docker Image
$ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml up

# GraalVM based Native Apache Kafka Docker Image
$ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml up

To produce messages using client scripts (Ensure that java version >= 17):

# Run from root of the repo
$ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties

Note that the examples are meant to be tried one at a time, make sure you close an example server before trying out the other to avoid conflicts.