|  | Kafka Docker Image Usage Guide | 
|  | ============================== | 
|  |  | 
|  | Introduction | 
|  | ------------ | 
|  |  | 
|  | This document contains usage guide as well as examples for Docker image. | 
|  | Docker Compose files are provided in this directory for the example use cases. | 
|  |  | 
|  | Kafka server can be started using following ways: | 
|  |  | 
|  | - Default configs | 
|  | - File input | 
|  | - Environment variables | 
|  |  | 
|  | Installation Preparation | 
|  | ------------ | 
|  |  | 
|  | Note that the `Docker` version **must be >= 20.10.4**. | 
|  |  | 
|  | The prior Docker versions may cause permission errors when running the Kafka container, as they do not correctly set directory permissions when creating container paths like `/opt/kafka/config`. | 
|  |  | 
|  | If you are using the prior version, you may encounter the following error during container startup: | 
|  | ```text | 
|  | ===> User | 
|  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) | 
|  | ===> Setting default values of environment variables if not already set. | 
|  | ===> Configuring … | 
|  | Running in KRaft mode… | 
|  | /opt/kafka/config/ file not writable | 
|  | ``` | 
|  |  | 
|  | To avoid this, **please upgrade Docker to 20.10.4 or later**. | 
|  |  | 
|  | Running on default configs | 
|  | -------------------------- | 
|  |  | 
|  | If no user provided configuration (file input or environment variables) is passed to the Docker container, the default KRaft configuration for single combined-mode node will be used. | 
|  | This default configuration is packaged in the Kafka tarball. | 
|  |  | 
|  | Use input file for providing configs | 
|  | ------------------------------------ | 
|  |  | 
|  | - This method requires users to provide path to a local folder which contains the Kafka property files and mount it to Docker container using Docker volume. | 
|  | - It replaces the default KRaft configuration file present in Docker container. | 
|  | - The Command `docker run --volume /path/to/property/folder:/mnt/shared/config -p 9092:9092 apache/kafka:latest` can be used to mount the folder containing the property files. | 
|  | - Property files will be only read by the Docker container. | 
|  |  | 
|  | Using Environment Variables | 
|  | --------------------------- | 
|  |  | 
|  | When using the environment variables, you need to set all properties required to start the KRaft node. | 
|  | Therefore, the recommended way to use environment variables is via Docker Compose, which allows users to set all the properties that are needed. | 
|  | It is also possible to use the input file to have a common set of configurations, and then override specific node properties using the environment variables. | 
|  |  | 
|  | - Kafka property defined via environment variables will override the value of that property defined in the user provided property file. | 
|  | - If properties are provided via environment variables only, all required properties must be specified. | 
|  | - The following rules must be used to construct the environment variable key name: | 
|  | - Replace `.` with `_` | 
|  | - Replace `_` with `__` (double underscore) | 
|  | - Replace `-` with `___` (triple underscore) | 
|  | - Prefix the result with `KAFKA_` | 
|  | - Examples: | 
|  | - For `abc.def`, use `KAFKA_ABC_DEF` | 
|  | - For `abc-def`, use `KAFKA_ABC___DEF` | 
|  | - For `abc_def`, use `KAFKA_ABC__DEF` | 
|  |  | 
|  | - To provide configs to log4j property files, following points should be considered: | 
|  | - log4j properties provided via environment variables will be appended to the default properties file (log4j properties files bundled with Kafka). | 
|  | - `KAFKA_LOG4J_ROOT_LOGLEVEL` can be provided to set the value of `log4j.rootLogger` in `log4j2.yaml` and `tools-log4j2.yaml`. | 
|  | - log4j loggers can be added to `log4j2.yaml` by setting them in `KAFKA_LOG4J_LOGGERS` environment variable in a single comma separated string. | 
|  | - Example: | 
|  | - Assuming that `KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2'` environment variable is provided to Docker container. | 
|  | - `log4j.logger.property1=value1` and `log4j.logger.property2=value2` will be added to the `log4j2.yaml` file inside Docker container. | 
|  |  | 
|  | Running in SSL mode | 
|  | ------------------- | 
|  |  | 
|  | - Recommended way to run in ssl mode is by mounting secrets on `/etc/kafka/secrets` in Docker container and providing configs following through environment variables (`KAFKA_SSL_KEYSTORE_FILENAME`, `KAFKA_SSL_KEYSTORE_CREDENTIALS`, `KAFKA_SSL_KEY_CREDENTIALS`, `KAFKA_SSL_TRUSTSTORE_FILENAME` and `KAFKA_SSL_TRUSTSTORE_CREDENTIALS`) to let the Docker image scripts extract passwords and populate correct paths in `server.properties`. | 
|  | - Please ensure appropriate `KAFKA_ADVERTISED_LISTENERS` are provided through environment variables to enable SSL mode in Kafka server, i.e. it should contain an `SSL` listener. | 
|  | - Alternatively property file input can be used to provide ssl properties. | 
|  | - Make sure you set location of truststore and keystore correctly when using file input. See example for file input in `docker-compose-files/single-node/file-input` for better clarity. | 
|  | - Note that advertised.listeners property needs to be provided along with SSL properties in file input and cannot be provided through environment variable separately. | 
|  | - In conclusion, ssl properties with advertised.listeners should be treated as a group and provided in file input or environment variables in it's entirety. | 
|  | - In case ssl properties are provided both through file input and environment variables, environment variable properties will override the file input properties, just as mentioned in the beginning of this section. | 
|  |  | 
|  | Examples | 
|  | -------- | 
|  |  | 
|  | - `docker-compose-files` directory contains Docker Compose files for some example configs to run `apache/kafka` OR `apache/kafka-native` Docker image. | 
|  | - Pass the `IMAGE` variable with the Docker Compose file to specify which Docker image to use for bringing up the containers. | 
|  | ``` | 
|  | # to bring up containers using apache/kafka docker image | 
|  | IMAGE=apache/kafka:latest <docker compose command> | 
|  |  | 
|  | # to bring up containers using apache/kafka-native docker image | 
|  | IMAGE=apache/kafka-native:latest <docker compose command> | 
|  | ``` | 
|  | - Run the commands from root of the repository. | 
|  | - Checkout `single-node` examples for quick small examples to play around with. | 
|  | - `cluster` contains multi node examples, for `combined` mode as well as `isolated` mode. | 
|  | - Kafka server running on Docker container can be accessed using cli scripts or your own client code. | 
|  | - Make sure jars are built, if you decide to use cli scripts of this repo. | 
|  |  | 
|  | Single Node | 
|  | ----------- | 
|  |  | 
|  | - These examples are for understanding various ways inputs can be provided and kafka can be configured in Docker container. | 
|  | - Examples are present inside `docker-compose-files/single-node` directory. | 
|  | - Plaintext: | 
|  | - This is the simplest compose file. | 
|  | - We are using environment variables purely for providing configs. | 
|  | - `KAFKA_LISTENERS` is getting supplied. But if it was not provided, defaulting would have kicked in and we would have used `KAFKA_ADVERTISED_LISTENERS` to generate `KAFKA_LISTENERS`, by replacing the host with `0.0.0.0`. | 
|  | - Note that we have provided a `CLUSTER_ID`, but it's not mandatory as there is a default cluster id present in container. | 
|  | - We had to provide `KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR` and set it explicitly to 1, because if we don't provide it default value provided by kafka will be taken which is 3. | 
|  | - We have also set hostname of the container. | 
|  | It can be kept same as the container name for clarity. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml up | 
|  | ``` | 
|  | - To produce messages using client scripts: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092 | 
|  | ``` | 
|  | - SSL: | 
|  | - Note that here we are using environment variables to pass configs. | 
|  | - Notice how secrets folder is mounted to docker container. | 
|  | - In case of environment variable it is mandatory to keep the files in `/etc/kafka/secrets` folder in docker container, given that the path of the files will be derived from that, as we are just providing file names in other SSL configs. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/ssl/docker-compose.yml up | 
|  | ``` | 
|  | - To produce messages using client scripts (Ensure that java version >= 17): | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties | 
|  | ``` | 
|  | - File Input: | 
|  | - Here ssl configs are provided via file input. | 
|  | - Notice that now full file path is needed for the configs. | 
|  | - Note that there is extra volume mount now. | 
|  | - Configs provided via environment variable will override the file input configs. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/single-node/file-input/docker-compose.yml up | 
|  | ``` | 
|  | - To produce messages using client scripts (Ensure that java version >= 17): | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties | 
|  | ``` | 
|  |  | 
|  | Multi Node Cluster | 
|  | ------------------ | 
|  |  | 
|  | - These examples are for real world use cases where multiple nodes of kafka are required. | 
|  | - Combined: | 
|  | - Examples are present in `docker-compose-files/cluster/combined` directory. | 
|  | - Plaintext: | 
|  | - Each broker must expose a unique port to host machine. | 
|  | - For example broker-1, broker2 and broker3 are listening on port 9092, they're exposing it to the host via ports 29092, 39092 and 49092 respectively. | 
|  | - Here important thing to note is that to ensure that kafka brokers are accessible both to clients as well as to each other we have introduced an additional listener. | 
|  | - PLAINTEXT is supposed to be listener accessible to other brokers. | 
|  | - The inter broker listener advertised by the brokers is exposed on container's hostname. This is done so that brokers can find each other in Docker network. | 
|  | - PLAINTEXT_HOST is supposed to be listener accessible to the clients. | 
|  | - The port advertised for host machine is done on localhost, as this is the hostname (in this example) that client will use to connect with kafka running inside Docker container. | 
|  | - Here we take advantage of hostname set for each broker and set the listener accordingly. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/plaintext/docker-compose.yml up | 
|  | ``` | 
|  | - To access using client script: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092 | 
|  | ``` | 
|  | - SSL: | 
|  | - Similar to Plaintext example, for inter broker communication in SSL mode, SSL-INTERNAL is required and for client to broker communication, SSL is required. | 
|  | - Note that `KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM` is set to empty as hostname was not set in credentials. | 
|  | This won't be needed in production use cases. | 
|  | - Also note that for example we have used the same credentials for all brokers. | 
|  | Make sure each broker has it's own secrets. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/ssl/docker-compose.yml up | 
|  | ``` | 
|  | - To produce messages using client scripts (Ensure that java version >= 17): | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties | 
|  | ``` | 
|  | - Isolated: | 
|  | - Examples are present in `docker-compose-files/cluster/isolated` directory. | 
|  | - Plaintext: | 
|  | - Here controllers and brokers are configured separately. | 
|  | - It's a good practice to define that brokers depend on controllers. | 
|  | - In this case also we have same listeners setup as mentioned in combined case. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/plaintext/docker-compose.yml up | 
|  | ``` | 
|  | - To access using client script: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:29092 | 
|  | ``` | 
|  | - SSL: | 
|  | - Pretty much same as combined example, with controllers and brokers separated now. | 
|  | - Note that `SSL-INTERNAL` is only for inter broker communication and controllers are using `PLAINTEXT`. | 
|  | - To run the example: | 
|  | ``` | 
|  | # Run from root of the repo | 
|  |  | 
|  | # JVM based Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up | 
|  |  | 
|  | # GraalVM based Native Apache Kafka Docker Image | 
|  | $ IMAGE=apache/kafka-native:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/ssl/docker-compose.yml up | 
|  | ``` | 
|  | - To produce messages using client scripts (Ensure that java version >= 17): | 
|  | ``` | 
|  | # Run from root of the repo | 
|  | $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties | 
|  | ``` | 
|  |  | 
|  | - Note that the examples are meant to be tried one at a time, make sure you close an example server before trying out the other to avoid conflicts. |