GraalVM Based Native Apache Kafka Docker Image

Introduction

  • The Native Apache Kafka Docker Image can launch brokers with sub-second startup time and minimal memory footprint by leveraging native Kafka executable.
  • The native Kafka executable is built by compiling Apache Kafka code ahead-of-time using the GraalVM native-image tool.
  • This image is experimental and intended for local development and testing purposes only; it is not recommended for production use.
  • This is introduced with KIP-974.

Native-Image reachability metadata

The native-image tool performs static analysis while building a native binary to determine the dynamic features(the dynamic language features of the JVM, including reflection and resource handling, compute the dynamically-accessed program elements such as invoked methods or resource URLs at runtime), but it cannot always exhaustively predict all uses. To ensure inclusion of these elements into the native binary, reachability metadata should be provided to the native-image builder. These metadata configs are provided in the native-image-configs directory.

How to generate reachability metadata

GraalVM provide an option to create these configs automatically by running the application normally with the native-image agent attached. The configs in the native-image-configs directory are generated by running existing Apache Kafka System Tests(using GraalVM JIT and attaching the native-image agent to the process) as they are quite exhaustive.

To update reachability metadata, users can set following environment variable and run related jar files. New metadata configs will be generated in the native-image-configs directory.

export JAVA_TOOL_OPTIONS="-agentlib:native-image-agent=config-merge-dir=/path/to/kafka/docker/native/native-image-configs"

Limitations of Native Apache Kafka executable

  • Dynamic Features: For any new dynamic feature added or modified in Apache Kafka, the corresponding metadata configurations must be added or updated in the native-image-configs. As of now these configs are static.
  • Runtime Jars: Native Kafka Executable does not support capabilities that require a new runtime jar provided by the user, as the jar's information is not available during build time. In such cases, a new native Kafka binary must be built, including that jar in the classpath during the build.
  • Garbage Collection: This implementation uses the community edition of GraalVM, which supports only serial GC. Therefore, Native Apache Kafka supports serial GC and not G1 GC.

How to use this image in docker container

Check out this guide.

How to build, test and release this docker image

Check out this guide.

How to run system tests on native Apache Kafka

Check out this guide.