KafkaIO contains I/O transforms which allow you to read/write messages from/to Apache Kafka.
To use KafkaIO you must first add a dependency on beam-sdks-java-io-kafka
. KafkaIO supports multiple versions of Kafka clients at run time. It does not pull a specific version kafka-clients
transitively. You need to include a compatible version of kafka-clients
as runtime dependency. Usually current and recent versions of Kafka are supported, please see JavaDoc for KafkaIO for complete list.
<dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-sdks-java-io-kafka</artifactId> <version>...</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>a_recent_version</version> <scope>runtime</scope> </dependency>
The documentation is maintained in JavaDoc for KafkaIO class. It includes usage examples and primary concepts.
This recreates the proto descriptor set included in this resource directory.
protoc \ -Isdks/java/io/kafka/src/test/resources/ \ --descriptor_set_out=sdks/java/io/kafka/src/test/resources/proto_byte/file_descriptor/proto_byte_utils.pb \ sdks/java/io/kafka/src/test/resources/proto_byte/proto_byte_utils.proto