Google Git
Sign in
apache / beam / refs/heads/master / . / sdks / java / io / kafka
tree: 24fc117c3f3f494db169c807aa5b17d7a460f10d [path history] [tgz]
  1. kafka-01103/
  2. kafka-100/
  3. kafka-111/
  4. kafka-201/
  5. kafka-211/
  6. kafka-222/
  7. kafka-231/
  8. kafka-241/
  9. kafka-251/
  10. src/
  11. upgrade/
  12. build.gradle
  13. kafka-integration-test.gradle
  14. README.md
sdks/java/io/kafka/README.md

KafkaIO contains I/O transforms which allow you to read/write messages from/to Apache Kafka.

Dependencies

To use KafkaIO you must first add a dependency on beam-sdks-java-io-kafka. KafkaIO supports multiple versions of Kafka clients at run time. It does not pull a specific version kafka-clients transitively. You need to include a compatible version of kafka-clients as runtime dependency. Usually current and recent versions of Kafka are supported, please see JavaDoc for KafkaIO for complete list.

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-io-kafka</artifactId>
    <version>...</version>
</dependency>

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>a_recent_version</version>
  <scope>runtime</scope>
</dependency>

Documentation

The documentation is maintained in JavaDoc for KafkaIO class. It includes usage examples and primary concepts.

Protobuf tests

This recreates the proto descriptor set included in this resource directory.

protoc \
 -Isdks/java/io/kafka/src/test/resources/ \
 --descriptor_set_out=sdks/java/io/kafka/src/test/resources/proto_byte/file_descriptor/proto_byte_utils.pb \
 sdks/java/io/kafka/src/test/resources/proto_byte/proto_byte_utils.proto