Interact with Apache Kafka streams from Flink's APIs.
Flink provides special Kafka Connectors for reading and writing data to Kafka topics. The Flink Kafka Consumer integrates with Flink's checkpointing mechanisms to provide different processing guarantees (most importantly exactly-once guarantees).
For exactly-once processing Flink can not rely on the auto-commit capabilities of the Kafka consumers. The Kafka consumer might commit offsets to Kafka which have not been processed successfully.
Flink provides different connector implementations for different use-cases and environments.
Please pick a package (maven artifact id) and class name for your use-case and environment. For most users, the flink-connector-kafka-083 package and the FlinkKafkaConsumer082 class are appropriate.
| Package | Supported Since | Class | Kafka Version | Allows exactly once processing | Notes | | ------------- |-------------| -----| ------ | ------ | | flink-connector-kafka | 0.9, 0.10 | KafkaSource | 0.8.1, 0.8.2 | No, does not participate in checkpointing at all. | Uses the old, high level KafkaConsumer API, autocommits to ZK by Kafka | | flink-connector-kafka | 0.9, 0.10 | PersistentKafkaSource | 0.8.1, 0.8.2 | No, does not guarantee exactly-once processing, element order or strict partition assignment | Uses the old, high level KafkaConsumer API, offsets are committed into ZK manually | | flink-connector-kafka-083 | 0.9.1 0.10 | FlinkKafkaConsumer081 | 0.8.1 | yes | Uses the SimpleConsumer API of Kafka internally. Offsets are committed to ZK manually | | flink-connector-kafka-083 | 0.9.1 0.10 | FlinkKafkaConsumer082 | 0.8.2 | yes | Uses the SimpleConsumer API of Kafka internally. Offsets are committed to ZK manually |