Google Git
Sign in
apache / flink-kubernetes-operator / 003e594c9a52ac67fb64ef67588291fcbc85c99e
commit003e594c9a52ac67fb64ef67588291fcbc85c99e[log] [tgz]
authorMaximilian Michels <mxm@apache.org>Fri Sep 19 13:27:15 2025 +0200
committerGitHub <noreply@github.com>Fri Sep 19 13:27:15 2025 +0200
tree66ba435b647cd93175f6353f2b8440a1bdbd1c2f
parentb820577c670411674fc969d805456fb28d38ea2c [diff]
[FLINK-38381] Enumerate Kafka partitions across Kafka clusters (#1030)

When using the DynamicKafkaSink, topics can be spread across multiple
clusters. This used to work fine, but a regression has been added which
considers partitions across different clusters to be identical. This limits the
scale out of the source operator.

Here is an example:

```
"1.Source__Kafka_Source_(testTopic).kafkaCluster.my-cluster-1.KafkaSourceReader.topic.testTopic.partition.0.currentOffset",
"1.Source__Kafka_Source_(testTopic).kafkaCluster.my-cluster-2.KafkaSourceReader.topic.testTopic.partition.0.currentOffset"
```

Those would result be treated as one partition, but there are two partitions from separate kafka clusters.
2 files changed
tree: 66ba435b647cd93175f6353f2b8440a1bdbd1c2f
  1. .github/
  2. docs/
  3. e2e-tests/
  4. examples/
  5. flink-autoscaler/
  6. flink-autoscaler-plugin-jdbc/
  7. flink-autoscaler-standalone/
  8. flink-kubernetes-docs/
  9. flink-kubernetes-operator/
  10. flink-kubernetes-operator-api/
  11. flink-kubernetes-standalone/
  12. flink-kubernetes-webhook/
  13. helm/
  14. licenses/
  15. tools/
  16. .asf.yaml
  17. .dockerignore
  18. .gitignore
  19. .gitmodules
  20. docker-entrypoint.sh
  21. Dockerfile
  22. LICENSE
  23. NOTICE
  24. pom.xml
  25. README.md
README.md

Apache Flink Kubernetes Operator

A Kubernetes operator for Apache Flink, implemented in Java. It allows users to manage Flink applications and their lifecycle through native k8s tooling like kubectl.

Documentation & Getting Started

Please check out the full documentation, hosted by the ASF, for detailed information and user guides.

Check our quick-start guide for simple setup instructions to get you started with the operator.

Features at a glance

  • Deploy and monitor Flink Application, Session and Job deployments
  • Upgrade, suspend and delete deployments
  • Full logging and metrics integration
  • Flexible deployments and native integration with Kubernetes tooling
  • Flink Job Autoscaler

For the complete feature-set please refer to our documentation.

Project Status

Project status: Production Ready

Current API version: v1beta1

To download the latest stable version please visit the Flink Downloads Page. The official operator images are also available on Dockerhub.

Please check out our docs to read about the upgrade process and our backward compatibility guarantees.

Support

Don’t hesitate to ask!

Contact the developers and community on the mailing lists if you need any help.

Open an issue if you found a bug in Flink.

Contributing

You can learn more about how to contribute in the Apache Flink website. For code contributions, please read carefully the Contributing Code section for an overview of ongoing community work.

License

The code in this repository is licensed under the Apache Software License 2.