Google Git
Sign in
apache/doris-kafka-connector

Kafka Connector for Apache Doris

Mirrored from https://github.com/apache/doris-kafka-connector.git
Clone this repo:

Branches

Tags

  1. a369bd9 [Feature] Support S3 TVF sink mode (#101) by wudi · 16 hours ago master
  2. efc32bb [Feature] Support TLS connections to Doris (#100) by wudi · 2 weeks ago
  3. 6e614d8 [improve] Cache available BE in BackendUtils to avoid per-load HTTP probe (#99) by diggle · 4 weeks ago
  4. d8acdb9 [Chore] Update copyright year in NOTICE.txt (#97) by wudi · 5 months ago release-26.0.0
  5. 543fd92 [fix] Disable gzip compression in schema evolution e2e test (#96) by wudi · 5 months ago

Doris-Kafka-Connector

Kafka Connect is a scalable and reliable tool for data transmission between Apache Kafka and other systems. Connectors can be defined Move large amounts of data in and out of Kafka.

Doris provides the Sink Connector plugin, which can write data from Kafka topics to Doris. More information about usage, please visit Doris Kafka Connector

License

Apache License, Version 2.0

How to Build

After running the following command, the jar package of Doris-Kafka-Connector will be generated in the dist directory.

sh build.sh

Note: Confluent archive zip file can be packaged through sh build.sh --confluent

Code formatting

Doris-Kafka-Connector uses the AOSP style in google-java-format version 1.7 as the formatting style of the project code.

When you need to format your code, you have two formatting options:

  • Execute sh format.sh under the project
  • Execute mvn spotless:apply under the project

After executing the above formatting command, you can use mvn spotless:check to check whether the code format meets the requirements.

TLS configuration

The connector supports opt-in one-way TLS for Doris HTTP and MySQL/JDBC connections. TLS is disabled by default. When it is enabled, the connector does not probe plaintext endpoints or fall back to plaintext after a TLS failure.

PropertyDefaultDescription
doris.enable.tlsfalseEnables TLS for Doris HTTP and MySQL/JDBC connections.
doris.tls.ca-certificate-pathemptyWorker-local PEM CA certificate or certificate-chain path. When empty, the JVM default trust store is used.
doris.tls.skip-hostname-verificationfalseDisables hostname verification while retaining CA and certificate validation.
doris.tls.excluded-protocolsemptyComma-separated protocols that remain plaintext. Supported values are http and mysql.

Example:

doris.urls=doris-fe.example.com
doris.http.port=8040
doris.query.port=9030
doris.enable.tls=true
doris.tls.ca-certificate-path=/etc/kafka-connect/certs/doris-ca.pem
doris.tls.skip-hostname-verification=false
doris.tls.excluded-protocols=

doris.urls continues to contain host names only; do not add http://, https://, or a port. The configured CA file must exist at the same local path on every Kafka Connect worker that may run the task. Client certificates and private keys are not supported. Copy-into uploads to external object storage use an independent JVM-system trust context and do not inherit a private Doris CA. Plaintext Doris URLs are rejected when HTTP TLS is enabled. MySQL TLS requires the bundled Connector/J 8 driver and never falls back to the legacy driver.

Help

For additional help, please file an issue in the repository or raise a question in Apache Doris Public Slack.