Google Git
Sign in
apache/pulsar-connectors

Apache Pulsar Connectors - source and sink connectors for Apache Pulsar

Mirrored from https://github.com/apache/pulsar-connectors.git
Clone this repo:

Branches

  1. e689bc3 [fix][io] JDBC sink: prevent OOM from unbounded queue on connection failure (#9) by Harangozó Péter · 9 days ago master
  2. c762d7e [improve][build] Improve the gradle build to use similar best practices as apache/pulsar build (#10) by Lari Hotari · 10 days ago
  3. 62b8773 Remove default_branch key and jira_options from .asf.yaml by Lari Hotari · 11 days ago
  4. 7a810f3 Add .asf.yaml for GitHub repository configuration by Matteo Merli · 11 days ago
  5. 32d9445 Add CI, integration test infrastructure, and testcontainer-based tests (#8) by Matteo Merli · 11 days ago

Apache Pulsar Connectors

This repository contains the IO connectors for Apache Pulsar.

Connectors are packaged as NAR files and can be deployed into any Pulsar installation by placing them in the connectors directory or mounting them into the apachepulsar/pulsar Docker image.

Available Connectors

Sources

ConnectorDescription
CanalMySQL binlog via Alibaba Canal
Debezium (MySQL, PostgreSQL, MongoDB, MSSQL, Oracle)CDC via Debezium
DynamoDBAmazon DynamoDB Streams
FileLocal filesystem
KafkaApache Kafka
KinesisAmazon Kinesis Data Streams
MongoDBMongoDB change streams
NSQNSQ messaging
RabbitMQRabbitMQ / AMQP

Sinks

ConnectorDescription
AerospikeAerospike database
AlluxioAlluxio distributed storage
Azure Data ExplorerAzure Data Explorer (Kusto)
CassandraApache Cassandra
Elasticsearch / OpenSearchElasticsearch and OpenSearch
HBaseApache HBase
HDFS3Hadoop HDFS
HTTPHTTP endpoint
InfluxDBInfluxDB time-series database
JDBC (PostgreSQL, MariaDB, ClickHouse, SQLite, OpenMLDB)JDBC databases
KafkaApache Kafka
KinesisAmazon Kinesis Data Streams
MongoDBMongoDB
RedisRedis
SolrApache Solr

Adaptor

ConnectorDescription
Kafka Connect AdaptorRun Kafka Connect connectors on Pulsar

Prerequisites

Note: This project includes a Gradle Wrapper so no separate Gradle installation is needed. Use ./gradlew on Linux/macOS and gradlew.bat on Windows.

Building

Compile and assemble all modules:

./gradlew assemble

NAR files are produced under each connector's build/libs/ directory.

Build a specific connector:

./gradlew :elastic-search:assemble

Build the distribution package containing all connector NARs:

./gradlew :distribution:pulsar-io-distribution:assemble

Check source code license headers:

./gradlew rat spotlessCheck

Auto-fix license headers:

./gradlew spotlessApply

Running Tests

# All unit tests
./gradlew test

# Specific connector
./gradlew :elastic-search:test

# Specific test class
./gradlew :elastic-search:test --tests "ElasticSearchSinkTests"

Using Connectors

With Docker

Mount connector NARs into the Pulsar container:

docker run -v /path/to/connectors:/pulsar/connectors apachepulsar/pulsar

Manual Installation

Copy NAR files to the connectors/ directory of your Pulsar installation:

cp elastic-search/build/libs/pulsar-io-elastic-search-*.nar $PULSAR_HOME/connectors/

Versioning

This repository follows its own release cadence, independent from Apache Pulsar releases. All connectors are released together as a single release. The initial release version matches the Pulsar version at the time of the split.

Each release specifies which Pulsar versions it is compatible with. The pulsar-io-core API has been stable for years, so connectors are generally compatible across Pulsar 4.x releases.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md in the main Pulsar repository for guidelines.

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.