| = Camel-Kafka-connector Google Pubsub Sink Example |
| |
| This is an example for Camel-Kafka-connector Google PubSub Sink Source Example |
| |
| == Standalone |
| |
| === What is needed |
| |
| - A google cloud account |
| - A service account for PubSub service |
| - A project in google cloud |
| - A PubSub topic |
| - A PubSub topic subscription |
| |
| === Setting up PubSub |
| |
| You need to go here: https://console.cloud.google.com/cloudpubsub |
| - Create a service account with permission to operate on PubSub service |
| - Export a service account key file somewhere on your filesystem (for example /home/connectors/ServiceAccountPubsub.json) |
| - Follow the instruction to create a topic (call it 'test' for example) |
| - Follow the instruction to create a subscription (call it 'test-sub' for example) |
| |
| Take note of project id and subscription name. If for example your topic name is 'projects/testpubsub-306306/topics/test' |
| the project id will be 'testpubsub-306306'. |
| |
| === Running Kafka |
| |
| [source] |
| ---- |
| $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties |
| $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties |
| $KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic |
| ---- |
| |
| === Download the connector package |
| |
| Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/` |
| |
| [source] |
| ---- |
| > cd /home/oscerd/connectors/ |
| > wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.9.0/camel-google-pubsub-kafka-connector-0.9.0-package.tar.gz |
| > untar.gz camel-google-pubsub-kafka-connector-0.9.0-package.tar.gz |
| ---- |
| |
| === Configuring Kafka Connect |
| |
| You'll need to set up the `plugin.path` property in your kafka |
| |
| Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location: |
| |
| [source] |
| ---- |
| ... |
| plugin.path=/home/oscerd/connectors |
| ... |
| ---- |
| |
| === Setup the connectors |
| |
| Open the Google Pubsub Sink connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties` |
| |
| [source] |
| ---- |
| name=CamelGoogle-pubsubSinkConnector |
| connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector |
| tasks.max=1 |
| key.converter=org.apache.kafka.connect.storage.StringConverter |
| value.converter=org.apache.kafka.connect.storage.StringConverter |
| |
| topics=mytopic |
| |
| camel.sink.path.destinationName=test |
| camel.sink.path.projectId=testpubsub-306306 |
| camel.sink.endpoint.synchronousPull=true |
| ---- |
| |
| For this example we'll use a Google PubSub Source connector too to get the messages: |
| |
| Open the Google Pubsub source connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-source/config/CamelGooglepubsubSourceConnector.properties` |
| |
| [source] |
| ---- |
| name=CamelGoogle-pubsubSourceConnector |
| connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector |
| tasks.max=1 |
| key.converter=org.apache.kafka.connect.storage.StringConverter |
| value.converter=org.apache.kafka.connect.converters.ByteArrayConverter |
| |
| topics=mytopic-source |
| |
| camel.source.path.destinationName=test-sub |
| camel.source.path.projectId=testpubsub-306306 |
| camel.source.endpoint.synchronousPull=true |
| ---- |
| |
| |
| === Running the example |
| |
| On the same terminal you'll run the connector run the following command: |
| |
| [source] |
| ---- |
| export GOOGLE_APPLICATION_CREDENTIALS=/home/connectors/ServiceAccountPubsub.json |
| ---- |
| |
| pointing to the service account key you created. |
| |
| Run the kafka connect with the Google Pubsub Source connector: |
| |
| [source] |
| ---- |
| $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties |
| ---- |
| |
| In a terminal with kafkacat run the following command: |
| [source] |
| ---- |
| > echo "Camel Kafka Connector is great!" | ./kafkacat -b localhost:9092 -t mytopic |
| % Reached end of topic mytopic [0] at offset 1 |
| ---- |
| |
| On a different terminal run the kafkacat consumer |
| |
| [source] |
| ---- |
| > ./kafkacat -b localhost:9092 -t mytopic-source |
| Camel Kafka Connector is great! |
| % Reached end of topic mytopic [0] at offset 1 |
| ---- |
| |
| The message in the test topic has been consumed by the source connector now. |