= Camel-Kafka-connector Google Pubsub Sink Example

This is an example for Camel-Kafka-connector Google PubSub Sink Source Example

== Standalone

=== What is needed

- A google cloud account
- A service account for PubSub service
- A project in google cloud
- A PubSub topic
- A PubSub topic subscription

=== Setting up PubSub

You need to go here: https://console.cloud.google.com/cloudpubsub
- Create a service account with permission to operate on PubSub service
- Export a service account key file somewhere on your filesystem (for example /home/connectors/ServiceAccountPubsub.json)
- Follow the instruction to create a topic (call it 'test' for example)
- Follow the instruction to create a subscription (call it 'test-sub' for example)

Take note of project id and subscription name. If for example your topic name is 'projects/testpubsub-306306/topics/test'
the project id will be 'testpubsub-306306'.

=== Running Kafka

[source]
----
$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
----

=== Download the connector package

Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.11.5/camel-google-pubsub-kafka-connector-0.11.5-package.tar.gz
> untar.gz camel-google-pubsub-kafka-connector-0.11.5-package.tar.gz
----

=== Configuring Kafka Connect

You'll need to set up the `plugin.path` property in your kafka

Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:

[source]
----
...
plugin.path=/home/oscerd/connectors
...
----

=== Setup the connectors

Open the Google Pubsub Sink connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties`

[source]
----
name=CamelGoogle-pubsubSinkConnector
connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

topics=mytopic

camel.sink.path.destinationName=test
camel.sink.path.projectId=testpubsub-306306
camel.sink.endpoint.synchronousPull=true
----

For this example we'll use a Google PubSub Source connector too to get the messages:

Open the Google Pubsub source connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-source/config/CamelGooglepubsubSourceConnector.properties`

[source]
----
name=CamelGoogle-pubsubSourceConnector
connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

topics=mytopic-source

camel.source.path.destinationName=test-sub
camel.source.path.projectId=testpubsub-306306
camel.source.endpoint.synchronousPull=true
----


=== Running the example

On the same terminal you'll run the connector run the following command:

[source]
----
export GOOGLE_APPLICATION_CREDENTIALS=/home/connectors/ServiceAccountPubsub.json
----

pointing to the service account key you created.

Run the kafka connect with the Google Pubsub Source connector:

[source]
----
$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
----

In a terminal with kafkacat run the following command:
[source]
----
> echo "Camel Kafka Connector is great!" | ./kafkacat -b localhost:9092 -t mytopic
% Reached end of topic mytopic [0] at offset 1
----

On a different terminal run the kafkacat consumer

[source]
----
> ./kafkacat -b localhost:9092 -t mytopic-source
Camel Kafka Connector is great!
% Reached end of topic mytopic [0] at offset 1
----

The message in the test topic has been consumed by the source connector now.