There are couple of ways to use Samza SQL
Samza SQL console tool documented here uses Samza standalone to run the Samza SQL on your local machine. This is the quickest way to play with Samza SQL. Please follow the instructions here to get access to the Samza tools on your machine.
Please follow the instructions from the Kafka quickstart to start the zookeeper and Kafka server.
The below sql statements requires a topic named ProfileChangeStream to be created on the Kafka broker. You can follow the instructions in the Kafka quick start guide to create a topic named “ProfileChangeStream”.
./deploy/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ProfileChangeStream
Use generate-kafka-events from Samza tools to generate events into the ProfileChangeStream
cd samza-tools-<version> ./scripts/generate-kafka-events.sh -t ProfileChangeStream -e ProfileChange
Below are some of the sql queries that you can execute using the samza-sql-console tool from Samza tools package.
# This command just prints out all the events in the Kafka topic ProfileChangeStream into console output as a json serialized payload. ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select * from kafka.ProfileChangeStream" # This command prints out the fields that are selected into the console output as a json serialized payload. ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select Name, OldCompany, NewCompany from kafka.ProfileChangeStream" # This command showcases the RegexMatch udf and filtering capabilities. ./scripts/samza-sql-console.sh --sql "insert into log.consoleoutput select Name as __key__, Name, NewCompany, RegexMatch('.*soft', OldCompany) from kafka.ProfileChangeStream where NewCompany = 'LinkedIn'"
The hello-samza project is an example project designed to help you run your first Samza application. It has examples of applications using the Low Level Task API, High Level Streams API as well as Samza SQL.
This tutorial demonstrates a simple Samza application that uses SQL to perform stream processing.
Please follow the instructions from hello-samza-high-level-yarn on how to build the hello-samza repository and start the yarn grid.
Please follow the steps in the section “Create ProfileChangeStream Kafka topic” and “Generate events into ProfileChangeStream topic” above.
Before you can run a Samza application, you need to build a package for it. Please follow the instructions from hello-samza-high-level-yarn on how to build the hello-samza application package.
After you've built your Samza package, you can start the app on the grid using the run-app.sh script.
./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties
The app executes the following SQL command :
insert into kafka.NewLinkedInEmployees select Name from ProfileChangeStream where NewCompany = 'LinkedIn'
This SQL performs the following
Give the job a minute to startup, and then tail the Kafka topic:
./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic NewLinkedInEmployees
Congratulations! You've now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza SQL application on it.
To shutdown the app, use the same run-app.sh script with an extra --operation=kill argument
./deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/page-view-filter-sql.properties --operation=kill
Please follow the instructions from Hello Samza High Level API - YARN Deployment on how to shutdown and cleanup the app.