| commit | 7c177f4e6eea13ac13ec72eaa1fc19a85a6983b2 | [log] [tgz] |
|---|---|---|
| author | Martin Kleppmann <mkleppmann@linkedin.com> | Tue Mar 18 02:01:15 2014 +0000 |
| committer | Martin Kleppmann <mkleppmann@linkedin.com> | Tue Mar 18 21:13:42 2014 +0000 |
| tree | afabe043c1c33083aee19350804e9f3658a9589a | |
| parent | a37cf4a349ced326ab2b4192ee7212d201a62474 [diff] |
SAMZA-190 Reviewboard configuration for hello-samza
Hello Samza is a starter project for Samza jobs.
Please see Hello Samza to get started.
If you'd like to use Vagrant to get up and running, follow these instructions.
Then once that is done (or if done already) clone this repository and boot the virtual machine up.
cd hello-samza vagrant up
This will take ~ 10-15 minutes to install Kafka, Hadoop/YARN, Samza, configure everything together and launch the jobs.
Once the VM is launched and you are back at a command prompt go into the virtual machine and see whats running.
vagrant ssh cd /vagrant
The wikipedia-feed Samza job that is running is consuming a feed of real-time edits from Wikipedia, and producing them to a Kafka topic called “wikipedia-raw”. You can view this in real-time by using the Kafka console consumer to view the topic.
deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw
The wikipedia-parser Samza job is then parsing the messages in wikipedia-raw, and extracting information about the size of the edit, who made the change, etc. It outputs these counts to the wikipedia-edits topic.
deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits
The wikipedia-stats Samza job reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It outputs these counts to the wikipedia-stats topic.
deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats
You can view the Samza jobs running in the YARN UI http://192.168.80.20:8088/cluster/apps too.
To see how this was setup and works look at vagrant/bootstrap.sh and Hello Samza.