| <?xml version="1.0" encoding="UTF-8"?> |
| <!-- |
| Copyright 2002-2004 The Apache Software Foundation |
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN" |
| "http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd"> |
| <article id="ar_Tutorial"> |
| <title>Programming with ZooKeeper - A basic tutorial</title> |
| |
| <articleinfo> |
| <legalnotice> |
| <para>Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. You may |
| obtain a copy of the License at <ulink |
| url="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</ulink>.</para> |
| |
| <para>Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an "AS IS" |
| BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
| implied. See the License for the specific language governing permissions |
| and limitations under the License.</para> |
| </legalnotice> |
| |
| <abstract> |
| <para>This article contains sample Java code for simple implementations of barrier |
| and consumers queues..</para> |
| |
| </abstract> |
| </articleinfo> |
| |
| <section id="ch_Introduction"> |
| <title>Introduction</title> |
| |
| <para>In this tutorial, we show simple implementations of barriers and |
| producer-consumer queues using ZooKeeper. We call the respective classes Barrier and Queue. |
| These examples assume that you have at least one ZooKeeper server running.</para> |
| |
| <para>Both primitives use the following common excerpt of code:</para> |
| |
| <programlisting> |
| static ZooKeeper zk = null; |
| static Integer mutex; |
| |
| String root; |
| |
| SyncPrimitive(String address) { |
| if(zk == null){ |
| try { |
| System.out.println("Starting ZK:"); |
| zk = new ZooKeeper(address, 3000, this); |
| mutex = new Integer(-1); |
| System.out.println("Finished starting ZK: " + zk); |
| } catch (IOException e) { |
| System.out.println(e.toString()); |
| zk = null; |
| } |
| } |
| } |
| |
| synchronized public void process(WatchedEvent event) { |
| synchronized (mutex) { |
| mutex.notify(); |
| } |
| } |
| </programlisting> |
| |
| <para>Both classes extend SyncPrimitive. In this way, we execute steps that are |
| common to all primitives in the constructor of SyncPrimitive. To keep the examples |
| simple, we create a ZooKeeper object the first time we instantiate either a barrier |
| object or a queue object, and we declare a static variable that is a reference |
| to this object. The subsequent instances of Barrier and Queue check whether a |
| ZooKeeper object exists. Alternatively, we could have the application creating a |
| ZooKeeper object and passing it to the constructor of Barrier and Queue.</para> |
| <para> |
| We use the process() method to process notifications triggered due to watches. |
| In the following discussion, we present code that sets watches. A watch is internal |
| structure that enables ZooKeeper to notify a client of a change to a node. For example, |
| if a client is waiting for other clients to leave a barrier, then it can set a watch and |
| wait for modifications to a particular node, which can indicate that it is the end of the wait. |
| This point becomes clear once we go over the examples. |
| </para> |
| </section> |
| |
| <section id="sc_barriers"><title>Barriers</title> |
| |
| <para> |
| A barrier is a primitive that enables a group of processes to synchronize the |
| beginning and the end of a computation. The general idea of this implementation |
| is to have a barrier node that serves the purpose of being a parent for individual |
| process nodes. Suppose that we call the barrier node "/b1". Each process "p" then |
| creates a node "/b1/p". Once enough processes have created their corresponding |
| nodes, joined processes can start the computation. |
| </para> |
| |
| <para>In this example, each process instantiates a Barrier object, and its constructor takes as parameters:</para> |
| |
| <itemizedlist><listitem><para>the address of a ZooKeeper server (e.g., "zoo1.foo.com:2181")</para></listitem> |
| <listitem><para>the path of the barrier node on ZooKeeper (e.g., "/b1")</para></listitem> |
| <listitem><para>the size of the group of processes</para></listitem> |
| </itemizedlist> |
| |
| <para>The constructor of Barrier passes the address of the Zookeeper server to the |
| constructor of the parent class. The parent class creates a ZooKeeper instance if |
| one does not exist. The constructor of Barrier then creates a |
| barrier node on ZooKeeper, which is the parent node of all process nodes, and |
| we call root (<emphasis role="bold">Note:</emphasis> This is not the ZooKeeper root "/").</para> |
| |
| <programlisting> |
| /** |
| * Barrier constructor |
| * |
| * @param address |
| * @param root |
| * @param size |
| */ |
| Barrier(String address, String root, int size) { |
| super(address); |
| this.root = root; |
| this.size = size; |
| |
| // Create barrier node |
| if (zk != null) { |
| try { |
| Stat s = zk.exists(root, false); |
| if (s == null) { |
| zk.create(root, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT); |
| } |
| } catch (KeeperException e) { |
| System.out |
| .println("Keeper exception when instantiating queue: " |
| + e.toString()); |
| } catch (InterruptedException e) { |
| System.out.println("Interrupted exception"); |
| } |
| } |
| |
| // My node name |
| try { |
| name = new String(InetAddress.getLocalHost().getCanonicalHostName().toString()); |
| } catch (UnknownHostException e) { |
| System.out.println(e.toString()); |
| } |
| |
| } |
| </programlisting> |
| <para> |
| To enter the barrier, a process calls enter(). The process creates a node under |
| the root to represent it, using its host name to form the node name. It then wait |
| until enough processes have entered the barrier. A process does it by checking |
| the number of children the root node has with "getChildren()", and waiting for |
| notifications in the case it does not have enough. To receive a notification when |
| there is a change to the root node, a process has to set a watch, and does it |
| through the call to "getChildren()". In the code, we have that "getChildren()" |
| has two parameters. The first one states the node to read from, and the second is |
| a boolean flag that enables the process to set a watch. In the code the flag is true. |
| </para> |
| |
| <programlisting> |
| /** |
| * Join barrier |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| |
| boolean enter() throws KeeperException, InterruptedException{ |
| zk.create(root + "/" + name, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.EPHEMERAL_SEQUENTIAL); |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| |
| if (list.size() < size) { |
| mutex.wait(); |
| } else { |
| return true; |
| } |
| } |
| } |
| } |
| </programlisting> |
| <para> |
| Note that enter() throws both KeeperException and InterruptedException, so it is |
| the reponsability of the application to catch and handle such exceptions.</para> |
| |
| <para> |
| Once the computation is finished, a process calls leave() to leave the barrier. |
| First it deletes its corresponding node, and then it gets the children of the root |
| node. If there is at least one child, then it waits for a notification (obs: note |
| that the second parameter of the call to getChildren() is true, meaning that |
| ZooKeeper has to set a watch on the the root node). Upon reception of a notification, |
| it checks once more whether the root node has any child.</para> |
| |
| <programlisting> |
| /** |
| * Wait until all reach barrier |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| |
| boolean leave() throws KeeperException, InterruptedException{ |
| zk.delete(root + "/" + name, 0); |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| if (list.size() > 0) { |
| mutex.wait(); |
| } else { |
| return true; |
| } |
| } |
| } |
| } |
| } |
| </programlisting> |
| </section> |
| <section id="sc_producerConsumerQueues"><title>Producer-Consumer Queues</title> |
| <para> |
| A producer-consumer queue is a distributed data estructure thata group of processes |
| use to generate and consume items. Producer processes create new elements and add |
| them to the queue. Consumer processes remove elements from the list, and process them. |
| In this implementation, the elements are simple integers. The queue is represented |
| by a root node, and to add an element to the queue, a producer process creates a new node, |
| a child of the root node. |
| </para> |
| |
| <para> |
| The following excerpt of code corresponds to the constructor of the object. As |
| with Barrier objects, it first calls the constructor of the parent class, SyncPrimitive, |
| that creates a ZooKeeper object if one doesn't exist. It then verifies if the root |
| node of the queue exists, and creates if it doesn't. |
| </para> |
| <programlisting> |
| /** |
| * Constructor of producer-consumer queue |
| * |
| * @param address |
| * @param name |
| */ |
| Queue(String address, String name) { |
| super(address); |
| this.root = name; |
| // Create ZK node name |
| if (zk != null) { |
| try { |
| Stat s = zk.exists(root, false); |
| if (s == null) { |
| zk.create(root, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT); |
| } |
| } catch (KeeperException e) { |
| System.out |
| .println("Keeper exception when instantiating queue: " |
| + e.toString()); |
| } catch (InterruptedException e) { |
| System.out.println("Interrupted exception"); |
| } |
| } |
| } |
| </programlisting> |
| |
| <para> |
| A producer process calls "produce()" to add an element to the queue, and passes |
| an integer as an argument. To add an element to the queue, the method creates a |
| new node using "create()", and uses the SEQUENCE flag to instruct ZooKeeper to |
| append the value of the sequencer counter associated to the root node. In this way, |
| we impose a total order on the elements of the queue, thus guaranteeing that the |
| oldest element of the queue is the next one consumed. |
| </para> |
| |
| <programlisting> |
| /** |
| * Add element to the queue. |
| * |
| * @param i |
| * @return |
| */ |
| |
| boolean produce(int i) throws KeeperException, InterruptedException{ |
| ByteBuffer b = ByteBuffer.allocate(4); |
| byte[] value; |
| |
| // Add child with value i |
| b.putInt(i); |
| value = b.array(); |
| zk.create(root + "/element", value, Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT_SEQUENTIAL); |
| |
| return true; |
| } |
| </programlisting> |
| <para> |
| To consume an element, a consumer process obtains the children of the root node, |
| reads the node with smallest counter value, and returns the element. Note that |
| if there is a conflict, then one of the two contending processes won't be able to |
| delete the node and the delete operation will throw an exception.</para> |
| |
| <para> |
| A call to getChildren() returns the list of children in lexicographic order. |
| As lexicographic order does not necessary follow the numerical order of the counter |
| values, we need to decide which element is the smallest. To decide which one has |
| the smallest counter value, we traverse the list, and remove the prefix "element" |
| from each one.</para> |
| |
| <programlisting> |
| /** |
| * Remove first element from the queue. |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| int consume() throws KeeperException, InterruptedException{ |
| int retvalue = -1; |
| Stat stat = null; |
| |
| // Get the first element available |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| if (list.size() == 0) { |
| System.out.println("Going to wait"); |
| mutex.wait(); |
| } else { |
| Integer min = new Integer(list.get(0).substring(7)); |
| for(String s : list){ |
| Integer tempValue = new Integer(s.substring(7)); |
| //System.out.println("Temporary value: " + tempValue); |
| if(tempValue < min) min = tempValue; |
| } |
| System.out.println("Temporary value: " + root + "/element" + min); |
| byte[] b = zk.getData(root + "/element" + min, |
| false, stat); |
| zk.delete(root + "/element" + min, 0); |
| ByteBuffer buffer = ByteBuffer.wrap(b); |
| retvalue = buffer.getInt(); |
| |
| return retvalue; |
| } |
| } |
| } |
| } |
| } |
| </programlisting> |
| |
| </section> |
| <section id="sc_sourceListing"><title>Complete Source Listing</title> |
| <example id="eg_SyncPrimitive_java"> |
| <title>SyncPrimitive.Java</title> |
| <programlisting> |
| import java.io.IOException; |
| import java.net.InetAddress; |
| import java.net.UnknownHostException; |
| import java.nio.ByteBuffer; |
| import java.util.List; |
| import java.util.Random; |
| |
| import org.apache.zookeeper.CreateMode; |
| import org.apache.zookeeper.KeeperException; |
| import org.apache.zookeeper.WatchedEvent; |
| import org.apache.zookeeper.Watcher; |
| import org.apache.zookeeper.ZooKeeper; |
| import org.apache.zookeeper.ZooDefs.Ids; |
| import org.apache.zookeeper.data.Stat; |
| |
| public class SyncPrimitive implements Watcher { |
| |
| static ZooKeeper zk = null; |
| static Integer mutex; |
| |
| String root; |
| |
| SyncPrimitive(String address) { |
| if(zk == null){ |
| try { |
| System.out.println("Starting ZK:"); |
| zk = new ZooKeeper(address, 3000, this); |
| mutex = new Integer(-1); |
| System.out.println("Finished starting ZK: " + zk); |
| } catch (IOException e) { |
| System.out.println(e.toString()); |
| zk = null; |
| } |
| } |
| //else mutex = new Integer(-1); |
| } |
| |
| synchronized public void process(WatchedEvent event) { |
| synchronized (mutex) { |
| //System.out.println("Process: " + event.getType()); |
| mutex.notify(); |
| } |
| } |
| |
| /** |
| * Barrier |
| */ |
| static public class Barrier extends SyncPrimitive { |
| int size; |
| String name; |
| |
| /** |
| * Barrier constructor |
| * |
| * @param address |
| * @param root |
| * @param size |
| */ |
| Barrier(String address, String root, int size) { |
| super(address); |
| this.root = root; |
| this.size = size; |
| |
| // Create barrier node |
| if (zk != null) { |
| try { |
| Stat s = zk.exists(root, false); |
| if (s == null) { |
| zk.create(root, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT); |
| } |
| } catch (KeeperException e) { |
| System.out |
| .println("Keeper exception when instantiating queue: " |
| + e.toString()); |
| } catch (InterruptedException e) { |
| System.out.println("Interrupted exception"); |
| } |
| } |
| |
| // My node name |
| try { |
| name = new String(InetAddress.getLocalHost().getCanonicalHostName().toString()); |
| } catch (UnknownHostException e) { |
| System.out.println(e.toString()); |
| } |
| |
| } |
| |
| /** |
| * Join barrier |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| |
| boolean enter() throws KeeperException, InterruptedException{ |
| zk.create(root + "/" + name, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.EPHEMERAL_SEQUENTIAL); |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| |
| if (list.size() < size) { |
| mutex.wait(); |
| } else { |
| return true; |
| } |
| } |
| } |
| } |
| |
| /** |
| * Wait until all reach barrier |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| |
| boolean leave() throws KeeperException, InterruptedException{ |
| zk.delete(root + "/" + name, 0); |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| if (list.size() > 0) { |
| mutex.wait(); |
| } else { |
| return true; |
| } |
| } |
| } |
| } |
| } |
| |
| /** |
| * Producer-Consumer queue |
| */ |
| static public class Queue extends SyncPrimitive { |
| |
| /** |
| * Constructor of producer-consumer queue |
| * |
| * @param address |
| * @param name |
| */ |
| Queue(String address, String name) { |
| super(address); |
| this.root = name; |
| // Create ZK node name |
| if (zk != null) { |
| try { |
| Stat s = zk.exists(root, false); |
| if (s == null) { |
| zk.create(root, new byte[0], Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT); |
| } |
| } catch (KeeperException e) { |
| System.out |
| .println("Keeper exception when instantiating queue: " |
| + e.toString()); |
| } catch (InterruptedException e) { |
| System.out.println("Interrupted exception"); |
| } |
| } |
| } |
| |
| /** |
| * Add element to the queue. |
| * |
| * @param i |
| * @return |
| */ |
| |
| boolean produce(int i) throws KeeperException, InterruptedException{ |
| ByteBuffer b = ByteBuffer.allocate(4); |
| byte[] value; |
| |
| // Add child with value i |
| b.putInt(i); |
| value = b.array(); |
| zk.create(root + "/element", value, Ids.OPEN_ACL_UNSAFE, |
| CreateMode.PERSISTENT_SEQUENTIAL); |
| |
| return true; |
| } |
| |
| |
| /** |
| * Remove first element from the queue. |
| * |
| * @return |
| * @throws KeeperException |
| * @throws InterruptedException |
| */ |
| int consume() throws KeeperException, InterruptedException{ |
| int retvalue = -1; |
| Stat stat = null; |
| |
| // Get the first element available |
| while (true) { |
| synchronized (mutex) { |
| List<String> list = zk.getChildren(root, true); |
| if (list.size() == 0) { |
| System.out.println("Going to wait"); |
| mutex.wait(); |
| } else { |
| Integer min = new Integer(list.get(0).substring(7)); |
| for(String s : list){ |
| Integer tempValue = new Integer(s.substring(7)); |
| //System.out.println("Temporary value: " + tempValue); |
| if(tempValue < min) min = tempValue; |
| } |
| System.out.println("Temporary value: " + root + "/element" + min); |
| byte[] b = zk.getData(root + "/element" + min, |
| false, stat); |
| zk.delete(root + "/element" + min, 0); |
| ByteBuffer buffer = ByteBuffer.wrap(b); |
| retvalue = buffer.getInt(); |
| |
| return retvalue; |
| } |
| } |
| } |
| } |
| } |
| |
| public static void main(String args[]) { |
| if (args[0].equals("qTest")) |
| queueTest(args); |
| else |
| barrierTest(args); |
| |
| } |
| |
| public static void queueTest(String args[]) { |
| Queue q = new Queue(args[1], "/app1"); |
| |
| System.out.println("Input: " + args[1]); |
| int i; |
| Integer max = new Integer(args[2]); |
| |
| if (args[3].equals("p")) { |
| System.out.println("Producer"); |
| for (i = 0; i < max; i++) |
| try{ |
| q.produce(10 + i); |
| } catch (KeeperException e){ |
| |
| } catch (InterruptedException e){ |
| |
| } |
| } else { |
| System.out.println("Consumer"); |
| |
| for (i = 0; i < max; i++) { |
| try{ |
| int r = q.consume(); |
| System.out.println("Item: " + r); |
| } catch (KeeperException e){ |
| i--; |
| } catch (InterruptedException e){ |
| |
| } |
| } |
| } |
| } |
| |
| public static void barrierTest(String args[]) { |
| Barrier b = new Barrier(args[1], "/b1", new Integer(args[2])); |
| try{ |
| boolean flag = b.enter(); |
| System.out.println("Entered barrier: " + args[2]); |
| if(!flag) System.out.println("Error when entering the barrier"); |
| } catch (KeeperException e){ |
| |
| } catch (InterruptedException e){ |
| |
| } |
| |
| // Generate random integer |
| Random rand = new Random(); |
| int r = rand.nextInt(100); |
| // Loop for rand iterations |
| for (int i = 0; i < r; i++) { |
| try { |
| Thread.sleep(100); |
| } catch (InterruptedException e) { |
| |
| } |
| } |
| try{ |
| b.leave(); |
| } catch (KeeperException e){ |
| |
| } catch (InterruptedException e){ |
| |
| } |
| System.out.println("Left barrier"); |
| } |
| } |
| </programlisting></example> |
| </section> |
| |
| </article> |
| |
| |