Distributed locks are used to synchronize accesses shared resources. Most applications use Zookeeper to model the distributed locks.
The simplest way to model a lock using zookeeper is (See Zookeeper leader recipe for an exact and more advanced solution)
This is good enough if there is only one lock. But in practice, an application will need many such locks. Distributing and managing the locks among difference process becomes challenging. Extending such a solution to many locks will result in
In other words we want a system to satisfy the following requirements.
Helix provides a simple and elegant solution to this problem. Simply specify the number of locks and Helix will ensure that above constraints are satisfied.
To quickly see this working run the lock-manager-demo script where 12 locks are evenly distributed among three nodes, and when a node fails, the locks get re-distributed among remaining two nodes. Note that Helix does not re-shuffle the locks completely, instead it simply distributes the locks relinquished by dead node among 2 remaining nodes evenly.
This version starts multiple threads with in same process to simulate a multi node deployment. Try the long version to get a better idea of how it works.
git clone https://git-wip-us.apache.org/repos/asf/helix.git cd helix mvn clean install package -DskipTests cd recipes/distributed-lock-manager/target/distributed-lock-manager-pkg/bin chmod +x * ./lock-manager-demo
./lock-manager-demo STARTING localhost_12000 STARTING localhost_12002 STARTING localhost_12001 STARTED localhost_12000 STARTED localhost_12002 STARTED localhost_12001 localhost_12001 acquired lock:lock-group_3 localhost_12000 acquired lock:lock-group_8 localhost_12001 acquired lock:lock-group_2 localhost_12001 acquired lock:lock-group_4 localhost_12002 acquired lock:lock-group_1 localhost_12002 acquired lock:lock-group_10 localhost_12000 acquired lock:lock-group_7 localhost_12001 acquired lock:lock-group_5 localhost_12002 acquired lock:lock-group_11 localhost_12000 acquired lock:lock-group_6 localhost_12002 acquired lock:lock-group_0 localhost_12000 acquired lock:lock-group_9 lockName acquired By ====================================== lock-group_0 localhost_12002 lock-group_1 localhost_12002 lock-group_10 localhost_12002 lock-group_11 localhost_12002 lock-group_2 localhost_12001 lock-group_3 localhost_12001 lock-group_4 localhost_12001 lock-group_5 localhost_12001 lock-group_6 localhost_12000 lock-group_7 localhost_12000 lock-group_8 localhost_12000 lock-group_9 localhost_12000 Stopping localhost_12000 localhost_12000Interrupted localhost_12001 acquired lock:lock-group_9 localhost_12001 acquired lock:lock-group_8 localhost_12002 acquired lock:lock-group_6 localhost_12002 acquired lock:lock-group_7 lockName acquired By ====================================== lock-group_0 localhost_12002 lock-group_1 localhost_12002 lock-group_10 localhost_12002 lock-group_11 localhost_12002 lock-group_2 localhost_12001 lock-group_3 localhost_12001 lock-group_4 localhost_12001 lock-group_5 localhost_12001 lock-group_6 localhost_12002 lock-group_7 localhost_12002 lock-group_8 localhost_12001 lock-group_9 localhost_12001
This provides more details on how to setup the cluster and where to plugin application code.
./start-standalone-zookeeper 2199
./helix-admin --zkSvr localhost:2199 --addCluster lock-manager-demo
Create a lock group and specify the number of locks in the lock group.
./helix-admin --zkSvr localhost:2199 --addResource lock-manager-demo lock-group 6 OnlineOffline FULL_AUTO
Create a Lock class that handles the callbacks.
public class Lock extends StateModel { private String lockName; public Lock(String lockName) { this.lockName = lockName; } public void lock(Message m, NotificationContext context) { System.out.println(" acquired lock:"+ lockName ); } public void release(Message m, NotificationContext context) { System.out.println(" releasing lock:"+ lockName ); } }
LockFactory that creates the lock
public class LockFactory extends StateModelFactory<Lock>{ /* Instantiates the lock handler, one per lockName*/ public Lock create(String lockName) { return new Lock(lockName); } }
At node start up, simply join the cluster and helix will invoke the appropriate call backs on Lock instance. One can start any number of nodes and Helix detects that a new node has joined the cluster and re-distributes the locks automatically.
public class LockProcess{ public static void main(String args){ String zkAddress= "localhost:2199"; String clusterName = "lock-manager-demo"; //Give a unique id to each process, most commonly used format hostname_port String instanceName ="localhost_12000"; ZKHelixAdmin helixAdmin = new ZKHelixAdmin(zkAddress); //configure the instance and provide some metadata InstanceConfig config = new InstanceConfig(instanceName); config.setHostName("localhost"); config.setPort("12000"); admin.addInstance(clusterName, config); //join the cluster HelixManager manager; manager = HelixManagerFactory.getHelixManager(clusterName, instanceName, InstanceType.PARTICIPANT, zkAddress); manager.getStateMachineEngine().registerStateModelFactory("OnlineOffline", modelFactory); manager.connect(); Thread.currentThread.join(); } }
Controller can be started either as a separate process or can be embedded within each node process
This is recommended when number of nodes in the cluster >100. For fault tolerance, you can run multiple controllers on different boxes.
./run-helix-controller --zkSvr localhost:2199 --cluster mycluster 2>&1 > /tmp/controller.log &
This is recommended when the number of nodes in the cluster is less than 100. To start a controller from each process, simply add the following lines to MyClass
public class LockProcess{ public static void main(String args){ String zkAddress= "localhost:2199"; String clusterName = "lock-manager-demo"; . . manager.connect(); HelixManager controller; controller = HelixControllerMain.startHelixController(zkAddress, clusterName, "controller", HelixControllerMain.STANDALONE); Thread.currentThread.join(); } }