blob: b427afd2ec3fbe71e9b5f066f6e42a8f57cf1ba7 [file] [log] [blame]
{"body":"What is HELIX\r\n--------------\r\nHelix is a generic cluster management framework used for automatic management of partitioned, replicated and distributed resources hosted on a group of nodes(cluster). Helix provides the following features \r\n\r\n1. Automatic assignment of resource/partition to nodes\r\n2. Node Failure detection and recovery\r\n3. Dynamic addition of Resources \r\n4. Dynamic Addition of nodes to the cluster\r\n5. Pluggable distributed state machine to manage the state of a resource via state transitions\r\n6. Automatic Load balancing and throttling of transitions \r\n\r\n-----\r\n\r\nOverview\r\n--------\r\nHelix uses terms that are commonly used to describe distributed data system concepts. \r\n\r\n1. Cluster: A logical set of Instances that perform similar set of activities. \r\n2. Instance/Node: Instance is logical entity in a cluster. Node is the physical entity in the cluster. A node can have one or more logical Instances. \r\nNode and Instance are used interchangeably. Each Instance in the cluster is identified by a unique id. \r\nMost commonly used convention is hostname\\_port where hostname is the FQDN(Fully qualified Domain Name) and port is the port where the service is hosted. \r\n3. Resource: A resource represents the logical entity hosted by the distributed system. It can be a database name, index or a task group name \r\n4. Partition: A resource is generally split into one or partitions.\r\n5. Replica: Each partition can have one or more replicas\r\n6. State: Each replicas can have state associated with it. For example: Master, Slave, Leader, Stand By, Offline, Online etc. \r\n\r\nTo summarize, A resource (Database, Index or any task) in general is partitioned, replicated and distributed among the Instance/nodes in the cluster and each partition has a state associated with it. \r\n\r\nHelix manages the state of a resource by supporting a pluggable distributed state machine. One can define the state machine table along with the constraints for each state. \r\n\r\nHere are some common state models used\r\n\r\n1. Master, Slave\r\n2. Online, Offline\r\n3. Leader, Standby.\r\n\r\nFor example in case of a MasterSlave state model one can specify the state machine as follows. The table says given a start state and end state what should be the next state. \r\nFor example, if the current state is Offline and target state is master, the table says next state is Slave. \r\n\r\n<pre><code>\r\n OFFLINE | SLAVE | MASTER \r\n _____________________________\r\n | | | |\r\nOFFLINE | N/A | SLAVE | SLAVE |\r\n |__________|________|_________|\r\n | | | |\r\nSLAVE | OFFLINE | N/A | MASTER |\r\n |__________|________|_________|\r\n | | | |\r\nMASTER | SLAVE | SLAVE | N/A |\r\n |__________|________|_________|\r\n\r\n</code></pre>\r\n\r\nHelix also supports the ability to provide constraints on each state. For example in a MasterSlave state model with a replication factor of 3 one can say \r\n\r\n MASTER:1 \r\n SLAVE:2\r\n\r\nHelix will automatically maintain 1 Master and 2 Slaves by initiating appropriate state transitions on each instance in the cluster. \r\n\r\nEach transition results in a partition moving from its CURRENT state state to an NEW state. These transitions are triggered on changes in the cluster state like \r\n\r\n* Node start up\r\n* Node soft and hard failures \r\n* Addition of resources\r\n* Addition of nodes\r\n\r\nIn simple words, Helix is a distributed state machine with support for constraints on each state.\r\n\r\nHelix framework can be used to build distributed, scalable, elastic and fault tolerant systems by configuring the distributed state machine and its constraints based on application requirement. Application has to provide the implementation for handling state transitions appropriately. Example \r\n\r\nOnce the state machine and constraints are configured through Helix, application will have the provide implementation to handle the transitions appropriately. \r\n\r\n<pre><code>\r\nMasterSlaveStateModel extends HelixStateModel {\r\n\r\n void onOfflineToSlave(Message m, NotificationContext context){\r\n print(\"Transitioning from Offline to Slave for resource:\"+ m.getResourceName() + \" and partition:\"+ m.getPartitionName());\r\n }\r\n void onSlaveToMaster(Message m, NotificationContext context){\r\n print(\"Transitioning from Slave to Master for resource:\"+ m.getResourceName() + \" and partition:\"+ m.getPartitionName());\r\n }\r\n void onMasterToSlave(Message m, NotificationContext context){\r\n print(\"Transitioning from Master to Slave for resource:\"+ m.getResourceName() + \" and partition:\"+ m.getPartitionName());\r\n }\r\n void onSlaveToOffline(Message m, NotificationContext context){\r\n print(\"Transitioning from Slave to Offline for resource:\"+ m.getResourceName() + \" and partition:\"+ m.getPartitionName());\r\n }\r\n}\r\n</code></pre>\r\n\r\nOnce state machine is configured, the framework allows one to \r\n\r\n* Dynamically add nodes to the cluster\r\n* Automatically modify the topology(rebalance partitions) of the cluster \r\n* Dynamically add resources to the cluster\r\n* Enable/disable partition/instances for software upgrade without impacting availability.\r\n\r\nHelix uses Zookeeper for maintaining the cluster state and change notifications.\r\n\r\nWhy HELIX\r\n-------------\r\nThe approach of using a distributed state machine with constraints on state and transitions has many benefits like\r\n\r\n* Abstract cluster management aspects from the core functionality of distributed system. Using a state machine to describe the process flow improves the overall design of the system.\r\n* Each node in DDS is not aware of the global state since they simply have to follow instructions of the controller. This significantly reduces the code complexity \r\n* Since goal of Controller is to satisfy state machine constraints at all times, scenerios like cluster startup, node failure, cluster expansion are solved in a similar way.\r\n\r\nAt LinkedIn, we have used Helix to manage [3 different distributed systems](https://github.com/linkedin/helix/wiki/UseCases) \r\n\r\n----------------\r\n\r\nAdditional Info\r\n---------------\r\n\r\nDocumentation: [Home](https://github.com/linkedin/helix/wiki/Home) \r\nSample App using Helix: [rabbitmq-consumer-group] (https://github.com/linkedin/helix/wiki/Sample_App) \r\nQuickstart guide: [How to build and run a mock example](https://github.com/linkedin/helix/wiki/Quickstart) \r\nArchitecture: [Helix Architecture](https://github.com/linkedin/helix/wiki/Architecture) \r\nFeatures: [Helix Features](https://github.com/linkedin/helix/wiki/Features) \r\nApiUsage: http://linkedin.github.com/helix/apidocs/\r\nUseCases: [Helix LinkedIn Usecases](https://github.com/linkedin/helix/wiki/UseCases) \r\n\r\n\r\n\r\n\r\n \r\n","google":"","note":"Don't delete this file! It's used internally to help with page regeneration.","tagline":"A generic cluster management framework used for automatic management of partitioned, replicated and distributed resources hosted on a group of nodes( cluster)","name":"Helix"}