| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> |
| <html> |
| <head> |
| <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> |
| <meta content="Apache Forrest" name="Generator"> |
| <meta name="Forrest-version" content="0.8"> |
| <meta name="Forrest-skin-name" content="pelt"> |
| <title>ZooKeeper Administrator's Guide</title> |
| <link type="text/css" href="skin/basic.css" rel="stylesheet"> |
| <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet"> |
| <link media="print" type="text/css" href="skin/print.css" rel="stylesheet"> |
| <link type="text/css" href="skin/profile.css" rel="stylesheet"> |
| <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script> |
| <link rel="shortcut icon" href="images/favicon.ico"> |
| </head> |
| <body onload="init()"> |
| <script type="text/javascript">ndeSetTextSize();</script> |
| <div id="top"> |
| <!--+ |
| |breadtrail |
| +--> |
| <div class="breadtrail"> |
| <a href="http://www.apache.org/">Apache</a> > <a href="http://hadoop.apache.org/">Hadoop</a> > <a href="http://hadoop.apache.org/zookeeper/">ZooKeeper</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script> |
| </div> |
| <!--+ |
| |header |
| +--> |
| <div class="header"> |
| <!--+ |
| |start group logo |
| +--> |
| <div class="grouplogo"> |
| <a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a> |
| </div> |
| <!--+ |
| |end group logo |
| +--> |
| <!--+ |
| |start Project Logo |
| +--> |
| <div class="projectlogo"> |
| <a href="http://hadoop.apache.org/zookeeper/"><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a> |
| </div> |
| <!--+ |
| |end Project Logo |
| +--> |
| <!--+ |
| |start Search |
| +--> |
| <div class="searchbox"> |
| <form action="http://www.google.com/search" method="get" class="roundtopsmall"> |
| <input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google"> |
| <input name="Search" value="Search" type="submit"> |
| </form> |
| </div> |
| <!--+ |
| |end search |
| +--> |
| <!--+ |
| |start Tabs |
| +--> |
| <ul id="tabs"> |
| <li> |
| <a class="unselected" href="http://hadoop.apache.org/zookeeper/">Project</a> |
| </li> |
| <li> |
| <a class="unselected" href="http://wiki.apache.org/hadoop/ZooKeeper">Wiki</a> |
| </li> |
| <li class="current"> |
| <a class="selected" href="index.html">ZooKeeper 3.4 Documentation</a> |
| </li> |
| </ul> |
| <!--+ |
| |end Tabs |
| +--> |
| </div> |
| </div> |
| <div id="main"> |
| <div id="publishedStrip"> |
| <!--+ |
| |start Subtabs |
| +--> |
| <div id="level2tabs"></div> |
| <!--+ |
| |end Endtabs |
| +--> |
| <script type="text/javascript"><!-- |
| document.write("Last Published: " + document.lastModified); |
| // --></script> |
| </div> |
| <!--+ |
| |breadtrail |
| +--> |
| <div class="breadtrail"> |
| |
| |
| </div> |
| <!--+ |
| |start Menu, mainarea |
| +--> |
| <!--+ |
| |start Menu |
| +--> |
| <div id="menu"> |
| <div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Overview</div> |
| <div id="menu_1.1" class="menuitemgroup"> |
| <div class="menuitem"> |
| <a href="index.html">Welcome</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperOver.html">Overview</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperStarted.html">Getting Started</a> |
| </div> |
| <div class="menuitem"> |
| <a href="releasenotes.html">Release Notes</a> |
| </div> |
| </div> |
| <div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Developer</div> |
| <div id="menu_1.2" class="menuitemgroup"> |
| <div class="menuitem"> |
| <a href="api/index.html">API Docs</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperProgrammers.html">Programmer's Guide</a> |
| </div> |
| <div class="menuitem"> |
| <a href="javaExample.html">Java Example</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a> |
| </div> |
| <div class="menuitem"> |
| <a href="recipes.html">Recipes</a> |
| </div> |
| </div> |
| <div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">BookKeeper</div> |
| <div id="menu_1.3" class="menuitemgroup"> |
| <div class="menuitem"> |
| <a href="bookkeeperStarted.html">Getting started</a> |
| </div> |
| <div class="menuitem"> |
| <a href="bookkeeperOverview.html">Overview</a> |
| </div> |
| <div class="menuitem"> |
| <a href="bookkeeperConfig.html">Setup guide</a> |
| </div> |
| <div class="menuitem"> |
| <a href="bookkeeperProgrammer.html">Programmer's guide</a> |
| </div> |
| </div> |
| <div onclick="SwitchMenu('menu_selected_1.4', 'skin/')" id="menu_selected_1.4Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Admin & Ops</div> |
| <div id="menu_selected_1.4" class="selectedmenuitemgroup" style="display: block;"> |
| <div class="menupage"> |
| <div class="menupagetitle">Administrator's Guide</div> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperQuotas.html">Quota Guide</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperJMX.html">JMX</a> |
| </div> |
| <div class="menuitem"> |
| <a href="zookeeperObservers.html">Observers Guide</a> |
| </div> |
| </div> |
| <div onclick="SwitchMenu('menu_1.5', 'skin/')" id="menu_1.5Title" class="menutitle">Contributor</div> |
| <div id="menu_1.5" class="menuitemgroup"> |
| <div class="menuitem"> |
| <a href="zookeeperInternals.html">ZooKeeper Internals</a> |
| </div> |
| </div> |
| <div onclick="SwitchMenu('menu_1.6', 'skin/')" id="menu_1.6Title" class="menutitle">Miscellaneous</div> |
| <div id="menu_1.6" class="menuitemgroup"> |
| <div class="menuitem"> |
| <a href="http://wiki.apache.org/hadoop/ZooKeeper">Wiki</a> |
| </div> |
| <div class="menuitem"> |
| <a href="http://wiki.apache.org/hadoop/ZooKeeper/FAQ">FAQ</a> |
| </div> |
| <div class="menuitem"> |
| <a href="http://hadoop.apache.org/zookeeper/mailing_lists.html">Mailing Lists</a> |
| </div> |
| </div> |
| <div id="credit"></div> |
| <div id="roundbottom"> |
| <img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div> |
| <!--+ |
| |alternative credits |
| +--> |
| <div id="credit2"></div> |
| </div> |
| <!--+ |
| |end Menu |
| +--> |
| <!--+ |
| |start content |
| +--> |
| <div id="content"> |
| <div title="Portable Document Format" class="pdflink"> |
| <a class="dida" href="zookeeperAdmin.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br> |
| PDF</a> |
| </div> |
| <h1>ZooKeeper Administrator's Guide</h1> |
| <h3>A Guide to Deployment and Administration</h3> |
| <div id="minitoc-area"> |
| <ul class="minitoc"> |
| <li> |
| <a href="#ch_deployment">Deployment</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#sc_systemReq">System Requirements</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#sc_supportedPlatforms">Supported Platforms</a> |
| </li> |
| <li> |
| <a href="#sc_requiredSoftware">Required Software </a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#sc_zkMulitServerSetup">Clustered (Multi-Server) Setup</a> |
| </li> |
| <li> |
| <a href="#sc_singleAndDevSetup">Single Server and Developer Setup</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#ch_administration">Administration</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#sc_designing">Designing a ZooKeeper Deployment</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#sc_CrossMachineRequirements">Cross Machine Requirements</a> |
| </li> |
| <li> |
| <a href="#Single+Machine+Requirements">Single Machine Requirements</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#sc_provisioning">Provisioning</a> |
| </li> |
| <li> |
| <a href="#sc_strengthsAndLimitations">Things to Consider: ZooKeeper Strengths and Limitations</a> |
| </li> |
| <li> |
| <a href="#sc_administering">Administering</a> |
| </li> |
| <li> |
| <a href="#sc_maintenance">Maintenance</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#Ongoing+Data+Directory+Cleanup">Ongoing Data Directory Cleanup</a> |
| </li> |
| <li> |
| <a href="#Debug+Log+Cleanup+%28log4j%29">Debug Log Cleanup (log4j)</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#sc_supervision">Supervision</a> |
| </li> |
| <li> |
| <a href="#sc_monitoring">Monitoring</a> |
| </li> |
| <li> |
| <a href="#sc_logging">Logging</a> |
| </li> |
| <li> |
| <a href="#sc_troubleshooting">Troubleshooting</a> |
| </li> |
| <li> |
| <a href="#sc_configuration">Configuration Parameters</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#sc_minimumConfiguration">Minimum Configuration</a> |
| </li> |
| <li> |
| <a href="#sc_advancedConfiguration">Advanced Configuration</a> |
| </li> |
| <li> |
| <a href="#sc_clusterOptions">Cluster Options</a> |
| </li> |
| <li> |
| <a href="#sc_authOptions">Authentication & Authorization Options</a> |
| </li> |
| <li> |
| <a href="#Unsafe+Options">Unsafe Options</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#sc_zkCommands">ZooKeeper Commands: The Four Letter Words</a> |
| </li> |
| <li> |
| <a href="#sc_dataFileManagement">Data File Management</a> |
| <ul class="minitoc"> |
| <li> |
| <a href="#The+Data+Directory">The Data Directory</a> |
| </li> |
| <li> |
| <a href="#The+Log+Directory">The Log Directory</a> |
| </li> |
| <li> |
| <a href="#sc_filemanagement">File Management</a> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <a href="#sc_commonProblems">Things to Avoid</a> |
| </li> |
| <li> |
| <a href="#sc_bestPractices">Best Practices</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| |
| |
| |
| |
| |
| |
| |
| <a name="N1000B"></a><a name="ch_deployment"></a> |
| <h2 class="h3">Deployment</h2> |
| <div class="section"> |
| <p>This section contains information about deploying Zookeeper and |
| covers these topics:</p> |
| <ul> |
| |
| <li> |
| |
| <p> |
| <a href="#sc_systemReq">System Requirements</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_zkMulitServerSetup">Clustered (Multi-Server) Setup</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_singleAndDevSetup">Single Server and Developer Setup</a> |
| </p> |
| |
| </li> |
| |
| </ul> |
| <p>The first two sections assume you are interested in installing |
| ZooKeeper in a production environment such as a datacenter. The final |
| section covers situations in which you are setting up ZooKeeper on a |
| limited basis - for evaluation, testing, or development - but not in a |
| production environment.</p> |
| <a name="N10032"></a><a name="sc_systemReq"></a> |
| <h3 class="h4">System Requirements</h3> |
| <a name="N10038"></a><a name="sc_supportedPlatforms"></a> |
| <h4>Supported Platforms</h4> |
| <ul> |
| |
| <li> |
| |
| <p>GNU/Linux is supported as a development and production |
| platform for both server and client.</p> |
| |
| </li> |
| |
| <li> |
| |
| <p>Sun Solaris is supported as a development and production |
| platform for both server and client.</p> |
| |
| </li> |
| |
| <li> |
| |
| <p>FreeBSD is supported as a development and production |
| platform for clients only. Java NIO selector support in |
| the FreeBSD JVM is broken.</p> |
| |
| </li> |
| |
| <li> |
| |
| <p>Win32 is supported as a <em>development |
| platform</em> only for both server and client.</p> |
| |
| </li> |
| |
| <li> |
| |
| <p>MacOSX is supported as a <em>development |
| platform</em> only for both server and client.</p> |
| |
| </li> |
| |
| </ul> |
| <a name="N10066"></a><a name="sc_requiredSoftware"></a> |
| <h4>Required Software </h4> |
| <p>ZooKeeper runs in Java, release 1.6 or greater (JDK 6 or |
| greater). It runs as an <em>ensemble</em> of |
| ZooKeeper servers. Three ZooKeeper servers is the minimum |
| recommended size for an ensemble, and we also recommend that |
| they run on separate machines. At Yahoo!, ZooKeeper is |
| usually deployed on dedicated RHEL boxes, with dual-core |
| processors, 2GB of RAM, and 80GB IDE hard drives.</p> |
| <a name="N10074"></a><a name="sc_zkMulitServerSetup"></a> |
| <h3 class="h4">Clustered (Multi-Server) Setup</h3> |
| <p>For reliable ZooKeeper service, you should deploy ZooKeeper in a |
| cluster known as an <em>ensemble</em>. As long as a majority |
| of the ensemble are up, the service will be available. Because Zookeeper |
| requires a majority, it is best to use an |
| odd number of machines. For example, with four machines ZooKeeper can |
| only handle the failure of a single machine; if two machines fail, the |
| remaining two machines do not constitute a majority. However, with five |
| machines ZooKeeper can handle the failure of two machines. </p> |
| <p>Here are the steps to setting a server that will be part of an |
| ensemble. These steps should be performed on every host in the |
| ensemble:</p> |
| <ol> |
| |
| <li> |
| |
| <p>Install the Java JDK. You can use the native packaging system |
| for your system, or download the JDK from:</p> |
| |
| |
| <p> |
| <a href="http://java.sun.com/javase/downloads/index.jsp">http://java.sun.com/javase/downloads/index.jsp</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>Set the Java heap size. This is very important to avoid |
| swapping, which will seriously degrade ZooKeeper performance. To |
| determine the correct value, use load tests, and make sure you are |
| well below the usage limit that would cause you to swap. Be |
| conservative - use a maximum heap size of 3GB for a 4GB |
| machine.</p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>Install the ZooKeeper Server Package. It can be downloaded |
| from: |
| </p> |
| |
| <p> |
| |
| <a href="http://hadoop.apache.org/zookeeper/releases.html"> |
| http://hadoop.apache.org/zookeeper/releases.html |
| </a> |
| |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>Create a configuration file. This file can be called anything. |
| Use the following settings as a starting point:</p> |
| |
| |
| <pre class="code"> |
| tickTime=2000 |
| dataDir=/var/zookeeper/ |
| clientPort=2181 |
| initLimit=5 |
| syncLimit=2 |
| server.1=zoo1:2888:3888 |
| server.2=zoo2:2888:3888 |
| server.3=zoo3:2888:3888</pre> |
| |
| |
| <p>You can find the meanings of these and other configuration |
| settings in the section <a href="#sc_configuration">Configuration Parameters</a>. A word |
| though about a few here:</p> |
| |
| |
| <p>Every machine that is part of the ZooKeeper ensemble should know |
| about every other machine in the ensemble. You accomplish this with |
| the series of lines of the form <strong>server.id=host:port:port</strong>. The parameters <strong>host</strong> and <strong>port</strong> are straightforward. You attribute the |
| server id to each machine by creating a file named |
| <span class="codefrag filename">myid</span>, one for each server, which resides in |
| that server's data directory, as specified by the configuration file |
| parameter <strong>dataDir</strong>.</p> |
| </li> |
| |
| |
| <li> |
| <p>The myid file |
| consists of a single line containing only the text of that machine's |
| id. So <span class="codefrag filename">myid</span> of server 1 would contain the text |
| "1" and nothing else. The id must be unique within the |
| ensemble and should have a value between 1 and 255.</p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>If your configuration file is set up, you can start a |
| ZooKeeper server:</p> |
| |
| |
| <p> |
| <span class="codefrag computeroutput">$ java -cp zookeeper.jar:lib/log4j-1.2.15.jar:conf \ |
| org.apache.zookeeper.server.quorum.QuorumPeerMain zoo.cfg |
| </span> |
| </p> |
| |
| |
| <p>QuorumPeerMain starts a ZooKeeper server, |
| <a href="http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/">JMX</a> |
| management beans are also registered which allows |
| management through a JMX management console. |
| The <a href="zookeeperJMX.html">ZooKeeper JMX |
| document</a> contains details on managing ZooKeeper with JMX. |
| </p> |
| |
| |
| <p>See the script <em>bin/zkServer.sh</em>, |
| which is included in the release, for an example |
| of starting server instances.</p> |
| |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>Test your deployment by connecting to the hosts:</p> |
| |
| |
| <ul> |
| |
| <li> |
| |
| <p>In Java, you can run the following command to execute |
| simple operations:</p> |
| |
| |
| <p> |
| <span class="codefrag computeroutput">$ java -cp zookeeper.jar:src/java/lib/log4j-1.2.15.jar:conf:src/java/lib/jline-0.9.94.jar \ |
| org.apache.zookeeper.ZooKeeperMain -server 127.0.0.1:2181</span> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>In C, you can compile either the single threaded client or |
| the multithreaded client: or n the c subdirectory in the |
| ZooKeeper sources. This compiles the single threaded |
| client:</p> |
| |
| |
| <p> |
| <span class="codefrag computeroutput">$ make cli_st</span> |
| </p> |
| |
| |
| <p>And this compiles the mulithreaded client:</p> |
| |
| |
| <p> |
| <span class="codefrag computeroutput">$ make cli_mt</span> |
| </p> |
| |
| </li> |
| |
| </ul> |
| |
| |
| <p>Running either program gives you a shell in which to execute |
| simple file-system-like operations. To connect to ZooKeeper with the |
| multithreaded client, for example, you would run:</p> |
| |
| |
| <p> |
| <span class="codefrag computeroutput">$ cli_mt 127.0.0.1:2181</span> |
| </p> |
| |
| </li> |
| |
| </ol> |
| <a name="N1011B"></a><a name="sc_singleAndDevSetup"></a> |
| <h3 class="h4">Single Server and Developer Setup</h3> |
| <p>If you want to setup ZooKeeper for development purposes, you will |
| probably want to setup a single server instance of ZooKeeper, and then |
| install either the Java or C client-side libraries and bindings on your |
| development machine.</p> |
| <p>The steps to setting up a single server instance are the similar |
| to the above, except the configuration file is simpler. You can find the |
| complete instructions in the <a href="zookeeperStarted.html#sc_InstallingSingleMode">Installing and |
| Running ZooKeeper in Single Server Mode</a> section of the <a href="zookeeperStarted.html">ZooKeeper Getting Started |
| Guide</a>.</p> |
| <p>For information on installing the client side libraries, refer to |
| the <a href="zookeeperProgrammers.html#Bindings">Bindings</a> |
| section of the <a href="zookeeperProgrammers.html">ZooKeeper |
| Programmer's Guide</a>.</p> |
| </div> |
| |
| |
| <a name="N1013C"></a><a name="ch_administration"></a> |
| <h2 class="h3">Administration</h2> |
| <div class="section"> |
| <p>This section contains information about running and maintaining |
| ZooKeeper and covers these topics: </p> |
| <ul> |
| |
| <li> |
| |
| <p> |
| <a href="#sc_designing">Designing a ZooKeeper Deployment</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_provisioning">Provisioning</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_strengthsAndLimitations">Things to Consider: ZooKeeper Strengths and Limitations</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_administering">Administering</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_maintenance">Maintenance</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_supervision">Supervision</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_monitoring">Monitoring</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_logging">Logging</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_troubleshooting">Troubleshooting</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_configuration">Configuration Parameters</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_zkCommands">ZooKeeper Commands: The Four Letter Words</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_dataFileManagement">Data File Management</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_commonProblems">Things to Avoid</a> |
| </p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <a href="#sc_bestPractices">Best Practices</a> |
| </p> |
| |
| </li> |
| |
| </ul> |
| <a name="N101B8"></a><a name="sc_designing"></a> |
| <h3 class="h4">Designing a ZooKeeper Deployment</h3> |
| <p>The reliablity of ZooKeeper rests on two basic assumptions.</p> |
| <ol> |
| |
| <li> |
| <p> Only a minority of servers in a deployment |
| will fail. <em>Failure</em> in this context |
| means a machine crash, or some error in the network that |
| partitions a server off from the majority.</p> |
| |
| </li> |
| |
| <li> |
| <p> Deployed machines operate correctly. To |
| operate correctly means to execute code correctly, to have |
| clocks that work properly, and to have storage and network |
| components that perform consistently.</p> |
| |
| </li> |
| |
| </ol> |
| <p>The sections below contain considerations for ZooKeeper |
| administrators to maximize the probability for these assumptions |
| to hold true. Some of these are cross-machines considerations, |
| and others are things you should consider for each and every |
| machine in your deployment.</p> |
| <a name="N101D4"></a><a name="sc_CrossMachineRequirements"></a> |
| <h4>Cross Machine Requirements</h4> |
| <p>For the ZooKeeper service to be active, there must be a |
| majority of non-failing machines that can communicate with |
| each other. To create a deployment that can tolerate the |
| failure of F machines, you should count on deploying 2xF+1 |
| machines. Thus, a deployment that consists of three machines |
| can handle one failure, and a deployment of five machines can |
| handle two failures. Note that a deployment of six machines |
| can only handle two failures since three machines is not a |
| majority. For this reason, ZooKeeper deployments are usually |
| made up of an odd number of machines.</p> |
| <p>To achieve the highest probability of tolerating a failure |
| you should try to make machine failures independent. For |
| example, if most of the machines share the same switch, |
| failure of that switch could cause a correlated failure and |
| bring down the service. The same holds true of shared power |
| circuits, cooling systems, etc.</p> |
| <a name="N101E1"></a><a name="Single+Machine+Requirements"></a> |
| <h4>Single Machine Requirements</h4> |
| <p>If ZooKeeper has to contend with other applications for |
| access to resourses like storage media, CPU, network, or |
| memory, its performance will suffer markedly. ZooKeeper has |
| strong durability guarantees, which means it uses storage |
| media to log changes before the operation responsible for the |
| change is allowed to complete. You should be aware of this |
| dependency then, and take great care if you want to ensure |
| that ZooKeeper operations aren’t held up by your media. Here |
| are some things you can do to minimize that sort of |
| degradation: |
| </p> |
| <ul> |
| |
| <li> |
| |
| <p>ZooKeeper's transaction log must be on a dedicated |
| device. (A dedicated partition is not enough.) ZooKeeper |
| writes the log sequentially, without seeking Sharing your |
| log device with other processes can cause seeks and |
| contention, which in turn can cause multi-second |
| delays.</p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p>Do not put ZooKeeper in a situation that can cause a |
| swap. In order for ZooKeeper to function with any sort of |
| timeliness, it simply cannot be allowed to swap. |
| Therefore, make certain that the maximum heap size given |
| to ZooKeeper is not bigger than the amount of real memory |
| available to ZooKeeper. For more on this, see |
| <a href="#sc_commonProblems">Things to Avoid</a> |
| below. </p> |
| |
| </li> |
| |
| </ul> |
| <a name="N101FF"></a><a name="sc_provisioning"></a> |
| <h3 class="h4">Provisioning</h3> |
| <p></p> |
| <a name="N10208"></a><a name="sc_strengthsAndLimitations"></a> |
| <h3 class="h4">Things to Consider: ZooKeeper Strengths and Limitations</h3> |
| <p></p> |
| <a name="N10211"></a><a name="sc_administering"></a> |
| <h3 class="h4">Administering</h3> |
| <p></p> |
| <a name="N1021A"></a><a name="sc_maintenance"></a> |
| <h3 class="h4">Maintenance</h3> |
| <p>Little long term maintenance is required for a ZooKeeper |
| cluster however you must be aware of the following:</p> |
| <a name="N10223"></a><a name="Ongoing+Data+Directory+Cleanup"></a> |
| <h4>Ongoing Data Directory Cleanup</h4> |
| <p>The ZooKeeper <a href="#var_datadir">Data |
| Directory</a> contains files which are a persistent copy |
| of the znodes stored by a particular serving ensemble. These |
| are the snapshot and transactional log files. As changes are |
| made to the znodes these changes are appended to a |
| transaction log, occasionally, when a log grows large, a |
| snapshot of the current state of all znodes will be written |
| to the filesystem. This snapshot supercedes all previous |
| logs. |
| </p> |
| <p>A ZooKeeper server <strong>will not remove |
| old snapshots and log files</strong>, this is the |
| responsibility of the operator. Every serving environment is |
| different and therefore the requirements of managing these |
| files may differ from install to install (backup for example). |
| </p> |
| <p>The PurgeTxnLog utility implements a simple retention |
| policy that administrators can use. The <a href="api/index.html">API docs</a> contains details on |
| calling conventions (arguments, etc...). |
| </p> |
| <p>In the following example the last count snapshots and |
| their corresponding logs are retained and the others are |
| deleted. The value of <count> should typically be |
| greater than 3 (although not required, this provides 3 backups |
| in the unlikely event a recent log has become corrupted). This |
| can be run as a cron job on the ZooKeeper server machines to |
| clean up the logs daily.</p> |
| <pre class="code"> java -cp zookeeper.jar:log4j.jar:conf org.apache.zookeeper.server.PurgeTxnLog <dataDir> <snapDir> -n <count></pre> |
| <a name="N10244"></a><a name="Debug+Log+Cleanup+%28log4j%29"></a> |
| <h4>Debug Log Cleanup (log4j)</h4> |
| <p>See the section on <a href="#sc_logging">logging</a> in this document. It is |
| expected that you will setup a rolling file appender using the |
| in-built log4j feature. The sample configuration file in the |
| release tar's conf/log4j.properties provides an example of |
| this. |
| </p> |
| <a name="N10253"></a><a name="sc_supervision"></a> |
| <h3 class="h4">Supervision</h3> |
| <p>You will want to have a supervisory process that manages |
| each of your ZooKeeper server processes (JVM). The ZK server is |
| designed to be "fail fast" meaning that it will shutdown |
| (process exit) if an error occurs that it cannot recover |
| from. As a ZooKeeper serving cluster is highly reliable, this |
| means that while the server may go down the cluster as a whole |
| is still active and serving requests. Additionally, as the |
| cluster is "self healing" the failed server once restarted will |
| automatically rejoin the ensemble w/o any manual |
| interaction.</p> |
| <p>Having a supervisory process such as <a href="http://cr.yp.to/daemontools.html">daemontools</a> or |
| <a href="http://en.wikipedia.org/wiki/Service_Management_Facility">SMF</a> |
| (other options for supervisory process are also available, it's |
| up to you which one you would like to use, these are just two |
| examples) managing your ZooKeeper server ensures that if the |
| process does exit abnormally it will automatically be restarted |
| and will quickly rejoin the cluster.</p> |
| <a name="N10268"></a><a name="sc_monitoring"></a> |
| <h3 class="h4">Monitoring</h3> |
| <p>The ZooKeeper service can be monitored in one of two |
| primary ways; 1) the command port through the use of <a href="#sc_zkCommands">4 letter words</a> and 2) <a href="zookeeperJMX.html">JMX</a>. See the appropriate section for |
| your environment/requirements.</p> |
| <a name="N1027A"></a><a name="sc_logging"></a> |
| <h3 class="h4">Logging</h3> |
| <p>ZooKeeper uses <strong>log4j</strong> version 1.2 as |
| its logging infrastructure. The ZooKeeper default <span class="codefrag filename">log4j.properties</span> |
| file resides in the <span class="codefrag filename">conf</span> directory. Log4j requires that |
| <span class="codefrag filename">log4j.properties</span> either be in the working directory |
| (the directory from which ZooKeeper is run) or be accessible from the classpath.</p> |
| <p>For more information, see |
| <a href="http://logging.apache.org/log4j/1.2/manual.html#defaultInit">Log4j Default Initialization Procedure</a> |
| of the log4j manual.</p> |
| <a name="N1029A"></a><a name="sc_troubleshooting"></a> |
| <h3 class="h4">Troubleshooting</h3> |
| <dl> |
| |
| <dt> |
| <term> Server not coming up because of file corruption</term> |
| </dt> |
| <dd> |
| <p>A server might not be able to read its database and fail to come up because of |
| some file corruption in the transaction logs of the ZooKeeper server. You will |
| see some IOException on loading ZooKeeper database. In such a case, |
| make sure all the other servers in your ensemble are up and working. Use "stat" |
| command on the command port to see if they are in good health. After you have verified that |
| all the other servers of the ensemble are up, you can go ahead and clean the database |
| of the corrupt server. Delete all the files in datadir/version-2 and datalogdir/version-2/. |
| Restart the server. |
| </p> |
| </dd> |
| |
| </dl> |
| <a name="N102AB"></a><a name="sc_configuration"></a> |
| <h3 class="h4">Configuration Parameters</h3> |
| <p>ZooKeeper's behavior is governed by the ZooKeeper configuration |
| file. This file is designed so that the exact same file can be used by |
| all the servers that make up a ZooKeeper server assuming the disk |
| layouts are the same. If servers use different configuration files, care |
| must be taken to ensure that the list of servers in all of the different |
| configuration files match.</p> |
| <a name="N102B4"></a><a name="sc_minimumConfiguration"></a> |
| <h4>Minimum Configuration</h4> |
| <p>Here are the minimum configuration keywords that must be defined |
| in the configuration file:</p> |
| <dl> |
| |
| <dt> |
| <term>clientPort</term> |
| </dt> |
| <dd> |
| <p>the port to listen for client connections; that is, the |
| port that clients attempt to connect to.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>dataDir</term> |
| </dt> |
| <dd> |
| <p>the location where ZooKeeper will store the in-memory |
| database snapshots and, unless specified otherwise, the |
| transaction log of updates to the database.</p> |
| <div class="note"> |
| <div class="label">Note</div> |
| <div class="content"> |
| |
| <p>Be careful where you put the transaction log. A |
| dedicated transaction log device is key to consistent good |
| performance. Putting the log on a busy device will adversely |
| effect performance.</p> |
| |
| </div> |
| </div> |
| </dd> |
| |
| |
| <dt> |
| <term>tickTime</term> |
| </dt> |
| <dd> |
| <p>the length of a single tick, which is the basic time unit |
| used by ZooKeeper, as measured in milliseconds. It is used to |
| regulate heartbeats, and timeouts. For example, the minimum |
| session timeout will be two ticks.</p> |
| </dd> |
| |
| </dl> |
| <a name="N102DB"></a><a name="sc_advancedConfiguration"></a> |
| <h4>Advanced Configuration</h4> |
| <p>The configuration settings in the section are optional. You can |
| use them to further fine tune the behaviour of your ZooKeeper servers. |
| Some can also be set using Java system properties, generally of the |
| form <em>zookeeper.keyword</em>. The exact system |
| property, when available, is noted below.</p> |
| <dl> |
| |
| <dt> |
| <term>dataLogDir</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>This option will direct the machine to write the |
| transaction log to the <strong>dataLogDir</strong> rather than the <strong>dataDir</strong>. This allows a dedicated log |
| device to be used, and helps avoid competition between logging |
| and snaphots.</p> |
| <div class="note"> |
| <div class="label">Note</div> |
| <div class="content"> |
| |
| <p>Having a dedicated log device has a large impact on |
| throughput and stable latencies. It is highly recommened to |
| dedicate a log device and set <strong>dataLogDir</strong> to point to a directory on |
| that device, and then make sure to point <strong>dataDir</strong> to a directory |
| <em>not</em> residing on that device.</p> |
| |
| </div> |
| </div> |
| </dd> |
| |
| |
| <dt> |
| <term>globalOutstandingLimit</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>zookeeper.globalOutstandingLimit.</strong>)</p> |
| <p>Clients can submit requests faster than ZooKeeper can |
| process them, especially if there are a lot of clients. To |
| prevent ZooKeeper from running out of memory due to queued |
| requests, ZooKeeper will throttle clients so that there is no |
| more than globalOutstandingLimit outstanding requests in the |
| system. The default limit is 1,000.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>preAllocSize</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>zookeeper.preAllocSize</strong>)</p> |
| <p>To avoid seeks ZooKeeper allocates space in the |
| transaction log file in blocks of preAllocSize kilobytes. The |
| default block size is 64M. One reason for changing the size of |
| the blocks is to reduce the block size if snapshots are taken |
| more often. (Also, see <strong>snapCount</strong>).</p> |
| </dd> |
| |
| |
| <dt> |
| <term>snapCount</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>zookeeper.snapCount</strong>)</p> |
| <p>ZooKeeper logs transactions to a transaction |
| log. After snapCount transactions are written to a log |
| file a snapshot is started and a new transaction log |
| file is created. The default snapCount is |
| 100,000.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>traceFile</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>requestTraceFile</strong>)</p> |
| <p>If this option is defined, requests will be will logged to |
| a trace file named traceFile.year.month.day. Use of this option |
| provides useful debugging information, but will impact |
| performance. (Note: The system property has no zookeeper prefix, |
| and the configuration variable name is different from the system |
| property. Yes - it's not consistent, and it's annoying.)</p> |
| </dd> |
| |
| |
| <dt> |
| <term>maxClientCnxns</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Limits the number of concurrent connections (at the socket |
| level) that a single client, identified by IP address, may make |
| to a single member of the ZooKeeper ensemble. This is used to |
| prevent certain classes of DoS attacks, including file |
| descriptor exhaustion. The default is 10. Setting this to 0 |
| entirely removes the limit on concurrent connections.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>clientPortBindAddress</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> the |
| address (ipv4, ipv6 or hostname) to listen for client |
| connections; that is, the address that clients attempt |
| to connect to. This is optional, by default we bind in |
| such a way that any connection to the <strong>clientPort</strong> for any |
| address/interface/nic on the server will be |
| accepted.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>minSessionTimeout</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p> |
| <strong>New in 3.3.0:</strong> the |
| minimum session timeout in milliseconds that the server |
| will allow the client to negotiate. Defaults to 2 times |
| the <strong>tickTime</strong>.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>maxSessionTimeout</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p> |
| <strong>New in 3.3.0:</strong> the |
| maximum session timeout in milliseconds that the server |
| will allow the client to negotiate. Defaults to 20 times |
| the <strong>tickTime</strong>.</p> |
| </dd> |
| |
| </dl> |
| <a name="N1036C"></a><a name="sc_clusterOptions"></a> |
| <h4>Cluster Options</h4> |
| <p>The options in this section are designed for use with an ensemble |
| of servers -- that is, when deploying clusters of servers.</p> |
| <dl> |
| |
| <dt> |
| <term>electionAlg</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Election implementation to use. A value of "0" corresponds |
| to the original UDP-based version, "1" corresponds to the |
| non-authenticated UDP-based version of fast leader election, "2" |
| corresponds to the authenticated UDP-based version of fast |
| leader election, and "3" corresponds to TCP-based version of |
| fast leader election. Currently, algorithm 3 is the default</p> |
| <div class="note"> |
| <div class="label">Note</div> |
| <div class="content"> |
| |
| <p> The implementations of leader election |
| 1 and 2 are currently not supported, and we have the intention |
| of deprecating them in the near future. Implementations 0 and 3 are |
| currently supported, and we plan to keep supporting them in the near future. |
| To avoid having to support multiple versions of leader election unecessarily, |
| we may eventually consider deprecating algorithm 0 as well, but we will plan |
| according to the needs of the community. |
| </p> |
| |
| </div> |
| </div> |
| </dd> |
| |
| |
| <dt> |
| <term>initLimit</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Amount of time, in ticks (see <a href="#id_tickTime">tickTime</a>), to allow followers to |
| connect and sync to a leader. Increased this value as needed, if |
| the amount of data managed by ZooKeeper is large.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>leaderServes</term> |
| </dt> |
| <dd> |
| <p>(Java system property: zookeeper.<strong>leaderServes</strong>)</p> |
| <p>Leader accepts client connections. Default value is "yes". |
| The leader machine coordinates updates. For higher update |
| throughput at thes slight expense of read throughput the leader |
| can be configured to not accept clients and focus on |
| coordination. The default to this option is yes, which means |
| that a leader will accept client connections.</p> |
| <div class="note"> |
| <div class="label">Note</div> |
| <div class="content"> |
| |
| <p>Turning on leader selection is highly recommended when |
| you have more than three ZooKeeper servers in an ensemble.</p> |
| |
| </div> |
| </div> |
| </dd> |
| |
| |
| <dt> |
| <term>server.x=[hostname]:nnnnn[:nnnnn], etc</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>servers making up the ZooKeeper ensemble. When the server |
| starts up, it determines which server it is by looking for the |
| file <span class="codefrag filename">myid</span> in the data directory. That file |
| contains the server number, in ASCII, and it should match |
| <strong>x</strong> in <strong>server.x</strong> in the left hand side of this |
| setting.</p> |
| <p>The list of servers that make up ZooKeeper servers that is |
| used by the clients must match the list of ZooKeeper servers |
| that each ZooKeeper server has.</p> |
| <p>There are two port numbers <strong>nnnnn</strong>. |
| The first followers use to connect to the leader, and the second is for |
| leader election. The leader election port is only necessary if electionAlg |
| is 1, 2, or 3 (default). If electionAlg is 0, then the second port is not |
| necessary. If you want to test multiple servers on a single machine, then |
| different ports can be used for each server.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>syncLimit</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Amount of time, in ticks (see <a href="#id_tickTime">tickTime</a>), to allow followers to sync |
| with ZooKeeper. If followers fall too far behind a leader, they |
| will be dropped.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>group.x=nnnnn[:nnnnn]</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Enables a hierarchical quorum construction."x" is a group identifier |
| and the numbers following the "=" sign correspond to server identifiers. |
| The left-hand side of the assignment is a colon-separated list of server |
| identifiers. Note that groups must be disjoint and the union of all groups |
| must be the ZooKeeper ensemble. </p> |
| <p> You will find an example <a href="zookeeperHierarchicalQuorums.html">here</a> |
| |
| </p> |
| </dd> |
| |
| |
| <dt> |
| <term>weight.x=nnnnn</term> |
| </dt> |
| <dd> |
| <p>(No Java system property)</p> |
| <p>Used along with "group", it assigns a weight to a server when |
| forming quorums. Such a value corresponds to the weight of a server |
| when voting. There are a few parts of ZooKeeper that require voting |
| such as leader election and the atomic broadcast protocol. By default |
| the weight of server is 1. If the configuration defines groups, but not |
| weights, then a value of 1 will be assigned to all servers. |
| </p> |
| <p> You will find an example <a href="zookeeperHierarchicalQuorums.html">here</a> |
| |
| </p> |
| </dd> |
| |
| </dl> |
| <p></p> |
| <a name="N103EC"></a><a name="sc_authOptions"></a> |
| <h4>Authentication & Authorization Options</h4> |
| <p>The options in this section allow control over |
| authentication/authorization performed by the service.</p> |
| <dl> |
| |
| <dt> |
| <term>zookeeper.DigestAuthenticationProvider.superDigest</term> |
| </dt> |
| <dd> |
| <p>(Java system property only: <strong>zookeeper.DigestAuthenticationProvider.superDigest</strong>)</p> |
| <p>By default this feature is <strong>disabled</strong> |
| </p> |
| <p> |
| <strong>New in 3.2:</strong> |
| Enables a ZooKeeper ensemble administrator to access the |
| znode hierarchy as a "super" user. In particular no ACL |
| checking occurs for a user authenticated as |
| super.</p> |
| <p>org.apache.zookeeper.server.auth.DigestAuthenticationProvider |
| can be used to generate the superDigest, call it with |
| one parameter of "super:<password>". Provide the |
| generated "super:<data>" as the system property value |
| when starting each server of the ensemble.</p> |
| <p>When authenticating to a ZooKeeper server (from a |
| ZooKeeper client) pass a scheme of "digest" and authdata |
| of "super:<password>". Note that digest auth passes |
| the authdata in plaintext to the server, it would be |
| prudent to use this authentication method only on |
| localhost (not over the network) or over an encrypted |
| connection.</p> |
| </dd> |
| |
| </dl> |
| <a name="N1040F"></a><a name="Unsafe+Options"></a> |
| <h4>Unsafe Options</h4> |
| <p>The following options can be useful, but be careful when you use |
| them. The risk of each is explained along with the explanation of what |
| the variable does.</p> |
| <dl> |
| |
| <dt> |
| <term>forceSync</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>zookeeper.forceSync</strong>)</p> |
| <p>Requires updates to be synced to media of the transaction |
| log before finishing processing the update. If this option is |
| set to no, ZooKeeper will not require updates to be synced to |
| the media.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>jute.maxbuffer:</term> |
| </dt> |
| <dd> |
| <p>(Java system property:<strong> |
| jute.maxbuffer</strong>)</p> |
| <p>This option can only be set as a Java system property. |
| There is no zookeeper prefix on it. It specifies the maximum |
| size of the data that can be stored in a znode. The default is |
| 0xfffff, or just under 1M. If this option is changed, the system |
| property must be set on all servers and clients otherwise |
| problems will arise. This is really a sanity check. ZooKeeper is |
| designed to store data on the order of kilobytes in size.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>skipACL</term> |
| </dt> |
| <dd> |
| <p>(Java system property: <strong>zookeeper.skipACL</strong>)</p> |
| <p>Skips ACL checks. This results in a boost in throughput, |
| but opens up full access to the data tree to everyone.</p> |
| </dd> |
| |
| </dl> |
| <a name="N10441"></a><a name="sc_zkCommands"></a> |
| <h3 class="h4">ZooKeeper Commands: The Four Letter Words</h3> |
| <p>ZooKeeper responds to a small set of commands. Each command is |
| composed of four letters. You issue the commands to ZooKeeper via telnet |
| or nc, at the client port.</p> |
| <p>Three of the more interesting commands: "stat" gives some |
| general information about the server and connected clients, |
| while "srvr" and "cons" give extended details on server and |
| connections respectively.</p> |
| <dl> |
| |
| <dt> |
| <term>conf</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Print |
| details about serving configuration.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>cons</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> List |
| full connection/session details for all clients connected |
| to this server. Includes information on numbers of packets |
| received/sent, session id, operation latencies, last |
| operation performed, etc...</p> |
| </dd> |
| |
| |
| <dt> |
| <term>crst</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Reset |
| connection/session statistics for all connections.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>dump</term> |
| </dt> |
| <dd> |
| <p>Lists the outstanding sessions and ephemeral nodes. This |
| only works on the leader.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>envi</term> |
| </dt> |
| <dd> |
| <p>Print details about serving environment</p> |
| </dd> |
| |
| |
| <dt> |
| <term>ruok</term> |
| </dt> |
| <dd> |
| <p>Tests if server is running in a non-error state. The server |
| will respond with imok if it is running. Otherwise it will not |
| respond at all.</p> |
| <p>A response of "imok" does not necessarily indicate that the |
| server has joined the quorum, just that the server process is active |
| and bound to the specified client port. Use "stat" for details on |
| state wrt quorum and client connection information.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>srst</term> |
| </dt> |
| <dd> |
| <p>Reset server statistics.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>srvr</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Lists |
| full details for the server.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>stat</term> |
| </dt> |
| <dd> |
| <p>Lists brief details for the server and connected |
| clients.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>wchs</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Lists |
| brief information on watches for the server.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>wchc</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Lists |
| detailed information on watches for the server, by |
| session. This outputs a list of sessions(connections) |
| with associated watches (paths). Note, depending on the |
| number of watches this operation may be expensive (ie |
| impact server performance), use it carefully.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>wchp</term> |
| </dt> |
| <dd> |
| <p> |
| <strong>New in 3.3.0:</strong> Lists |
| detailed information on watches for the server, by path. |
| This outputs a list of paths (znodes) with associated |
| sessions. Note, depending on the number of watches this |
| operation may be expensive (ie impact server performance), |
| use it carefully.</p> |
| </dd> |
| |
| </dl> |
| <p>Here's an example of the <strong>ruok</strong> |
| command:</p> |
| <pre class="code">$ echo ruok | nc 127.0.0.1 5111 |
| imok |
| </pre> |
| <a name="N104BE"></a><a name="sc_dataFileManagement"></a> |
| <h3 class="h4">Data File Management</h3> |
| <p>ZooKeeper stores its data in a data directory and its transaction |
| log in a transaction log directory. By default these two directories are |
| the same. The server can (and should) be configured to store the |
| transaction log files in a separate directory than the data files. |
| Throughput increases and latency decreases when transaction logs reside |
| on a dedicated log devices.</p> |
| <a name="N104C7"></a><a name="The+Data+Directory"></a> |
| <h4>The Data Directory</h4> |
| <p>This directory has two files in it:</p> |
| <ul> |
| |
| <li> |
| |
| <p> |
| <span class="codefrag filename">myid</span> - contains a single integer in |
| human readable ASCII text that represents the server id.</p> |
| |
| </li> |
| |
| |
| <li> |
| |
| <p> |
| <span class="codefrag filename">snapshot.<zxid></span> - holds the fuzzy |
| snapshot of a data tree.</p> |
| |
| </li> |
| |
| </ul> |
| <p>Each ZooKeeper server has a unique id. This id is used in two |
| places: the <span class="codefrag filename">myid</span> file and the configuration file. |
| The <span class="codefrag filename">myid</span> file identifies the server that |
| corresponds to the given data directory. The configuration file lists |
| the contact information for each server identified by its server id. |
| When a ZooKeeper server instance starts, it reads its id from the |
| <span class="codefrag filename">myid</span> file and then, using that id, reads from the |
| configuration file, looking up the port on which it should |
| listen.</p> |
| <p>The <span class="codefrag filename">snapshot</span> files stored in the data |
| directory are fuzzy snapshots in the sense that during the time the |
| ZooKeeper server is taking the snapshot, updates are occurring to the |
| data tree. The suffix of the <span class="codefrag filename">snapshot</span> file names |
| is the <em>zxid</em>, the ZooKeeper transaction id, of the |
| last committed transaction at the start of the snapshot. Thus, the |
| snapshot includes a subset of the updates to the data tree that |
| occurred while the snapshot was in process. The snapshot, then, may |
| not correspond to any data tree that actually existed, and for this |
| reason we refer to it as a fuzzy snapshot. Still, ZooKeeper can |
| recover using this snapshot because it takes advantage of the |
| idempotent nature of its updates. By replaying the transaction log |
| against fuzzy snapshots ZooKeeper gets the state of the system at the |
| end of the log.</p> |
| <a name="N10503"></a><a name="The+Log+Directory"></a> |
| <h4>The Log Directory</h4> |
| <p>The Log Directory contains the ZooKeeper transaction logs. |
| Before any update takes place, ZooKeeper ensures that the transaction |
| that represents the update is written to non-volatile storage. A new |
| log file is started each time a snapshot is begun. The log file's |
| suffix is the first zxid written to that log.</p> |
| <a name="N1050D"></a><a name="sc_filemanagement"></a> |
| <h4>File Management</h4> |
| <p>The format of snapshot and log files does not change between |
| standalone ZooKeeper servers and different configurations of |
| replicated ZooKeeper servers. Therefore, you can pull these files from |
| a running replicated ZooKeeper server to a development machine with a |
| stand-alone ZooKeeper server for trouble shooting.</p> |
| <p>Using older log and snapshot files, you can look at the previous |
| state of ZooKeeper servers and even restore that state. The |
| LogFormatter class allows an administrator to look at the transactions |
| in a log.</p> |
| <p>The ZooKeeper server creates snapshot and log files, but |
| never deletes them. The retention policy of the data and log |
| files is implemented outside of the ZooKeeper server. The |
| server itself only needs the latest complete fuzzy snapshot |
| and the log files from the start of that snapshot. See the |
| <a href="#sc_maintenance">maintenance</a> section in |
| this document for more details on setting a retention policy |
| and maintenance of ZooKeeper storage. |
| </p> |
| <a name="N10522"></a><a name="sc_commonProblems"></a> |
| <h3 class="h4">Things to Avoid</h3> |
| <p>Here are some common problems you can avoid by configuring |
| ZooKeeper correctly:</p> |
| <dl> |
| |
| <dt> |
| <term>inconsistent lists of servers</term> |
| </dt> |
| <dd> |
| <p>The list of ZooKeeper servers used by the clients must match |
| the list of ZooKeeper servers that each ZooKeeper server has. |
| Things work okay if the client list is a subset of the real list, |
| but things will really act strange if clients have a list of |
| ZooKeeper servers that are in different ZooKeeper clusters. Also, |
| the server lists in each Zookeeper server configuration file |
| should be consistent with one another.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>incorrect placement of transasction log</term> |
| </dt> |
| <dd> |
| <p>The most performance critical part of ZooKeeper is the |
| transaction log. ZooKeeper syncs transactions to media before it |
| returns a response. A dedicated transaction log device is key to |
| consistent good performance. Putting the log on a busy device will |
| adversely effect performance. If you only have one storage device, |
| put trace files on NFS and increase the snapshotCount; it doesn't |
| eliminate the problem, but it should mitigate it.</p> |
| </dd> |
| |
| |
| <dt> |
| <term>incorrect Java heap size</term> |
| </dt> |
| <dd> |
| <p>You should take special care to set your Java max heap size |
| correctly. In particular, you should not create a situation in |
| which ZooKeeper swaps to disk. The disk is death to ZooKeeper. |
| Everything is ordered, so if processing one request swaps the |
| disk, all other queued requests will probably do the same. the |
| disk. DON'T SWAP.</p> |
| <p>Be conservative in your estimates: if you have 4G of RAM, do |
| not set the Java max heap size to 6G or even 4G. For example, it |
| is more likely you would use a 3G heap for a 4G machine, as the |
| operating system and the cache also need memory. The best and only |
| recommend practice for estimating the heap size your system needs |
| is to run load tests, and then make sure you are well below the |
| usage limit that would cause the system to swap.</p> |
| </dd> |
| |
| </dl> |
| <a name="N10546"></a><a name="sc_bestPractices"></a> |
| <h3 class="h4">Best Practices</h3> |
| <p>For best results, take note of the following list of good |
| Zookeeper practices:</p> |
| <p>For multi-tennant installations see the <a href="zookeeperProgrammers.html#ch_zkSessions">section</a> |
| detailing ZooKeeper "chroot" support, this can be very useful |
| when deploying many applications/services interfacing to a |
| single ZooKeeper cluster.</p> |
| </div> |
| |
| <p align="right"> |
| <font size="-2"></font> |
| </p> |
| </div> |
| <!--+ |
| |end content |
| +--> |
| <div class="clearboth"> </div> |
| </div> |
| <div id="footer"> |
| <!--+ |
| |start bottomstrip |
| +--> |
| <div class="lastmodified"> |
| <script type="text/javascript"><!-- |
| document.write("Last Published: " + document.lastModified); |
| // --></script> |
| </div> |
| <div class="copyright"> |
| Copyright © |
| 2008 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a> |
| </div> |
| <!--+ |
| |end bottomstrip |
| +--> |
| </div> |
| </body> |
| </html> |