| </div></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo-black.svg" alt="Apache Pulsar logo" class="themedImage_ToTc themedImage--light_HNdA" height="25" width="127"><img src="/img/logo-black.svg" alt="Apache Pulsar logo" class="themedImage_ToTc themedImage--dark_i4oU" height="25" width="127"></div><b class="navbar__title text--truncate"></b></a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Get Started</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/3.2.x/concepts-overview/">Concepts</a></li><li><a class="dropdown__link" href="/docs/3.2.x/">Quickstart</a></li><li><a class="dropdown__link" href="/ecosystem/">Ecosystem</a></li></ul></div><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/docs/next/">Docs</a><a class="navbar__item navbar__link" href="/features/">Features</a><a class="navbar__item navbar__link" href="/use-cases/">Use Cases</a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link community-dropdown">Community</a><ul class="dropdown__menu"><li><a class="dropdown__link scroll-link scroll-welcome" id="scroll-welcome" href="/community/">Welcome</a></li><li><a class="dropdown__link scroll-link scroll-discussions" id="scroll-discussions" href="/community/#section-discussions">Discussions</a></li><li><a class="dropdown__link scroll-link" id="scroll-governance" href="/community/#section-governance">Governance</a></li><li><a class="dropdown__link scroll-link" id="scroll-community" href="/community/#section-community">Meet the Community</a></li><li><a class="dropdown__link scroll-link" id="scroll-contribute" href="/community/#section-contribute">Contribute</a></li><li><a class="dropdown__link" href="/contribute/">Contribution Guide</a></li><li><a href="https://github.com/apache/pulsar/wiki" target="_blank" rel="noopener noreferrer" class="dropdown__link">Wiki</a></li><li><a href="https://github.com/apache/pulsar/issues" target="_blank" rel="noopener noreferrer" class="dropdown__link">Issue Tracking</a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Learn</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/blog/">Blog</a></li><li><a class="dropdown__link" href="/books/">Books</a></li><li><a class="dropdown__link" href="/case-studies/">Case Studies</a></li><li><a class="dropdown__link" href="/articles/">Articles</a></li><li><a class="dropdown__link" href="/presentations/">Presentations</a></li><li><a class="dropdown__link" href="/events/">Events</a></li></ul></div></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link navbar_download_button" href="/download/">Download</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20" aria-hidden="true"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebarViewport_Xe31"><div class="sidebar_mhZE"><div class="sidebarVersionSwitch_EHpo">Version:<div class="navbar__item dropdown dropdown--hoverable"><a aria-current="page" class="navbar__link active" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/next/">Next</a><ul class="dropdown__menu"><li><a class="dropdown__link">Next</a></li><li><a class="dropdown__link">3.2.x</a></li><li><a class="dropdown__link">3.0.x LTS</a></li><li><a class="dropdown__link">Others</a></li></ul></div></div><nav aria-label="Docs sidebar" class="menu thin-scrollbar menu_SIkG menuWithAnnouncementBar_GW3s"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/next/">About</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/next/getting-started-home/">Get Started</a><button aria-label="Toggle the collapsible sidebar category 'Get Started'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/concepts-overview/">Concepts and Architecture</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/schema-overview/">Pulsar Schema</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/functions-overview/">Pulsar Functions</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/io-overview/">Pulsar IO</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/sql-overview/">Pulsar SQL</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/tiered-storage-overview/">Tiered Storage</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/txn-why/">Transactions</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--active" aria-expanded="true" href="/docs/next/install-deploy-upgrade-landing/">Deployment</a><button aria-label="Toggle the collapsible sidebar category 'Deployment'" type="button" class="clean-btn menu__caret"></button></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/next/deploy-aws/">Terraform (AWS)</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/next/deploy-ibm/">IBM Cloud Services</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" tabindex="0" href="/docs/next/deploy-kubernetes/">Kubernetes</a><button aria-label="Toggle the collapsible sidebar category 'Kubernetes'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/next/deploy-bare-metal/">Bare metal</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/next/deploy-bare-metal-multi-cluster/">Bare metal multi-cluster</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/next/deploy-docker/">Docker</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/administration-zk-bk/">Administration</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/administration-stats/">Observability</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/security-overview/">Security</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/performance-pulsar-perf/">Performance</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/client-libraries/">Client Libraries</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/admin-api-overview/">Admin API</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/next/adaptors-kafka/">Adaptors</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/next/how-to-landing/">Tutorials</a><button aria-label="Toggle the collapsible sidebar category 'Tutorials'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/next/developers-landing/">Development</a><button aria-label="Toggle the collapsible sidebar category 'Development'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/next/reference-landing/">Reference</a><button aria-label="Toggle the collapsible sidebar category 'Reference'" type="button" class="clean-btn menu__caret"></button></div></li></ul></nav></div></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_YNFT"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item"><a class="breadcrumbs__link" itemprop="item" href="/docs/next/install-deploy-upgrade-landing/"><span itemprop="name">Deployment</span></a><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Bare metal multi-cluster</span><meta itemprop="position" content="2"></li></ul></nav><span class="theme-doc-version-badge badge badge--secondary">Version: Next</span><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Deploying a multi-cluster on bare metal</h1></header><div class="theme-admonition theme-admonition-tip alert alert--success admonition_LlT9"><div class="admonitionHeading_tbUL"><span class="admonitionIcon_kALy"><svg viewBox="0 0 12 16"><path fill-rule="evenodd" d="M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"></path></svg></span>tip</div><div class="admonitionContent_S0QG"><ol><li>You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or a single team. If you need to run a multi-cluster Pulsar instance, see the <a href="/docs/next/deploy-bare-metal-multi-cluster/">guide</a>.</li><li>If you want to use all built-in <a href="/docs/next/io-overview/">Pulsar IO</a> connectors, you need to download <code>apache-pulsar-io-connectors</code> package and install <code>apache-pulsar-io-connectors</code> under <code>connectors</code> directory in the pulsar directory on every broker node or every function-worker node if you have run a separate cluster of function workers for <a href="/docs/next/functions-overview/">Pulsar Functions</a>.</li><li>If you want to use <a href="/docs/next/concepts-tiered-storage/">Tiered Storage</a> feature in your Pulsar deployment, you need to download <code>apache-pulsar-offloaders</code> package and install <code>apache-pulsar-offloaders</code> under <code>offloaders</code> directory in the Pulsar directory on every broker node. For more details on how to configure this feature, you can refer to the <a href="/docs/next/cookbooks-tiered-storage/">Tiered storage cookbook</a>.</li></ol></div></div><p>A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using <a href="/docs/next/administration-geo/">geo-replication</a>.</p><blockquote><h4 class="anchor anchorWithStickyNavbar_LWe7" id="run-pulsar-locally-or-on-kubernetes">Run Pulsar locally or on Kubernetes?<a href="#run-pulsar-locally-or-on-kubernetes" class="hash-link" aria-label="Direct link to Run Pulsar locally or on Kubernetes?" title="Direct link to Run Pulsar locally or on Kubernetes?"></a></h4><p>This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the <a href="/docs/next/getting-started-standalone/">Setting up a local cluster</a> guide. If you want to run Pulsar on <a href="https://kubernetes.io" target="_blank" rel="noopener noreferrer">Kubernetes</a>, see the <a href="/docs/next/deploy-kubernetes/">Pulsar on Kubernetes</a> guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services.</p></blockquote><p>Deploying a multi-cluster Pulsar instance on bare metal consists of the following steps.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="system-requirement">System requirement<a href="#system-requirement" class="hash-link" aria-label="Direct link to System requirement" title="Direct link to System requirement"></a></h2><p>Currently, Pulsar is available for 64-bit <strong>macOS</strong> and <strong>Linux</strong>. See <a href="/docs/next/getting-started-docker/">Run Pulsar In Docker</a> if you want to run Pulsar on <strong>Windows</strong>.</p><p>Also, you need the proper 64-bit JRE/JDK version installed. Please refer to <a href="https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation" target="_blank" rel="noopener noreferrer">Pulsar Runtime Java Version Recommendation</a>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="install-pulsar">Install Pulsar<a href="#install-pulsar" class="hash-link" aria-label="Direct link to Install Pulsar" title="Direct link to Install Pulsar"></a></h2><p>To get started running Pulsar, download a binary tarball release in one of the following ways:</p><ul><li><p>by clicking the link below and downloading the release from an Apache mirror:</p><ul><li><a href="https://archive.apache.org/dist/pulsar/pulsar-3.2.2/apache-pulsar-3.2.2-bin.tar.gz" target="_blank" rel="noopener noreferrer" download="">Pulsar 3.2.2 binary release</a></li></ul></li><li><p>from the Pulsar <a href="/download/">downloads page</a></p></li><li><p>from the Pulsar <a href="https://github.com/apache/pulsar/releases/latest" target="_blank" rel="noopener noreferrer">releases page</a></p></li><li><p>using <a href="https://www.gnu.org/software/wget" target="_blank" rel="noopener noreferrer">wget</a>:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">wget</span><span class="token plain"> </span><span class="token string" style="color:rgb(255, 121, 198)">'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=pulsar/pulsar-3.2.2/apache-pulsar-3.2.2-bin.tar.gz'</span><span class="token plain"> -O apache-pulsar-3.2.2-bin.tar.gz</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></li></ul><p>Once you download the tarball, untar it and <code>cd</code> into the resulting directory:</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">tar</span><span class="token plain"> xvfz apache-pulsar-3.2.2-bin.tar.gz</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">cd</span><span class="token plain"> apache-pulsar-3.2.2</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>The Pulsar binary package initially contains the following directories:</p><table><thead><tr><th align="left">Directory</th><th align="left">Contains</th></tr></thead><tbody><tr><td align="left"><code>bin</code></td><td align="left"><a href="/docs/next/reference-cli-tools/">Command-line tools</a> of Pulsar, such as <a href="/docs/next/reference-cli-tools/"><code>pulsar</code></a> and <a href="/reference/#/next/pulsar-admin/" target="_blank" rel="noopener noreferrer"><code>pulsar-admin</code></a></td></tr><tr><td align="left"><code>conf</code></td><td align="left">Configuration files for Pulsar, including for <a href="/docs/next/reference-configuration/#broker">broker configuration</a>, <a href="/docs/next/reference-configuration/#zookeeper">ZooKeeper configuration</a>, and more</td></tr><tr><td align="left"><code>examples</code></td><td align="left">A Java JAR file containing example <a href="/docs/next/functions-overview/">Pulsar Functions</a></td></tr><tr><td align="left"><code>lib</code></td><td align="left">The <a href="https://en.wikipedia.org/wiki/JAR_(file_format)" target="_blank" rel="noopener noreferrer">JAR</a> files that Pulsar uses</td></tr><tr><td align="left"><code>licenses</code></td><td align="left">License files, in <code>.txt</code> form, for various components of the Pulsar codebase</td></tr></tbody></table><p>The following directories are created once you begin running Pulsar:</p><table><thead><tr><th align="left">Directory</th><th align="left">Contains</th></tr></thead><tbody><tr><td align="left"><code>data</code></td><td align="left">The data storage directory that ZooKeeper and BookKeeper use</td></tr><tr><td align="left"><code>instances</code></td><td align="left">Artifacts created for <a href="/docs/next/functions-overview/">Pulsar Functions</a></td></tr><tr><td align="left"><code>logs</code></td><td align="left">Logs that the installation creates</td></tr></tbody></table><h2 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-deploy-zookeeper">Step 1: Deploy ZooKeeper<a href="#step-1-deploy-zookeeper" class="hash-link" aria-label="Direct link to Step 1: Deploy ZooKeeper" title="Direct link to Step 1: Deploy ZooKeeper"></a></h2><p>Each Pulsar instance relies on two separate ZooKeeper quorums.</p><ul><li>Local ZooKeeper operates at the cluster level and provides cluster-specific configuration management and coordination. Each Pulsar cluster needs a dedicated ZooKeeper cluster.</li><li>Configuration Store operates at the instance level and provides configuration management for the entire system (and thus across clusters). An independent cluster of machines or the same machines that local ZooKeeper uses can provide the configuration store quorum.</li></ul><p>You can use an independent cluster of machines or the same machines used by local ZooKeeper to provide the configuration store quorum.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="deploy-local-zookeeper">Deploy local ZooKeeper<a href="#deploy-local-zookeeper" class="hash-link" aria-label="Direct link to Deploy local ZooKeeper" title="Direct link to Deploy local ZooKeeper"></a></h3><p>ZooKeeper manages a variety of essential coordination-related and configuration-related tasks for Pulsar.</p><p>You need to stand up one local ZooKeeper cluster per Pulsar cluster for deploying a Pulsar instance.</p><p>To begin, add all ZooKeeper servers to the quorum configuration specified in the <a href="/docs/next/reference-configuration/#zookeeper"><code>conf/zookeeper.conf</code></a> file. Add a <code>server.N</code> line for each node in the cluster to the configuration, where <code>N</code> is the number of the ZooKeeper node. The following is an example of a three-node cluster:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.1</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-west.example.com:2888:3888</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.2</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.us-west.example.com:2888:3888</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.3</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.us-west.example.com:2888:3888</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>On each host, you need to specify the ID of the node in the <code>myid</code> file of each node, which is in <code>data/zookeeper</code> folder of each server by default (you can change the file location via the <a href="/docs/next/reference-configuration/#zookeeper-dataDir"><code>dataDir</code></a> parameter).</p><div class="theme-admonition theme-admonition-tip alert alert--success admonition_LlT9"><div class="admonitionHeading_tbUL"><span class="admonitionIcon_kALy"><svg viewBox="0 0 12 16"><path fill-rule="evenodd" d="M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"></path></svg></span>tip</div><div class="admonitionContent_S0QG"><p>See the <a href="https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup" target="_blank" rel="noopener noreferrer">Multi-server setup guide</a> in the ZooKeeper documentation for detailed information on <code>myid</code> and more.</p></div></div><p>On a ZooKeeper server at <code>zk1.us-west.example.com</code>, for example, you could set the <code>myid</code> value like this:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">mkdir</span><span class="token plain"> -p data/zookeeper</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">echo</span><span class="token plain"> </span><span class="token number">1</span><span class="token plain"> </span><span class="token operator">></span><span class="token plain"> data/zookeeper/myid</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>On <code>zk2.us-west.example.com</code> the command looks like <code>echo 2 > data/zookeeper/myid</code> and so on.</p><p>Once you add each server to the <code>zookeeper.conf</code> configuration and each server has the appropriate <code>myid</code> entry, you can start ZooKeeper on all hosts (in the background, using nohup) with the <a href="/docs/next/reference-cli-tools/"><code>pulsar-daemon</code></a> CLI tool:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-daemon start zookeeper</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="deploy-the-configuration-store">Deploy the configuration store<a href="#deploy-the-configuration-store" class="hash-link" aria-label="Direct link to Deploy the configuration store" title="Direct link to Deploy the configuration store"></a></h3><p>The ZooKeeper cluster configured and started up in the section above is a local ZooKeeper cluster that you can use to manage a single Pulsar cluster. In addition to a local cluster, however, a full Pulsar instance also requires a configuration store for handling some instance-level configuration and coordination tasks.</p><p>If you deploy a single-cluster instance, you do not need a separate cluster for the configuration store. If, however, you deploy a multi-cluster instance, you should stand up a separate ZooKeeper cluster for configuration tasks.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="single-cluster-pulsar-instance">Single-cluster Pulsar instance<a href="#single-cluster-pulsar-instance" class="hash-link" aria-label="Direct link to Single-cluster Pulsar instance" title="Direct link to Single-cluster Pulsar instance"></a></h4><p>If your Pulsar instance consists of just one cluster, then you can deploy a configuration store on the same machines as the local ZooKeeper quorum but run on different TCP ports.</p><p>To deploy a ZooKeeper configuration store in a single-cluster instance, add the same ZooKeeper servers to the local quorum. You need to use the configuration file in <a href="/docs/next/reference-configuration/#configuration-store"><code>conf/global_zookeeper.conf</code></a> using the same method for <a href="#deploy-local-zookeeper">local ZooKeeper</a>, but make sure to use a different port (2181 is the default for ZooKeeper). The following is an example that uses port 2184 for a three-node ZooKeeper cluster:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token key attr-name" style="color:rgb(241, 250, 140)">clientPort</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">2184</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.1</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-west.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.2</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.us-west.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.3</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.us-west.example.com:2185:2186</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>As before, create the <code>myid</code> files for each server on <code>data/global-zookeeper/myid</code>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="multi-cluster-pulsar-instance">Multi-cluster Pulsar instance<a href="#multi-cluster-pulsar-instance" class="hash-link" aria-label="Direct link to Multi-cluster Pulsar instance" title="Direct link to Multi-cluster Pulsar instance"></a></h4><p>When you deploy a global Pulsar instance, with clusters distributed across different geographical regions, the configuration store serves as a highly available and strongly consistent metadata store that can tolerate failures and partitions spanning whole regions.</p><p>The key here is to make sure the ZK quorum members are spread across at least 3 regions, and other regions run as observers.</p><p>Again, given the very low expected load on the configuration store servers, you can share the same hosts used for the local ZooKeeper quorum.</p><p>For example, assume a Pulsar instance with the following clusters <code>us-west</code>, <code>us-east</code>, <code>us-central</code>, <code>eu-central</code>, <code>ap-south</code>. Also assume, each cluster has its own local ZK servers named such as the following:</p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">zk[1-3].${CLUSTER}.example.com</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In this scenario, if you want to pick the quorum participants from a few clusters and let all the others be ZK observers. For example, to form a 7 servers quorum, you can pick 3 servers from <code>us-west</code>, 2 from <code>us-central</code> and 2 from <code>us-east</code>.</p><p>This method guarantees that writing to the configuration store is possible even if one of these regions is unreachable.</p><p>The ZK configuration in all the servers looks like this:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token key attr-name" style="color:rgb(241, 250, 140)">clientPort</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">2184</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.1</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-west.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.2</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.us-west.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.3</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.us-west.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.4</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-central.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.5</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.us-central.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.6</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.us-central.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.7</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-east.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.8</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.us-east.example.com:2185:2186</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.9</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.us-east.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.10</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.eu-central.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.11</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.eu-central.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.12</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.eu-central.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.13</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.ap-south.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.14</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk2.ap-south.example.com:2185:2186:observer</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">server.15</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk3.ap-south.example.com:2185:2186:observer</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Additionally, ZK observers need to have the following parameters:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token key attr-name" style="color:rgb(241, 250, 140)">peerType</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">observer</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h5 class="anchor anchorWithStickyNavbar_LWe7" id="start-the-service">Start the service<a href="#start-the-service" class="hash-link" aria-label="Direct link to Start the service" title="Direct link to Start the service"></a></h5><p>Once your configuration store configuration is in place, you can start up the service using <a href="/docs/next/reference-cli-tools/"><code>pulsar-daemon</code></a></p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-daemon start configuration-store</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-cluster-metadata-initialization">Step 2: Cluster metadata initialization<a href="#step-2-cluster-metadata-initialization" class="hash-link" aria-label="Direct link to Step 2: Cluster metadata initialization" title="Direct link to Step 2: Cluster metadata initialization"></a></h2><p>Once you set up the cluster-specific ZooKeeper and configuration store quorums for your instance, you need to write some metadata to ZooKeeper for each cluster in your instance. <strong>you only need to write these metadata once</strong>.</p><p>You can initialize this metadata using the <a href="/docs/next/reference-cli-tools/"><code>initialize-cluster-metadata</code></a> command of the <a href="/docs/next/reference-cli-tools/"><code>pulsar</code></a> CLI tool. The following is an example:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar initialize-cluster-metadata </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --cluster us-west </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --metadata-store zk:zk1.us-west.example.com:2181,zk2.us-west.example.com:2181/my-chroot-path </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --configuration-metadata-store zk:zk1.us-west.example.com:2181,zk2.us-west.example.com:2181/my-chroot-path </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --web-service-url http://pulsar.us-west.example.com:8080/ </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --web-service-url-tls https://pulsar.us-west.example.com:8443/ </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --broker-service-url pulsar://pulsar.us-west.example.com:6650/ </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651/</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>As you can see from the example above, you need to specify the following:</p><ul><li>The name of the cluster</li><li>The local metadata store connection string for the cluster</li><li>The configuration store connection string for the entire instance</li><li>The web service URL for the cluster</li><li>A broker service URL enabling interaction with the <a href="/docs/next/reference-terminology/#broker">brokers</a> in the cluster</li></ul><p>If you use <a href="/docs/next/security-tls-transport/">TLS</a>, you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster.</p><p>Make sure to run <code>initialize-cluster-metadata</code> for each cluster in your instance.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="step-3-deploy-bookkeeper">Step 3: Deploy BookKeeper<a href="#step-3-deploy-bookkeeper" class="hash-link" aria-label="Direct link to Step 3: Deploy BookKeeper" title="Direct link to Step 3: Deploy BookKeeper"></a></h2><p>BookKeeper provides <a href="/docs/next/concepts-architecture-overview/#persistent-storage">persistent message storage</a> for Pulsar.</p><p>Each Pulsar broker needs its own cluster of bookies. The BookKeeper cluster shares a local ZooKeeper quorum with the Pulsar cluster.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="configure-bookies">Configure bookies<a href="#configure-bookies" class="hash-link" aria-label="Direct link to Configure bookies" title="Direct link to Configure bookies"></a></h3><p>You can configure BookKeeper bookies using the <a href="/docs/next/reference-configuration/#bookkeeper"><code>conf/bookkeeper.conf</code></a> configuration file. The most important aspect of configuring each bookie is ensuring that the <a href="/docs/next/reference-configuration/#bookkeeper-zkServers"><code>zkServers</code></a> parameter is set to the connection string for the local ZooKeeper of Pulsar cluster.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="start-bookies">Start bookies<a href="#start-bookies" class="hash-link" aria-label="Direct link to Start bookies" title="Direct link to Start bookies"></a></h3><p>You can start a bookie in two ways: in the foreground or as a background daemon.</p><p>To start a bookie in the background, use the <a href="/docs/next/reference-cli-tools/"><code>pulsar-daemon</code></a> CLI tool:</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-daemon start bookie</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>You can verify that the bookie works properly using the <code>bookiesanity</code> command for the <a href="/docs/next/reference-cli-tools/">BookKeeper shell</a>:</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/bookkeeper shell bookiesanity</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>This command creates a new ledger on the local bookie, writes a few entries, reads them back and finally deletes the ledger.</p><p>After you have started all bookies, you can use the <code>simpletest</code> command for <a href="/docs/next/reference-cli-tools/">BookKeeper shell</a> on any bookie node, to verify that all bookies in the cluster are running.</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/bookkeeper shell simpletest --ensemble </span><span class="token operator"><</span><span class="token plain">num-bookies</span><span class="token operator">></span><span class="token plain"> --writeQuorum </span><span class="token operator"><</span><span class="token plain">num-bookies</span><span class="token operator">></span><span class="token plain"> --ackQuorum </span><span class="token operator"><</span><span class="token plain">num-bookies</span><span class="token operator">></span><span class="token plain"> --numEntries </span><span class="token operator"><</span><span class="token plain">num-entries</span><span class="token operator">></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Bookie hosts are responsible for storing message data on disk. In order for bookies to provide optimal performance, having a suitable hardware configuration is essential for the bookies. The following are key dimensions for bookie hardware capacity.</p><ul><li>Disk I/O capacity read/write</li><li>Storage capacity</li></ul><p>Message entries written to bookies are always synced to disk before returning an acknowledgment to the Pulsar broker. To ensure low write latency, BookKeeper is |
| designed to use multiple devices:</p><ul><li>A <strong>journal</strong> to ensure durability. For sequential writes, having fast <a href="https://linux.die.net/man/2/fsync" target="_blank" rel="noopener noreferrer">fsync</a> operations on bookie hosts is critical. Typically, small and fast <a href="https://en.wikipedia.org/wiki/Solid-state_drive" target="_blank" rel="noopener noreferrer">solid-state drives</a> (SSDs) should suffice, or <a href="https://en.wikipedia.org/wiki/Hard_disk_drive" target="_blank" rel="noopener noreferrer">hard disk drives</a> (HDDs) with a <a href="https://en.wikipedia.org/wiki/RAID" target="_blank" rel="noopener noreferrer">RAID</a> controller and a battery-backed write cache. Both solutions can reach fsync latency of ~0.4 ms.</li><li>A <strong>ledger storage device</strong> is where data is stored until all consumers acknowledge the message. Writes happen in the background, so writing I/O is not a big concern. Reads happen sequentially most of the time and the backlog is drained only in case of consumer drain. To store large amounts of data, a typical configuration involves multiple HDDs with a RAID controller.</li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="step-4-deploy-brokers">Step 4: Deploy brokers<a href="#step-4-deploy-brokers" class="hash-link" aria-label="Direct link to Step 4: Deploy brokers" title="Direct link to Step 4: Deploy brokers"></a></h2><p>Once you set up ZooKeeper, initialize cluster metadata, and spin up BookKeeper bookies, you can deploy brokers.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="broker-configuration">Broker configuration<a href="#broker-configuration" class="hash-link" aria-label="Direct link to Broker configuration" title="Direct link to Broker configuration"></a></h3><p>You can configure brokers using the <a href="/docs/next/reference-configuration/#broker"><code>conf/broker.conf</code></a> configuration file.</p><p>The most important element of broker configuration is ensuring that each broker is aware of its local ZooKeeper quorum as well as the configuration store quorum. Make sure that you set the <a href="/docs/next/reference-configuration/#broker"><code>metadataStoreUrl</code></a> parameter to reflect the local quorum and the <a href="/docs/next/reference-configuration/#broker"><code>configurationMetadataStoreUrl</code></a> parameter to reflect the configuration store quorum (although you need to specify only those ZooKeeper servers located in the same cluster).</p><p>You also need to specify the name of the <a href="/docs/next/reference-terminology/#cluster">cluster</a> to which the broker belongs using the <a href="/docs/next/reference-configuration/#broker-clusterName"><code>clusterName</code></a> parameter. In addition, you need to match the broker and web service ports provided when you initialize the metadata (especially when you use a different port from default) of the cluster.</p><p>The following is an example configuration:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)"># Local ZooKeeper servers</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">metadataStoreUrl</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Configuration store quorum connection string.</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">configurationMetadataStoreUrl</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">zk1.us-west.example.com:2184,zk2.us-west.example.com:2184,zk3.us-west.example.com:2184</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">clusterName</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">us-west</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Broker data port</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">brokerServicePort</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">6650</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Broker data port for TLS</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">brokerServicePortTls</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">6651</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Port to use to server HTTP request</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">webServicePort</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">8080</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)"># Port to use to server HTTPS request</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token key attr-name" style="color:rgb(241, 250, 140)">webServicePortTls</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">8443</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="broker-hardware">Broker hardware<a href="#broker-hardware" class="hash-link" aria-label="Direct link to Broker hardware" title="Direct link to Broker hardware"></a></h3><p>Pulsar brokers do not require any special hardware since they do not use the local disk. You had better choose fast CPUs and 10Gbps <a href="https://en.wikipedia.org/wiki/Network_interface_controller" target="_blank" rel="noopener noreferrer">NIC</a> so that the software can take full advantage of that.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="start-the-broker-service">Start the broker service<a href="#start-the-broker-service" class="hash-link" aria-label="Direct link to Start the broker service" title="Direct link to Start the broker service"></a></h3><p>You can start a broker in the background by using <a href="https://en.wikipedia.org/wiki/Nohup" target="_blank" rel="noopener noreferrer">nohup</a> with the <a href="/docs/next/reference-cli-tools/"><code>pulsar-daemon</code></a> CLI tool:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-daemon start broker</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>You can also start brokers in the foreground by using <a href="/docs/next/reference-cli-tools/"><code>pulsar broker</code></a>:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar broker</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="service-discovery">Service discovery<a href="#service-discovery" class="hash-link" aria-label="Direct link to Service discovery" title="Direct link to Service discovery"></a></h2><p><a href="/docs/next/client-libraries/">Clients</a> connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL.</p><p>You can use your own service discovery system, and you only need to satisfy just one requirement: when a client performs an HTTP request to an <a href="/docs/next/reference-configuration/">endpoint</a> for a Pulsar cluster, such as <code>http://pulsar.us-west.example.com:8080</code>, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means.</p><blockquote><p><strong>Service discovery already provided by many scheduling systems</strong> |
| Many large-scale deployment systems, such as <a href="/docs/next/deploy-kubernetes/">Kubernetes</a>, have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism.</p></blockquote><h2 class="anchor anchorWithStickyNavbar_LWe7" id="admin-client-and-verification">Admin client and verification<a href="#admin-client-and-verification" class="hash-link" aria-label="Direct link to Admin client and verification" title="Direct link to Admin client and verification"></a></h2><p>At this point, your Pulsar instance should be ready to use. You can now configure client machines that can serve as <a href="/docs/next/admin-api-overview/">administrative clients</a> for each cluster. You can use the <a href="/docs/next/reference-configuration/#client"><code>conf/client.conf</code></a> configuration file to configure admin clients.</p><p>The most important thing is that you point the <a href="/docs/next/reference-configuration/#client-serviceUrl"><code>serviceUrl</code></a> parameter to the correct service URL for the cluster:</p><div class="language-properties codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-properties codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token key attr-name" style="color:rgb(241, 250, 140)">serviceUrl</span><span class="token punctuation" style="color:rgb(248, 248, 242)">=</span><span class="token value attr-value">http://pulsar.us-west.example.com:8080/</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="provision-new-tenants">Provision new tenants<a href="#provision-new-tenants" class="hash-link" aria-label="Direct link to Provision new tenants" title="Direct link to Provision new tenants"></a></h2><p>Pulsar is built as a fundamentally multi-tenant system.</p><p>If a new tenant wants to use the system, you need to create a new one. You can create a new tenant by using the <a href="/reference/#/next/pulsar-admin/" target="_blank" rel="noopener noreferrer"><code>pulsar-admin</code></a> CLI tool:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-admin tenants create test-tenant </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--allowed-clusters us-west </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--admin-roles test-admin-role</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>In this command, users who identify with <code>test-admin-role</code> role can administer the configuration for the <code>test-tenant</code> tenant. The <code>test-tenant</code> tenant can only use the <code>us-west</code> cluster. From now on, this tenant can manage its resources.</p><p>Once you create a tenant, you need to create <a href="/docs/next/reference-terminology/#namespace">namespaces</a> for topics within that tenant.</p><p>The first step is to create a namespace. A namespace is an administrative unit that can contain many topics. A common practice is to create a namespace for each different use case from a single tenant.</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-admin namespaces create test-tenant/ns1</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h5 class="anchor anchorWithStickyNavbar_LWe7" id="test-producer-and-consumer">Test producer and consumer<a href="#test-producer-and-consumer" class="hash-link" aria-label="Direct link to Test producer and consumer" title="Direct link to Test producer and consumer"></a></h5><p>Everything is now ready to send and receive messages. The quickest way to test the system is through the <a href="/docs/next/reference-cli-tools/"><code>pulsar-perf</code></a> client tool.</p><p>You can use a topic in the namespace that you have just created. Topics are automatically created the first time when a producer or a consumer tries to use them.</p><p>The topic name, in this case, can be:</p><div class="language-http codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-http codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token header header-name keyword" style="color:rgb(189, 147, 249);font-style:italic">persistent</span><span class="token header punctuation" style="color:rgb(248, 248, 242)">:</span><span class="token header header-value">//test-tenant/ns1/my-topic</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Start a consumer that creates a subscription on the topic and waits for messages:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-perf consume persistent://test-tenant/ns1/my-topic</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>Start a producer that publishes messages at a fixed rate and reports stats every 10 seconds:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-perf produce persistent://test-tenant/ns1/my-topic</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>To report the topic stats:</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">bin/pulsar-admin topics stats persistent://test-tenant/ns1/my-topic</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/pulsar-site/edit/main/docs/deploy-bare-metal-multi-cluster.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_Z9Sw" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_vwxv"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><a class="pagination-nav__link pagination-nav__link--prev" href="/docs/next/deploy-bare-metal/"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Bare metal</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/docs/next/deploy-docker/"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Docker</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_jeP5 thin-scrollbar theme-doc-toc-desktop"><div class="border"><div style="color:var(--ifm-toc-link-color)">Was this helpful?</div><div style="border-width:1px;padding:3px;display:flex"><div style="justify-content:center;display:flex;border-radius:99999px;width:2.5rem;height:2.5rem;cursor:pointer;background:;color:"><svg style="width:initial;height:initial" width="12" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M10.086 1.594A1 1 0 0 1 11 1a4 4 0 0 1 4 4v3h4.655a3 3 0 0 1 2.994 3.45l-1.38 9A3.002 3.002 0 0 1 18.275 23H4a3 3 0 0 1-3-3v-7a3 3 0 0 1 3-3h2.35l3.736-8.406ZM8 11.212l3.608-8.117A2 2 0 0 1 13 5v4a1 1 0 0 0 1 1h5.671a1 1 0 0 1 1 1.15l-1.38 9a1 1 0 0 1-1 .85H8v-9.788ZM6 21v-9H4a1 1 0 0 0-1 1v7a1 1 0 0 0 1 1h2Z" fill="currentColor"></path></svg></div><div style="justify-content:center;display:flex;border-radius:99999px;width:2.5rem;height:2.5rem;cursor:pointer;background:;color:"><svg style="width:initial;height:initial" width="12" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M20.563 3.316A1.31 1.31 0 0 0 19.687 3h-1.688v9h1.688a1.31 1.31 0 0 0 1.312-1.077V4.077a1.31 1.31 0 0 0-.436-.761ZM16 12.788l-3.608 8.117A1.999 1.999 0 0 1 11 19v-4a1 1 0 0 0-1-1H4.328a1.002 1.002 0 0 1-1-1.15l1.38-9a1 1 0 0 1 1-.85h10.291v9.788ZM19.661 1a3.31 3.31 0 0 1 3.329 2.866c.006.044.01.09.01.134v7c0 .045-.004.09-.01.134A3.31 3.31 0 0 1 19.661 14h-2.012l-3.736 8.406a1 1 0 0 1-.914.594 4 4 0 0 1-4-4v-3H4.344a3 3 0 0 1-2.994-3.45l1.38-9A3.002 3.002 0 0 1 5.724 1h13.937Z" fill="currentColor"></path></svg></div></div><div class="Actions_uugI"><a target="_blank" class="Action_iBHd" href="https://github.com/apache/pulsar/issues/new?assignees=&labels=doc-required&projects=&template=doc.yml&title=%5BDoc%5D+">💡 Suggest changes</a><a target="_blank" class="Action_iBHd" href="https://github.com/apache/pulsar/discussions/new?category=q-a">🛟 Get support</a></div></div><ul class="table-of-contents table-of-contents__left-border"><li><a href="#system-requirement" class="table-of-contents__link toc-highlight">System requirement</a></li><li><a href="#install-pulsar" class="table-of-contents__link toc-highlight">Install Pulsar</a></li><li><a href="#step-1-deploy-zookeeper" class="table-of-contents__link toc-highlight">Step 1: Deploy ZooKeeper</a><ul><li><a href="#deploy-local-zookeeper" class="table-of-contents__link toc-highlight">Deploy local ZooKeeper</a></li><li><a href="#deploy-the-configuration-store" class="table-of-contents__link toc-highlight">Deploy the configuration store</a></li></ul></li><li><a href="#step-2-cluster-metadata-initialization" class="table-of-contents__link toc-highlight">Step 2: Cluster metadata initialization</a></li><li><a href="#step-3-deploy-bookkeeper" class="table-of-contents__link toc-highlight">Step 3: Deploy BookKeeper</a><ul><li><a href="#configure-bookies" class="table-of-contents__link toc-highlight">Configure bookies</a></li><li><a href="#start-bookies" class="table-of-contents__link toc-highlight">Start bookies</a></li></ul></li><li><a href="#step-4-deploy-brokers" class="table-of-contents__link toc-highlight">Step 4: Deploy brokers</a><ul><li><a href="#broker-configuration" class="table-of-contents__link toc-highlight">Broker configuration</a></li><li><a href="#broker-hardware" class="table-of-contents__link toc-highlight">Broker hardware</a></li><li><a href="#start-the-broker-service" class="table-of-contents__link toc-highlight">Start the broker service</a></li></ul></li><li><a href="#service-discovery" class="table-of-contents__link toc-highlight">Service discovery</a></li><li><a href="#admin-client-and-verification" class="table-of-contents__link toc-highlight">Admin client and verification</a></li><li><a href="#provision-new-tenants" class="table-of-contents__link toc-highlight">Provision new tenants</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="footer__link-item">Foundation<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/events/current-event.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Events<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="footer__link-item">License<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/foundation/thanks" target="_blank" rel="noopener noreferrer" class="footer__link-item">Thanks<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/foundation/sponsorship" target="_blank" rel="noopener noreferrer" class="footer__link-item">Sponsorship<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/security/">Security</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Privacy<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a class="footer__link-item" href="/contact/">Contact</a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"> |
