| </div></div><nav aria-label="Main" class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/"><div class="navbar__logo"><img src="/img/logo-black.svg" alt="Apache Pulsar logo" class="themedImage_ToTc themedImage--light_HNdA" height="25" width="127"><img src="/img/logo-black.svg" alt="Apache Pulsar logo" class="themedImage_ToTc themedImage--dark_i4oU" height="25" width="127"></div><b class="navbar__title text--truncate"></b></a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Get Started</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/docs/3.2.x/concepts-overview/">Concepts</a></li><li><a class="dropdown__link" href="/docs/3.2.x/">Quickstart</a></li><li><a class="dropdown__link" href="/ecosystem/">Ecosystem</a></li></ul></div><a aria-current="page" class="navbar__item navbar__link navbar__link--active" href="/docs/3.2.x/">Docs</a><a class="navbar__item navbar__link" href="/features/">Features</a><a class="navbar__item navbar__link" href="/use-cases/">Use Cases</a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link community-dropdown">Community</a><ul class="dropdown__menu"><li><a class="dropdown__link scroll-link scroll-welcome" id="scroll-welcome" href="/community/">Welcome</a></li><li><a class="dropdown__link scroll-link scroll-discussions" id="scroll-discussions" href="/community/#section-discussions">Discussions</a></li><li><a class="dropdown__link scroll-link" id="scroll-governance" href="/community/#section-governance">Governance</a></li><li><a class="dropdown__link scroll-link" id="scroll-community" href="/community/#section-community">Meet the Community</a></li><li><a class="dropdown__link scroll-link" id="scroll-contribute" href="/community/#section-contribute">Contribute</a></li><li><a class="dropdown__link" href="/contribute/">Contribution Guide</a></li><li><a href="https://github.com/apache/pulsar/wiki" target="_blank" rel="noopener noreferrer" class="dropdown__link">Wiki</a></li><li><a href="https://github.com/apache/pulsar/issues" target="_blank" rel="noopener noreferrer" class="dropdown__link">Issue Tracking</a></li></ul></div><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Learn</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/blog/">Blog</a></li><li><a class="dropdown__link" href="/books/">Books</a></li><li><a class="dropdown__link" href="/case-studies/">Case Studies</a></li><li><a class="dropdown__link" href="/articles/">Articles</a></li><li><a class="dropdown__link" href="/presentations/">Presentations</a></li><li><a class="dropdown__link" href="/events/">Events</a></li></ul></div></div><div class="navbar__items navbar__items--right"><a class="navbar__item navbar__link navbar_download_button" href="/download/">Download</a><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20" aria-hidden="true"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="Scroll back to top" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebarViewport_Xe31"><div class="sidebar_mhZE"><div class="sidebarVersionSwitch_EHpo">Version:<div class="navbar__item dropdown dropdown--hoverable"><a aria-current="page" class="navbar__link active" aria-haspopup="true" aria-expanded="false" role="button" href="/docs/3.2.x/">3.2.x</a><ul class="dropdown__menu"><li><a class="dropdown__link">Next</a></li><li><a class="dropdown__link">3.2.x</a></li><li><a class="dropdown__link">3.0.x LTS</a></li><li><a class="dropdown__link">Others</a></li></ul></div></div><nav aria-label="Docs sidebar" class="menu thin-scrollbar menu_SIkG menuWithAnnouncementBar_GW3s"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/docs/3.2.x/">About</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/3.2.x/getting-started-home/">Get Started</a><button aria-label="Toggle the collapsible sidebar category 'Get Started'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/docs/3.2.x/concepts-overview/">Concepts and Architecture</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-overview/">Overview</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-messaging/">Messaging</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-architecture-overview/">Architecture</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-clients/">Clients</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-2 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-overview/">Broker load balancing</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-overview/">Overview</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-use-cases/">Use cases</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-features/">Features</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-benefits/">Benefits</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-concepts/">Concepts</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-types/">Types</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-quick-start/">Quick start</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-3 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-broker-load-balancing-migration/">Migration</a></li></ul></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-replication/">Geo Replication</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-cluster-level-failover/">Cluster-level failover</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-multi-tenancy/">Multi Tenancy</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-authentication/">Authentication and Authorization</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-topic-compaction/">Topic Compaction</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-throttling/">Message throttling</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-proxy-sni-routing/">Proxy support with SNI routing</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/docs/3.2.x/concepts-multiple-advertised-listeners/">Multiple advertised listeners</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/schema-overview/">Pulsar Schema</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/functions-overview/">Pulsar Functions</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/io-overview/">Pulsar IO</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/sql-overview/">Pulsar SQL</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/tiered-storage-overview/">Tiered Storage</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/txn-why/">Transactions</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/3.2.x/install-deploy-upgrade-landing/">Deployment</a><button aria-label="Toggle the collapsible sidebar category 'Deployment'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/administration-zk-bk/">Administration</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/administration-stats/">Observability</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/security-overview/">Security</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/performance-pulsar-perf/">Performance</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/client-libraries/">Client Libraries</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/admin-api-overview/">Admin API</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/docs/3.2.x/adaptors-kafka/">Adaptors</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/3.2.x/how-to-landing/">Tutorials</a><button aria-label="Toggle the collapsible sidebar category 'Tutorials'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/3.2.x/developers-landing/">Development</a><button aria-label="Toggle the collapsible sidebar category 'Development'" type="button" class="clean-btn menu__caret"></button></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist" aria-expanded="false" href="/docs/3.2.x/reference-landing/">Reference</a><button aria-label="Toggle the collapsible sidebar category 'Reference'" type="button" class="clean-btn menu__caret"></button></div></li></ul></nav></div></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="Breadcrumbs"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="Home page" class="breadcrumbs__link" href="/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_YNFT"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Concepts and Architecture</span><meta itemprop="position" content="1"></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">Broker load balancing</span><meta itemprop="position" content="2"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Concepts</span><meta itemprop="position" content="3"></li></ul></nav><span class="theme-doc-version-badge badge badge--secondary">Version: 3.2.x</span><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Concepts</h1></header><p>Pulsar provides robust support for load balancing to ensure efficient utilization of resources across Pulsar clusters. Load balancing in Pulsar involves distributing messages and partitions evenly among brokers and consumers to prevent hotspots and optimize performance.</p><p>Before getting started with load balancing, it's important to review the key components to ensure that resources are utilized efficiently and varying workloads can be handled by the system effectively.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="brokers">Brokers<a href="#brokers" class="hash-link" aria-label="Direct link to Brokers" title="Direct link to Brokers"></a></h2><p>In a Pulsar cluster, <a href="/docs/3.2.x/reference-terminology/#broker">brokers</a> are responsible for serving messages for different topics and partitions. Broker load balancing ensures that each broker handles a proportional share of the load. </p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="producers">Producers<a href="#producers" class="hash-link" aria-label="Direct link to Producers" title="Direct link to Producers"></a></h2><p><a href="/docs/3.2.x/reference-terminology/#producer">Producers</a> in Pulsar are responsible for publishing messages to topics. Pulsar clients (producers) connect to brokers to publish messages. Producer load balancing (i.e., connection pooling mechanism in Pulsar) ensures that producers are distributed across brokers to avoid overwhelming a single broker with too many connections.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="consumers">Consumers<a href="#consumers" class="hash-link" aria-label="Direct link to Consumers" title="Direct link to Consumers"></a></h2><p><a href="/docs/3.2.x/reference-terminology/#consumer">Consumers</a> in Pulsar are responsible for consuming messages from topics. Depending on how consumer load balancing is configured (i.e., using exclusive or shared consumers or auto-rebalancing), you can ensure even load distribution.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="topics">Topics<a href="#topics" class="hash-link" aria-label="Direct link to Topics" title="Direct link to Topics"></a></h2><p><a href="/docs/3.2.x/reference-terminology/#topic">Topics</a> are the basic units for clients to publish and consume messages. Related topics are logically grouped into a namespace. To efficiently manage metadata and keep track of all of them moving through the system, Pulsar uses a strategy of grouping topics by partitioning on a namespace to create topic bundles.</p><p><img loading="lazy" alt="Relationships of topics and namespaces" src="/assets/images/broker-load-balancing-1-bbb79076076734e16152ec6c68fdfe2a.png" width="578" height="290" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="bundles">Bundles<a href="#bundles" class="hash-link" aria-label="Direct link to Bundles" title="Direct link to Bundles"></a></h2><p><a href="/docs/3.2.x/reference-terminology/#namespace-bundle">Bundles</a> represent a range of partitions for a particular namespace in Pulsar, comprising a portion of the overall hash range of the namespace. </p><p>Bundle is introduced in Pulsar to represent a middle-layer group. Each bundle is an <strong>assignment unit</strong>, which means topics are assigned to brokers at the <strong>bundle</strong> level rather than the topic level.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="broker-load-balancing">Broker load balancing<a href="#broker-load-balancing" class="hash-link" aria-label="Direct link to Broker load balancing" title="Direct link to Broker load balancing"></a></h2><p>The broker load balancer component is like a "traffic cop" sitting between clients and brokers. It balances topic sessions across brokers based on dynamic load data, such as broker resource usage (e.g., CPU, memory, network IO) and topic/bundle loads (e.g., throughput).</p><p>When properly balanced, the brokers can handle increased traffic and ensure that the system can scale seamlessly to accommodate growing workloads. Load balancing helps prevent bottlenecks and ensures that the resources of the cluster are utilized optimally, leading to better throughput and reduced message processing latency.</p><p><img loading="lazy" alt="Concept of broker load balancer" src="/assets/images/broker-load-balancing-2-64cb741da795712329dd766cc1c53d21.png" width="1064" height="646" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="topic-bundling">Topic bundling<a href="#topic-bundling" class="hash-link" aria-label="Direct link to Topic bundling" title="Direct link to Topic bundling"></a></h2><p>Topic bundling refers to the process of grouping topics into bundles. Pulsar organizes topics into bundles within a namespace. Each bundle is a range of partitions, and Pulsar can automatically distribute these bundles across brokers to achieve load balancing. This allows the cluster to scale more efficiently as brokers can independently manage their assigned bundles.</p><p>For example, </p><ul><li><p>Topic load statistics (e.g., message rates) are aggregated at the <strong>bundle</strong> layer, which reduces the cardinality of load samples to monitor.</p></li><li><p>For dynamic topic-broker assignments, Pulsar persists these mappings at the <strong>bundle</strong> level, which decreases the space for storing dynamic topic-broker ownerships.</p></li></ul><p>Pulsar allows you to dynamically scale the number of brokers, producers, and consumers to adapt to changing workloads. As brokers are added or removed, Pulsar handles the redistribution of partitions and bundles automatically.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="workflow">Workflow<a href="#workflow" class="hash-link" aria-label="Direct link to Workflow" title="Direct link to Workflow"></a></h3><p>Below is the workflow for grouping topics into bundles.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-shard-namespaces-into-bundles">Step 1: shard namespaces into bundles<a href="#step-1-shard-namespaces-into-bundles" class="hash-link" aria-label="Direct link to Step 1: shard namespaces into bundles" title="Direct link to Step 1: shard namespaces into bundles"></a></h4><p>Internally, when a namespace is created, the namespace is sharded into a list of bundles. </p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-assign-topics-to-bundles">Step 2: assign topics to bundles<a href="#step-2-assign-topics-to-bundles" class="hash-link" aria-label="Direct link to Step 2: assign topics to bundles" title="Direct link to Step 2: assign topics to bundles"></a></h4><p>When a topic is created or looked up for pub/sub sessions, brokers map the topic to a particular bundle by taking the hash of the topic name (for example, hash("my-topic") = 0x0000000F) and checking in which bundle the hash falls. </p><p>Here "topic" means either a <strong>non-partitioned topic</strong> or <strong>one partition of a partitioned topic</strong>. For partitioned topics, Pulsar internally considers partitions as separate topics, hence different partitions can be assigned to different bundles and brokers. </p><p><img loading="lazy" alt="Workflow of topic bundling" src="/assets/images/broker-load-balancing-3-b62cf1287cba8a48e4d627269475e7e4.png" width="1066" height="528" class="img_ev3q"></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="bundle-assignment">Bundle assignment<a href="#bundle-assignment" class="hash-link" aria-label="Direct link to Bundle assignment" title="Direct link to Bundle assignment"></a></h2><p>Bundle assignment refers to assigning bundles to brokers dynamically based on changing conditions. </p><p>For example, based on broker resource usage (e.g., CPU, memory, network IO) and bundle loads (e.g., throughput), a bundle is dynamically assigned to a particular broker. Each bundle is independent of the others and thus is independently assigned to different brokers. Each broker takes ownership of a bundle (aka, a subset of the topics for a namespace).</p><p>Bundle assignment plays a crucial role in achieving efficient load distribution and scalability within a Pulsar cluster. The purpose of bundle assignments is to ensure balanced resource utilization and facilitate dynamic scaling within the Pulsar architecture.</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="workflow-1">Workflow<a href="#workflow-1" class="hash-link" aria-label="Direct link to Workflow" title="Direct link to Workflow"></a></h3><p>Below is the workflow for dynamic bundle assignment.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-assign-bundles-to-brokers-dynamically">Step 1: assign bundles to brokers dynamically<a href="#step-1-assign-bundles-to-brokers-dynamically" class="hash-link" aria-label="Direct link to Step 1: assign bundles to brokers dynamically" title="Direct link to Step 1: assign bundles to brokers dynamically"></a></h4><p>When a client starts using new topics (bundles) that are not assigned to any broker, a process is triggered to choose the best-suited broker to acquire ownership of these bundles according to the load conditions.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-reassign-bundles-to-other-brokers-optional">Step 2: reassign bundles to other brokers (optional)<a href="#step-2-reassign-bundles-to-other-brokers-optional" class="hash-link" aria-label="Direct link to Step 2: reassign bundles to other brokers (optional)" title="Direct link to Step 2: reassign bundles to other brokers (optional)"></a></h4><p>If a broker owning a bundle crashes, the bundle (topic) is reassigned to another available broker.</p><p><img loading="lazy" alt="Workflow of dynamic bundle assignment" src="/assets/images/broker-load-balancing-4-84486092a1d2e3a13218e8e776796d82.png" width="936" height="714" class="img_ev3q"></p><p>To discover the current bundle-broker ownership for a given topic, Pulsar uses a server-side discovery mechanism that redirects clients to the owner brokers' URLs. This discovery logic requires:</p><ul><li><p>Bundle key ranges for a given namespace, to map a topic to a bundle.</p></li><li><p>Bundle-broker ownership mapping, to direct the client to the current owner or to trigger a new ownership acquisition in case there is no broker assigned.</p></li><li><p>All bundle ranges and broker-bundle ownership mappings are stored in a metadata space, and brokers look up them when clients try to discover owner brokers. For performance reasons, these data are cached at the broker in-memory layer too.</p></li></ul><h2 class="anchor anchorWithStickyNavbar_LWe7" id="bundle-splitting">Bundle splitting<a href="#bundle-splitting" class="hash-link" aria-label="Direct link to Bundle splitting" title="Direct link to Bundle splitting"></a></h2><p>Bundle splitting refers to the process of identifying and splitting overloaded bundles, which helps reduce hot spots, achieve more granular load balancing, improve resource utilization, and enable finer-grained horizontal scaling within the Pulsar cluster.</p><p>The bundle splitting process involves breaking down the original bundle into smaller bundles, each containing a subset of the original partitions. This allows for better distribution of the message and processing load across brokers in the cluster.</p><p>You can split bundles in the following ways:</p><ul><li><p>Automatic: enable Pulsar's automatic bundle splitting process when a namespace has a significant increase in workload or the number of partitions exceeds the optimal capacity for a single bundle.</p></li><li><p>Manual: trigger bundle splitting manually, to divide an existing bundle into multiple smaller bundles. </p></li></ul><table><thead><tr><th>Bundle splitting methods</th><th>Definition</th><th>When to use</th></tr></thead><tbody><tr><td>Automatic</td><td>Bundles are split automatically based on different <a href="#bundle-splitting-algorithms">bundle splitting algorithms</a>.</td><td>Automatic bundle splitting is most commonly used.<br><br>You can use this method in various scenarios, such as when a bundle remains hot for a long time.</td></tr><tr><td>Manual</td><td>Bundles are split manually based on specified positions.</td><td>Manual bundle splitting serves as a supplementary approach to automatic bundle splitting.<br><br>You can use this method in various scenarios, such as: <br><br> - If automatic bundle splitting is enabled, but there are still bundles that remain hot for a long time. <br><br> - If you want to split bundles and redistribute traffic evenly before having any broker overloaded.</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="workflow-2">Workflow<a href="#workflow-2" class="hash-link" aria-label="Direct link to Workflow" title="Direct link to Workflow"></a></h3><p>Below is the workflow for splitting bundles automatically or manually.</p><div class="tabs-container tabList__CuJ"><ul role="tablist" aria-orientation="horizontal" class="tabs"><li role="tab" tabindex="0" aria-selected="true" class="tabs__item tabItem_LNqP tabs__item--active">Automatic bundle splitting</li><li role="tab" tabindex="-1" aria-selected="false" class="tabs__item tabItem_LNqP">Manual bundle splitting</li></ul><div class="margin-top--md"><div role="tabpanel" class="tabItem_Ymn6"><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-find-target-bundles">Step 1: find target bundles<a href="#step-1-find-target-bundles" class="hash-link" aria-label="Direct link to Step 1: find target bundles" title="Direct link to Step 1: find target bundles"></a></h4><p>If the auto bundle split is enabled, </p><ul><li><p>For the modular load balancer, the leader broker will check if any bundle's load is beyond the threshold.</p></li><li><p>For the extensible load balancer, the load manager will check the bundle's load in each owner broker.</p></li></ul><p>Bundle splitting threshold can be set based on various conditions. Any existing bundle that exceeds any of the thresholds is a candidate to be split. The load balancer assigns the newly split bundles to other brokers, to facilitate the traffic distribution.</p><p>For how to enable bundle split and set bundle split thresholds automatically, see TBD (the docs is WIP, stay tuned!).</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-compute-bundle-splitting-boundaries">Step 2: compute bundle splitting boundaries<a href="#step-2-compute-bundle-splitting-boundaries" class="hash-link" aria-label="Direct link to Step 2: compute bundle splitting boundaries" title="Direct link to Step 2: compute bundle splitting boundaries"></a></h4><p>Now the target bundles which need to be split are found. Before splitting, the owner broker needs to compute the splitting positions based on <a href="#bundle-splitting-algorithms">bundle splitting algorithms</a>. </p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-3-split-bundles-by-boundaries">Step 3: split bundles by boundaries<a href="#step-3-split-bundles-by-boundaries" class="hash-link" aria-label="Direct link to Step 3: split bundles by boundaries" title="Direct link to Step 3: split bundles by boundaries"></a></h4><p>Now the owner broker starts splitting the target bundles and then repartition them. </p><p>After the split, the owner broker updates the bundle ownerships and ranges in the metadata space. The newly split bundles can be automatically unloaded from the owner broker.</p><p>For example, if the bundle partition is <!-- -->[0x0000, 0x8000, 0xFFFF]<!-- -->, and the splitting boundary is <!-- -->[0x4000]<!-- --> on the target bundle range, [0x0000, 0x8000).</p><p>Then the bundle partitions after split is <!-- -->[0x0000, 0x4000, 0x8000, 0xFFFF]<!-- -->.</p><p>Then the bundle ranges after split is [0x0000, 0x4000), [0x4000, 0x8000), and <!-- -->[0x8000, 0xFFFF]<!-- -->.</p></div><div role="tabpanel" class="tabItem_Ymn6" hidden=""><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-find-target-bundles-1">Step 1: find target bundles<a href="#step-1-find-target-bundles-1" class="hash-link" aria-label="Direct link to Step 1: find target bundles" title="Direct link to Step 1: find target bundles"></a></h4><p>Based on the broker resource usage (for example, the number of topics or sessions, message rates, or bandwidth), you can choose a hot bundle to split.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-compute-bundle-splitting-position-boundaries">Step 2: compute bundle splitting position boundaries<a href="#step-2-compute-bundle-splitting-position-boundaries" class="hash-link" aria-label="Direct link to Step 2: compute bundle splitting position boundaries" title="Direct link to Step 2: compute bundle splitting position boundaries"></a></h4><ul><li><p>If you want to use the specified_positions_divide algorithm, you need to specify a splitting boundary.</p></li><li><p>If you want to use other <a href="#bundle-splitting-algorithms">bundle splitting algorithms</a> except for the specified_positions_divide algorithm, those algorithms will calculate the position automatically.</p></li></ul><p>Step 3: split bundles at the specific boundaries from step 2.</p><p>For how to split bundles manually, see TBD (the docs is WIP, stay tuned!).</p></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="bundle-splitting-algorithms">Bundle splitting algorithms<a href="#bundle-splitting-algorithms" class="hash-link" aria-label="Direct link to Bundle splitting algorithms" title="Direct link to Bundle splitting algorithms"></a></h3><p>Bundle splitting positions can be calculated using different bundle splitting algorithms.</p><p>Below is a brief summary of bundle splitting algorithms.</p><table><thead><tr><th>Bundle splitting algorithm</th><th>Definition</th><th>When to use</th><th>Available in automatic or manual method?</th><th>Available version</th></tr></thead><tbody><tr><td>range_equally_divide</td><td>Split a bundle into two parts with the same hash range size.</td><td>This is the <strong>default</strong> bundle splitting algorithm. <br><br> Use when there are a large number of topics.</td><td>- Automatic <br> - Manual</td><td>Pulsar 1.7 and later versions</td></tr><tr><td>topic_count_equally_divide</td><td>Split a bundle into two parts with the same number of topics.</td><td>Use when there are a small number of topics.</td><td>- Automatic <br> - Manual</td><td>Pulsar 2.6 and later versions</td></tr><tr><td>specified_positions_divide</td><td>Split a bundle into several parts by the specified positions.</td><td>Use when the automatic bundle splitting is turned off, or a bundle is not split even if the automatic bundle splitting is turned on. <br><br> <strong>Note</strong>: Be cautious when using this algorithm. For example, if bundles are split into <strong>too many small parts</strong>, then these bundles could not be hit by the hash key. Currently, <strong>bundle compaction is not supported</strong>.</td><td>- Manual</td><td>Pulsar 2.11 and later versions</td></tr><tr><td>flow_or_qps_equally_divide</td><td>Split a bundle into several parts based on message rate and throughput.</td><td>Use when splitting bundles proportional to traffic.</td><td>- Automatic <br> - Manual</td><td>Pulsar 3.0 and later versions</td></tr></tbody></table><h4 class="anchor anchorWithStickyNavbar_LWe7" id="range_equally_divide">range_equally_divide<a href="#range_equally_divide" class="hash-link" aria-label="Direct link to range_equally_divide" title="Direct link to range_equally_divide"></a></h4><p>range_equally_divide splits a bundle into two parts with the same hash range size.</p><p>For example, if the target bundle to split is (0x00000000, 0x80000000), then the bundle split boundary is <!-- -->[0x40000000]<!-- -->.</p><p><img loading="lazy" alt="concept of range_equally_divide" src="/assets/images/broker-load-balancing-5-9aaca4795accfd547b0ba5383727b31c.png" width="936" height="636" class="img_ev3q"></p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="topic_count_equally_divide">topic_count_equally_divide<a href="#topic_count_equally_divide" class="hash-link" aria-label="Direct link to topic_count_equally_divide" title="Direct link to topic_count_equally_divide"></a></h4><p>topic_count_equally_divide splits a bundle into two parts with the same number of topics.</p><p>For example, if there are 6 topics in the target bundle [0x00000000, 0x80000000), then you can set the bundle splitting boundary at 0x50000000 to make the left and right sides of the number of topics the same. </p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic1) = 0x10000000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic2) = 0x20000000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic3) = 0x35000000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic4) = 0x65000000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic5) = 0x70000000</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">hash(topic6) = 0x75000000</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>That is, the target bundle to split is [0x00000000, 0x80000000), and the bundle split boundary is <!-- -->[0x50000000]<!-- -->.</p><p><img loading="lazy" alt="concept of topic_count_equally_divide" src="/assets/images/broker-load-balancing-6-c37e5496244f826fdfbe895870939233.png" width="936" height="732" class="img_ev3q"></p><p>For implementation details, see <a href="https://github.com/apache/pulsar/pull/6241" target="_blank" rel="noopener noreferrer">PR-6241: support evenly distribute topics count when splitting bundles</a>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="specified_positions_divide">specified_positions_divide<a href="#specified_positions_divide" class="hash-link" aria-label="Direct link to specified_positions_divide" title="Direct link to specified_positions_divide"></a></h4><p>specified_positions_divide splits bundles into several parts by specified positions.</p><p>For example, if you have 2 large topics and there are on the same bundle. Topic1 is at 0x30000000, topic2 is at 0x35000000, and the bundle range is [0x00000000, 0x40000000), then you can set the bundle split boundary as 0x33000000.</p><p>For implementation details, see <a href="https://github.com/apache/pulsar/issues/13761" target="_blank" rel="noopener noreferrer">PIP-143: support split bundle by specified boundaries</a>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="flow_or_qps_equally_divide">flow_or_qps_equally_divide<a href="#flow_or_qps_equally_divide" class="hash-link" aria-label="Direct link to flow_or_qps_equally_divide" title="Direct link to flow_or_qps_equally_divide"></a></h4><p>flow_or_qps_equally_divide splits bundles into several parts based on <strong>message rate</strong> (controlled by <code>loadBalancerNamespaceBundleMaxMsgRate</code>) or <strong>message throughput</strong> (controlled by <code>loadBalancerNamespaceBundleMaxBandwidthMbytes</code>). The split position is determined by reaching the threshold of either message rate or message throughput. </p><p>For example, suppose that you have 6 topics on a bundle range <!-- -->[0x00000000 to 0x80000000]<!-- --> as below. </p><table><thead><tr><th>Topic name</th><th>Hash code</th><th>Message rate</th><th>Message throughput</th></tr></thead><tbody><tr><td>t1</td><td>0x10000000</td><td>100/s</td><td>10M/s</td></tr><tr><td>t2</td><td>0x15000000</td><td>200/s</td><td>20M/s</td></tr><tr><td>t3</td><td>0x24000000</td><td>300/s</td><td>30M/s</td></tr><tr><td>t4</td><td>0x39000000</td><td>400/s</td><td>40M/s</td></tr><tr><td>t5</td><td>0x58000000</td><td>500/s</td><td>50M/s</td></tr><tr><td>t6</td><td>0x76000000</td><td>600/s</td><td>60M/s</td></tr></tbody></table><p>The split position varies depending on the values of message rate or message throughput.</p><h5 class="anchor anchorWithStickyNavbar_LWe7" id="case-1-split-position-is-determined-by-message-rate-as-it-reaches-the-threshold-earlier">Case 1: split position is determined by message rate as it reaches the threshold earlier<a href="#case-1-split-position-is-determined-by-message-rate-as-it-reaches-the-threshold-earlier" class="hash-link" aria-label="Direct link to Case 1: split position is determined by message rate as it reaches the threshold earlier" title="Direct link to Case 1: split position is determined by message rate as it reaches the threshold earlier"></a></h5><p>If you set</p><p><code>loadBalancerNamespaceBundleMaxMsgRate</code>=450/s</p><p><code>loadBalancerNamespaceBundleMaxBandwidthMbytes</code>=200M/s</p><p>Since </p><p>100/s+ 200/s < 450/s</p><p>100/s+ 200/s + 300/s > 450/s </p><p>So the split boundary is between t2 and t3, that is:</p><p>splitStartPosition = 0x15000000</p><p>splitEndPosition = 0x24000000</p><p>splitPosition = (0x15000000 + 0x24000000) / 2 = 0x19500000</p><p>Note that the <strong>bundle split will be continued</strong> as below:</p><p>------ 2nd bundle splitting ------</p><p>Since </p><p>300/s ＜ 450/s</p><p>300/s + 400/s > 450/s</p><p>So the split boundary is between t3 and t4, that is</p><p>splitStartPosition = 0x24000000</p><p>splitEndPosition = 0x39000000</p><p>splitPosition = 31500000 = (0x24000000 + 0x39000000) / 2</p><p>------ 3rd bundle splitting ------</p><p>Since </p><p>400/s ＜ 450/s</p><p>400/s + 500/s > 450/s</p><p>So the split boundary is between t4 and t5, that is</p><p>splitStartPosition = 0x39000000</p><p>splitEndPosition = 0x58000000</p><p>splitPosition = 48500000 = (0x39000000 + 0x58000000) /2</p><p>------ 4th bundle splitting ------</p><p>Since</p><p>500/s > 450/s</p><p>600/s > 450/s</p><p>So the split boundary is between t5 and t6, that is</p><p>splitStartPosition = 0x58000000</p><p>splitEndPosition = 0x76000000</p><p>splitPosition = 67000000 = (0x58000000 + 0x76000000) / 2</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="case-2-split-position-is-determined-by-message-throughput-as-it-reaches-the-threshold-earlier">Case 2: split position is determined by message throughput as it reaches the threshold earlier<a href="#case-2-split-position-is-determined-by-message-throughput-as-it-reaches-the-threshold-earlier" class="hash-link" aria-label="Direct link to Case 2: split position is determined by message throughput as it reaches the threshold earlier" title="Direct link to Case 2: split position is determined by message throughput as it reaches the threshold earlier"></a></h4><p>If you set</p><p><code>loadBalancerNamespaceBundleMaxMsgRate</code>=1900/s</p><p><code>loadBalancerNamespaceBundleMaxBandwidthMbytes</code>=90M/s</p><p>Since </p><p>10M/s+ 20M/s + 30M/s < 90M/s</p><p>10M/s+ 20M/s + 30M/s + 40M/s > 90M/s </p><p>So the split boundary is between t3 and t4, that is:</p><p>splitStartPosition = 0x24000000</p><p>splitEndPosition = 0x39000000</p><p>splitPosition = (0x24000000 + 0x39000000) / 2 = 0x31500000</p><p>Note that the <strong>bundle split will be continued</strong>: </p><p>------ 2nd bundle splitting ------</p><p>Since </p><p>40 + 50 ≤ 90</p><p>40 +50 + 60 > 90</p><p>So the split boundary is between t5 and t6, that is:</p><p>splitStartPosition = 0x58000000</p><p>splitEndPosition = 0x76000000</p><p>splitPosition = (0x58000000 + 0x0x76000000) / 2 = 0x67000000</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="case-3-split-position-is-determined-by-both-message-rate-and-message-throughput-as-they-reach-the-thresholds-at-the-same-time">Case 3: split position is determined by both message rate and message throughput as they reach the thresholds at the same time<a href="#case-3-split-position-is-determined-by-both-message-rate-and-message-throughput-as-they-reach-the-thresholds-at-the-same-time" class="hash-link" aria-label="Direct link to Case 3: split position is determined by both message rate and message throughput as they reach the thresholds at the same time" title="Direct link to Case 3: split position is determined by both message rate and message throughput as they reach the thresholds at the same time"></a></h4><p>If you set</p><p><code>loadBalancerNamespaceBundleMaxMsgRate</code>=1100</p><p><code>loadBalancerNamespaceBundleMaxBandwidthMbytes</code>=110</p><ul><li><p>From the <strong>message rate</strong> side:</p><p> Since </p><p> 100/s+ 200/s + 300/s + 400/s < 1100/s </p><p> 100/s+ 200/s + 300/s + 400/s + 500/s > 1100/s</p><p> So the split boundary is between t4 and t5, that is:</p><p> splitStartPosition = 0x39000000</p><p> splitEndPosition = 0x58000000</p><p> splitPosition = (0x39000000 + 0x58000000) / 2 = 0x48500000</p></li><li><p>From the <strong>message throughput</strong> side:</p><p> Since </p><p> 10M/s+ 20M/s + 30M/s + 40M/s < 110M/s </p><p> 0M/s+ 20M/s + 30M/s + 40M/s + 50M/s > 110M/s</p><p> So the split boundary is between t4 and t5, that is:</p><p> splitStartPosition = 0x39000000</p><p> splitEndPosition = 0x58000000</p><p> splitPosition = (0x39000000 + 0x58000000) / 2 = 0x48500000</p></li></ul><p>In summary, the split position is 0x48500000.</p><p>For implementation details, see <a href="https://github.com/apache/pulsar/issues/16782" target="_blank" rel="noopener noreferrer">PIP-169: support split bundle by flow or QPS</a>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="bundle-unloading">Bundle unloading<a href="#bundle-unloading" class="hash-link" aria-label="Direct link to Bundle unloading" title="Direct link to Bundle unloading"></a></h2><p>Bundle unloading (shedding) refers to the process of closing bundles (topics), releasing ownership, and reassigning bundles (topics) to a less-loaded broker from overloaded brokers, based on load conditions. </p><p>Bundle unloading balances the workload across brokers and optimizes resource utilization in the cluster. For example, when a Pulsar cluster experiences changing workloads or scaling events (e.g., adding or removing brokers), bundle unloading ensures that the partitions are evenly distributed and no broker becomes overloaded. </p><p>You can unload bundles in the following ways:</p><ul><li><p>Automatic: enable Pulsar's automatic bundle unloading process when a broker is overloaded.</p></li><li><p>Manual: trigger bundle splitting manually, to unload a bundle from one broker to another broker within a Pulsar cluster. </p></li></ul><table><thead><tr><th>Bundle unloading methods</th><th>Definition</th><th>When to use</th></tr></thead><tbody><tr><td>Automatic</td><td>When the load balancer recognizes a particular broker is overloaded, it forcefully unloads some bundles' traffic from the overloaded broker, so that the unloaded bundles (topics) can be reassigned to less-loaded brokers by the assignment process.</td><td>Use when there is fluctuating traffic that varies over time.</td></tr><tr><td>Manual</td><td>You can manually trigger bundle unloading at any time.</td><td>Manual bundle unloading serves as a <strong>supplementary</strong> approach to automatic bundle unloading. <br><br>If the automatic unloading does not kick in (e.g., due to misconfiguration), you can trigger manual unloading to mitigate the load-imbalance issue. To avoid manual unload operations, you need to correctly tune load balance configs according to the cluster's traffic.</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="workflow-3">Workflow<a href="#workflow-3" class="hash-link" aria-label="Direct link to Workflow" title="Direct link to Workflow"></a></h3><p>Below is the workflow for unloading bundles automatically or manually.</p><div class="tabs-container tabList__CuJ"><ul role="tablist" aria-orientation="horizontal" class="tabs"><li role="tab" tabindex="0" aria-selected="true" class="tabs__item tabItem_LNqP tabs__item--active">Automatic bundle unloading</li><li role="tab" tabindex="-1" aria-selected="false" class="tabs__item tabItem_LNqP">Manual bundle unloading</li></ul><div class="margin-top--md"><div role="tabpanel" class="tabItem_Ymn6"><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-find-target-brokers">Step 1: find target brokers<a href="#step-1-find-target-brokers" class="hash-link" aria-label="Direct link to Step 1: find target brokers" title="Direct link to Step 1: find target brokers"></a></h4><p>With the broker load information collected from all brokers, the leader broker computes the resource usage of each broker based on the <a href="/docs/3.2.x/concepts-broker-load-balancing-concepts/tbd/">bundle unloading strategies</a>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-unload-bundles">Step 2: unload bundles<a href="#step-2-unload-bundles" class="hash-link" aria-label="Direct link to Step 2: unload bundles" title="Direct link to Step 2: unload bundles"></a></h4><p>If the lead broker finds a broker is overloaded, it will calculate the overloaded bundles, ask the overloaded broker to unload some bundles of topics, remove the target bundles' ownerships, and close the topic sessions and the client connections. </p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-3-assign-new-owners">Step 3: assign new owners<a href="#step-3-assign-new-owners" class="hash-link" aria-label="Direct link to Step 3: assign new owners" title="Direct link to Step 3: assign new owners"></a></h4><p>The unloaded bundles are assigned to less loaded brokers, and the clients connect to them. </p><ul><li><p>For the modular load balancer, bundles will be post-assigned to available brokers when clients send lookup requests.</p></li><li><p>For the extensible load balancer, bundles will be pre-assigned to available brokers when unloading.</p></li></ul><p>When unloading happens, the client experiences a small latency blip while the topic is reassigned. </p><p>For how to unload bundles automatically, see TBD (the docs is WIP, stay tuned!).</p></div><div role="tabpanel" class="tabItem_Ymn6" hidden=""><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-1-find-target-bundles-2">Step 1: find target bundles<a href="#step-1-find-target-bundles-2" class="hash-link" aria-label="Direct link to Step 1: find target bundles" title="Direct link to Step 1: find target bundles"></a></h4><p>Based on the broker resource usage (for example, CPU, network, and memory usage), you can choose hot bundles to unload.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="step-2-unload-hot-bundles">Step 2: unload hot bundles<a href="#step-2-unload-hot-bundles" class="hash-link" aria-label="Direct link to Step 2: unload hot bundles" title="Direct link to Step 2: unload hot bundles"></a></h4><p>Unload hot bundles to available brokers. Target bundles' ownerships will be transferred, and topic connections will be closed.</p><p>For how to unload bundles manually, see TBD (the docs is WIP, stay tuned!).</p></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="bundle-unloading-strategies">Bundle unloading strategies<a href="#bundle-unloading-strategies" class="hash-link" aria-label="Direct link to Bundle unloading strategies" title="Direct link to Bundle unloading strategies"></a></h3><p>You can choose different bundle unloading strategies based on your needs. </p><p>Below is a quick summary of bundle unloading strategies, which are <strong>only applicable</strong> for unloading bundles<strong> automatically</strong>.</p><table><thead><tr><th>Bundle unloading strategy</th><th>Definition</th><th>When to use</th><th>Available version</th></tr></thead><tbody><tr><td>OverloadShedder</td><td>Unload bundles on brokers if a <strong>broker's maximum resource usage</strong> exceeds the configured threshold.</td><td>Use when you want to set broker usage below a threshold.</td><td>Pulsar 1.18 and later versions.<br><br> This strategy is <strong>only available</strong> in the <strong>modular</strong> load balancer.</td></tr><tr><td>ThresholdShedder</td><td>Unload bundles if a broker's average usage is greater than the<strong> cluster average usage</strong> plus <strong>configured threshold</strong>.</td><td>Use when you want to evenly spread loads across all brokers base on cluster average usage.</td><td>Pulsar 2.6 and later versions.<br><br> This strategy is <strong>only available</strong> in the <strong>modular</strong> load balancer.</td></tr><tr><td>UniformLoadShedder</td><td>Distribute load uniformly across all brokers, based on <strong>minimal</strong> and <strong>maximum</strong> load.</td><td>Use when you want to compare the minimal and maximum loaded brokers.</td><td>Pulsar 2.10.0 and later versions.<br><br> This strategy is <strong>only available</strong> in the <strong>modular</strong> load balancer.</td></tr><tr><td>TransferShedder</td><td>Unload bundles from the <strong>highest</strong> load brokers to the <strong>lowest</strong> load brokers until the standard deviation of the broker load distribution is below the configured threshold.</td><td>This is the <strong>default</strong> strategy for the <strong>extensible</strong> load balancer. <br><br>It pre-assigns destination brokers when unloading.</td><td>Pulsar 3.0 and later versions.<br><br>This strategy is <strong>only available</strong> in the <strong>extensible</strong> load balancer.</td></tr></tbody></table><h4 class="anchor anchorWithStickyNavbar_LWe7" id="overloadshedder">OverloadShedder<a href="#overloadshedder" class="hash-link" aria-label="Direct link to OverloadShedder" title="Direct link to OverloadShedder"></a></h4><p>OverloadShedder strategy sheds bundles on brokers if a broker's maximum resource usage exceeds the configured threshold (<code>loadBalancerBrokerOverloadedThresholdPercentage</code>).</p><p><img loading="lazy" alt="Concept of OverloadShedder" src="/assets/images/shedding-strategy-overloadshedder-82646099fbc5728e78d7d4d3eb151e18.svg" width="1790" height="625" class="img_ev3q"> </p><p>If a broker is overloaded and has at least two bundles assigned. At the same time, if that broker has at least one bundle that has not been unloaded, then this (these) bundle(s) will be unloaded. Bundles with higher message throughput will be unloaded before those with lower message throughput.</p><p>The determination of when a broker is overloaded is based on the threshold of CPU, network, and memory usage. Whenever either of those metrics reaches the threshold (the <strong>default value</strong> is 85%), the system triggers the bundle unloading.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="thresholdshedder">ThresholdShedder<a href="#thresholdshedder" class="hash-link" aria-label="Direct link to ThresholdShedder" title="Direct link to ThresholdShedder"></a></h4><p>ThresholdShedder strategy sheds the bundles if a <strong>broker's average usage</strong> is greater than the <strong>cluster average usage</strong> plus <strong>configured threshold</strong>. </p><p><img loading="lazy" alt="Concept of ThresholdShedder" src="/assets/images/shedding-strategy-thresholdshedder-69bc0835a1752056e0cc058d6c7c72f1.svg" width="1776" height="683" class="img_ev3q"> </p><h5 class="anchor anchorWithStickyNavbar_LWe7" id="workflow-4">Workflow<a href="#workflow-4" class="hash-link" aria-label="Direct link to Workflow" title="Direct link to Workflow"></a></h5><ol><li><p>ThresholdShedder first computes the average resource usage of brokers for the whole cluster (that is, <strong>cluster average usage</strong>). </p><ul><li><p>The resource usage for each broker is calculated using the method <code>LocalBrokerData#getMaxResourceUsageWithWeight</code>.</p></li><li><p>Historical observations are included in the running average based on the broker's setting for <code>loadBalancerHistoryResourcePercentage</code>. </p></li></ul></li><li><p>ThresholdShedder compares each broker's average resource usage (based on current and historical resource usage) to the cluster average usage:</p><p>a. If a <strong>broker's resource usage</strong> is greater than the <strong>cluster average usage</strong> plus the <strong>configured threshold</strong>, ThresholdShedder proposes removing enough bundles to bring the unloaded broker 5% below the <strong>cluster average usage</strong>. <strong>Note</strong> that recently unloaded bundles are <strong>not unloaded again</strong>.</p><p>b. If a <strong>broker's resource usage</strong> is smaller than the <strong>cluster average usage</strong>, or smaller than the <strong>cluster average usage</strong> plus the <strong>configured threshold</strong>, no bundle will be unloaded.</p></li></ol><p>For example, assume that you have 3 brokers and each broker’s average usage is as below.</p><table><thead><tr><th>Broker name</th><th>Broker's average usage</th></tr></thead><tbody><tr><td>broker1</td><td>40%</td></tr><tr><td>broker2</td><td>10%</td></tr><tr><td>broker3</td><td>10%</td></tr></tbody></table><p>So the cluster average usage is 20% = (40% + 10% + 10%) / 3. </p><p>If you set the threshold (<code>loadBalancerBrokerThresholdShedderPercentage</code>) to 10%, |
| then only broker1's certain bundles get unloaded, because the broker1's resource usage (40%) is greater than the sum (30%) of the <strong>cluster average usage</strong> (20%) plus <strong>configured threshold</strong> (10%).</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="uniformloadshedder">UniformLoadShedder<a href="#uniformloadshedder" class="hash-link" aria-label="Direct link to UniformLoadShedder" title="Direct link to UniformLoadShedder"></a></h4><p>UniformLoadShedder strategy distributes load uniformly across all brokers. It checks the load difference between the broker with the <strong>highest</strong> load and the broker with the <strong>lowest</strong> load. If the difference is higher than configured thresholds, either <strong>message rate</strong> (controlled by)<code>loadBalancerMsgRateDifferenceShedderThreshold</code> or <strong>throughput</strong> (controlled by <code>loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold</code>), then it finds out bundles that can be unloaded to distribute traffic evenly across all brokers.</p><p><img loading="lazy" alt="Concept of UniformLoadShedder" src="/assets/images/shedding-strategy-uniformLoadshedder-a521b90620882e7ce650d4ee6d0561bc.svg" width="1786" height="635" class="img_ev3q"> </p><p>For implementation details, see <a href="https://github.com/apache/pulsar/pull/12902" target="_blank" rel="noopener noreferrer">PR-12902: add uniform load shedder strategy to distribute traffic uniformly across brokers</a>.</p><h4 class="anchor anchorWithStickyNavbar_LWe7" id="transfershedder">TransferShedder<a href="#transfershedder" class="hash-link" aria-label="Direct link to TransferShedder" title="Direct link to TransferShedder"></a></h4><p>TransferShedder strategy unloads bundles from the <strong>highest</strong> load brokers to the <strong>lowest</strong> load brokers until all of the following are true:</p><ul><li><p>The standard deviation of the broker load distribution is below the configured threshold (loadBalancerBrokerLoadTargetStd, default value is 0.25).</p></li><li><p>There are no significant underloaded brokers.</p><ul><li><p>No broker receives 0 traffic.</p></li><li><p>No broker's load < avgLoad * min(0.5, loadBalancerBrokerLoadTargetStd / 2)</p></li></ul></li><li><p>There is no significant overloaded brokers</p><ul><li>No broker’s load > loadBalancerBrokerOverloadedThresholdPercentage && load > avgLoad + loadBalancerBrokerLoadTargetStd</li></ul></li></ul><p>Pulsar introduced TransferShedder to utilize the bundle transfer protocol from the extensible load balancer. With this bundle transfer protocol, the bundle ownership can be gracefully transferred from the source broker to the destination broker. This means that TransferShedder pre-assigns the destination brokers at the unloading time instead of client lookups. Hence, after unloading, clients can bypass the assignment process as the new owner is already assigned.</p><p>For implementation details, see <a href="https://github.com/apache/pulsar/issues/18215" target="_blank" rel="noopener noreferrer">PIP-220: TransferShedder</a>.</p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="related-topics">Related topics<a href="#related-topics" class="hash-link" aria-label="Direct link to Related topics" title="Direct link to Related topics"></a></h2><ul><li><p>To get a comprehensive understanding and discover the key insights, see <a href="/docs/3.2.x/concepts-broker-load-balancing-overview/">Broker load balancing | Overview</a>. </p></li><li><p>To discover different usage scenarios, see <a href="/docs/3.2.x/concepts-broker-load-balancing-use-cases/">Broker load balancing | Use cases</a>.</p></li><li><p>To explore functionalities, see <a href="/docs/3.2.x/concepts-broker-load-balancing-features/">Broker load balancing | Features</a>.</p></li><li><p>To understand advantages, see <a href="/docs/3.2.x/concepts-broker-load-balancing-benefits/">Broker load balancing | Benefits</a>.</p></li><li><p>To review various versions of broker load balancers, see <a href="/docs/3.2.x/concepts-broker-load-balancing-types/">Broker load balancing | Types</a>.</p></li><li><p>To get up quickly, see <a href="/docs/3.2.x/concepts-broker-load-balancing-quick-start/">Broker load balancing | Quick start</a>.</p></li><li><p>To migrate one broker load balancer type to another, see <a href="/docs/3.2.x/concepts-broker-load-balancing-migration/">Broker load balancing | Migration</a>.</p></li></ul></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/pulsar-site/edit/main/versioned_docs/version-3.2.x/concepts-broker-load-balancing-concepts.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_Z9Sw" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>Edit this page</a></div><div class="col lastUpdated_vwxv"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><a class="pagination-nav__link pagination-nav__link--prev" href="/docs/3.2.x/concepts-broker-load-balancing-benefits/"><div class="pagination-nav__sublabel">Previous</div><div class="pagination-nav__label">Benefits</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/docs/3.2.x/concepts-broker-load-balancing-types/"><div class="pagination-nav__sublabel">Next</div><div class="pagination-nav__label">Types</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_jeP5 thin-scrollbar theme-doc-toc-desktop"><div class="border"><div style="color:var(--ifm-toc-link-color)">Was this helpful?</div><div style="border-width:1px;padding:3px;display:flex"><div style="justify-content:center;display:flex;border-radius:99999px;width:2.5rem;height:2.5rem;cursor:pointer;background:;color:"><svg style="width:initial;height:initial" width="12" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M10.086 1.594A1 1 0 0 1 11 1a4 4 0 0 1 4 4v3h4.655a3 3 0 0 1 2.994 3.45l-1.38 9A3.002 3.002 0 0 1 18.275 23H4a3 3 0 0 1-3-3v-7a3 3 0 0 1 3-3h2.35l3.736-8.406ZM8 11.212l3.608-8.117A2 2 0 0 1 13 5v4a1 1 0 0 0 1 1h5.671a1 1 0 0 1 1 1.15l-1.38 9a1 1 0 0 1-1 .85H8v-9.788ZM6 21v-9H4a1 1 0 0 0-1 1v7a1 1 0 0 0 1 1h2Z" fill="currentColor"></path></svg></div><div style="justify-content:center;display:flex;border-radius:99999px;width:2.5rem;height:2.5rem;cursor:pointer;background:;color:"><svg style="width:initial;height:initial" width="12" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M20.563 3.316A1.31 1.31 0 0 0 19.687 3h-1.688v9h1.688a1.31 1.31 0 0 0 1.312-1.077V4.077a1.31 1.31 0 0 0-.436-.761ZM16 12.788l-3.608 8.117A1.999 1.999 0 0 1 11 19v-4a1 1 0 0 0-1-1H4.328a1.002 1.002 0 0 1-1-1.15l1.38-9a1 1 0 0 1 1-.85h10.291v9.788ZM19.661 1a3.31 3.31 0 0 1 3.329 2.866c.006.044.01.09.01.134v7c0 .045-.004.09-.01.134A3.31 3.31 0 0 1 19.661 14h-2.012l-3.736 8.406a1 1 0 0 1-.914.594 4 4 0 0 1-4-4v-3H4.344a3 3 0 0 1-2.994-3.45l1.38-9A3.002 3.002 0 0 1 5.724 1h13.937Z" fill="currentColor"></path></svg></div></div><div class="Actions_uugI"><a target="_blank" class="Action_iBHd" href="https://github.com/apache/pulsar/issues/new?assignees=&labels=doc-required&projects=&template=doc.yml&title=%5BDoc%5D+">💡 Suggest changes</a><a target="_blank" class="Action_iBHd" href="https://github.com/apache/pulsar/discussions/new?category=q-a">🛟 Get support</a></div></div><ul class="table-of-contents table-of-contents__left-border"><li><a href="#brokers" class="table-of-contents__link toc-highlight">Brokers</a></li><li><a href="#producers" class="table-of-contents__link toc-highlight">Producers</a></li><li><a href="#consumers" class="table-of-contents__link toc-highlight">Consumers</a></li><li><a href="#topics" class="table-of-contents__link toc-highlight">Topics</a></li><li><a href="#bundles" class="table-of-contents__link toc-highlight">Bundles</a></li><li><a href="#broker-load-balancing" class="table-of-contents__link toc-highlight">Broker load balancing</a></li><li><a href="#topic-bundling" class="table-of-contents__link toc-highlight">Topic bundling</a><ul><li><a href="#workflow" class="table-of-contents__link toc-highlight">Workflow</a></li></ul></li><li><a href="#bundle-assignment" class="table-of-contents__link toc-highlight">Bundle assignment</a><ul><li><a href="#workflow-1" class="table-of-contents__link toc-highlight">Workflow</a></li></ul></li><li><a href="#bundle-splitting" class="table-of-contents__link toc-highlight">Bundle splitting</a><ul><li><a href="#workflow-2" class="table-of-contents__link toc-highlight">Workflow</a></li><li><a href="#bundle-splitting-algorithms" class="table-of-contents__link toc-highlight">Bundle splitting algorithms</a></li></ul></li><li><a href="#bundle-unloading" class="table-of-contents__link toc-highlight">Bundle unloading</a><ul><li><a href="#workflow-3" class="table-of-contents__link toc-highlight">Workflow</a></li><li><a href="#bundle-unloading-strategies" class="table-of-contents__link toc-highlight">Bundle unloading strategies</a></li></ul></li><li><a href="#related-topics" class="table-of-contents__link toc-highlight">Related topics</a></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="footer__link-item">Foundation<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/events/current-event.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Events<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="footer__link-item">License<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/foundation/thanks" target="_blank" rel="noopener noreferrer" class="footer__link-item">Thanks<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a href="https://www.apache.org/foundation/sponsorship" target="_blank" rel="noopener noreferrer" class="footer__link-item">Sponsorship<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/security/">Security</a></li><li class="footer__item"><a href="https://www.apache.org/foundation/policies/privacy.html" target="_blank" rel="noopener noreferrer" class="footer__link-item">Privacy<svg width="13.5" height="13.5" aria-hidden="true" viewBox="0 0 24 24" class="iconExternalLink_nPIU"><path fill="currentColor" d="M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z"></path></svg></a></li><li class="footer__item"><a class="footer__link-item" href="/contact/">Contact</a></li></ul></div><div class="col footer__col"><div class="footer__title"></div><ul class="footer__items clean-list"><li class="footer__item"> |
