A Pulsar instance consists of multiple Pulsar {% popover clusters %} working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using geo-replication. Deploying a multi-cluster Pulsar instance involves the following basic steps:
If you're deploying a single Pulsar cluster, see the Clusters and Brokers guide.
{% include admonition.html type=“info” title=‘Running Pulsar locally or on Kubernetes?’ content=" This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you‘d like to run a standalone Pulsar cluster on a single machine for development purposes, see the Setting up a local cluster guide. If you’re looking to run Pulsar on Kubernetes, see the Pulsar on Kubernetes guide, which includes sections on running Pulsar on Kubernetes on Google Container Engine and on Amazon Web Services. " %}
{% include explanations/install-package.md %}
{% include explanations/deploying-zk.md %}
Once you've set up local and global ZooKeeper for your instance, there is some metadata that needs to be written to ZooKeeper for each cluster in your instance. It only needs to be written once.
You can initialize this metadata using the initialize-cluster-metadata command of the pulsar CLI tool. Here's an example:
$ bin/pulsar initialize-cluster-metadata \ --cluster us-west \ --zookeeper zk1.us-west.example.com:2181 \ --global-zookeeper zk1.us-west.example.com:2184 \ --web-service-url http://pulsar.us-west.example.com:8080/ \ --web-service-url-tls https://pulsar.us-west.example.com:8443/ \ --broker-service-url pulsar://pulsar.us-west.example.com:6650/ \ --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651/
As you can see from the example above, the following needs to be specified:
{% include admonition.html type=“info” title=“Global cluster” content=' In each Pulsar instance, there is a global cluster that you can administer just like other clusters. The global cluster enables you to do things like create global topics. ' %}
If you‘re using TLS, you’ll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster.
Make sure to run initialize-cluster-metadata for each cluster in your instance.
{% include explanations/deploying-bk.md %}
Once you've set up ZooKeeper, initialized cluster metadata, and spun up BookKeeper {% popover bookies %}, you can deploy {% popover brokers %}.
Brokers can be configured using the conf/broker.conf configuration file.
The most important element of broker configuration is ensuring that each broker is aware of its local ZooKeeper quorum as well as the global ZooKeeper quorum. Make sure that you set the zookeeperServers parameter to reflect the local quorum and the globalZookeeperServers parameter to reflect the global quorum (although you'll need to specify only those global ZooKeeper servers located in the same cluster).
You also need to specify the name of the {% popover cluster %} to which the broker belongs using the clusterName parameter.
Here's an example configuration:
# Local ZooKeeper servers zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181 # Global Zookeeper quorum connection string. globalZookeeperServers=zk1.us-west.example.com:2184,zk2.us-west.example.com:2184,zk3.us-west.example.com:2184 clusterName=us-west
Pulsar brokers do not require any special hardware since they don't use the local disk. Fast CPUs and 10Gbps NIC are recommended since the software can take full advantage of that.
You can start a broker in the background using nohup with the pulsar-daemon CLI tool:
$ bin/pulsar-daemon start broker
You can also start brokers in the foreground using pulsar broker:
$ bin/pulsar broker
{% include explanations/service-discovery-setup.md %}
At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as administrative clients for each cluster. You can use the conf/client.conf configuration file to configure admin clients.
The most important thing is that you point the serviceUrl parameter to the correct service URL for the cluster:
serviceUrl=http://pulsar.us-west.example.com:8080/
Pulsar was built as a fundamentally {% popover multi-tenant %} system. New tenants can be provisioned as Pulsar {% popover properties %}. Properties can be
To allow a new tenant to use the system, we need to create a new {% popover property %}. You can create a new property using the pulsar-admin CLI tool:
$ bin/pulsar-admin properties create test-prop \ --allowed-clusters us-west \ --admin-roles test-admin-role
This will allow users who identify with role test-admin-role to administer the configuration for the property test which will only be allowed to use the cluster us-west. From now on, this tenant will be able to self-manage its resources.
Once a tenant has been created, you will need to create {% popover namespaces %} for topics within that property.
The first step is to create a namespace. A namespace is an administrative unit that can contain many topic. Common practice is to create a namespace for each different use case from a single tenant.
$ bin/pulsar-admin namespaces create test/us-west/ns1
Everything is now ready to send and receive messages. The quickest way to test the system is through the pulsar-perf client tool.
Let's use a topic in the namespace we just created. Topics are automatically created the first time a producer or a consumer tries to use them.
The topic name in this case could be:
{% include topic.html p=“test” c=“us-west” n=“ns1” t=“my-topic” %}
Start a consumer that will create a subscription on the topic and will wait for messages:
$ bin/pulsar-perf consume persistent://test/us-west/ns1/my-topic
Start a producer that publishes messages at a fixed rate and report stats every 10 seconds:
$ bin/pulsar-perf produce persistent://test/us-west/ns1/my-topic
To report the topic stats:
$ bin/pulsar-admin persistent stats persistent://test/us-west/ns1/my-topic