import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
Apache Pulsar is comprised of multiple components, ZooKeeper, bookies, and brokers. These components are either stateful or stateless. You do not have to upgrade ZooKeeper nodes unless you have special requirements. While you upgrade, you need to pay attention to bookies (stateful), brokers, and proxies (stateless).
Read the following guidelines before upgrading a Pulsar cluster.
autorecovery
is enabled, you need to disable autorecovery
in the upgrade process, and re-enable it after completing the process.:::note
Currently, Apache Pulsar is compatible between versions.
:::
To upgrade an Apache Pulsar cluster, follow the upgrade sequence.
Upgrade ZooKeeper (optional).
Upgrade bookies.
Canary test: test an upgraded version in one or a small set of bookies.
Rolling upgrade:
autorecovery
with the following command.
bin/bookkeeper shell autorecovery -disable
autorecovery
with the following command.
bin/bookkeeper shell autorecovery -enable
Upgrade brokers.
Upgrade proxies.
While you upgrade ZooKeeper servers, you can do a canary test first, and then upgrade all ZooKeeper servers in the cluster.
You can test an upgraded version in one of ZooKeeper servers before upgrading all ZooKeeper servers in your cluster.
To upgrade a ZooKeeper server to a new version, complete the following steps:
pulsar zookeeper-shell
to connect to the newly upgraded ZooKeeper server and run a few commands to verify if it works as expected.:::tip
If issues occur during the canary test, you can shut down the problematic ZooKeeper node, revert the binary and configuration, and restart the ZooKeeper with the reverted binary.
:::
After the canary test to upgrade one ZooKeeper in your cluster, you can upgrade all ZooKeeper servers in your cluster.
You can upgrade all ZooKeeper servers one by one by following the steps in the canary test.
While you upgrade bookies, you can do a canary test first, and then upgrade all bookies in the cluster. For more details, you can read Apache BookKeeper Upgrade guide.
You can test an upgraded version in one or a small set of bookies before upgrading all bookies in your cluster.
To upgrade a bookie to a new version, complete the following steps:
Stop the bookie.
Upgrade the binary and configuration files.
Start the bookie in ReadOnly
mode to verify if the bookie of this new version runs well for reading workload.
bin/pulsar bookie --readOnly
When the bookie runs successfully in ReadOnly
mode, stop the bookie and restart it in Write/Read
mode.
bin/pulsar bookie
Observe and make sure the cluster serves both write and read traffic.
:::tip
If issues occur during the canary test, you can shut down the problematic bookie node. Other bookies in the cluster replace this problematic bookie node with auto-recovery.
:::
After the canary test to upgrade some bookies in your cluster, you can upgrade all bookies in your cluster.
Before upgrading, you have to decide whether to upgrade the whole cluster at once, including downtime and rolling upgrade scenarios.
In a rolling upgrade scenario, upgrade one bookie at a time. In a downtime upgrade scenario, shut down the entire cluster, upgrade each bookie, and then start the cluster.
While you upgrade in both scenarios, the procedure is the same for each bookie.
:::tip
When you upgrade a large BookKeeper cluster in a rolling upgrade scenario, upgrading one bookie at a time is slow. If you configure a rack-aware or region-aware placement policy, you can upgrade bookies rack by rack or region by region, which speeds up the whole upgrade process.
:::
The upgrade procedure for brokers and proxies is the same. Brokers and proxies are stateless
, so upgrading the two services is easy.
You can test an upgraded version in one or a small set of nodes before upgrading all nodes in your cluster.
To upgrade a broker (or proxy) to a new version, complete the following steps:
:::tip
If issues occur during the canary test, you can shut down the problematic broker (or proxy) node. Revert to the old version and restart the broker (or proxy).
:::
After the canary test to upgrade some brokers or proxies in your cluster, you can upgrade all brokers or proxies in your cluster.
Before upgrading, you have to decide whether to upgrade the whole cluster at once, including downtime and rolling upgrade scenarios.
In a rolling upgrade scenario, you can upgrade one broker or one proxy at a time if the size of the cluster is small. If your cluster is large, you can upgrade brokers or proxies in batches. When you upgrade a batch of brokers or proxies, make sure the remaining brokers and proxies in the cluster have enough capacity to handle the traffic during the upgrade.
In a downtime upgrade scenario, shut down the entire cluster, upgrade each broker or proxy, and then start the cluster.
While you upgrade in both scenarios, the procedure is the same for each broker or proxy.
:::tip
To check the health of the broker, you can use the following command or API.
<Tabs groupId="api-choice" defaultValue="Admin CLI" values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"}]}> <TabItem value="Admin CLI"> ```bash $ pulsar-admin brokers healthcheck ``` </TabItem> <TabItem value="REST API"> Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v2/brokers/health|operation/healthCheck?version=@pulsar:version_number@} </TabItem> </Tabs>
:::