For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary RocksDB and BookKeeper components running inside of a single Java Virtual Machine (JVM) process.
Pulsar in production?
If you're looking to run a full production Pulsar installation, see the Deploying a Pulsar instance guide.
This tutorial guides you through every step of installing Pulsar locally.
Currently, Pulsar is available for 64-bit macOS, Linux, and Windows. To use Pulsar, you need to install 64-bit JRE/JDK. For the runtime Java version, see Pulsar Runtime Java Version Recommendation according to your target Pulsar version.
:::tip
By default, Pulsar allocates 2G JVM heap memory to start. It can be changed in conf/pulsar_env.sh
file under PULSAR_MEM
. This is an extra option passed into JVM.
:::
:::note
Broker is only supported on 64-bit JVM.
:::
In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar.
One of the ways to easily install an x86 JDK is to use SDKMan. Follow instructions on the SDKMan website.
~/.sdkman/etc/config
and changing the following property from false
to true
.
sdkman_rosetta2_compatible=true
sdk list java|grep installed
Example output:
| >>> | 17.0.3.6.1 | amzn | installed | 17.0.3.6.1-amzn
If you have any Java 17 version installed, uninstall it.
sdk uinstall java 17.0.3.6.1
sdk install java 17.0.3.6.1-amzn
To get started with Pulsar, download a binary tarball release in one of the following ways:
download from the Apache mirror (Pulsar @pulsar:version@ binary release)
download from the Pulsar downloads page
download from the Pulsar releases page
use wget:
$ wget pulsar:binary_release_url
After you download the tarball, untar it and use the cd
command to navigate to the resulting directory:
$ tar xvfz apache-pulsar-@pulsar:version@-bin.tar.gz $ cd apache-pulsar-@pulsar:version@
The Pulsar binary package initially contains the following directories:
Directory | Contains |
---|---|
bin | Pulsar's command-line tools, such as pulsar and pulsar-admin . |
conf | Configuration files for Pulsar, including broker configuration and more. Note: Pulsar standalone uses RocksDB as the local metadata store and its configuration file path metadataStoreConfigPath is configurable in the standalone.conf file. For more information about the configurations of RocksDB, see here and related documentation. |
examples | A Java JAR file containing Pulsar Functions example. |
instances | Artifacts created for Pulsar Functions. |
lib | The JAR files used by Pulsar. |
licenses | License files, in the.txt form, for various components of the Pulsar codebase. |
These directories are created once you begin running Pulsar.
Directory | Contains |
---|---|
data | The data storage directory used by RocksDB and BookKeeper. |
logs | Logs created by the installation. |
:::tip
If you want to use built-in connectors and tiered storage offloaders, you can install them according to the following instructions:
:::
Since 2.1.0-incubating
release, Pulsar releases a separate binary distribution, containing all the builtin
connectors. To enable those builtin
connectors, you can download the connectors tarball release in one of the following ways:
download from the Apache mirror Pulsar IO Connectors @pulsar:version@ release
download from the Pulsar downloads page
download from the Pulsar releases page
use wget:
$ wget pulsar:connector_release_url/{connector}-@pulsar:version@.nar
After you download the NAR file, copy the file to the connectors
directory in the pulsar directory. For example, if you download the pulsar-io-aerospike-@pulsar:version@.nar
connector file, enter the following commands:
$ mkdir connectors $ mv pulsar-io-aerospike-@pulsar:version@.nar connectors $ ls connectors pulsar-io-aerospike-@pulsar:version@.nar ...
:::note
connectors
tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions).apachepulsar/pulsar-all
image instead of the apachepulsar/pulsar
image. apachepulsar/pulsar-all
image has already bundled all builtin connectors.:::
:::tip
2.2.0
release, Pulsar releases a separate binary distribution, containing the tiered storage offloaders.:::
To get started with tiered storage offloaders, you need to download the offloaders tarball release on every broker node in one of the following ways:
download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release
download from the Pulsar downloads page
download from the Pulsar releases page
use wget:
$ wget pulsar:offloader_release_url
After you download the tarball, untar the offloaders package and copy the offloaders as offloaders
in the pulsar directory:
$ tar xvfz apache-pulsar-offloaders-@pulsar:version@-bin.tar.gz // you will find a directory named `apache-pulsar-offloaders-@pulsar:version@` in the pulsar directory // then copy the offloaders $ mv apache-pulsar-offloaders-@pulsar:version@/offloaders offloaders $ ls offloaders tiered-storage-jcloud-@pulsar:version@.nar
For more information on how to configure tiered storage, see Tiered storage cookbook.
:::note
offloaders
tarball is unzipped in every broker's pulsar directory.apachepulsar/pulsar-all
image instead of the apachepulsar/pulsar
image. apachepulsar/pulsar-all
image has already bundled tiered storage offloaders.:::
Once you have an up-to-date local copy of the release, you can start a local cluster using the pulsar
command, which is stored in the bin
directory, and specifying that you want to start Pulsar in standalone mode.
$ bin/pulsar standalone
If you have started Pulsar successfully, you will see INFO
-level log messages like this:
21:59:29.327 [DLM-/stream/storage-OrderedScheduler-3-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully started storage container (0). 21:59:34.576 [main] INFO org.apache.pulsar.broker.authentication.AuthenticationService - Authentication is disabled 21:59:34.576 [main] INFO org.apache.pulsar.websocket.WebSocketService - Pulsar WebSocket Service started
:::tip
bin/pulsar-daemon start standalone
command. For more information, see pulsar-daemon.bin/pulsar-admin brokers healthcheck
command. For more information, see Pulsar-admin docs.public/default
namespace is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see Topics.:::
Pulsar provides a CLI tool called pulsar-client
. The pulsar-client tool enables you to consume and produce messages to a Pulsar topic in a running cluster.
The following command consumes a message with the subscription name first-subscription
to the my-topic
topic:
$ bin/pulsar-client consume my-topic -s "first-subscription"
If the message has been successfully consumed, you will see a confirmation like the following in the pulsar-client
logs:
22:17:16.781 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully consumed
:::tip
As you have noticed that we do not explicitly create the my-topic
topic, to which we consume the message. When you consume a message on a topic that does not yet exist, Pulsar creates that topic for you automatically. Producing a message to a topic that does not exist will automatically create that topic for you as well.
:::
The following command produces a message saying hello-pulsar
to the my-topic
topic:
$ bin/pulsar-client produce my-topic --messages "hello-pulsar"
If the message has been successfully published to the topic, you will see a confirmation like the following in the pulsar-client
logs:
22:21:08.693 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced
Press Ctrl+C
to stop a local standalone Pulsar.
:::tip
If the service runs as a background process using the bin/pulsar-daemon start standalone
command, then use the bin/pulsar-daemon stop standalone
command to stop the service. For more information, see pulsar-daemon.
:::