blob: 1d6ad923fd4d500ed7dfbaca5470b4fc6d7214bd [file] [log] [blame] [view]
---
id: kubernetes-helm
title: Get started in Kubernetes
sidebar_label: "Run Pulsar in Kubernetes"
---
:::tip
This page has been deprecated and is no longer updated. For the latest and complete information about running Pulsar in Kubernetes, see the [quick start guide](getting-started-helm.md).
:::
This section guides you through every step of installing and running Apache Pulsar with Helm on Kubernetes quickly, including the following sections:
- Install the Apache Pulsar on Kubernetes using Helm
- Start and stop Apache Pulsar
- Create topics using `pulsar-admin`
- Produce and consume messages using Pulsar clients
- Monitor Apache Pulsar status with Prometheus and Grafana
For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md).
## Prerequisite
- Kubernetes server 1.18.0+
- kubectl 1.18.0+
- Helm 3.0+
:::tip
For the following steps, step 2 and step 3 are for **developers** and step 4 and step 5 are for **administrators**.
:::
## Step 0: Prepare a Kubernetes cluster
Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster.
We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps:
1. Create a Kubernetes cluster on Minikube.
```bash
minikube start --memory=8192 --cpus=4 --kubernetes-version=<k8s-version>
```
The `<k8s-version>` can be any [Kubernetes version supported by your Minikube installation](https://minikube.sigs.k8s.io/docs/reference/configuration/kubernetes/), such as `v1.16.1`.
2. Set `kubectl` to use Minikube.
```bash
kubectl config use-context minikube
```
3. To use the [Kubernetes Dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) with the local Kubernetes cluster on Minikube, enter the command below:
```bash
minikube dashboard
```
The command automatically triggers opening a webpage in your browser.
## Step 1: Install Pulsar Helm chart
1. Add Pulsar charts repo.
```bash
helm repo add apache https://pulsar.apache.org/charts
```
```bash
helm repo update
```
2. Clone the Pulsar Helm chart repository.
```bash
git clone https://github.com/apache/pulsar-helm-chart
cd pulsar-helm-chart
```
3. Run the script `prepare_helm_release.sh` to create secrets required for installing the Apache Pulsar Helm chart. The username `pulsar` and password `pulsar` are used for logging into the Grafana dashboard and Pulsar Manager.
```bash
./scripts/pulsar/prepare_helm_release.sh \
-n pulsar \
-k pulsar-mini \
-c
```
4. Use the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.
```bash
helm install \
--values examples/values-minikube.yaml \
--namespace pulsar \
pulsar-mini apache/pulsar
```
5. Check the status of all pods.
```bash
kubectl get pods -n pulsar
```
If all pods start up successfully, you can see that the `STATUS` is changed to `Running` or `Completed`.
**Output**
```bash
NAME READY STATUS RESTARTS AGE
pulsar-mini-bookie-0 1/1 Running 0 9m27s
pulsar-mini-bookie-init-5gphs 0/1 Completed 0 9m27s
pulsar-mini-broker-0 1/1 Running 0 9m27s
pulsar-mini-grafana-6b7bcc64c7-4tkxd 1/1 Running 0 9m27s
pulsar-mini-prometheus-5fcf5dd84c-w8mgz 1/1 Running 0 9m27s
pulsar-mini-proxy-0 1/1 Running 0 9m27s
pulsar-mini-pulsar-init-t7cqt 0/1 Completed 0 9m27s
pulsar-mini-pulsar-manager-9bcbb4d9f-htpcs 1/1 Running 0 9m27s
pulsar-mini-toolset-0 1/1 Running 0 9m27s
pulsar-mini-zookeeper-0 1/1 Running 0 9m27s
```
6. Check the status of all services in the namespace `pulsar`.
```bash
kubectl get services -n pulsar
```
**Output**
```bash
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pulsar-mini-bookie ClusterIP None <none> 3181/TCP,8000/TCP 11m
pulsar-mini-broker ClusterIP None <none> 8080/TCP,6650/TCP 11m
pulsar-mini-grafana LoadBalancer 10.106.141.246 <pending> 3000:31905/TCP 11m
pulsar-mini-prometheus ClusterIP None <none> 9090/TCP 11m
pulsar-mini-proxy LoadBalancer 10.97.240.109 <pending> 80:32305/TCP,6650:31816/TCP 11m
pulsar-mini-pulsar-manager LoadBalancer 10.103.192.175 <pending> 9527:30190/TCP 11m
pulsar-mini-toolset ClusterIP None <none> <none> 11m
pulsar-mini-zookeeper ClusterIP None <none> 2888/TCP,3888/TCP,2181/TCP 11m
```
## Step 2: Use pulsar-admin to create Pulsar tenants/namespaces/topics
`pulsar-admin` is the CLI (command-Line Interface) tool for Pulsar. In this step, you can use `pulsar-admin` to create resources, including tenants, namespaces, and topics.
1. Enter the `toolset` container.
```bash
kubectl exec -it -n pulsar pulsar-mini-toolset-0 -- /bin/bash
```
2. In the `toolset` container, create a tenant named `apache`.
```bash
bin/pulsar-admin tenants create apache
```
Then you can list the tenants to see if the tenant is created successfully.
```bash
bin/pulsar-admin tenants list
```
You should see a similar output as below. The tenant `apache` has been successfully created.
```bash
"apache"
"public"
"pulsar"
```
3. In the `toolset` container, create a namespace named `pulsar` in the tenant `apache`.
```bash
bin/pulsar-admin namespaces create apache/pulsar
```
Then you can list the namespaces of tenant `apache` to see if the namespace is created successfully.
```bash
bin/pulsar-admin namespaces list apache
```
You should see a similar output as below. The namespace `apache/pulsar` has been successfully created.
```bash
"apache/pulsar"
```
4. In the `toolset` container, create a topic `test-topic` with `4` partitions in the namespace `apache/pulsar`.
```bash
bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4
```
5. In the `toolset` container, list all the partitioned topics in the namespace `apache/pulsar`.
```bash
bin/pulsar-admin topics list-partitioned-topics apache/pulsar
```
Then you can see all the partitioned topics in the namespace `apache/pulsar`.
```bash
"persistent://apache/pulsar/test-topic"
```
## Step 3: Use Pulsar client to produce and consume messages
You can use the Pulsar client to create producers and consumers to produce and consume messages.
By default, the Pulsar Helm chart exposes the Pulsar cluster through a Kubernetes `LoadBalancer`. In Minikube, you can use the following command to check the proxy service.
```bash
kubectl get services -n pulsar | grep pulsar-mini-proxy
```
You will see a similar output as below.
```bash
pulsar-mini-proxy LoadBalancer 10.97.240.109 <pending> 80:32305/TCP,6650:31816/TCP 28m
```
This output tells what are the node ports that Pulsar cluster's binary port and HTTP port are mapped to. The port after `80:` is the HTTP port while the port after `6650:` is the binary port.
Then you can find the IP address and exposed ports of your Minikube server by running the following command.
```bash
minikube service pulsar-mini-proxy -n pulsar
```
**Output**
```bash
|-----------|-------------------|-------------|-------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------------|-------------|-------------------------|
| pulsar | pulsar-mini-proxy | http/80 | http://172.17.0.4:32305 |
| | | pulsar/6650 | http://172.17.0.4:31816 |
|-----------|-------------------|-------------|-------------------------|
🏃 Starting tunnel for service pulsar-mini-proxy.
|-----------|-------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------------|-------------|------------------------|
| pulsar | pulsar-mini-proxy | | http://127.0.0.1:61853 |
| | | | http://127.0.0.1:61854 |
|-----------|-------------------|-------------|------------------------|
```
At this point, you can get the service URLs to connect to your Pulsar client. Here are URL examples:
```conf
webServiceUrl=http://127.0.0.1:61853/
brokerServiceUrl=pulsar://127.0.0.1:61854/
```
Then you can proceed with the following steps:
1. Download the Apache Pulsar tarball from the [downloads page](/download/).
2. Decompress the tarball based on your download file.
```bash
tar -xf <file-name>.tar.gz
```
3. Expose `PULSAR_HOME`.
(1) Enter the directory of the decompressed download file.
(2) Expose `PULSAR_HOME` as the environment variable.
```bash
export PULSAR_HOME=$(pwd)
```
4. Configure the Pulsar client.
In the `${PULSAR_HOME}/conf/client.conf` file, replace `webServiceUrl` and `brokerServiceUrl` with the service URLs you get from the above steps.
5. Create a subscription to consume messages from `apache/pulsar/test-topic`.
```bash
bin/pulsar-client consume -s sub apache/pulsar/test-topic -n 0
```
6. Open a new terminal. In the new terminal, create a producer and send 10 messages to the `test-topic` topic.
```bash
bin/pulsar-client produce apache/pulsar/test-topic -m "---------hello apache pulsar-------" -n 10
```
7. Verify the results.
- From the producer side
**Output**
The messages have been produced successfully.
```bash
18:15:15.489 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced
```
- From the consumer side
**Output**
At the same time, you can receive the messages as below.
```bash
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
```
## Step 4: Use Pulsar Manager to manage the cluster
[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar.
1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command:
```bash
minikube service -n pulsar pulsar-mini-pulsar-manager
```
2. The Pulsar Manager UI will be open in your browser. You can use the username `pulsar` and password `pulsar` to log into Pulsar Manager.
3. In Pulsar Manager UI, you can create an environment.
- Click `New Environment` button in the top-left corner.
- Type `pulsar-mini` for the field `Environment Name` in the popup window.
- Type `http://pulsar-mini-broker:8080` for the field `Service URL` in the popup window.
- Click `Confirm` button in the popup window.
4. After successfully creating an environment, you are redirected to the `tenants` page of that environment. Then you can create `tenants`, `namespaces` and `topics` using the Pulsar Manager.
## Step 5: Use Prometheus and Grafana to monitor cluster
Grafana is an open-source visualization tool, which can be used for visualizing time series data into dashboards.
1. By default, the Grafana is exposed as a separate `LoadBalancer`. You can open the Grafana UI using the following command:
```bash
minikube service pulsar-mini-grafana -n pulsar
```
2. The Grafana UI is open in your browser. You can use the username `pulsar` and password `pulsar` to log into the Grafana Dashboard.
3. You can view dashboards for different components of a Pulsar cluster.