commit | e9ebf5d567930bdbefaaea1fb9bda99ef6898dc0 | [log] [tgz] |
---|---|---|
author | Enrico Olivelli <eolivelli@gmail.com> | Mon Jun 22 13:15:51 2020 +0200 |
committer | GitHub <noreply@github.com> | Mon Jun 22 19:15:51 2020 +0800 |
tree | 51a5bcf582aeb9331bd9010b63f6e26a0b61a3ba | |
parent | d24e5055c0d452cb0d702afe5f45e5fd80d4995e [diff] |
Upgrade HerdDB to 0.16, Pulsar Client to 2.5.2, add demo config for diskless-cluster mode (#303) ### Motivation Since HerdDB 0.16 you can store all of the data on BookKeeper, this is called "diskless-cluster" mode. So you will be able to run PulsarManager + HerdDB on a stateless pod, all of the data will be stored on the same BK+ZK cluster as your Pulsar installation. ### Modifications - Upgrade HerdDB to 0.16.1. - Upgrade Apache Pulsar Admin Client to 2.5.2 (that officially supports BK 4.10) - Use pulsar-client-admin-original instead of pulsar-client-admin, because pulsar-client-admin contains a shaded version of CirceChecksum library from BK that is not compatibile with BK 4.10, needed by HerdDB in order to run in diskless-cluster mode. - Configure gradle to put application.properties inside pulsar-manager.tar, this way it is super easy to change the configuration of pulsar-manager backend without recompiling the package (no need to use Gradle for System administrator) - Add an example JDBC URI to boot HerdDB in diskless-cluster mode ### Verifying this change - [ ] Make sure that the change passes the `./gradlew build` checks.
Apache Pulsar manager is a web-based GUI management tool for managing and monitoring Pulsar.
A Pulsar Manager is capable of managing multiple Pulsar clusters. A Pulsar instance or a group of Pulsar clusters is defined as an Environment
in Pulsar Manager. You can create as many environments as you can. The following diagram illustrates the overall architecture of a Pulsar Manager.
A running Pulsar Manager instance is comprised of two components: one is the frontend
which provides the GUI to manage and monitor Pulsar clusters, the other one is the backend
which interacts with Pulsar clusters for collecting metrics, forwarding admin requests to the requested Pulsar cluster or brokers.
Since Pulsar Manager directly interacts with Pulsar brokers and bookies for management, it is required to deploy the Pulsar Manager into the same network as your Pulsar clusters. So the Pulsar Manager backend is able to talk to the brokers and bookies in your Pulsar cluster.
Start Pulsar standalone.
docker pull apachepulsar/pulsar:latest docker run -d -it \ -p 6650:6650 \ -p 8080:8080 \ -v pulsardata:/pulsar/data \ -v pulsarconf:/pulsar/conf \ --name pulsar-standalone \ apachepulsar/pulsar:latest \ bin/pulsar standalone
Start Pulsar Manager in a separate container.
NOTE: the command links the pulsar-manager container with the pulsar standalone container so they are in the same network.
``` docker pull apachepulsar/pulsar-manager:v0.1.0 docker run -it \ -p 9527:9527 \ -e REDIRECT_HOST=127.0.0.1 \ -e REDIRECT_PORT=9527 \ -e DRIVER_CLASS_NAME=org.postgresql.Driver \ -e URL='jdbc:postgresql://127.0.0.1:5432/pulsar_manager' \ -e USERNAME=pulsar \ -e PASSWORD=pulsar \ -e LOG_LEVEL=DEBUG \ -v pulsarui:/data \ --link pulsar-standalone \ apachepulsar/pulsar-manager:v0.1.0 ``` * `REDIRECT_HOST`: the IP address of the front-end server. * `REDIRECT_PORT`: the port of the front-end server. * `DRIVER_CLASS_NAME`: the driver class name of PostgreSQL. * `URL`: the jdbc url of your PostgreSQL database, example: jdbc:postgresql://127.0.0.1:5432/pulsar_manager. The docker image automatically start a local instance of PostgresSQL database. * `USERNAME`: the username of PostgreSQL. * `PASSWORD`: the password of PostgreSQL. * `LOG_LEVEL`: level of log. * `pulsarui`: docker volume to store PostgresSQL data.
Alternatively, you can start a standalone and Pulsar Manager using the docker compose file provided in this repository.
Clone this git repoistory.
https://github.com/apache/pulsar-manager
Go to the docker directory.
cd docker
Start the docker composed cluster.
docker-compose up
The Pulsar Manager can be deployed as part of Pulsar Helm Chart.
Install the Pulsar cluster with Pulsar Manager
helm repo add streamnative https://charts.streamnative.io helm repo update helm install <release-name> streamnative/pulsar
Access the Pulsar Manager.
The default values will create a ClusterIP
for the Pulsar Manager you can use to interact with cluster. To find the IP address of the Pulsar Manager, use the following command:
kubectl get service -n <k8s-namespace>
After find the ip address of the Pulsar Manager, you can access the Pulsar Manager at http://${pulsar-manager-cluster-ip}/#/environments
.
Clone the source code.
git clone https://github.com/apache/pulsar-manager
Build and start the backend.
cd pulsar-manager ./gradlew build -x test cd build/distributions unzip pulsar-manager.zip or tar -zxvf pulsar-manager.tar ./pulsar-manager/bin/pulsar-manager
Open a new terminal, build and start the front end.
cd pulsar-manager/front-end npm install --save npm run dev
After running these steps, the Pulsar Manager is running locally at http://127.0.0.1/#/environments.
http://${frontend-end-ip}/#/environments
.If you started Pulsar Manager using docker or docker-compose, the Pulsar Manager is running at port 9527. You can access the Pulsar Manager UI at http://127.0.0.1/#/environments.
If you are deploying Pulsar Manager 0.1.0 using the released container, you can log in the Pulsar Manager UI using the following credentials.
pulsar
pulsar
If you are deploying Pulsar Manager using the latest code, you can create a super-user using the following command. Then you can use the super user credentials to log in the Pulsar Manager UI.
```$xslt curl -H "Content-Type: application/json" \ -X PUT http://backend-service:7750/pulsar-manager/users/superuser \ -d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username@test.org"}' ```
backend-service
: The IP address or domain name of the backend service.
Create an environment.
An environment represents a Pulsar instance or a group of clusters you want to manage. A Pulsar Manager is capable of managing multiple environments.
http://pulsar-standalone:8080
.For more information about the back end, see pulsar-manager-backend.
For more information about the front end, see pulsar-manager-frontend.
Use the default account (pulsar
) and the default password (pulsar
) to log in.
The puslar-manager supports multiple environment configurations and can manage multiple environments conveniently.
The pulsar-manager can monitor topics and subscriptions.
Pulsar Manager bundles JDBC Drivers for HerdDB. The default configuration starts and embedded in-memory only HerdDB database.
HerdDB can be used in production, you just have to use the correct JDBC URL. Follow the instructions in application.properties to switch the connection to a standalone HerdDB service or cluster.
The JDBC URL will look like this: jdbc:herddb:server:localhost:7000
In cluster mode HerdDB uses Apache BookKeeper and Apache ZooKeeper to store data and metadata, you can share your ZooKeeper cluster and the Bookies bundled with Pulsar.
The JDBC URL will look like this: jdbc:herddb:zookeeper:localhost:2181/herddb
In order to start and setup an HerdDB database follow the instructions on the HerdDB documentation.