Below is a list configurations for the Presto Pulsar connector and instruction on how to deploy a cluster.
There are several configurations for the Presto Pulsar Connector. The properties file that contain these configurations can be found at ${project.root}/conf/presto/catalog/pulsar.properties. The configurations for the connector and its default values are described below.
# name of the connector to be displayed in the catalog connector.name=pulsar # the url of Pulsar broker service pulsar.broker-service-url=http://localhost:8080 # URI of Zookeeper cluster pulsar.zookeeper-uri=localhost:2181 # minimum number of entries to read at a single time pulsar.entry-read-batch-size=100 # default number of splits to use per query pulsar.target-num-splits=4
If you already have an existing Presto cluster, you can copy Presto Pulsar connector plugin to your existing cluster. You can download the archived plugin package via:
$ wget pulsar:binary_release_url
Please note that the Getting Started guide shows you how to easily setup a standalone single node environment to experiment with.
Pulsar SQL is powered by Presto thus many of the configurations for deployment is the same for the Pulsar SQL worker.
You can use the same CLI args as the Presto launcher:
$ ./bin/pulsar sql-worker --help Usage: launcher [options] command Commands: run, start, stop, restart, kill, status Options: -h, --help show this help message and exit -v, --verbose Run verbosely --etc-dir=DIR Defaults to INSTALL_PATH/etc --launcher-config=FILE Defaults to INSTALL_PATH/bin/launcher.properties --node-config=FILE Defaults to ETC_DIR/node.properties --jvm-config=FILE Defaults to ETC_DIR/jvm.config --config=FILE Defaults to ETC_DIR/config.properties --log-levels-file=FILE Defaults to ETC_DIR/log.properties --data-dir=DIR Defaults to INSTALL_PATH --pid-file=FILE Defaults to DATA_DIR/var/run/launcher.pid --launcher-log-file=FILE Defaults to DATA_DIR/var/log/launcher.log (only in daemon mode) --server-log-file=FILE Defaults to DATA_DIR/var/log/server.log (only in daemon mode) -D NAME=VALUE Set a Java system property
There is a set of default configs for the cluster located in ${project.root}/conf/presto that will be used by default. You can change them to customize your deployment
You can also set the worker to read from a different configuration directory as well as set a different directory for writing its data:
$ ./bin/pulsar sql-worker run --etc-dir /tmp/incubator-pulsar/conf/presto --data-dir /tmp/presto-1
You can also start the worker as daemon process:
$ ./bin/pulsar sql-worker start
For example, if I wanted to deploy a Pulsar SQL/Presto cluster on 3 nodes, you can do the following:
First, copy the Pulsar binary distribution to all three nodes.
The first node, will run the Presto coordinator. The minimal configuration in ${project.root}/conf/presto/config.properties can be the following
coordinator=true node-scheduler.include-coordinator=true http-server.http.port=8080 query.max-memory=50GB query.max-memory-per-node=1GB discovery-server.enabled=true discovery.uri=<coordinator-url>
Also, modify pulsar.broker-service-url and pulsar.zookeeper-uri configs in ${project.root}/conf/presto/catalog/pulsar.properties on those nodes accordingly
Afterwards, you can start the coordinator by just running
$ ./bin/pulsar sql-worker run
For the other two nodes that will only serve as worker nodes, the configurations can be the following:
coordinator=false http-server.http.port=8080 query.max-memory=50GB query.max-memory-per-node=1GB discovery.uri=<coordinator-url>
Also, modify pulsar.broker-service-url and pulsar.zookeeper-uri configs in ${project.root}/conf/presto/catalog/pulsar.properties accordingly
You can also start the worker by just running: $ ./bin/pulsar sql-worker run
You can check the status of your cluster from the SQL CLI. To start the SQL CLI:
$ ./bin/pulsar sql --server <coordinate_url>
You can then run the following command to check the status of your nodes:
presto> SELECT * FROM system.runtime.nodes; node_id | http_uri | node_version | coordinator | state ---------+-------------------------+--------------+-------------+-------- 1 | http://192.168.2.1:8081 | testversion | true | active 3 | http://192.168.2.2:8081 | testversion | false | active 2 | http://192.168.2.3:8081 | testversion | false | active
For more information about deployment in Presto, please reference: