docker-pinot

This is a docker image of Apache Pinot.

How to build a docker image

There is a docker build script which will build a given Git repo/branch and tag the image.

Usage:

./docker-build.sh [Docker Tag] [Git Branch] [Pinot Git URL] [Kafka Version] [Java Version] [JDK Version] [OpenJDK Image ]

This script will check out Pinot Repo [Pinot Git URL] on branch [Git Branch] and build the docker image for that.

The docker image is tagged as [Docker Tag].

Docker Tag: Name and tag your docker image. Default is pinot:latest.

Git Branch: The Pinot branch to build. Default is master.

Pinot Git URL: The Pinot Git Repo to build, users can set it to their own fork. Please note that, the URL is https:// based, not git://. Default is the Apache Repo: https://github.com/apache/pinot.git.

Kafka Version: The Kafka Version to build pinot with. Default is 2.0

Java Version: The Java Build and Runtime image version. Default is 11

JDK Version: The JDK parameter to build pinot, set as part of maven build option: -Djdk.version=${JDK_VERSION}. Default is 11

OpenJDK Image: Base image to use for Pinot build and runtime. Default is openjdk.

• Example of building and tagging a snapshot on your own fork:

./docker-build.sh pinot_fork:snapshot-5.2 snapshot-5.2 https://github.com/your_own_fork/pinot.git

• Example of building a release version:

./docker-build.sh pinot:release-0.1.0 release-0.1.0 https://github.com/apache/pinot.git

Build image with arm64 base image

For users on Mac M1 chips, they need to build the images with arm64 base image, e.g. arm64v8/openjdk

• Example of building an arm64 image:

./docker-build.sh pinot:latest master https://github.com/apache/pinot.git 2.0 11 11 arm64v8/openjdk

or just run the docker build script directly

docker build -t pinot:latest --no-cache --network=host --build-arg PINOT_GIT_URL=https://github.com/apache/pinot.git --build-arg PINOT_BRANCH=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .

Note that if you are not on arm64 machine, you can still build the image by turning on the experimental feature of docker, and add --platform linux/arm64 into the docker build ... script, e.g.

docker build -t pinot:latest --platform linux/arm64 --no-cache --network=host --build-arg PINOT_GIT_URL=https://github.com/apache/pinot.git --build-arg PINOT_BRANCH=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .

How to publish a docker image

Script docker-push.sh publishes a given docker image to your docker registry.

In order to push to your own repo, the image needs to be explicitly tagged with the repo name.

• Example of publishing a image to apachepinot/pinot dockerHub repo.

./docker-push.sh apachepinot/pinot:latest

• Tag a built image, then push.

docker tag pinot:release-0.1.0 apachepinot/pinot:release-0.1.0
docker push apachepinot/pinot:release-0.1.0

Script docker-build-and-push.sh builds and publishes this docker image to your docker registry after build.

• Example of building and publishing a image to apachepinot/pinot dockerHub repo.

./docker-build-and-push.sh apachepinot/pinot:latest master https://github.com/apache/pinot.git

How to Run it

The entry point of docker image is pinot-admin.sh script.

Bring up Zookeeper

Example of bring up a local zookeeper in docker:

docker pull zookeeper
docker run --name  pinot-zookeeper --restart always -p 2181:2181  zookeeper

You can extract the zookeeper host from:

docker inspect pinot-zookeeper|grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.2",
                    "IPAddress": "172.17.0.2",

Please use local zookeeper path 172.17.0.2:2181 as -zkAddress parameter.

Pinot Controller

Example of bring up a local controller:

docker run -p 9000:9000 pinot:release-0.1.0 StartController -zkAddress 172.17.0.2:2181

Pinot Broker

Example of bring up a local broker:

docker run -p 8099:8099 pinot:release-0.1.0 StartBroker -zkAddress 172.17.0.2:2181

Pinot Server

Example of bring up a local server:

docker run -p 8098:8098 pinot:release-0.1.0 StartServer -zkAddress 172.17.0.2:2181

QuickStart

Use docker compose to bring up Pinot stack

Please note that this quickstart is based on Kafka 2.0.

Below is a script to use docker compose to bring up zookeeper/kafka/pinot-controller/pinot-broker/pinot-server

docker-compose -f docker-compose.yml up

Create table and load data from Kafka.

Below is the script to create airlineStats table

docker run --network=pinot_default apachepinot/pinot:release-0.7.1 AddTable -schemaFile examples/stream/airlineStats/airlineStats_schema.json -tableConfigFile examples/stream/airlineStats/docker/airlineStats_realtime_table_config.json -controllerHost pinot-controller -controllerPort 9000 -exec

Below is the script to ingest airplane stats data to Kafka

docker run --network=pinot_default apachepinot/pinot:release-0.7.1 StreamAvroIntoKafka -avroFile examples/stream/airlineStats/sample_data/airlineStats_data.avro -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092 -zkAddress zookeeper:2181

In order to query pinot, try to open localhost:9000 from your browser.