Here an example:
export DOCKER_HOST=tcp://containers-api.eu-gb.bluemix.net:8443
export DOCKER_CERT_PATH=/home/markus/.ice/certs/containers-api.eu-gb.bluemix.net/3c63cb44-86d8-4e89-9a40-f8f3f894a09f
export DOCKER_TLS_VERIFY=1
Now when proceeding to the next paragraph ‘build’ the docker commands build and run the image (provided you use ./docker-compose.sh instead of the native docker-compose command. Do not forget to allocate and bind a public IP address with the ‘cf ic ip’ set of commands afterwards.
Images are based on the official Java Alpine (OpenJDK 8) image. If you want to build the flink image run:
sh build.sh
or
docker build -t flink .
If you want to build the container for a specific version of flink/hadoop/scala you can configure it in the respective args:
build.sh --from-release --flink-version 1.8.0 --hadoop-version 2.8 --scala-version 2.11 --image-name <IMAGE_NAME>
Deploy cluster and see config/setup log output (best run in a screen session)
docker-compose up
Deploy as a daemon (and return)
docker-compose up -d
Scale the cluster up or down to N TaskManagers
docker-compose scale taskmanager=<N>
Access the Job Manager container
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
Kill the cluster
docker-compose kill
Upload jar to the cluster
docker cp <your_jar> $(docker ps --filter name=flink_jobmanager --format={{.ID}}):/<your_path>
Copy file to all the nodes in the cluster
for i in $(docker ps --filter name=flink --format={{.ID}}); do docker cp <your_file> $i:/<your_path> done
Run a topology
From the jobmanager:
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
If you have a local flink installation:
$FLINK_HOME/bin/flink run -m <jobmanager:port> <your_jar>
or
$FLINK_HOME/bin/flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
48081
6123
(default, not exposed to host)6122
(default, not exposed to host)6121
(default, not exposed to host)Edit the docker-compose.yml
file to edit port settings.