| // Licensed to the Apache Software Foundation (ASF) under one |
| // or more contributor license agreements. See the NOTICE file |
| // distributed with this work for additional information |
| // regarding copyright ownership. The ASF licenses this file |
| // to you under the Apache License, Version 2.0 (the |
| // "License"); you may not use this file except in compliance |
| // with the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, |
| // software distributed under the License is distributed on an |
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| // KIND, either express or implied. See the License for the |
| // specific language governing permissions and limitations |
| // under the License. |
| |
| = Kudu Docker Developer Documentation |
| |
| NOTE: All of this work is experimental and subject to change or removal. |
| |
| == Getting Started |
| |
| - Install docker following the instructions https://www.docker.com/get-started[here] |
| |
| == Building images |
| |
| NOTE: These sample commands assume running from the project root directory. |
| |
| Build all the images for the default OS: |
| [source,bash] |
| ---- |
| $ ./docker/docker-build.py |
| ---- |
| |
| NOTE: You can pass `--help` to the build script to see usage details and |
| all the available parameters. |
| |
| == Running an image |
| |
| Run an image with a bash prompt and remove it on exit: |
| [source,bash] |
| ---- |
| $ docker run -it --rm apache/kudu:latest /bin/bash |
| ---- |
| |
| == Running a Kudu cluster with docker-compose |
| |
| Below is a brief example of using Kudu with docker compose. For more advanced |
| guidance please see the docker-compose documentation |
| https://docs.docker.com/compose/[here]. |
| |
| Start a Kudu cluster with the following command: |
| (This cluster has 3 masters and 3 tablet servers) |
| [source,bash] |
| ---- |
| docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=3 -d |
| ---- |
| |
| Find the web UI url for kudu-master-1: |
| [source,bash] |
| ---- |
| docker ps -q --filter=name=.*kudu-master-1.* | xargs -L1 docker port \ |
| | grep "^8051" | cut -d":" -f2 | sed 's/.*/http\:\/\/localhost\:&/' |
| ---- |
| |
| You can adjust the number of tablet servers up or down by adjusting the scale: |
| [source,bash] |
| ---- |
| docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=<scale> -d |
| ---- |
| |
| Shutdown the cluster: |
| [source,bash] |
| ---- |
| docker-compose -f docker/docker-compose.yml down |
| ---- |
| |
| == Copying container files to the host |
| |
| It could be useful to copy files from a pre-built container to your host. |
| For example, pre-built thirdparty or kudu binaries. |
| |
| [source,bash] |
| ---- |
| $ SOURCE=`docker create kudu:thirdparty-xenial-latest` |
| $ docker cp $SOURCE:/kudu/thirdparty /local/kudu/thirdparty |
| ---- |
| |
| == Images |
| |
| === apache/kudu:[OS]-[VERSION] |
| A runtime image with the Kudu binaries and clients pre-installed |
| and an entrypoint script that enables easily starting Kudu |
| masters and tservers along with executing other commands. |
| Copies the built artifacts and files from the kudu:build image. |
| Uses the kudu:runtime image as a base. |
| |
| === apache/kudu:build-[OS]-[VERSION] |
| An image that has the Kudu source code pre-built. |
| Uses the kudu:thirdparty image as a base. |
| |
| === apache/kudu:thirdparty-[OS]-[VERSION] |
| An image that has Kudu's thirdparty dependencies built. |
| Uses the kudu:dev image as a base. |
| |
| === apache/kudu:dev-[OS]-[VERSION] |
| A base image that has all the development libraries for Kudu pre-installed. |
| |
| === apache/kudu:runtime-[OS]-[VERSION] |
| A base image that has all the runtime libraries for Kudu pre-installed. |
| |
| === apache/kudu:kudu-python-[VERSION] |
| Builds a runtime image with the Kudu python client pre-installed. |
| |
| == Tips and Troubleshooting |
| |
| === Ensure enough resources are allocated to docker |
| If your docker build fails with an unusual or unclear error a |
| good first step is to ensure docker has enough resources. |
| A good place to start is 4 CPUs, 4 GiB of memory, and 32 GiB of disk. |
| |
| === Free up disk space |
| If you don't clean up your environment periodically, you will likely |
| run out of disk space. Use the following commands clean up after builds. |
| |
| Clean all stopped containers, all dangling images, all dangling build cache |
| and all unused networks: |
| [source,bash] |
| ---- |
| $ docker system prune |
| ---- |
| |
| Remove all stopped containers: |
| [source,bash] |
| ---- |
| $ docker container prune |
| ---- |
| |
| Remove all dangling images: |
| [source,bash] |
| ---- |
| $ docker image prune |
| ---- |
| |
| Remove all Kudu images: |
| [source,bash] |
| ---- |
| $ docker rmi -f $(docker images -q apache/kudu) |
| ---- |
| |
| Remove specific images: |
| [source,bash] |
| ---- |
| $ docker rmi [IMAGE...] |
| ---- |
| |
| Remove images by tag pattern: |
| [source,bash] |
| ---- |
| $ TAG_PATTERN="apache/kudu:*" |
| $ docker rmi $(docker images -q "$TAG_PATTERN" --format "{{.Repository}}:{{.Tag}}") |
| ---- |
| |
| View and remove cache mounts (ccache and Gradle cache): |
| [source,bash] |
| ---- |
| $ docker system df -v | grep exec.cachemount |
| $ docker builder prune --filter type=exec.cachemount |
| ---- |
| |
| === Using the cache from pre-built images |
| You can tell docker to considered remote or local images in your build |
| as cache sources. This can be especially useful when the base or |
| thirdparty images have not changed as those builds can take a very |
| long time. |
| |
| To do this use the `--cache-from` flag when executing `docker build` |
| commands. |
| |
| === Use multiple workspaces |
| The docker builds can take a long time to run. During that |
| time you may want to continue other work in the git repo. Use |
| `git worktree` to specify a worktree in another directory and |
| run the docker build from there. Read more about `git worktree` |
| https://git-scm.com/docs/git-worktree[here]. |
| |
| |