docker/README.adoc

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

= Kudu Docker Developer Documentation

NOTE: All of this work is experimental and subject to change or removal.

== Getting Started

- Install docker following the instructions https://www.docker.com/get-started[here]

== Building images

NOTE: These sample commands assume running from the project root directory.

Build all the images for the default OS:
[source,bash]
----
$ ./docker/docker-build.py
----

NOTE: You can pass `--help` to the build script to see usage details and
all the available parameters.

== Running an image

Run an image with a bash prompt and remove it on exit:
[source,bash]
----
$ docker run -it --rm apache/kudu:latest /bin/bash
----

== Running a Kudu cluster with docker-compose

Below is a brief example of using Kudu with docker compose. For more advanced
guidance please see the docker-compose documentation
https://docs.docker.com/compose/[here].

Start a Kudu cluster with the following command:
(This cluster has 3 masters and 3 tablet servers)
[source,bash]
----
docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=3 -d
----

Find the web UI url for kudu-master-1:
[source,bash]
----
docker ps -q --filter=name=.*kudu-master-1.* | xargs -L1 docker port \
  | grep "^8051" | cut -d":" -f2 | sed 's/.*/http\:\/\/localhost\:&/'
----

You can adjust the number of tablet servers up or down by adjusting the scale:
[source,bash]
----
docker-compose -f docker/docker-compose.yml up --scale kudu-tserver=<scale> -d
----

Shutdown the cluster:
[source,bash]
----
docker-compose -f docker/docker-compose.yml down
----

== Copying container files to the host

It could be useful to copy files from a pre-built container to your host.
For example, pre-built thirdparty or kudu binaries.

[source,bash]
----
$ SOURCE=`docker create kudu:thirdparty-xenial-latest`
$ docker cp $SOURCE:/kudu/thirdparty /local/kudu/thirdparty
----

== Images

=== apache/kudu:[OS]-[VERSION]
A runtime image with the Kudu binaries and clients pre-installed
and an entrypoint script that enables easily starting Kudu
masters and tservers along with executing other commands.
Copies the built artifacts and files from the kudu:build image.
Uses the kudu:runtime image as a base.

=== apache/kudu:build-[OS]-[VERSION]
An image that has the Kudu source code pre-built.
Uses the kudu:thirdparty image as a base.

=== apache/kudu:thirdparty-[OS]-[VERSION]
An image that has Kudu's thirdparty dependencies built.
Uses the kudu:dev image as a base.

=== apache/kudu:dev-[OS]-[VERSION]
A base image that has all the development libraries for Kudu pre-installed.

=== apache/kudu:runtime-[OS]-[VERSION]
A base image that has all the runtime libraries for Kudu pre-installed.

=== apache/kudu:kudu-python-[VERSION]
Builds a runtime image with the Kudu python client pre-installed.

== Tips and Troubleshooting

=== Ensure enough resources are allocated to docker
If your docker build fails with an unusual or unclear error a
good first step is to ensure docker has enough resources.
A good place to start is 4 CPUs, 4 GiB of memory, and 32 GiB of disk.

=== Free up disk space
If you don't clean up your environment periodically, you will likely
run out of disk space. Use the following commands clean up after builds.

Clean all stopped containers, all dangling images, all dangling build cache
and all unused networks:
[source,bash]
----
$ docker system prune
----

Remove all stopped containers:
[source,bash]
----
$ docker container prune
----

Remove all dangling images:
[source,bash]
----
$ docker image prune
----

Remove all Kudu images:
[source,bash]
----
$ docker rmi -f $(docker images -q apache/kudu)
----

Remove specific images:
[source,bash]
----
$ docker rmi [IMAGE...]
----

Remove images by tag pattern:
[source,bash]
----
$ TAG_PATTERN="apache/kudu:*"
$ docker rmi $(docker images -q "$TAG_PATTERN" --format "{{.Repository}}:{{.Tag}}")
----

View and remove cache mounts (ccache and Gradle cache):
[source,bash]
----
$ docker system df -v | grep exec.cachemount
$ docker builder prune --filter type=exec.cachemount
----

=== Using the cache from pre-built images
You can tell docker to considered remote or local images in your build
as cache sources. This can be especially useful when the base or
thirdparty images have not changed as those builds can take a very
long time.

To do this use the `--cache-from` flag when executing `docker build`
commands.

=== Use multiple workspaces
The docker builds can take a long time to run. During that
time you may want to continue other work in the git repo. Use
`git worktree` to specify a worktree in another directory and
run the docker build from there. Read more about `git worktree`
https://git-scm.com/docs/git-worktree[here].