test-with-docker.py runs the Impala build and tests inside of Docker containers, parallelizing the test execution across test suites. See that file for more details.
This also contains infrastructure to build impala_base, catalogd, statestored, impalad_coordinator, impalad_executor and impalad_coord_exec container images from the output of an Impala build. The containers can be built via the CMake target docker_images. See CMakeLists.txt for the build targets.
Various docker-compose files in this directory provide a convenient way to run a basic Impala service with a single Impala Daemon and minimal set of supporting services. A Hive MetaStore service is used to manage metadata. All filesystem data is stored in Docker volumes. The default storage location for tables is in the impala-quickstart-warehouse volume, i.e. if you create a table in Impala, it will be stored in that volume by default.
A docker network called quickstart-network must be created, and the QUICKSTART_IP and QUICKSTART_LISTEN_ADDR environment variables must be set.
docker network create -d bridge quickstart-network export QUICKSTART_IP=$(docker network inspect quickstart-network -f '{{(index .IPAM.Config 0).Gateway}}') export QUICKSTART_LISTEN_ADDR=$QUICKSTART_IP
If you want the cluster to be open to connections from other hosts, you can set QUICKSTART_LISTEN_ADDR:
export QUICKSTART_LISTEN_ADDR=0.0.0.0
You can optionally set IMPALA_QUICKSTART_IMAGE_PREFIX to pull prebuilt images from a DockerHub repo. For example, the following will use images like apache/impala:81d5377c2-impalad_coordinator:
export IMPALA_QUICKSTART_IMAGE_PREFIX="apache/impala:81d5377c2-"
Leave IMPALA_QUICKSTART_IMAGE_PREFIX unset to use images built from a local Impala dev environment.
To start the base quickstart cluster without Kudu:
docker-compose -f docker/quickstart.yml up -d
To load data in background into Parquet and Kudu formats:
docker-compose -f docker/quickstart.yml -f docker/quickstart-kudu-minimal.yml \ -f docker/quickstart-load-data.yml up -d
To follow the data loading process, you can use the docker logs command, e.g.:
docker logs -f docker_data-loader_1
The impala service can be connected to $QUICKSTART_IP, or if you set QUICKSTART_LISTEN_ADDR=0.0.0.0, you can connect to it on localhost or your machine's host name.
docker run --network=quickstart-network -it \ ${IMPALA_QUICKSTART_IMAGE_PREFIX}impala_quickstart_client impala-shell
Or with a pre-installed impala-shell:
impala-shell -i ${QUICKSTART_IP}
If you want to directly interact with the contents of the warehouse in the impala-quickstart-warehouse Docker volume or copy data from the host into the quickstart warehouse, you can mount the volume in another container. E.g. to run an Ubuntu 18.04 container with the warehouse directory mounted at /user/hive/warehouse and your home directory mounted at /host_dir, you can run the following command:
docker run -v ~:/host_dir -v docker_impala-quickstart-warehouse:/user/hive/warehouse \ -it ubuntu:18.04 /bin/bash
In the container, you can find the external and managed tablespaces stored in the impala-quickstart-warehouse volume, for example:
root@377747c68bfa:/# ls /user/hive/warehouse/external/tpcds_raw/ call_center customer_demographics inventory store_returns web_sales catalog_page date_dim item store_sales web_site catalog_returns dbgen_version promotion time_dim catalog_sales generated reason warehouse customer household_demographics ship_mode web_page customer_address income_band store web_returns t@377747c68bfa:/# head -n2 /user/hive/warehouse/external/tpcds_raw/time_dim/time_dim.dat 0|AAAAAAAABAAAAAAA|0|0|0|0|AM|third|night|| 1|AAAAAAAACAAAAAAA|1|0|0|1|AM|third|night||
It is then possible to copy data files from the host into an external table. In impala-shell, create an external table:
create external table quickstart_example(s string) stored as textfile location '/user/hive/warehouse/external/quickstart_example';
Then in the host and container shells, create a text file and copy it into the external table directory.
# On host: echo 'hello world' > ~/hw.txt # In container: cp /host_dir/hw.txt /user/hive/warehouse/external/quickstart_example
You can then refresh the table to pick up the data file and query the table:
refresh quickstart_example; select * from quickstart_example;
The following environment variables influence the behaviour of the various quickstart docker compose files.
KUDU_QUICKSTART_VERSION - defaults to latest, can be overridden to a different tag to use different Kudu images.IMPALA_QUICKSTART_IMAGE_PREFIX - defaults to using local images, change to to a different prefix to pick up prebuilt images.QUICKSTART_LISTEN_ADDR - can be set to either $QUICKSTART_IP to listen on only the docker network interface, or 0.0.0.0 to listen on all interfaces.To publish the images you need to build locally then run publish_images_to_apache.sh to tag and push them to a docker repository. For example, to tag the images with the current commit hash and upload them to the default apache/impala Docker repository, you can run the following commands:
cd $IMPALA_HOME IMAGE_VERSION=$(git rev-parse --short HEAD) ./buildall.sh -release -noclean -ninja -skiptests -notests ninja docker_images quickstart_docker_images ./docker/publish_images_to_apache.sh -v ${IMAGE_VERSION} -
For official Impala releases you will want to use the release version instead.