layout: global title: Spark Docker Integration Tests license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Note that the integration test framework is using thirdparty Docker images to test functionalities of integration with other data sources, not running Spark itself in Docker containers, which is actually done by the Kubernetes Integration Tests module.
In order to run the Docker integration tests, the Docker engine needs to be installed and started on the machine. Additionally, the environment variable ENABLE_DOCKER_INTEGRATION_TESTS=1
shall be specified to enable the Docker integration tests.
export ENABLE_DOCKER_INTEGRATION_TESTS=1
or,
ENABLE_DOCKER_INTEGRATION_TESTS=1 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/test"
Testing the whole module of Docker Integration Tests might be time-consuming because of image pulling and the container bootstrapping. To run an individual Docker integration test, use the following command:
./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly <test class name>"
Besides the default Docker images, the integration tests can be run with custom Docker images. For example,
ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.4-slim-faststart ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"
The following environment variables can be used to specify the custom Docker images for different databases:
In certain scenarios, you may want to use a different docker context/endpoint instead of the default provided by Docker Desktop.
docker context ls NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR default * moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock desktop-linux moby Docker Desktop unix:///Users/.../docker.sock docker context use desktop-linux
Then you can run the integration tests as usual targeting the Docker endpoint named desktop-linux
.
This is useful when the upstream Docker image can not starton the Docker Desktop. For example, when the image you use only supports x86_64
architecture, but you are running on an Apple Silicon aarch64
machine, for which case you may want to use a custom context that is able to mock a x86_64
architecture, such as colima
. After colima installed, you can start a runtime with x86_64
support and run the integration tests as follows:
colima start --arch x86_64 --memory 8 --network-address docker context colima ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"
The following are the available properties that can be passed to optimize testing experience.
./build/sbt -Pdocker-integration-tests \ -Dspark.test.docker.keepContainer=true \ "testOnly *MariaDBKrbIntegrationSuite"