Google Git
Sign in
apache / spark-docker

Official Dockerfile for Apache Spark

Mirrored from https://github.com/apache/spark-docker.git
Clone this repo:

Branches

  1. 51acf3d Update example symbolic link creation in Dockerfile.template (#76) by Kent Yao · 5 weeks ago master
  2. 18c599e Publish 3.4.4 to docker registry (#75) by Dongjoon Hyun · 6 weeks ago
  3. cf333e1 [SPARK-45497][FOLLOWUP][3.5] Simplify symbolic link creation of spark-examples.jar (#74) by Kent Yao · 8 weeks ago
  4. 059a281 [SPARK-45497][FOLLOWUP] Simplify symbolic link creation of spark-examples.jar (#73) by Kent Yao · 9 weeks ago
  5. e498bfc [SPARK-49813] Publish Apache Spark 3.5.3 to docker registry by Haejoon Lee · 2 months ago

Apache Spark Official Dockerfiles

What is Apache Spark?

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

https://spark.apache.org/

Create a new version

Step 1 Add dockerfiles for a new version.

You can see 3.4.0 PR as reference.

  • 1.1 Add gpg key to tools/template.py

    This gpg key will be used by Dockerfiles (such as 3.4.0) to verify the signature of the Apache Spark tarball.

  • 1.2 Add image build workflow (such as 3.4.0 yaml)

    This file will be used by GitHub Actions to build the Docker image when you submit the PR to make sure dockerfiles are correct and pass all tests (build/standalone/kubernetes).

  • 1.3 Using ./add-dockerfiles.sh [version] to add Dockerfiles.

    You will get a new directory with the Dockerfiles for the specified version.

  • 1.4 Add version and tag info to versions.json, publish.yml and test.yml.

    This version file will be used by image build workflow (such as 3.4.0 reference) and docker official image.

Step 2. Publish apache/spark Images.

Click Publish (Java 21 only), Publish (Java 17 only) (such as 4.x) or Publish (such as 3.x) to publish images.

After this, the apache/spark docker images will be published.

Step 3. Publish spark Docker Official Images.

Submit the PR to docker-library/official-images, see (link)[https://github.com/docker-library/official-images/pull/15363] as reference.

You can type tools/manifest.py manifest to generate the content.

After this, the spark docker images will be published.

About images

Apache Spark ImageSpark Docker Official Image
Nameapache/sparkspark
MaintenanceReviewed, published by Apache Spark communityReviewed, published and maintained by Docker community
Update policyOnly build and push once when specific version releaseActively rebuild for updates and security fixes
Linkhttps://hub.docker.com/r/apache/sparkhttps://hub.docker.com/_/spark
sourceapache/spark-dockerapache/spark-docker and docker-library/official-images

We recommend using Spark Docker Official Image, the Apache Spark Image are provided in case of delays in the review process by Docker community.

About this repository

This repository contains the Dockerfiles used to build the Apache Spark Docker Image.

See more in SPARK-40513: SPIP: Support Docker Official Image for Spark.