[AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example. (#2408) ### Which issue does this PR close? Closes #2407 ### Rationale for this change The Docker build example generated by `auron-build.sh --help` used Spark 4.1 together with the `centos7` image. The `centos7` image only provides JDK 8, while Spark 4.x requires JDK 17 or later. The example also enabled Iceberg and Hudi for Spark versions with which those integrations are not compatible. As a result, users following the documented example encountered Java version or integration compatibility errors before completing the build. ### What changes are included in this PR? - Replace the single Docker build example with explicit examples for Spark 3.5, Spark 4.0, and Spark 4.1. - Use the `rockylinux8` image for these examples because it supports the JDK selection required by Spark 4.x. - Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x. - Include only compatible third-party integrations in each example: - Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15. - Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi. - Spark 4.1 excludes both Iceberg and Hudi. - Add a build compatibility matrix to `CONTRIBUTING.md`. - Document the Spark, Scala, JDK, Docker image, Flink, Celeborn, Uniffle, Paimon, Iceberg, and Hudi combinations. - Update the existing Docker build example in `CONTRIBUTING.md` to use `rockylinux8`. - Document that the compatibility matrix and Shell help examples should be updated together. ### Are there any user-facing changes? Yes. The help output now provides separate Docker build examples for Spark 3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala, JDK, Docker image, and third-party integration combination. Contributors can also consult the compatibility matrix in `CONTRIBUTING.md` before selecting build arguments. ### How was this patch tested? > ./auron-build.sh --help ``` Usage: ./auron-build.sh [OPTIONS] <maven build options> Build Auron project with specified Maven profiles Options: --pre Activate pre-release profile --release Activate release profile --clean <true|false> Clean before build (default: true) --skiptests <true|false> Skip unit tests (default: true) --sparktests <true|false> Run spark tests (default: false) --docker <true|false> Build in Docker environment (default: false) --threads <N|NC> Maven build threads (e.g. 1, 4, 1C). Default: local unset, docker 8 --image <NAME> Docker image to use (e.g. centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7) --sparkver <VERSION> Specify Spark version (e.g. 3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1) --flinkver <VERSION> Specify Flink version (e.g. 1.18) --scalaver <VERSION> Specify Scala version (e.g. 2.12,2.13) --celeborn <VERSION> Specify Celeborn version (e.g. 0.5,0.6) --uniffle <VERSION> Specify Uniffle version (e.g. 0.10) --paimon <VERSION> Specify Paimon version (e.g. 1.2) --iceberg <VERSION> Specify Iceberg version (e.g. 1.10.1) --hudi <VERSION> Specify Hudi version (e.g. 0.15) -h, --help Show this help message Examples: ./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative # Spark 3.5 with all supported third-party integrations ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15 # Spark 4.0 (Hudi is not supported) ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 # Spark 4.1 (Iceberg and Hudi are not supported) ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 ``` ### Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [x] No If yes, include: `Generated-by: <tool name and version>` ASF guidance: https://www.apache.org/legal/generative-tooling.html Signed-off-by: slfan1989 <slfan1989@apache.org>
The Auron accelerator for big data engines (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.
Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.
The key capabilities of Auron include:
Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:
We encourage you to extend DataFusion capability directly and add support in Auron with simple modifications in plan-serde and extension translation.
To build Auron from source, follow the steps below:
Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
We recommend using rustup for installation.
Auron is regularly tested on JDK 8, 11, and 17, and is also tested on JDK 21.
Make sure JAVA_HOME is properly set and points to your desired version.
Check out the source code.
Build the project.
You can build Auron either locally or inside Docker using one of the supported OS images via the unified script: auron-build.sh.
Run ./auron-build.sh --help to see all available options.
After the build completes, a fat JAR with all dependencies will be generated in either the target/ directory (for local builds) or target-docker/ directory (for Docker builds), depending on the selected build mode.
This section describes how to submit and configure a Spark Job with Auron support.
Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).
Add the following configs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:
spark.auron.enable true spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager spark.memory.offHeap.enabled false # suggested executor memory configuration spark.executor.memory 4g spark.executor.memoryOverhead 4096
spark-sql -f tpcds/q01.sql
TPC-DS 1TB Benchmark Results:
For methodology and additional results, please refer to benchmark documentation.
We also encourage you to benchmark Auron and share the results with us. 🤗
Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.
| Name | Scope | ||
|---|---|---|---|
| dev@auron.apache.org | Development-related discussions | Subscribe | Unsubscribe |
Interested in contributing to Auron? Please read our Contributing Guide for detailed information on how to get started.
Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.