[AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example. (#2408)

### Which issue does this PR close?

Closes #2407

### Rationale for this change

The Docker build example generated by `auron-build.sh --help` used Spark
4.1 together with the `centos7` image.

The `centos7` image only provides JDK 8, while Spark 4.x requires JDK 17
or later. The example also enabled Iceberg and Hudi for Spark versions
with which those integrations are not compatible.

As a result, users following the documented example encountered Java
version or integration compatibility errors before completing the build.

### What changes are included in this PR?

- Replace the single Docker build example with explicit examples for
  Spark 3.5, Spark 4.0, and Spark 4.1.
- Use the `rockylinux8` image for these examples because it supports the
  JDK selection required by Spark 4.x.
- Use Scala 2.12 for Spark 3.5 and Scala 2.13 for Spark 4.x.
- Include only compatible third-party integrations in each example:
  - Spark 3.5 includes Iceberg 1.10.1 and Hudi 0.15.
  - Spark 4.0 includes Iceberg 1.10.1 but excludes Hudi.
  - Spark 4.1 excludes both Iceberg and Hudi.
- Add a build compatibility matrix to `CONTRIBUTING.md`.
- Document the Spark, Scala, JDK, Docker image, Flink, Celeborn,
  Uniffle, Paimon, Iceberg, and Hudi combinations.
- Update the existing Docker build example in `CONTRIBUTING.md` to use
  `rockylinux8`.
- Document that the compatibility matrix and Shell help examples should
  be updated together.

### Are there any user-facing changes?

Yes.

The help output now provides separate Docker build examples for Spark
3.5, Spark 4.0, and Spark 4.1. Each example uses an appropriate Scala,
JDK, Docker image, and third-party integration combination.

Contributors can also consult the compatibility matrix in
`CONTRIBUTING.md` before selecting build arguments.

### How was this patch tested?

> ./auron-build.sh --help

```
Usage: ./auron-build.sh [OPTIONS] <maven build options>
Build Auron project with specified Maven profiles

Options:
  --pre                    Activate pre-release profile
  --release                Activate release profile
  --clean <true|false>     Clean before build (default: true)
  --skiptests <true|false> Skip unit tests (default: true)
  --sparktests <true|false> Run spark tests (default: false)
  --docker <true|false>    Build in Docker environment (default: false)
  --threads <N|NC>         Maven build threads (e.g. 1, 4, 1C). Default: local unset, docker 8
  --image <NAME>           Docker image to use (e.g. centos7,ubuntu24,rockylinux8,debian11,azurelinux3, default: centos7)
  --sparkver <VERSION>     Specify Spark version (e.g. 3.0,3.1,3.2,3.3,3.4,3.5,4.0,4.1)
  --flinkver <VERSION>     Specify Flink version (e.g. 1.18)
  --scalaver <VERSION>     Specify Scala version (e.g. 2.12,2.13)
  --celeborn <VERSION>     Specify Celeborn version (e.g. 0.5,0.6)
  --uniffle <VERSION>      Specify Uniffle version (e.g. 0.10)
  --paimon <VERSION>       Specify Paimon version (e.g. 1.2)
  --iceberg <VERSION>      Specify Iceberg version (e.g. 1.10.1)
  --hudi <VERSION>         Specify Hudi version (e.g. 0.15)
  -h, --help               Show this help message

Examples:
  ./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 -DskipBuildNative
  # Spark 3.5 with all supported third-party integrations
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 3.5 --scalaver 2.12 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1 --hudi 0.15
  # Spark 4.0 (Hudi is not supported)
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.0 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2 --iceberg 1.10.1
  # Spark 4.1 (Iceberg and Hudi are not supported)
  ./auron-build.sh --docker true --image rockylinux8 --clean true --skiptests true --release --sparkver 4.1 --scalaver 2.13 --flinkver 1.18 --celeborn 0.6 --uniffle 0.10 --paimon 1.2
```

### Was this patch authored or co-authored using generative AI tooling?
- [ ] Yes
- [x] No

If yes, include: `Generated-by: <tool name and version>`

ASF guidance: https://www.apache.org/legal/generative-tooling.html

Signed-off-by: slfan1989 <slfan1989@apache.org>
2 files changed
tree: 3cc8d70e456e30bfeccdd4fd1df4e43a3a3e024d
  1. .github/
  2. .idea/
  3. auron-core/
  4. auron-flink-extension/
  5. auron-spark-tests/
  6. auron-spark-ui/
  7. benchmark-results/
  8. build/
  9. common/
  10. dev/
  11. hadoop-shim/
  12. native-engine/
  13. spark-extension/
  14. spark-extension-shims-spark/
  15. spark-version-annotation-macros/
  16. thirdparty/
  17. .asf.yaml
  18. .gitattributes
  19. .gitignore
  20. .gitmodules
  21. .rat-excludes
  22. AGENTS.md
  23. auron-build.sh
  24. Cargo.lock
  25. Cargo.toml
  26. CONTRIBUTING.md
  27. DISCLAIMER
  28. LICENSE
  29. NOTICE
  30. pom.xml
  31. README.md
  32. rust-toolchain.toml
  33. rustfmt.toml
  34. scalafix.conf
  35. scalafmt.conf
README.md

Apache Auron (Incubating)

TPC-DS master-amd64-builds

The Auron accelerator for big data engines (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.

Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.

The key capabilities of Auron include:

  • Native execution: Implemented in Rust, eliminating JVM overhead and enabling predictable performance.
  • Vectorized computation: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing.
  • Pluggable architecture: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines.
  • Production-hardened optimizations: Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment.

Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:

  • Various object stores.
  • Operators.
  • Simple and Aggregate functions.
  • File formats.

We encourage you to extend DataFusion capability directly and add support in Auron with simple modifications in plan-serde and extension translation.

Build from source

To build Auron from source, follow the steps below:

  1. Install Rust

Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.

We recommend using rustup for installation.

  1. Install JDK

Auron is regularly tested on JDK 8, 11, and 17, and is also tested on JDK 21.

Make sure JAVA_HOME is properly set and points to your desired version.

  1. Check out the source code.

  2. Build the project.

You can build Auron either locally or inside Docker using one of the supported OS images via the unified script: auron-build.sh.

Run ./auron-build.sh --help to see all available options.

After the build completes, a fat JAR with all dependencies will be generated in either the target/ directory (for local builds) or target-docker/ directory (for Docker builds), depending on the selected build mode.

Run Spark Job with Auron Accelerator

This section describes how to submit and configure a Spark Job with Auron support.

  1. Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).

  2. Add the following configs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096
  1. submit a query with spark-sql, or other tools like spark-thriftserver:
spark-sql -f tpcds/q01.sql

Performance

TPC-DS 1TB Benchmark Results:

tpcds-benchmark-echarts.png

For methodology and additional results, please refer to benchmark documentation.

We also encourage you to benchmark Auron and share the results with us. 🤗

Community

Subscribe Mailing Lists

Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.

NameScope
dev@auron.apache.orgDevelopment-related discussionsSubscribeUnsubscribe

Contributing

Interested in contributing to Auron? Please read our Contributing Guide for detailed information on how to get started.

License

Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.