Google Git
Sign in
apache / auron / refs/heads/wl
commit3e726dbd95d96db5c2cdd4ce16cd214500116766[log] [tgz]
authorLei Wang <wanglei860213@gmail.com>Tue Aug 26 14:44:16 2025 +0800
committerLei Wang <wanglei860213@gmail.com>Tue Aug 26 14:44:16 2025 +0800
treef8bf960c29a4fefe376a47bbceae02246d41a642
parent300592d1b0bfdabb14012f33231ab929a6d11bfb [diff]
update READEME
1 file changed
tree: f8bf960c29a4fefe376a47bbceae02246d41a642
  1. .github/
  2. .idea/
  3. benchmark-results/
  4. build/
  5. dev/
  6. hadoop-shim/
  7. native-engine/
  8. spark-extension/
  9. spark-extension-shims-spark3/
  10. spark-version-annotation-macros/
  11. thirdparty/
  12. tpcds/
  13. .gitattributes
  14. .gitignore
  15. .gitmodules
  16. .rat-excludes
  17. auron-build.sh
  18. Cargo.lock
  19. Cargo.toml
  20. LICENSE.txt
  21. pom.xml
  22. README.md
  23. release-docker.sh
  24. RELEASES.md
  25. rust-toolchain.toml
  26. rustfmt.toml
  27. scalafix.conf
  28. scalafmt.conf
README.md

AURON

TPC-DS master-ce7-builds

The Auron accelerator for big data engine (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.

Auron takes a fully optimized physical plan from distrbuted computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.

The key Key capabilities of Auron include:

  • Native execution: Implemented in Rust, eliminating JVM overhead and enabling predictable performance.
  • Vectorized computation: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing.
  • Pluggable architecture:: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines.
  • Production-hardened optimizations: Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment.

Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:

  • Various object stores.
  • Operators.
  • Simple and Aggregate functions.
  • File formats.

We encourage you to extend DataFusion capability directly and add the supports in Auron with simple modifications in plan-serde and extension translation.

Build from source

To build Auron, please follow the steps below:

  1. Install Rust

The native execution lib is written in Rust. So you're required to install Rust (nightly) first for compilation. We recommend you to use rustup.

  1. Install JDK

Auron has been well tested on jdk8/11/17.

  1. Check out the source code.

  2. Build the project.

use ./auron-build.sh for building the project. execute ./auron-build.sh --help for help.

After the build is finished, a fat Jar package that contains all the dependencies will be generated in the target directory.

Build with docker

You can use the following command to build a centos-7 compatible release:

SHIM=spark-3.3 MODE=release JAVA_VERSION=8 SCALA_VERSION=2.12 ./release-docker.sh

Run Spark Job with Auron Accelerator

This section describes how to submit and configure a Spark Job with Auron support.

  1. move auron jar package to spark client classpath (normally spark-xx.xx.xx/jars/).

  2. add the follow confs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096
  1. submit a query with spark-sql, or other tools like spark-thriftserver:
spark-sql -f tpcds/q01.sql

Performance

TPC-DS 1TB Benchmark (for details, see https://auron-project.github.io/documents/benchmarks.html):

tpcds-benchmark-echarts.png

We also encourage you to benchmark Auron and share the results with us. 🤗

Community

Subscribe Mailing Lists

Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.

NameScope
dev@auron.apache.orgDevelopment-related discussionsSubscribeUnsubscribe

Report Issues or Submit Pull Request

If you meet any questions, connect us and fix it by submitting a 🔗Pull Request.

License

Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.