The Auron accelerator for big data engine (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.
Auron takes a fully optimized physical plan from distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.
The key capabilities of Auron include:
Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:
We encourage you to extend DataFusion capability directly and add the supports in Auron with simple modifications in plan-serde and extension translation.
To build Auron from source, follow the steps below:
Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.
We recommend using rustup for installation.
Auron has been well tested with JDK 8, 11, and 17.
Make sure JAVA_HOME
is properly set and points to your desired version.
Check out the source code.
Build the project.
You can build Auron either locally or inside Docker with CentOS7 using a unified script: auron-build.sh
.
Run ./auron-build.sh --help
to see all available options.
After the build completes, a fat JAR with all dependencies will be generated in either the target/
directory (for local builds) or target-docker/
directory (for Docker builds), depending on the selected build mode.
This section describes how to submit and configure a Spark Job with Auron support.
Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).
Add the following configs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf
:
spark.auron.enable true spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager spark.memory.offHeap.enabled false # suggested executor memory configuration spark.executor.memory 4g spark.executor.memoryOverhead 4096
spark-sql -f tpcds/q01.sql
TPC-DS 1TB Benchmark Results:
For methodology and additional results, please refer to benchmark documentation.
We also encourage you to benchmark Auron and share the results with us. 🤗
Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.
Name | Scope | ||
---|---|---|---|
dev@auron.apache.org | Development-related discussions | Subscribe | Unsubscribe |
If you meet any questions, connect us and fix it by submitting a 🔗Pull Request.
Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.