Google Git
Sign in
apache / auron / refs/tags/v4.0.0
commit6a68b113dfc8d0f0990c13875339a60f47392ed2[log] [tgz]
authorzhangli20 <zhangli20@kuaishou.com>Wed Sep 18 15:08:25 2024 +0800
committerzhangli20 <zhangli20@kuaishou.com>Wed Oct 09 12:15:48 2024 +0800
treeb6c3f7585fac85e984486f467595efac3b434d46
parented846af7569bc7b3d53778fa3356bc026eb49a9f [diff]
update to datafusion-v42/arrow-v53/arrow-java-v16
56 files changed
tree: b6c3f7585fac85e984486f467595efac3b434d46
  1. .github/
  2. benchmark-results/
  3. dev/
  4. native-engine/
  5. spark-extension/
  6. spark-extension-shims-spark3/
  7. tpcds/
  8. .gitattributes
  9. .gitignore
  10. .gitmodules
  11. build-native.sh
  12. Cargo.lock
  13. Cargo.toml
  14. LICENSE.txt
  15. pom.xml
  16. README.md
  17. release-docker.sh
  18. RELEASES.md
  19. rust-toolchain.toml
  20. rustfmt.toml
  21. scalafix.conf
  22. scalafmt.conf
README.md

BLAZE

TPC-DS master-ce7-builds

The Blaze accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the Spark distributed computing framework.

Blaze takes a fully optimized physical plan from Spark, mapping it into DataFusion's execution plan, and performs native plan computation in Spark executors.

Blaze is composed of the following high-level components:

  • Spark Extension: hooks the whole accelerator into Spark execution lifetime.
  • Spark Shims: specialized codes for different versions of spark.
  • Native Engine: implements the native engine in rust, including:
    • ExecutionPlan protobuf specification
    • JNI gateway
    • Customized operators, expressions, functions

Based on the inherent well-defined extensibility of DataFusion, Blaze can be easily extended to support:

  • Various object stores.
  • Operators.
  • Simple and Aggregate functions.
  • File formats.

We encourage you to extend DataFusion capability directly and add the supports in Blaze with simple modifications in plan-serde and extension translation.

Build from source

To build Blaze, please follow the steps below:

  1. Install Rust

The native execution lib is written in Rust. So you're required to install Rust (nightly) first for compilation. We recommend you to use rustup.

  1. Install Protobuf

Ensure protoc is available in PATH environment. protobuf can be installed via linux system package manager (or Homebrew on mac), or manually download and build from https://github.com/protocolbuffers/protobuf/releases .

  1. Install JDK+Maven

Blaze has been well tested on jdk8 and maven3.5, should work fine with higher versions.

  1. Check out the source code.
git clone git@github.com:kwai/blaze.git
cd blaze
  1. Build the project.

Specify shims package of which spark version that you would like to run on.

Currently we have supported these shims:

  • spark-3.0 - for spark3.0.x
  • spark-3.1 - for spark3.1.x
  • spark-3.2 - for spark3.2.x
  • spark-3.3 - for spark3.3.x
  • spark-3.4 - for spark3.4.x
  • spark-3.5 - for spark3.5.x.

You could either build Blaze in pre mode for debugging or in release mode to unlock the full potential of Blaze.

SHIM=spark-3.3 # or spark-3.0/spark-3.1/spark-3.2/spark-3.3/spark-3.4/spark-3.5
MODE=release # or pre
mvn package -P"${SHIM}" -P"${MODE}"

After the build is finished, a fat Jar package that contains all the dependencies will be generated in the target directory.

Build with docker

You can use the following command to build a centos-7 compatible release:

SHIM=spark-3.3 MODE=release ./release-docker.sh

Run Spark Job with Blaze Accelerator

This section describes how to submit and configure a Spark Job with Blaze support.

  1. move blaze jar package to spark client classpath (normally spark-xx.xx.xx/jars/).

  2. add the follow confs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:

spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096
  1. submit a query with spark-sql, or other tools like spark-thriftserver:
spark-sql -f tpcds/q01.sql

Integrate with Apache Celeborn

Blaze has supported Celeborn integration now, use the following configurations to enable shuffling with Celeborn:


# change celeborn endpoint and storage directory to the correct location
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleManager
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.celeborn.master.endpoints localhost:9097
spark.celeborn.client.spark.shuffle.writer hash
spark.celeborn.client.push.replicate.enabled false
spark.celeborn.storage.availableTypes HDFS
spark.celeborn.storage.hdfs.dir hdfs:///home/celeborn
spark.sql.adaptive.localShuffleReader.enabled false

Performance

Check Benchmark Results with the latest date for the performance comparison with vanilla Spark 3.3.3. The benchmark result shows that Blaze save about 50% time on TPC-DS/TPC-H 1TB datasets. Stay tuned and join us for more upcoming thrilling numbers.

TPC-DS Query time: (How can I run TPC-DS benchmark?) 20240701-query-time-tpcds

TPC-H Query time: 20240701-query-time-tpch

We also encourage you to benchmark Blaze and share the results with us. 🤗

Community

We're using Discussions to connect with other members of our community. We hope that you:

  • Ask questions you're wondering about.
  • Share ideas.
  • Engage with other community members.
  • Welcome others who are open-minded. Remember that this is a community we build together 💪 .

License

Blaze is licensed under the Apache 2.0 License. A copy of the license can be found here.