---

title: Introduction

Overview

The Apache Auron (Incubating) accelerator for Apache Spark leverages native vectorized execution to accelerate query processing. It combines the power of the Apache Arrow-DataFusion library and the scale of the Spark distributed computing framework.

Introduction

Problem Statement

Apache Spark is a popular distributed computing framework for handling large-scale data processing tasks. However, as the data size increases, traditional row-based processing can lead to significant CPU latencies and become a performance bottleneck. To overcome this challenge, vectorized execution technology has been introduced as an optimization method for Spark.

Apache Auron (Incubating)’s Solution

Vectorized execution technology operates by processing data in batches rather than rows, reducing function calls and improving computation performance with SIMD instructions. Apache Auron (Incubating) leverages this technology by integrating the Apache Arrow-DataFusion library with the Spark framework.

Apache Auron (Incubating) checks and translates supported operators in the Spark’s physical plan and generates an equivalent native execution plan, then it passes the generated execution plan to the underlying native engine through JNI calls. The native engine executes the plan with DataFusion framework, which benefits from vectorized execution and has better performance comparing to Spark’s JVM based execution.

Target User

Apache Auron (Incubating)’s target users are those who want to accelerate Spark SQL/DataFrame queries. Users can install Apache Auron (Incubating) as a Spark client extension. After installing, most SQL queries should run faster without modifying, and save cluster resources.

Architecture

The architecture design of Apache Auron (Incubating) is as follows. Apache Auron (Incubating) takes a fully optimized physical plan from Spark, mapping it into equivalent execution plan implemented in native engine, and executes in Spark distributed environment.

Apache Auron (Incubating) is composed of the following high-level components:

• Spark Extension: hooks the whole accelerator into Spark execution lifetime.
• Spark Shims: specialized codes for different versions of Spark.
• Native Engine: implements the native engine in rust, including:
  • ExecutionPlan protobuf specification.
  • JNI gateway.
  • Customized operators, expressions, functions with DataFusion framework
  • Other common mods like memory management, fallback framework, HDFS-integrating, etc.

The architecture diagram of the native engine is as follows:

Currently Supported Native Operators/Expressions

All supported operators in Apache Auron (Incubating) are listed below. Apache Auron (Incubating) does support fallbacking an operator to Spark execution which has not been implemented, so SQLs containing unsupported operators can still be executed successfully. However, fallbacks takes extra costs, too many fallbacks will slow down the execution.

Most Spark builtin expressions are supported in Apache Auron (Incubating) (by translating to DataFusion-physical-exprs). Apache Auron (Incubating) also supports expression-level fallbacking, which can fallback a single unsupported expression to Spark execution. so SQLs containing some unsupported expressions like UDF/UDTFs can still be optimized.

Join the Community

Source Code

Please see Apache Auron (Incubating) source code for more information.

Cooperators

Apache Auron (Incubating) currently has some users and contributors:

Subscribe Mailing Lists

Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.

Name	Scope
dev@auron.apache.org	Development-related discussions	Subscribe	Unsubscribe

Report Issues or Submit Pull Request

If you meet any questions, connect us and fix it by submitting a 🔗Pull Request.