Ballista is a distributed query execution engine that enhances Apache DataFusion by enabling the parallelized execution of workloads across multiple nodes in a distributed environment.
It is also a lighter, Rust-native alternative to both batch engines such as Apache Spark and interactive MPP query engines such as Trino and Presto, with two execution modes that share one cluster:
On a 1 TB benchmark derived from TPC-H, Ballista is 2.2x faster than Spark in staged mode and 3.4x faster in pipelined mode. A session can switch modes between queries, so one engine can serve both ETL and interactive queries.
An existing DataFusion application can be distributed with a few lines of code changed:
+use ballista::prelude::*; use datafusion::prelude::*; #[tokio::main] async fn main() -> datafusion::error::Result<()> { - let ctx = SessionContext::new(); + // create SessionContext with ballista support + // standalone context will start all required + // ballista infrastructure in the background as well + let ctx = SessionContext::standalone().await?;
// register the table ctx.register_csv("example", "tests/data/example.csv", CsvReadOptions::new()) .await?; // create a plan to run a SQL query let df = ctx .sql("SELECT a, MIN(b) FROM example WHERE a <= b GROUP BY a LIMIT 100") .await?; // execute and print results df.show().await?; Ok(()) }
For documentation or more examples, please refer to the Ballista User Guide.
Ballista is measured against Apache Spark 4.1.3 on the same cluster shape: Kubernetes on AWS, with 32 executors of 8 vCPUs running the 22 TPC-H-derived queries over 1 TB of Parquet in S3. Each Ballista executor had 64 GiB of memory, and each Spark executor 74 GiB.
Derived from TPC-H and not an official TPC-H result. Times are the mean of two runs, including the cold first run. The Spark numbers come from an earlier run with separate scripts and its own phrasing of the queries. See the benchmarking guide for versions, settings, the full results and how to reproduce them.
DataFrame.cache() keeps intermediate results on the executors for reuse within a session (opt-in)Ballista serves several distinct audiences:
These audiences are documented in more detail, along with the guarantees each relies on, in the User Personas guide.
A Ballista cluster consists of one or more scheduler processes and one or more executor processes. These processes can be run as native binaries and are also available as Docker Images, which can be easily deployed with Docker Compose or Kubernetes.
Clients submit query plans to the scheduler over gRPC, or SQL over Arrow Flight SQL. The scheduler breaks each query into stages and sends tasks to the executors, which report progress back. Executors exchange data with each other over Arrow Flight, reading shuffle files in staged mode or streaming record batches directly in pipelined mode, and return results to the client the same way.
See the architecture guide for more details.
A lakehouse aims to merge the data warehouse and the ETL platform, but it is often still served by two kinds of engine: a batch engine for long ETL jobs, built to survive task failures, and an interactive engine for short analytical queries, built for low latency. Ballista's two execution modes let one engine serve both, from the same scheduler and executors. A session selects its mode with ballista.execution.mode and can change it between queries, so the same SQL and DataFrame code serves either workload.
The modes trade fault tolerance for latency. In staged mode, a failed task is re-run from the shuffle files its input stages wrote, which suits long-running jobs and busy shared clusters. In pipelined mode, results stream to the client as they are produced, but there are no shuffle files to recover from, so a failed task fails the whole job, and the cluster needs room to run every task of the job at once. See the Pipelined Execution Guide for details.
Sharing one pool of executors can also take less capacity than running two clusters. Batch and interactive load are often uncorrelated, so the capacity one leaves idle is available to the other, and the pool can be sized for the peak of the combined load rather than for the sum of two separate peaks.
Sharing a cluster this way is still early. Pipelined execution is experimental, and the scheduler does not prioritize one job over another, so on a busy cluster an interactive query can wait for vcores that batch jobs hold, as the limitations of pipelined execution describe. Prioritizing interactive queries over batch jobs in the scheduler is what a shared cluster still needs.
The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.
Ballista provides a browser-based Web TUI for monitoring a running cluster. It exposes the TUI views for jobs, executors, metrics, and scheduler information directly in a web browser.
When the scheduler HTTP endpoint is available, opening the scheduler address in a browser, for example http://localhost:50050, redirects to a hosted Web TUI.
For more information, including how to run the Web TUI locally, see the Ballista CLI documentation.
Ballista is part of the Apache DataFusion project at the Apache Software Foundation. It ships about every two months alongside DataFusion. Adopters include Spice AI and Coralogix.
Changes are tested on real clusters before they merge:
Ballista supports a wide range of SQL, including CTEs, joins and subqueries, and can execute complex queries at scale, but there is still a gap between DataFusion and Ballista, which may bring incompatibilities. The community is actively working to close it. Refer to the DataFusion SQL Reference for more information on supported SQL.
Ballista uses Cargo features to enable optional functionality.
| Feature | Default | Description |
|---|---|---|
standalone | Yes | Enables standalone mode with in-process scheduler and executor |
| Feature | Default | Description |
|---|---|---|
arrow-ipc-optimizations | Yes | Enables Arrow IPC optimizations for better shuffle performance |
build-binary | No | Required for building binary executables (AWS S3 support, CLI parsing) |
force_hash_collisions | No | Testing-only: forces all values to hash to same value |
utoipa | No | Derives OpenAPI schemas for scheduler REST types; enabled by ballista-scheduler's rest-api |
| Feature | Default | Description |
|---|---|---|
build-binary | Yes | Builds the scheduler binary with CLI and logging |
substrait | No | Enables Substrait plan support |
flight-sql | No | Enables the Arrow Flight SQL frontend for ADBC and JDBC clients |
prometheus-metrics | No | Enables Prometheus metrics collection |
graphviz-support | No | Enables execution graph visualization |
keda-scaler | No | Kubernetes Event Driven Autoscaling integration |
rest-api | Yes | Enables REST API endpoints, the OpenAPI spec, and the history server |
| Feature | Default | Description |
|---|---|---|
arrow-ipc-optimizations | Yes | Enables Arrow IPC optimizations |
build-binary | Yes | Builds the executor binary with CLI and logging, using mimalloc as the allocator |
jemalloc | No | Uses jemalloc instead of mimalloc as the allocator (not supported on MSVC targets) |
| Feature | Default | Description |
|---|---|---|
cli | Yes | Builds the ballista-cli SQL shell binary |
tui | Yes | Enables a REST client with Terminal User Interface |
web | No | Builds the WebAssembly Web TUI (see the Ballista CLI documentation) |
# Build with standalone support (default) cargo build -p ballista # Build with Substrait support cargo build -p ballista-scheduler --features substrait
Please see the Contribution Guide for information about contributing to Ballista.