This directory contains examples for executing distributed queries with Ballista.
The standalone example is the easiest to get started with. Ballista supports a standalone mode where a scheduler and executor are started in-process.
cargo run --example standalone_sql
Source: examples/examples/standalone-sql.rs
cargo run --example standalone-substrait --features="ballista-scheduler/substrait"
Source: examples/examples/standalone-substrait.rs
For background information on the Ballista architecture, refer to the Ballista README.
The distributed examples need a running scheduler and at least one executor. Follow Ballista Quickstart to build the binaries and start them, then come back here.
The examples can be run using the cargo run --example syntax, from the examples directory.
cargo run --release --example remote-sql
Source: examples/examples/remote-sql.rs
cargo run --release --example remote-dataframe
Source: examples/examples/remote-dataframe.rs
Generic SQL clients can talk to the cluster directly, with no Ballista client library. Build a scheduler with the (non-default) Flight SQL frontend and start it with --flight-sql:
cargo build --release -p ballista-scheduler --features flight-sql RUST_LOG=info ./target/release/ballista-scheduler --flight-sql RUST_LOG=info ./target/release/ballista-executor -c 4 -p 50051
Then query it from Python over ADBC:
pip install adbc-driver-flightsql pyarrow python examples/python/adbc_flight_sql.py
See python/adbc_flight_sql.py and the Arrow Flight SQL user guide. Note that a scheduler started this way performs no authentication.
Every example in examples/examples/, including those not walked through above. Run each from this directory.
| Example | Needs | Command |
|---|---|---|
standalone-sql.rs | nothing, runs in-process | cargo run --example standalone_sql |
standalone-broadcast-join.rs | nothing, runs in-process | cargo run --example standalone_broadcast_join |
standalone-substrait.rs | nothing, runs in-process | cargo run --example standalone-substrait --features="ballista-scheduler/substrait" |
remote-sql.rs | a running cluster | cargo run --example remote-sql |
remote-dataframe.rs | a running cluster | cargo run --example remote-dataframe |
custom-scheduler.rs | nothing | cargo run --example custom-scheduler |
custom-executor.rs | custom-scheduler | cargo run --example custom-executor |
custom-client.rs | custom-scheduler, custom-executor, and MinIO (see the file) | cargo run --example custom-client |
mtls-cluster.rs | nothing; run certs, then scheduler, executor, client in separate terminals | cargo run --example mtls-cluster --features=tls -- <step> |
The --features flags are needed: an example's required-features in Cargo.toml stops it from building without those features, but does not turn them on. The in-process examples need ballista/standalone, which is on by default, so it only has to be named when building with --no-default-features.