tree: b2c7610f96a65808a07178ad98f6242d7866eb22 [search]
  1. examples/
  2. python/
  3. src/
  4. testdata/
  5. tests/
  6. Cargo.toml
  7. README.md
examples/README.md

Ballista Examples

This directory contains examples for executing distributed queries with Ballista.

Standalone Examples

The standalone example is the easiest to get started with. Ballista supports a standalone mode where a scheduler and executor are started in-process.

cargo run --example standalone_sql

Source: examples/examples/standalone-sql.rs

cargo run --example standalone-substrait --features="ballista-scheduler/substrait"

Source: examples/examples/standalone-substrait.rs

Distributed Examples

For background information on the Ballista architecture, refer to the Ballista README.

Start a cluster

The distributed examples need a running scheduler and at least one executor. Follow Ballista Quickstart to build the binaries and start them, then come back here.

Running the examples

The examples can be run using the cargo run --example syntax, from the examples directory.

Distributed SQL Example

cargo run --release --example remote-sql

Source: examples/examples/remote-sql.rs

Distributed DataFrame Example

cargo run --release --example remote-dataframe

Source: examples/examples/remote-dataframe.rs

Arrow Flight SQL / Python ADBC

Generic SQL clients can talk to the cluster directly, with no Ballista client library. Build a scheduler with the (non-default) Flight SQL frontend and start it with --flight-sql:

cargo build --release -p ballista-scheduler --features flight-sql
RUST_LOG=info ./target/release/ballista-scheduler --flight-sql
RUST_LOG=info ./target/release/ballista-executor -c 4 -p 50051

Then query it from Python over ADBC:

pip install adbc-driver-flightsql pyarrow
python examples/python/adbc_flight_sql.py

See python/adbc_flight_sql.py and the Arrow Flight SQL user guide. Note that a scheduler started this way performs no authentication.

All examples

Every example in examples/examples/, including those not walked through above. Run each from this directory.

ExampleNeedsCommand
standalone-sql.rsnothing, runs in-processcargo run --example standalone_sql
standalone-broadcast-join.rsnothing, runs in-processcargo run --example standalone_broadcast_join
standalone-substrait.rsnothing, runs in-processcargo run --example standalone-substrait --features="ballista-scheduler/substrait"
remote-sql.rsa running clustercargo run --example remote-sql
remote-dataframe.rsa running clustercargo run --example remote-dataframe
custom-scheduler.rsnothingcargo run --example custom-scheduler
custom-executor.rscustom-schedulercargo run --example custom-executor
custom-client.rscustom-scheduler, custom-executor, and MinIO (see the file)cargo run --example custom-client
mtls-cluster.rsnothing; run certs, then scheduler, executor, client in separate terminalscargo run --example mtls-cluster --features=tls -- <step>

The --features flags are needed: an example's required-features in Cargo.toml stops it from building without those features, but does not turn them on. The in-process examples need ballista/standalone, which is on by default, so it only has to be named when building with --no-default-features.