Introduction

We welcome and encourage contributions of all kinds, such as:

1. Tickets with issue reports of feature requests
2. Documentation improvements
3. Code (PR or PR Review)

In addition to submitting new PRs, we have a healthy tradition of community members helping review each other's PRs. Doing so is a great way to help the community as well as get more familiar with Rust and the relevant codebases.

You can find a curated good-first-issue list to help you get started.

Developer's Guide

This section describes how you can get started with Ballista development.

Bootstrap Environment

Ballista contains components implemented in the following programming languages:

• Rust (Scheduler and Executor processes, Client library)

Rust Environment

We use the standard Rust development tools.

• cargo build
• cargo fmt to format the code
• cargo test to test
• etc.

Testing setup:

• rustup update stable DataFusion uses the latest stable release of rust

Formatting instructions:

• ci/scripts/rust_fmt.sh
• ci/scripts/rust_clippy.sh
• ci/scripts/rust_toml_fmt.sh

or run them all at once:

• dev/rust_lint.sh

Configuration Documentation

The configuration tables in docs/source/user-guide/configs.md are generated from the CONFIG_ENTRIES registry in ballista/core/src/config.rs. After adding or changing a configuration key, regenerate them with:

./dev/update_config_docs.sh

and commit the result. Every registered key must be covered by one of the generated regions in that page, so a new key whose prefix matches none of them will fail the check until it is given a home. CI enforces this with ci/scripts/rust_config_docs_check.sh.

Python Environment

Refer to the instructions in the Python Bindings README

Integration Tests

The integration tests can be executed by running the following command from the root of the repository.

./dev/integration-tests.sh

The script generates partitioned TPC-H Parquet via tpchgen-cli, builds the binaries on the host, builds the docker images, brings up the docker-compose stack, and runs all 22 queries. Defaults to SF=10, 16 partitions, 3 iterations per query; override with SCALE_FACTOR, PARTITIONS, and ITERATIONS env vars. See benchmarks/README.md for details.

How to format .md document

We are using prettier to format .md files.

You can either use npm i -g prettier to install it globally or use npx to run it as a standalone binary. Using npx required a working node environment. Upgrading to the latest prettier is recommended (by adding --upgrade to the npm command).

$ prettier --version
2.3.0

After you've confirmed your prettier version, you can format all the .md files:

prettier -w README.md {ballista,ballista-cli,benchmarks,dev,docs,examples,python}/**/*.md