Google Git
Sign in
apache / datafusion-ballista-python / 0c38e8768fe3f979baa008f2513f5881c1906f92
commit0c38e8768fe3f979baa008f2513f5881c1906f92[log] [tgz]
authorAndy Grove <andygrove73@gmail.com>Sun May 14 10:06:03 2023 -0600
committerGitHub <noreply@github.com>Sun May 14 10:06:03 2023 -0600
tree039963872f5ba78ece7761e2da8e93eb5e76acf0
parente61bbcc97b1591d2705bbdc0698e26c456717735 [diff]
Update Python bindings to use latest Ballista, DataFusion 24.0.0 (#49)

* use latest ballista, DF 24

* fmt
5 files changed
tree: 039963872f5ba78ece7761e2da8e93eb5e76acf0
  1. .cargo/
  2. .github/
  3. ballista/
  4. ci/
  5. dev/
  6. docs/
  7. examples/
  8. src/
  9. .asf.yaml
  10. .dockerignore
  11. .flake8
  12. .gitignore
  13. Cargo.toml
  14. CHANGELOG.md
  15. CODE_OF_CONDUCT.md
  16. CONTRIBUTING.md
  17. LICENSE.txt
  18. pyproject.toml
  19. README.md
  20. requirements-310.txt
  21. requirements-37.txt
  22. requirements.in
  23. requirements.txt
README.md

Ballista Python Bindings (PyBallista)

This is a Python library that binds to Apache Arrow distributed query engine Ballista.

Status

What works?

  • Start Ballista schedulers and executors from Python
  • Execute distributed SQL queries (with DataFusion backend)
  • Use DataFrame API to read files and execute distributed queries (with DataFusion backend)
  • Support for CSV, Parquet, and Avro formats

What does not work?

  • Python UDFs
  • JSON

Roadmap

  • Support reading JSON
  • Support distributed Python UDFs and UDAFs
  • Support distributed query execution against Python DataFrame libraries such as Polars, Pandas, and cuDF, that are already supported by DataFusion's Python bindings (this will require new features in Ballista)

Examples

How to install (from pip)

pip install ballista
# or
python -m pip install ballista

How to develop

This assumes that you have rust and cargo installed. We use the workflow recommended by pyo3 and maturin.

Bootstrap:

# fetch this repo
git clone git@github.com:apache/arrow-ballista-python.git
# change to python directory
cd arrow-ballista-python
# prepare development environment (used to build wheel / install in development)
python3 -m venv venv
# activate the venv
source venv/bin/activate
# update pip itself if necessary
python -m pip install -U pip
# if python -V gives python 3.7
python -m pip install -r requirements-37.txt
# if python -V gives python 3.8/3.9/3.10
python -m pip install -r requirements-310.txt

Whenever rust code changes (your changes or via git pull):

# make sure you activate the venv using "source venv/bin/activate" first
maturin develop
python -m pytest

How to update dependencies

To change test dependencies, change the requirements.in and run

# install pip-tools (this can be done only once), also consider running in venv
python -m pip install pip-tools

# change requirements.in and then run
python -m piptools compile --generate-hashes -o requirements-37.txt
# or run this is you are on python 3.8/3.9/3.10
python -m piptools compile --generate-hashes -o requirements.txt

To update dependencies, run with -U

python -m piptools compile -U --generate-hashes -o requirements-310.txt

More details here