blob: d1ee13f55259f1efe28e5aa751b0c6b0a13555ac [file] [view]
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Ballista
Ballista support for DataFusion Python: run DataFusion Python queries on a
distributed Ballista cluster.
```bash
pip install ballista
```
User documentation is in the
[Python client user guide](https://datafusion.apache.org/ballista/user-guide/python/index.html).
This project is tracked under its own `Cargo.toml` and is intentionally not part of the default Cargo workspace so
that it doesn't cause overhead for maintainers of the main Ballista codebase. It is released separately from the Rust
crates, built against the `ballista` crates published on crates.io, once datafusion-python has published a matching
release. See the
[Python client release process](https://github.com/apache/datafusion-ballista/blob/main/python/dev/release/README.md).
## Creating a SessionContext
> [!IMPORTANT]
>
> Ballista supports the DataFusion Python API natively: `BallistaSessionContext`
> returns a standard `datafusion.SessionContext` and queries created from it
> are executed on the Ballista cluster.
>
> Some features are still being worked on, such as support for custom
> extension codecs. See the known limitations below.
>
> Progress has been tracked at [#2618](https://github.com/apache/datafusion-ballista/issues/2618)
>
Creates a new context which connects to a Ballista scheduler process.
```python
from datafusion import col, lit
from datafusion import DataFrame
# we do not need datafusion context
# it will be replaced by BallistaSessionContext
# from datafusion import SessionContext
from ballista import BallistaSessionContext
# Change from:
#
# ctx = SessionContext()
#
# to:
ctx = BallistaSessionContext("df://localhost:50050")
# all other classes and functions are from
# the datafusion module
ctx.sql("create external table t stored as parquet location './testdata/test.parquet'")
df : DataFrame = ctx.sql("select * from t limit 5")
df.show()
```
Session and Ballista settings are set with the `config` argument, either as a dictionary
or with `BallistaSessionConfig`. They are applied to the local session and propagated
to the cluster:
```python
from ballista import BallistaSessionConfig
ctx = BallistaSessionContext(
"df://localhost:50050",
config={"datafusion.execution.target_partitions": 256},
)
ctx = BallistaSessionContext(
"df://localhost:50050",
config=BallistaSessionConfig()
.with_job_name("daily report")
.with_target_partitions(256),
)
```
Python UDFs (scalar, aggregate and window) are serialised with the plan using `cloudpickle`, so
they do not need to be registered on the cluster. They require a scheduler and executors started
from Python (see [Scheduler and Executor](#scheduler-and-executor)), running the same Python minor
version as the client, with everything the UDF imports installed.
Known limitations and inefficiencies of the current approach:
- Anything requiring custom `datafusion_proto::logical_plan::LogicalExtensionCodec`.
- A Ballista connection will be created for each request.
- The scheduler address is only used when a query runs, so connection errors are raised by the
first query, not by `BallistaSessionContext`.
### Example DataFrame Usage
```python
ctx = BallistaSessionContext("df://localhost:50050")
df = ctx.read_parquet('./testdata/test.parquet').filter(col("id") > lit(4)).limit(5)
pyarrow_batches = df.collect()
```
See the [DataFusion Python documentation](https://datafusion.apache.org/python/) for more examples.
### Python Cluster Client
By default the native Rust client talks to the cluster. A client written in Python, `BallistaClient`,
can be used instead. It needs the `python-client` extra (`pip install "ballista[python-client]"`).
Set `ballista.python.client_impl` to `python` to use it, configured with the `ballista.*` options of `config`
(`ballista.python.client_impl` is not sent to the cluster and can not be changed with `SET`):
```python
from ballista import BallistaSessionContext
ctx = BallistaSessionContext(
"df://localhost:50050",
config={"ballista.python.client_impl": "python", "ballista.client.use_tls": True},
)
```
Pass a `BallistaClient` to customize how the cluster is reached, for example to add authentication
metadata:
```python
from ballista import BallistaClient, BallistaSessionContext
class AuthClient(BallistaClient):
def call_metadata(self):
return super().call_metadata() + [("authorization", "token")]
ctx = BallistaSessionContext(client=AuthClient("df://localhost:50050"))
```
## Jupyter Notebook Support
PyBallista provides first-class Jupyter notebook support with SQL magic commands and rich HTML rendering.
Install the Jupyter extras:
```bash
pip install "ballista[jupyter]"
```
DataFrames render as HTML tables, and the `ballista.jupyter` extension adds SQL magic commands
(`%ballista connect`, `%register`, `%sql`, `%%sql`), plan visualization (`explain_visual`) and
progress reporting (`collect_with_progress`):
```python
%load_ext ballista.jupyter
%ballista connect df://localhost:50050
%register parquet test_data ./testdata/test.parquet
%sql SELECT COUNT(*) FROM test_data
```
See [Jupyter Notebooks](https://datafusion.apache.org/ballista/user-guide/python/jupyter.html)
in the user guide for details.
### Example Notebooks
See the `examples/` directory for Jupyter notebooks demonstrating various features:
- `getting_started.ipynb` - Basic connection and queries
- `dataframe_api.ipynb` - DataFrame transformations
- `distributed_queries.ipynb` - Multi-stage distributed query examples
## Scheduler and Executor
Scheduler and executors can be configured and started from python code. They use
Python aware codecs, so unlike the Rust binaries and the Docker images they can run Python UDFs.
To start scheduler:
```python
from ballista import BallistaScheduler
scheduler = BallistaScheduler(bind_host="0.0.0.0", bind_port=50050)
scheduler.start()
scheduler.wait_for_termination()
```
For executor:
```python
from ballista import BallistaExecutor
executor = BallistaExecutor(
bind_host="0.0.0.0",
bind_port=50051,
scheduler_host="localhost",
scheduler_port=50050,
vcores=8,
work_dir="/tmp/ballista-executor",
)
executor.start()
executor.wait_for_termination()
```
All constructor arguments are optional. `close()` stops a started process.
`work_dir` holds the executor's shuffle and spill files; when omitted, the executor
uses a temporary directory.
For tests and notebooks, `setup_test_cluster()` starts a scheduler and an executor in
the current process and returns the scheduler `(host, port)`.
### Trust Model for Python UDFs
A scheduler or executor started from Python decodes Python UDFs in query plans with `cloudpickle.loads`, which can run arbitrary Python code. Anyone who can reach the scheduler or executor ports can therefore run arbitrary code on the cluster, as the user the processes
run as. This is the same trust model as PySpark.
Only expose such a cluster to trusted clients, for example by binding to `localhost` or a private network, or restricting access with a firewall.
Python UDFs only work on clusters started from Python. The `ballista-scheduler` and `ballista-executor` Rust binaries and the Docker images can not decode them.
## Development Process
Detailed development process explanation can be found in [datafusion python documentation](https://datafusion.apache.org/python/contributor-guide/introduction.html#).
[Improving build speed section](https://datafusion.apache.org/python/contributor-guide/introduction.html#improving-build-speed) can be relevant.
### Creating Virtual Environment
#### pip
```shell
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt
```
#### uv
```shell
uv sync --dev --no-install-package ballista
```
### Developing & Building
#### pip
```shell
maturin develop
```
Note that you can also run `maturin develop --release` to get a release build locally.
#### uv
```shell
uv run --no-project maturin develop --uv
```
Or `uv run --no-project maturin build --release --strip` to get a release build.
### Testing
#### pip
```shell
python3 -m pytest
```
#### uv
```shell
uv run --no-project pytest
```