Spark Connect Protobuf Definitions

This directory contains the .proto files that define the Spark Connect protocol.

After modifying any .proto file here, regenerate the Python stubs under python/pyspark/sql/connect/proto/ using one of the two methods below.

Method 1: Docker image (recommended)

This method does not require any local tool installation and produces a reproducible environment.

Build the image

From the root of the Spark repository:

docker build -t connect-cg \
  --build-context root=. \
  dev/spark-test-image/connect-gen-protos/

The root build context points at the Spark repository root so the Dockerfile can COPY pyproject.toml for dependency installation.

Run the image

From the root of the Spark repository:

docker run --cpus 1 -it --rm -v "$(pwd)":/spark connect-cg

The container mounts the repository at /spark, runs dev/connect-gen-protos.sh inside the container, and writes the generated files to python/pyspark/sql/connect/proto/ in your local checkout.

Method 2: Local Python environment

Prerequisites

Install the required tools:

buf — protobuf code generator
Python 3.12+

Install the required Python packages. Check pyproject.toml for the latest pinned versions of mypy, mypy-protobuf, and ruff, then run:

pip install 'mypy==<version>' 'mypy-protobuf==<version>' 'ruff==<version>'

For example, based on the current pyproject.toml:

pip install 'mypy==1.19.1' 'mypy-protobuf==3.3.0' 'ruff==0.14.8'

Generate

From the root of the Spark repository:

./dev/connect-gen-protos.sh

The generated Python files will be written to python/pyspark/sql/connect/proto/.

You can also generate to a custom output directory by passing a path:

./dev/connect-gen-protos.sh /tmp/my-proto-output