[SPARK-59432][CONNECT][PYTHON] Declare gil_used = false on the _pyspark module ### What changes were proposed in this pull request? Declare `#[pymodule(gil_used = false)]` on the `_pyspark` module in `crates/pyspark-rs/src/lib.rs`, opting the `pyspark-client-rust` extension into free-threaded (no-GIL) CPython. This is the first of the two steps identified in [SPARK-59432](https://issues.apache.org/jira/browse/SPARK-59432). The second — adding a free-threaded (`cp313t`/`cp314t`) wheel build leg — is a separate build/packaging change (abi3 and the free-threaded ABI are mutually exclusive) and is left as a follow-up. ### Why are the changes needed? The original justificationfor this PR is struck out below but as Hyukjin points out in his comment, pyo3 flipped the default of this variable to false so this is effectively a no-op! So this PR only serves to make it explicit in the code that no-gil is supported. ~On a free-threaded interpreter (CPython 3.13t/3.14t, PEP 703), CPython re-enables the GIL process-wide when it imports a PyO3 extension module that has not declared `gil_used = false`. A pure-Rust (tonic) Spark Connect client is well positioned to run with the GIL disabled — it pulls in no `grpcio`, which otherwise re-enables the GIL on import — so this declaration is a prerequisite for the extension to run GIL-off.~ ~The declaration is truthful: the extension is already thread-safe. details in Jira ticket.~ ### Does this PR introduce _any_ user-facing change? No. The attribute is a no-op on GIL-enabled interpreters. It has no runtime effect until a free-threaded wheel is published — the current wheels are `cp39-abi3`, which are not selectable on a free-threaded interpreter — and that wheel build leg is the tracked follow-up. ### How was this patch tested? Existing CI (`cargo fmt --all --check` and the workspace build); the attribute is inert on GIL-enabled builds and compiles under the current pyo3 0.28. End-to-end verification that the GIL is actually disabled (`sys._is_gil_enabled() is False`) will be added together with the free-threaded wheel build leg in the follow-up. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) Closes #100 from james-willis/spark-59432-gil-used-false. Authored-by: jameswillis <james@wherobots.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
A fast, native Rust client for Apache Spark Connect - and a drop-in pyspark replacement with 100% public-API parity with PySpark 4.2.0. It builds spark.connect protobuf plans, manages the gRPC channel, and decodes Arrow results in Rust, speaking the same protocol and returning the same results as the reference client.
Full documentation lives at apache.github.io/spark-connect-rust
The Rust API reference (generated from the crate) is on docs.rs/apache-spark-connect.
Python - a faster, drop-in replacement for the pyspark-client PyPI package (uninstall any existing pyspark / pyspark-client first):
pip install pyspark-client-rust
Your Spark Connect code then runs unchanged; use it exactly like PySpark.
Rust - the native crate:
[dependencies] apache-spark-connect = "4.2"
use spark_connect::{SparkSession, functions as f, lit}; fn main() -> Result<(), Box<dyn std::error::Error>> { let spark = SparkSession::builder() .remote("sc://localhost:15002") .get_or_create()?; let df = spark .range(1_000_000)? .select([(f::col("id") * lit(2)).alias("x")]) .filter((f::col("x") % lit(3)).eq(lit(0))); println!("count = {}", df.count()?); df.show(20)?; Ok(()) }
See the documentation for the full API, running a Spark Connect server, and more.
Write UDFs as plain Rust functions and call them directly — #[spark_wasm_udf] compiles them to WebAssembly and ships them to the executors:
#[spark_wasm_udf] mod udfs { pub fn add_one(x: i64) -> i64 { x + 1 } } spark.range(5)?.select([udf::add_one(col("id"))?]).show(20)?;
See the WASM UDF guide for the full setup, SQL registration, supported types, and more.
Issues are tracked in ASF JIRA under SPARK (GitHub Issues are disabled). See the contributing guide.
Apache License 2.0.
