feat: restructure existing modules, and add Column, Expression, and Functions modules (#1)
**feat**: restructure existing modules, and add Column, Expression, and Functions modules
* refactor existing modules to mirror the pyspark connector layout
* move logic plan creation into the `plan` module and out of the `dataframe` module
* create an `expression` module to handle `Expression` creation
* create `Column` struct as a representation of a `Column` object
* add traits for `alias` and `name`
* create `functions` module containing the first workings of pyspark functions
* add functions for `col`, `pi`, and `coalesce`
* **breaking change** the `select()` trait now only excepts `Vec<Column>`
* tests and examples are updated to match the change This project houses the experimental client for Spark Connect for Apache Spark written in Rust
Currently, the Spark Connect client for Rust is highly experimental and should not be used in any production setting. This is currently a “proof of concept” to identify the methods of interacting with Spark cluster from rust.
The spark-connect-rs aims to provide an entrypoint to Spark Connect, and provide similar DataFrame API interactions.
docker compose up --build -d
use spark_connect_rs; use spark_connect_rs::{SparkSession, SparkSessionBuilder}; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let spark = SparkSessionBuilder::remote("sc://127.0.0.1:15002/".to_string()) .build() .await?; let mut df = spark.sql("SELECT * FROM json.`/opt/spark/examples/src/main/resources/employees.json`"); df.filter("salary > 3000").show(Some(5), None, None).await?; Ok(()) }
git clone https://github.com/sjrusso8/spark-connect-rs.git git submodule update --init --recursive docker compose up --build -d cargo build && cargo test
The following section outlines some of the implemented functions that are working with the Spark Connect session.
better Error handling not so much use of clone()| SparkSession | API | Comment |
|---|---|---|
| range |  | |
| sql | | Does not include the new Spark Connect 3.5 feature with “position arguments” |
| read | | |
| createDataFrame | | |
| getActiveSession | | |
| many more!! |
| DataFrame | API | Comment |
|---|---|---|
| select | | |
| selectExpr | | Does not include the new Spark Connect 3.5 feature with “position arguments” |
| filter | | |
| limit | | |
| dropDuplicates | | |
| withColumnsRenamed | | |
| drop | | |
| sample | | |
| repartition | | |
| offset | | |
| schema | | The output needs to be handled better |
| explain | | The output needs to be handled better |
| show | | |
| tail | | |
| collect | | |
| withColumns | | |
| sort | | |
| groupBy | | |
| createTempView | | There is an error right now, and the functions are private till it's fixed |
| many more! | |
Spark Connect should respect the format as long as your cluster supports the specified type and has the required jars
| DataFrame | API | Comment |
|---|---|---|
| format | | |
| option | | |
| options | | |
| mode | | |
| bucketBy | | |
| sortBy | | |
| partitionBy | | |
| save | | |
| saveAsTable | | |
| insertInto | |
Spark Column type object and its implemented traits
| DataFrame | API | Comment |
|---|---|---|
| alias | | |
| asc | | |
| asc_nulls_first | | |
| asc_nulls_last | | |
| astype | | |
| between | | |
| cast | | |
| contains | | |
| desc | | |
| desc_nulls_first | | |
| desc_nulls_last | | |
| dropFields | | |
| endswith | | |
| ilike | | |
| isNotNull | | |
| isNull | | |
| isin | | |
| like | | |
| name | | |
| otherwise | | |
| rlike | | |
| startswith | | |
| substr | | |
| when | |