Google Git
Sign in
apache / spark-connect-go / 229c570bff7b503e267e4072acb55bc91886a30b
commit229c570bff7b503e267e4072acb55bc91886a30b[log] [tgz]
authorMartin Grund <martin.grund@databricks.com>Thu Jan 02 10:47:33 2025 +0100
committerMartin Grund <martin.grund@databricks.com>Thu Jan 02 10:47:33 2025 +0100
tree6bf7ace573157928835ccbedad4e30d07724a038
parent60b1610fc6c6597efedcb3d09293d9f71bd4a21b [diff]
#58 Adding `DropNa` and `df.na.*` methods

### What changes were proposed in this pull request?
This patch adds support for `df.DropNa()` and the corresponding `df.Na().drop()`-like functions.

* `df.Na().Drop()`
* `df.Na().Fill()`
* `df.Na().Replace()`

### Why are the changes needed?
Compatibility

### Does this PR introduce _any_ user-facing change?
New supported APIs

### How was this patch tested?
Added IT

Closes #99 from grundprinzip/dropna.

Authored-by: Martin Grund <martin.grund@databricks.com>
Signed-off-by: Martin Grund <martin.grund@databricks.com>
3 files changed
tree: 6bf7ace573157928835ccbedad4e30d07724a038
  1. .github/
  2. cmd/
  3. dev/
  4. internal/
  5. java/
  6. spark/
  7. sparksrc
  8. .asf.yaml
  9. .gitignore
  10. .gitmodules
  11. .golangci.yml
  12. buf.gen.yaml
  13. buf.work.yaml
  14. CONTRIBUTING.md
  15. go.mod
  16. go.sum
  17. LICENSE
  18. Makefile
  19. merge_connect_go_pr.py
  20. quick-start.md
  21. README.md
README.md

Apache Spark Connect Client for Golang

This project houses the experimental client for Spark Connect for Apache Spark written in Golang.

Current State of the Project

Currently, the Spark Connect client for Golang is highly experimental and should not be used in any production setting. In addition, the PMC of the Apache Spark project reserves the right to withdraw and abandon the development of this project if it is not sustainable.

Getting started

This section explains how to run Spark Connect Go locally.

Step 1: Install Golang: https://go.dev/doc/install.

Step 2: Ensure you have installed buf CLI installed, more info here

Step 3: Run the following commands to setup the Spark Connect client.

git clone https://github.com/apache/spark-connect-go.git
git submodule update --init --recursive

make gen && make test

Step 4: Setup the Spark Driver on localhost.

  1. Download Spark distribution (3.5.0+), unzip the package.

  2. Start the Spark Connect server with the following command (make sure to use a package version that matches your Spark distribution):

sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.2

Step 5: Run the example Go application.

go run cmd/spark-connect-example-spark-session/main.go

Runnning Spark Connect Go Application in a Spark Cluster

To run the Spark Connect Go application in a Spark Cluster, you need to build the Go application and submit it to the Spark Cluster. You can find a more detailed example runner and wrapper script in the java directory.

See the guide here: Sample Spark-Submit Wrapper.

How to write Spark Connect Go Application in your own project

See Quick Start Guide

High Level Design

The overall goal of the design is to find a good balance of principle of the least surprise for develoeprs that are familiar with the APIs of Apache Spark and idiomatic Go usage. The high-level structure of the packages follows roughly the PySpark giudance but with Go idioms.

Contributing

Please review the Contribution to Spark guide for information on how to get started contributing to the project.