| commit | f2c9478d33e92fbc2d376367709b573a2412514a | [log] [tgz] |
|---|---|---|
| author | hiboyang <14280154+hiboyang@users.noreply.github.com> | Fri Aug 18 12:20:20 2023 +0800 |
| committer | Ruifeng Zheng <ruifengz@apache.org> | Fri Aug 18 12:20:20 2023 +0800 |
| tree | b1e35d0d053da81d80bb9431b5f51570b3fe7d81 | |
| parent | b35574eb9cd5a86ef9a5e602f312cbdfaf1c55f8 [diff] |
[SPARK-44681] Fix issues when writing Go application code using Spark Connect Go client library ### What changes were proposed in this pull request? When trying to write Go application code using Spark Connect Go client and put the Go application code in users' own repo (e.g. https://github.com/user-foo/my-go-application), the Go application code cannot resolve Spark Connect Go client from `https://github.com/apache/spark-connect-go` correctly, due to two issues: 1. The name `github.com/apache/spark-connect-go/v_3_4` cannot resolve correctly to find the module, it complains not finding `go.mod` file under `github.com/apache/spark-connect-go/v_3_4`. After change the name to `github.com/apache/spark-connect-go/v34`, it is good. 2. The Go application code needs generated Go protobuf code as well. Thus need to commit the generated Go protobuf code into `github.com/apache/spark-connect-go` repo. ### Why are the changes needed? See above. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tested by run Go application code to use Spark Connect Go client. Closes #14 from hiboyang/bo-dev-05. Authored-by: hiboyang <14280154+hiboyang@users.noreply.github.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
This project houses the experimental client for Spark Connect for Apache Spark written in Golang.
Currently, the Spark Connect client for Golang is highly experimental and should not be used in any production setting. In addition, the PMC of the Apache Spark project reserves the right to withdraw and abandon the development of this project if it is not sustainable.
git clone https://github.com/apache/spark-connect-go.git git submodule update --init --recursive make gen && make test
Ensure you have installed
buf CLI; more info
A very simple example in Go looks like following:
func main() {
remote := "localhost:15002"
spark, _ := sql.SparkSession.Builder.Remote(remote).Build()
defer spark.Stop()
df, _ := spark.Sql("select 'apple' as word, 123 as count union all select 'orange' as word, 456 as count")
df.Show(100, false)
}
Following diagram shows main code in current prototype:
+-------------------+
| |
| dataFrameImpl |
| |
+-------------------+
|
|
+
+-------------------+
| |
| sparkSessionImpl |
| |
+-------------------+
|
|
+
+---------------------------+ +----------------+
| | | |
| SparkConnectServiceClient |--------------+| Spark Driver |
| | | |
+---------------------------+ +----------------+
SparkConnectServiceClient is GRPC client which talks to Spark Driver. sparkSessionImpl generates dataFrameImpl instances. dataFrameImpl uses the GRPC client in sparkSessionImpl to communicate with Spark Driver.
We will mimic the logic in Spark Connect Scala implementation, and adopt Go common practices, e.g. returning error object for error handling.
Install Golang: https://go.dev/doc/install.
Download Spark distribution (3.4.0+), unzip the folder.
Start Spark Connect server by running command:
sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.0
go run cmd/spark-connect-example-spark-session/main.go
Please review the Contribution to Spark guide for information on how to get started contributing to the project.