| commit | e1e8a327ce13d53ecaa4b7c2a5e8d69ffb822940 | [log] [tgz] |
|---|---|---|
| author | Dongjoon Hyun <dongjoon@apache.org> | Sun May 04 08:15:31 2025 -0700 |
| committer | Dongjoon Hyun <dongjoon@apache.org> | Sun May 04 08:15:31 2025 -0700 |
| tree | 232288811b646b91dfafe379946b0bdc37af94e8 | |
| parent | 568432552a8f5ff71dfc9c66911992023d3ed0ca [diff] |
[SPARK-51997] Mark `nodoc` to hide generated and internal classes from docs ### What changes were proposed in this pull request? This PR aims mark `nodoc` to hide generated and internal classes from docs. ### Why are the changes needed? Since #107, the documentation is visible here. - https://swiftpackageindex.com/apache/spark-connect-swift/main/documentation/sparkconnect We need to simplify the documentation to focus `Spark Connect for Swift` itself by hiding irrelevant information . For example, `Arrow`-related information will be removed eventually when we start to use the official `Apache Arrow` release. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #113 from dongjoon-hyun/SPARK-51997. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This is an experimental Swift library to show how to connect to a remote Apache Spark Connect Server and run SQL statements to manipulate remote data.
So far, this library project is tracking the upstream changes like the Apache Spark 4.0.0 RC4 release and Apache Arrow project's Swift-support.
Create a Swift project.
mkdir SparkConnectSwiftApp cd SparkConnectSwiftApp swift package init --name SparkConnectSwiftApp --type executable
Add SparkConnect package to the dependency like the following
$ cat Package.swift import PackageDescription let package = Package( name: "SparkConnectSwiftApp", platforms: [ .macOS(.v15) ], dependencies: [ .package(url: "https://github.com/apache/spark-connect-swift.git", branch: "main") ], targets: [ .executableTarget( name: "SparkConnectSwiftApp", dependencies: [.product(name: "SparkConnect", package: "spark-connect-swift")] ) ] )
Use SparkSession of SparkConnect module in Swift.
$ cat Sources/main.swift import SparkConnect let spark = try await SparkSession.builder.getOrCreate() print("Connected to Apache Spark \(await spark.version) Server") let statements = [ "DROP TABLE IF EXISTS t", "CREATE TABLE IF NOT EXISTS t(a INT) USING ORC", "INSERT INTO t VALUES (1), (2), (3)", ] for s in statements { print("EXECUTE: \(s)") _ = try await spark.sql(s).count() } print("SELECT * FROM t") try await spark.sql("SELECT * FROM t").cache().show() try await spark.range(10).filter("id % 2 == 0").write.mode("overwrite").orc("/tmp/orc") try await spark.read.orc("/tmp/orc").show() await spark.stop()
Run your Swift application.
$ swift run ... Connected to Apache Spark 4.0.0 Server EXECUTE: DROP TABLE IF EXISTS t EXECUTE: CREATE TABLE IF NOT EXISTS t(a INT) EXECUTE: INSERT INTO t VALUES (1), (2), (3) SELECT * FROM t +---+ | a | +---+ | 2 | | 1 | | 3 | +---+ +----+ | id | +----+ | 2 | | 6 | | 0 | | 8 | | 4 | +----+
You can find this example in the following repository.
Spark SQL REPL via Spark Connect for SwiftThis project also provides Spark SQL REPL. You can run it directly from this repository.
$ swift run ... Build of product 'SparkSQLRepl' complete! (2.33s) Connected to Apache Spark 4.0.0 Server spark-sql (default)> SHOW DATABASES; +---------+ |namespace| +---------+ | default| +---------+ Time taken: 30 ms spark-sql (default)> CREATE DATABASE db1; ++ || ++ ++ Time taken: 31 ms spark-sql (default)> USE db1; ++ || ++ ++ Time taken: 27 ms spark-sql (db1)> CREATE TABLE t1 AS SELECT * FROM RANGE(10); ++ || ++ ++ Time taken: 99 ms spark-sql (db1)> SELECT * FROM t1; +---+ | id| +---+ | 1| | 5| | 3| | 0| | 6| | 9| | 4| | 8| | 7| | 2| +---+ Time taken: 80 ms spark-sql (db1)> USE default; ++ || ++ ++ Time taken: 26 ms spark-sql (default)> DROP DATABASE db1 CASCADE; ++ || ++ ++ spark-sql (default)> exit;
Apache Spark 4 supports SQL Pipe Syntax.
$ swift run ... Build of product 'SparkSQLRepl' complete! (2.33s) Connected to Apache Spark 4.0.0 Server spark-sql (default)> FROM ORC.`/opt/spark/examples/src/main/resources/users.orc` |> AGGREGATE COUNT(*) cnt GROUP BY name |> ORDER BY cnt DESC, name ASC ; +------+---+ | name|cnt| +------+---+ |Alyssa| 1| | Ben| 1| +------+---+ Time taken: 159 ms
You can use SPARK_REMOTE to specify the Spark Connect connection string in order to provide more options.
SPARK_REMOTE=sc://localhost swift run