This example demonstrates the write lifecycle of the SDK; it is not a production template. It supports only POSIX filesystems and one write operation per destination at a time; see Limits of this example.
This write-only example implements DataSource, BatchWrite, StreamingWrite, DataWriterFactory, and DataWriter from the Rust native data source SDK. Its DataSource sets type ScanBuilder = Unsupported and type MicroBatchStream = Unsupported, and its batch and streaming writes declare truncation support with const SUPPORTS_TRUNCATE: bool = true; it exports export_data_source!(JsonSink) and implements no read traits. The SDK supplies the C ABI entry point, typed factory and commit message serialization, Arrow batch import, capability checks, and error handling.
The example targets version 1 of the C ABI of native data sources, spark_datasource.h, which Spark 4.4.0 introduces. Running it in Spark requires Java 22 or later on the driver and executors. The standalone Rust tests do not need Spark or a JVM.
The sink accepts exactly two columns, in this order: id BIGINT and label STRING. Both nullable and non-nullable input fields are accepted. Null values are preserved. Task writers encode batches incrementally as JSON lines, avoiding an additional file-format dependency in this small example.
| Option | Default | Meaning |
|---|---|---|
path | Required | Absolute directory shared by the driver and every executor |
readonly | false | Remove all the capabilities, which are those of writes, when true |
Unknown options are rejected, except checkpointLocation and queryName, which Spark passes to the sink of a streaming query. Source creation and capability discovery do not create files. The default instance advertises only native BATCH_WRITE, STREAMING_WRITE, BATCH_TRUNCATE, and STREAMING_TRUNCATE; it does not advertise reads. Spark performs its native capability check again with the operation's effective options.
Each write operation holds a .write-lock directory under path, rejecting another concurrent writer. It assigns an operation ID and stages tasks in .staging/<operation>. Each task attempt exclusively creates its own file, writes Arrow batches, flushes and syncs the file, then returns a typed message containing its operation, file, partition, and row count. Task commit alone does not publish data.
The driver receives the winning attempt messages, the message of partition i at position i. It validates those messages and renames the entire staging directory to objects/<operation>, preventing new tasks from opening files for that operation. It removes every file not named by a winning message, including speculative attempts and attempts whose messages never arrived. It then writes and syncs a new manifest and atomically renames it to manifest.json. That one manifest replacement is the commit point:
objects/<operation> or .staging/<operation>, and can make abort fail. Readers never see it, since they read only the files that the manifest lists.To inspect output, read only the files listed in manifest.json. Recursively reading the destination directory would also pick up data from previous truncate operations.
For example, after a successful write:
import json from pathlib import Path root = Path("/absolute/shared/path/native-sink") manifest = json.loads((root / "manifest.json").read_text()) files = [str(root / entry["path"]) for entry in manifest["files"]] if files: spark.read.schema("id LONG, label STRING").json(files).show()
A streaming query writes each micro-batch, or epoch, with the protocol above: Spark creates a JsonStreamingWrite for each micro-batch, which stages, seals, and publishes its files as a batch write does. The output mode “append” appends to the manifest, and “complete” replaces its files, as a truncate does.
The manifest also records the last epoch that each streaming query committed, by the ID of the query, which stays the same when the query restarts from its checkpoint. Spark may commit an epoch again after a restart, so a micro-batch whose epoch the manifest already records for its query publishes nothing: it removes what it staged and leaves the manifest unchanged. Each epoch of a query is therefore published once, and the queries that write to the same destination keep their epochs apart.
The example builds only in the Spark source tree, which binary distributions do not include, since it depends on the SDK under sql/native/rust. From the repository root:
cargo build --manifest-path sql/native/rust/Cargo.toml --workspace --locked cargo test --manifest-path sql/native/rust/Cargo.toml -p spark-native-sink-demo --locked python3 examples/src/main/rust/native-data-source/package.py \ --source rust_sdk_sink \ --library sql/native/rust/target/debug/libspark_datasource_rust_sdk_sink.so \ --output sql/native/rust/target/rust_sdk_sink.sparkpkg "$SPARK_HOME/bin/spark-submit" --master 'local[2]' \ examples/src/main/rust/native-data-sink/smoke.py \ --package sql/native/rust/target/rust_sdk_sink.sparkpkg
Use libspark_datasource_rust_sdk_sink.dylib for macOS. The smoke script uses a temporary local directory and requires a local Spark master. It checks append, overwrite, a failed overwrite that preserves published data and cleans staging, write rejection with readonly=true, and a streaming write whose restart without new input commits nothing. Rust tests additionally exercise losing attempts, missing and invalid commit messages, invalid manifests, interrupted tasks, a driver publication failure, lock cleanup, and epochs that a restarted query commits again.
An application can write with:
spark.addArtifact("sql/native/rust/target/rust_sdk_sink.sparkpkg") data = spark.range(10).selectExpr("id", "concat('row-', id) AS label") data.write.format("rust_sdk_sink").mode("append").save("/absolute/shared/path/native-sink") data.write.format("rust_sdk_sink").mode("overwrite").save("/absolute/shared/path/native-sink") stream = spark.readStream.schema("id BIGINT, label STRING").json("/absolute/shared/path/input") query = ( stream.writeStream.format("rust_sdk_sink") .option("checkpointLocation", "/absolute/shared/path/checkpoint") .start("/absolute/shared/path/native-sink") )
Use a local POSIX filesystem, or a shared filesystem with atomic directory renames, atomic file replacement, and coherent exclusive creation. The same absolute path must refer to the same storage on all hosts. Executor-local paths in a distributed cluster, object-store URLs, and Windows are unsupported. Only one write operation can own a destination at a time; the lock is not a distributed lease or fencing protocol.
This example demonstrates atomic visibility during normal execution and abort, and publishes each epoch of a streaming query once, not crash recovery or exactly-once publication of a retried batch write. It syncs files but does not implement a crash-durable metadata journal. Process or machine failure can leave a stale .write-lock, staged files, or unreferenced objects, and a killed task attempt that races the seal or the abort can leave staged files or unreferenced objects too. Recover the lock only after confirming no operation still owns it. Retained files from old manifests also need garbage collection with an appropriate reader-retention policy; this demo does not implement that policy.