| <!-- | |
| Licensed to the Apache Software Foundation (ASF) under one or more | |
| contributor license agreements. See the NOTICE file distributed with | |
| this work for additional information regarding copyright ownership. | |
| The ASF licenses this file to You under the Apache License, Version 2.0 | |
| (the "License"); you may not use this file except in compliance with | |
| the License. You may obtain a copy of the License at | |
| http://www.apache.org/licenses/LICENSE-2.0 | |
| Unless required by applicable law or agreed to in writing, software | |
| distributed under the License is distributed on an "AS IS" BASIS, | |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| See the License for the specific language governing permissions and | |
| limitations under the License. | |
| --> | |
| # Rust native sink | |
| This example demonstrates the write lifecycle of the SDK; it is not a production template. | |
| It supports only POSIX filesystems and one write operation per destination at a time; see | |
| [Limits of this example](#limits-of-this-example). | |
| This write-only example implements `DataSource`, `BatchWrite`, `StreamingWrite`, | |
| `DataWriterFactory`, and `DataWriter` from the [Rust native data source SDK][sdk]. Its | |
| `DataSource` sets `type ScanBuilder = Unsupported` and `type MicroBatchStream = Unsupported`, | |
| and its batch and streaming writes declare truncation support with | |
| `const SUPPORTS_TRUNCATE: bool = true`; it exports `export_data_source!(JsonSink)` and | |
| implements no read traits. The SDK supplies the C ABI entry point, typed factory and commit | |
| message serialization, Arrow batch import, capability checks, and error handling. | |
| The example targets version 1 of the C ABI of native data sources, `spark_datasource.h`, | |
| which Spark 4.4.0 introduces. Running it in Spark requires Java 22 or later on the driver | |
| and executors. The standalone Rust tests do not need Spark or a JVM. | |
| ## Supported data and options | |
| The sink accepts exactly two columns, in this order: `id BIGINT` and `label STRING`. | |
| Both nullable and non-nullable input fields are accepted. Null values are preserved. | |
| Task writers encode batches incrementally as JSON lines, avoiding an additional | |
| file-format dependency in this small example. | |
| | Option | Default | Meaning | | |
| | --- | --- | --- | | |
| | `path` | Required | Absolute directory shared by the driver and every executor | | |
| | `readonly` | `false` | Remove all the capabilities, which are those of writes, when `true` | | |
| Unknown options are rejected, except `checkpointLocation` and `queryName`, which Spark passes | |
| to the sink of a streaming query. Source creation and capability discovery do not create | |
| files. The default instance advertises only native `BATCH_WRITE`, `STREAMING_WRITE`, `BATCH_TRUNCATE`, and | |
| `STREAMING_TRUNCATE`; it does not advertise reads. Spark performs its native capability check | |
| again with the operation's effective options. | |
| ## Commit and abort protocol | |
| Each write operation holds a `.write-lock` directory under `path`, rejecting another | |
| concurrent writer. It assigns an operation ID and stages tasks in `.staging/<operation>`. | |
| Each task attempt exclusively creates its own file, writes Arrow batches, flushes and | |
| syncs the file, then returns a typed message containing its operation, file, partition, | |
| and row count. Task commit alone does not publish data. | |
| The driver receives the winning attempt messages, the message of partition `i` at | |
| position `i`. It validates those messages and renames the entire staging directory | |
| to `objects/<operation>`, preventing new tasks from opening files for that | |
| operation. It removes every file not named by a winning | |
| message, including speculative attempts and attempts whose messages never arrived. | |
| It then writes and syncs a new manifest and atomically renames it to `manifest.json`. | |
| That one manifest replacement is the commit point: | |
| - Append includes the previous manifest's files plus the winning new files. | |
| - Truncate includes only the winning new files, including an empty result. | |
| - Previously published data files are retained, so an existing manifest snapshot | |
| stays readable and a failed overwrite cannot delete the old data. | |
| - Abort removes the entire operation, including files with no commit message and | |
| a sealed directory whose manifest publication failed. It leaves the current | |
| manifest and its data unchanged. If the final rename itself reports an error, the | |
| outcome is unknown, so abort keeps the sealed directory that the new manifest may | |
| list. Dropping an unfinished coordinator also performs best-effort cleanup. | |
| Dropping a committed task writer leaves its file for the driver. | |
| - A task attempt that Spark killed, such as a losing speculative one, may still run | |
| after the driver commits or aborts. A file that it creates while the driver renames | |
| or removes the operation directory stays, unlisted, in `objects/<operation>` or | |
| `.staging/<operation>`, and can make abort fail. Readers never see it, since they | |
| read only the files that the manifest lists. | |
| To inspect output, read only the files listed in `manifest.json`. Recursively reading | |
| the destination directory would also pick up data from previous truncate operations. | |
| For example, after a successful write: | |
| ```python | |
| import json | |
| from pathlib import Path | |
| root = Path("/absolute/shared/path/native-sink") | |
| manifest = json.loads((root / "manifest.json").read_text()) | |
| files = [str(root / entry["path"]) for entry in manifest["files"]] | |
| if files: | |
| spark.read.schema("id LONG, label STRING").json(files).show() | |
| ``` | |
| ## Streaming writes | |
| A streaming query writes each micro-batch, or epoch, with the protocol above: Spark creates a | |
| `JsonStreamingWrite` for each micro-batch, which stages, seals, and publishes its files as a | |
| batch write does. The output mode "append" appends to the manifest, and "complete" replaces its | |
| files, as a truncate does. | |
| The manifest also records the last epoch that each streaming query committed, by the ID of the | |
| query, which stays the same when the query restarts from its checkpoint. Spark may commit an | |
| epoch again after a restart, so a micro-batch whose epoch the manifest already records for its | |
| query publishes nothing: it removes what it staged and leaves the manifest unchanged. Each epoch | |
| of a query is therefore published once, and the queries that write to the same destination keep | |
| their epochs apart. | |
| ## Build, package, and run | |
| The example builds only in the Spark source tree, which binary distributions do not include, | |
| since it depends on the SDK under `sql/native/rust`. From the repository root: | |
| ```sh | |
| cargo build --manifest-path sql/native/rust/Cargo.toml --workspace --locked | |
| cargo test --manifest-path sql/native/rust/Cargo.toml -p spark-native-sink-demo --locked | |
| python3 examples/src/main/rust/native-data-source/package.py \ | |
| --source rust_sdk_sink \ | |
| --library sql/native/rust/target/debug/libspark_datasource_rust_sdk_sink.so \ | |
| --output sql/native/rust/target/rust_sdk_sink.sparkpkg | |
| "$SPARK_HOME/bin/spark-submit" --master 'local[2]' \ | |
| examples/src/main/rust/native-data-sink/smoke.py \ | |
| --package sql/native/rust/target/rust_sdk_sink.sparkpkg | |
| ``` | |
| Use `libspark_datasource_rust_sdk_sink.dylib` for macOS. The smoke script uses a | |
| temporary local directory and requires a local Spark master. It checks append, | |
| overwrite, a failed overwrite that preserves published data and cleans staging, | |
| write rejection with `readonly=true`, and a streaming write whose restart without new | |
| input commits nothing. Rust tests additionally exercise losing attempts, missing and | |
| invalid commit messages, invalid manifests, interrupted tasks, a driver publication | |
| failure, lock cleanup, and epochs that a restarted query commits again. | |
| An application can write with: | |
| ```python | |
| spark.addArtifact("sql/native/rust/target/rust_sdk_sink.sparkpkg") | |
| data = spark.range(10).selectExpr("id", "concat('row-', id) AS label") | |
| data.write.format("rust_sdk_sink").mode("append").save("/absolute/shared/path/native-sink") | |
| data.write.format("rust_sdk_sink").mode("overwrite").save("/absolute/shared/path/native-sink") | |
| stream = spark.readStream.schema("id BIGINT, label STRING").json("/absolute/shared/path/input") | |
| query = ( | |
| stream.writeStream.format("rust_sdk_sink") | |
| .option("checkpointLocation", "/absolute/shared/path/checkpoint") | |
| .start("/absolute/shared/path/native-sink") | |
| ) | |
| ``` | |
| ## Limits of this example | |
| Use a local POSIX filesystem, or a shared filesystem with atomic directory renames, | |
| atomic file replacement, and coherent exclusive creation. The same absolute path must | |
| refer to the same storage on all hosts. Executor-local paths in a distributed cluster, | |
| object-store URLs, and Windows are unsupported. Only one write operation can own a | |
| destination at a time; the lock is not a distributed lease or fencing protocol. | |
| This example demonstrates atomic visibility during normal execution and abort, and | |
| publishes each epoch of a streaming query once, not crash recovery or exactly-once | |
| publication of a retried batch write. It syncs files but | |
| does not implement a crash-durable metadata journal. Process or machine failure can | |
| leave a stale `.write-lock`, staged files, or unreferenced objects, and a killed task | |
| attempt that races the seal or the abort can leave staged files or unreferenced | |
| objects too. Recover the lock | |
| only after confirming no operation still owns it. Retained files from old manifests | |
| also need garbage collection with an appropriate reader-retention policy; this demo | |
| does not implement that policy. | |
| [sdk]: ../../../../../sql/native/rust/README.md |