blob: c7c57e556cf63130005de575471de08c4a15239f [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Rust native sink
This example demonstrates the write lifecycle of the SDK; it is not a production template.
It supports only POSIX filesystems and one write operation per destination at a time; see
[Limits of this example](#limits-of-this-example).
This write-only example implements `DataSource`, `BatchWrite`, `StreamingWrite`,
`DataWriterFactory`, and `DataWriter` from the [Rust native data source SDK][sdk]. Its
`DataSource` sets `type ScanBuilder = Unsupported` and `type MicroBatchStream = Unsupported`,
and its batch and streaming writes declare truncation support with
`const SUPPORTS_TRUNCATE: bool = true`; it exports `export_data_source!(JsonSink)` and
implements no read traits. The SDK supplies the C ABI entry point, typed factory and commit
message serialization, Arrow batch import, capability checks, and error handling.
The example targets version 1 of the C ABI of native data sources, `spark_datasource.h`,
which Spark 4.4.0 introduces. Running it in Spark requires Java 22 or later on the driver
and executors. The standalone Rust tests do not need Spark or a JVM.
## Supported data and options
The sink accepts exactly two columns, in this order: `id BIGINT` and `label STRING`.
Both nullable and non-nullable input fields are accepted. Null values are preserved.
Task writers encode batches incrementally as JSON lines, avoiding an additional
file-format dependency in this small example.
| Option | Default | Meaning |
| --- | --- | --- |
| `path` | Required | Absolute directory shared by the driver and every executor |
| `readonly` | `false` | Remove all the capabilities, which are those of writes, when `true` |
Unknown options are rejected, except `checkpointLocation` and `queryName`, which Spark passes
to the sink of a streaming query. Source creation and capability discovery do not create
files. The default instance advertises only native `BATCH_WRITE`, `STREAMING_WRITE`, `BATCH_TRUNCATE`, and
`STREAMING_TRUNCATE`; it does not advertise reads. Spark performs its native capability check
again with the operation's effective options.
## Commit and abort protocol
Each write operation holds a `.write-lock` directory under `path`, rejecting another
concurrent writer. It assigns an operation ID and stages tasks in `.staging/<operation>`.
Each task attempt exclusively creates its own file, writes Arrow batches, flushes and
syncs the file, then returns a typed message containing its operation, file, partition,
and row count. Task commit alone does not publish data.
The driver receives the winning attempt messages, the message of partition `i` at
position `i`. It validates those messages and renames the entire staging directory
to `objects/<operation>`, preventing new tasks from opening files for that
operation. It removes every file not named by a winning
message, including speculative attempts and attempts whose messages never arrived.
It then writes and syncs a new manifest and atomically renames it to `manifest.json`.
That one manifest replacement is the commit point:
- Append includes the previous manifest's files plus the winning new files.
- Truncate includes only the winning new files, including an empty result.
- Previously published data files are retained, so an existing manifest snapshot
stays readable and a failed overwrite cannot delete the old data.
- Abort removes the entire operation, including files with no commit message and
a sealed directory whose manifest publication failed. It leaves the current
manifest and its data unchanged. If the final rename itself reports an error, the
outcome is unknown, so abort keeps the sealed directory that the new manifest may
list. Dropping an unfinished coordinator also performs best-effort cleanup.
Dropping a committed task writer leaves its file for the driver.
- A task attempt that Spark killed, such as a losing speculative one, may still run
after the driver commits or aborts. A file that it creates while the driver renames
or removes the operation directory stays, unlisted, in `objects/<operation>` or
`.staging/<operation>`, and can make abort fail. Readers never see it, since they
read only the files that the manifest lists.
To inspect output, read only the files listed in `manifest.json`. Recursively reading
the destination directory would also pick up data from previous truncate operations.
For example, after a successful write:
```python
import json
from pathlib import Path
root = Path("/absolute/shared/path/native-sink")
manifest = json.loads((root / "manifest.json").read_text())
files = [str(root / entry["path"]) for entry in manifest["files"]]
if files:
spark.read.schema("id LONG, label STRING").json(files).show()
```
## Streaming writes
A streaming query writes each micro-batch, or epoch, with the protocol above: Spark creates a
`JsonStreamingWrite` for each micro-batch, which stages, seals, and publishes its files as a
batch write does. The output mode "append" appends to the manifest, and "complete" replaces its
files, as a truncate does.
The manifest also records the last epoch that each streaming query committed, by the ID of the
query, which stays the same when the query restarts from its checkpoint. Spark may commit an
epoch again after a restart, so a micro-batch whose epoch the manifest already records for its
query publishes nothing: it removes what it staged and leaves the manifest unchanged. Each epoch
of a query is therefore published once, and the queries that write to the same destination keep
their epochs apart.
## Build, package, and run
The example builds only in the Spark source tree, which binary distributions do not include,
since it depends on the SDK under `sql/native/rust`. From the repository root:
```sh
cargo build --manifest-path sql/native/rust/Cargo.toml --workspace --locked
cargo test --manifest-path sql/native/rust/Cargo.toml -p spark-native-sink-demo --locked
python3 examples/src/main/rust/native-data-source/package.py \
--source rust_sdk_sink \
--library sql/native/rust/target/debug/libspark_datasource_rust_sdk_sink.so \
--output sql/native/rust/target/rust_sdk_sink.sparkpkg
"$SPARK_HOME/bin/spark-submit" --master 'local[2]' \
examples/src/main/rust/native-data-sink/smoke.py \
--package sql/native/rust/target/rust_sdk_sink.sparkpkg
```
Use `libspark_datasource_rust_sdk_sink.dylib` for macOS. The smoke script uses a
temporary local directory and requires a local Spark master. It checks append,
overwrite, a failed overwrite that preserves published data and cleans staging,
write rejection with `readonly=true`, and a streaming write whose restart without new
input commits nothing. Rust tests additionally exercise losing attempts, missing and
invalid commit messages, invalid manifests, interrupted tasks, a driver publication
failure, lock cleanup, and epochs that a restarted query commits again.
An application can write with:
```python
spark.addArtifact("sql/native/rust/target/rust_sdk_sink.sparkpkg")
data = spark.range(10).selectExpr("id", "concat('row-', id) AS label")
data.write.format("rust_sdk_sink").mode("append").save("/absolute/shared/path/native-sink")
data.write.format("rust_sdk_sink").mode("overwrite").save("/absolute/shared/path/native-sink")
stream = spark.readStream.schema("id BIGINT, label STRING").json("/absolute/shared/path/input")
query = (
stream.writeStream.format("rust_sdk_sink")
.option("checkpointLocation", "/absolute/shared/path/checkpoint")
.start("/absolute/shared/path/native-sink")
)
```
## Limits of this example
Use a local POSIX filesystem, or a shared filesystem with atomic directory renames,
atomic file replacement, and coherent exclusive creation. The same absolute path must
refer to the same storage on all hosts. Executor-local paths in a distributed cluster,
object-store URLs, and Windows are unsupported. Only one write operation can own a
destination at a time; the lock is not a distributed lease or fencing protocol.
This example demonstrates atomic visibility during normal execution and abort, and
publishes each epoch of a streaming query once, not crash recovery or exactly-once
publication of a retried batch write. It syncs files but
does not implement a crash-durable metadata journal. Process or machine failure can
leave a stale `.write-lock`, staged files, or unreferenced objects, and a killed task
attempt that races the seal or the abort can leave staged files or unreferenced
objects too. Recover the lock
only after confirming no operation still owns it. Retained files from old manifests
also need garbage collection with an appropriate reader-retention policy; this demo
does not implement that policy.
[sdk]: ../../../../../sql/native/rust/README.md