layout: global title: Native Data Sources displayTitle: Native Data Sources license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
A native data source is a data source implemented in native code, such as Rust or C++, without any JVM code. It is a shared library that implements the C interface declared in the header spark_datasource.h. Spark finds the library automatically when it is installed like other native libraries, or when a session adds it in a package, and loads it when a query uses the data source. Spark adapts it to Data Source V2, so it supports:
Data is exchanged only as columnar batches, through the Arrow C data interface, so Spark reads the data that a library returns without copying it. When a write gets the batches of a scan as they are, as in a plain copy from one native data source to another, Spark writes them without converting them to rows: it copies the buffers of their columns, since a library may keep the data that it writes until it closes its data writer, and converts value by value the columns of other Arrow types than those of the schema, such as utf8_view strings, and those that Arrow Java cannot slice, such as a column of a fixed width with 2 GiB of values or more. The copied values are written as the scan returned them, without the checks of a conversion, such as the precision of a decimal. The batches that Spark copies end with the batches of the scan, so a library may get smaller batches than spark.sql.execution.arrow.maxRecordsPerBatch rows then. The data of a query in which Spark also filters, projects or shuffles the rows is still converted to rows, and with adaptive query execution, so is the data of a query with a subquery. So is the data of every query when an extension of the session (spark.sql.extensions) has columnar rules, which may replace the conversions to rows with their own, such as those of an engine that runs queries in native code.
Spark calls native data sources with the Foreign Function and Memory API of Java, which is final from Java 22, so they require Java 22 or later on the driver and the executors. With an earlier version of Java, a query that uses a native data source fails with NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION. Spark exchanges the data with the C data interface of Arrow Java, whose JNI library is available for Linux with glibc and macOS on x86_64 and aarch64, and for Windows on x86_64. On other platforms, such as Linux on ppc64le or s390x, or Linux with musl, as Alpine Linux, a query that uses a native data source fails with NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM.
A native data source fits a source whose client exists only in native code, such as an in-house storage system written in C++ or the C SDK of a vendor, and that Spark needs to read and write. Such a source then needs no JVM connector: the same library works with all the releases of Spark that support its version of the interface, and with every Spark API, including Spark Connect clients in any language.
A native data source provides what Data Source V2 offers to data sources, except for the features that version 1 of the interface does not cover yet, such as row-level operations: see Limitations. Extensions of Spark outside of Data Source V2, such as table functions, SQL commands and optimizer rules, are out of the scope of this interface. A source that needs them is better served by a JVM connector, which can still read and write through a native library with the Arrow C stream interface.
A native data source library exports a single C function, spark_ds_init, which returns a table of function pointers, a SparkDsLibrary, that implements the data source: it needs no JVM headers or bindings. Each library has its own table, so any number of native data sources can be loaded side by side. When Spark loads a library, it calls spark_ds_init with a SparkDsHost, which gives the library the version of Spark, a function to log messages to the log of Spark, and a function that tells whether Spark logs the messages of a level, so that a library follows the log level of Spark.
A Spark distribution contains the header in its include directory, as $SPARK_HOME/include/spark_datasource.h, and the spark-sql JAR contains it as org/apache/spark/sql/datasource/spark_datasource.h. The header only depends on the C standard library, so a library can also copy it into its source tree, to build without Spark, for example from the source of a release:
curl -LO {{site.SPARK_GITHUB_URL}}/raw/v{{site.SPARK_VERSION_SHORT}}/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h
A library built against the header works with all the releases of Spark that support its version, SPARK_DS_ABI_VERSION: see Compatibility.
The schemas are Arrow schemas, with the types of version 1 that spark_datasource.h lists, such as int64 for bigint, utf8 for string and timestamp[us] for timestamp: Spark fails an operation whose schema has another type, such as an interval, before it calls the library. The data that a library returns must agree with the nullability of the schema to read.
Spark calls the functions of the table as follows:
create, which receives the name of the data source and the options of the user in a SparkDsCreateInfo, and returns the capabilities of the data source, such as SPARK_DS_BATCH_READ or SPARK_DS_STREAMING_WRITE, and closes it right after with close. Spark reports the operations that a data source does not declare as unsupported. It also creates a data source with the options of the table while it analyzes a query, to reject a batch write that the data source does not support before it computes the data to write. When the user does not specify a schema, Spark asks for it with schema.create_reader, pushes operations down with push_domains (see Filter Pushdown), push_limit and prune_columns, and plans the partitions with partitions, which also returns the state of the reader. Then it calls read on the executors for each partition, in tasks that may each read several partitions, for example after a coalesce or a union, with a SparkDsReadInfo that has the partition, the schema to read and the state of the reader, and read exports an ArrowArrayStream. Its callbacks can be called from any thread, but never concurrently. The schema to read has only the columns that the query needs, which prune_columns got, and the arrays of the stream have exactly these columns, also when the library does not implement prune_columns, which only lets it plan to read them.create_stream_reader, which gets the columns to read, which may be only those that the query needs, and the location of the checkpoint of the source in a SparkDsStreamReaderInfo. When the checkpoint does not have the initial offset of the source yet, Spark asks for it with initial_offset, and keeps it in the checkpoint, so a query that restarts before its first micro-batch is committed starts from the same offset. For each micro-batch, it asks for the latest_offset, with the starting progress boundary of the micro-batch, plans the partitions with stream_partitions, which also returns the state of the stream reader, and calls read on the executors. stream_partitions gets the starting and ending progress boundaries of the micro-batch, and the library defines how offsets map to records: if an offset is the next row to read, {"next":10} to {"next":13} reads rows 10, 11, and 12. Equal boundaries describe no data, and adjacent ranges must neither repeat nor skip records. With Trigger.AvailableNow, a run reads the data that was available when it started: Spark asks for the latest_offset without a limit once, before the first micro-batch that it plans, and reads up to it. Offsets are defined by the library, as compact JSON on a single line: Spark stores each offset as a line of the offset log of the checkpoint, and rejects an offset that is not valid JSON or contains a line break, such as pretty-printed JSON. Spark finds new data by comparing the latest offset with the start as strings, so the same position must always have the same JSON. Spark keeps what it needs in the checkpoint of the source under the names that start with _spark, which the library must not use. A data source with the capability SPARK_DS_ADMISSION_CONTROL limits the data of each micro-batch, for example with an option for a maximum number of rows: Spark asks for the limit once with default_read_limit, which sets a SparkDsReadLimit, and passes it to the latest_offset that ends each micro-batch, in the limit of its SparkDsLatestOffsetInfo, which then returns the end of the data of the next micro-batch within the limit. When there is data after the start, the latest offset must be after the start, even if the next unit of data, such as a file, is larger than the limit: Spark sees no new data otherwise. With Trigger.AvailableNow, the latest offset without a limit that Spark asks for first is the end of the run: Spark passes it as the run_end that the limited latest offsets must not go beyond, so that the run reads the data that was available when it started in micro-batches within the limit. A query that restarts starts a new run, with a new end. So with a limit, Spark calls latest_offset twice for some micro-batches: with Trigger.AvailableNow, for the first one, first without a limit for the end of the run, and with the other triggers, for each one, first with SPARK_DS_READ_ALL_AVAILABLE only for the progress of the query. Spark does not ask again with the limit when the latest offset without a limit is the same as the start, such as while a query waits for new data: there is no data to limit then. So a library must not count the calls as micro-batches. Trigger.Once ignores the limit, and so does Trigger.AvailableNow when another source of the query does not support it and spark.sql.streaming.triggerAvailableNowWrapper.enabled is false, as by default, where Spark reads all the available data in a single micro-batch. Spark still asks for the limit then, and fails the query if it is not valid.create_writer, which gets a SparkDsWriteInfo with the mode of the write, SPARK_DS_WRITE_APPEND or SPARK_DS_WRITE_TRUNCATE, and the ID of the query of a streaming write, and returns the state of the writer. With adaptive query execution, it first runs the shuffle stages of the data to write, and creates no writer if they fail, but the last stage runs in the tasks of the write, so a writer can still be aborted because computing the data failed: do irreversible work, such as removing the existing data for an overwrite, in commit. On the executors, each task creates a data writer with create_data_writer, which gets the schema of the data and the state of the writer in a SparkDsDataWriterInfo, writes the batches of its data with write, and calls commit_data_writer, or abort_data_writer if the task or the commit fails before commit_data_writer succeeded. Then the driver calls commit with the messages of the tasks, or abort if the write failed. A data writer whose task failed after commit_data_writer succeeded is neither aborted nor passed to commit, so commit or abort cleans up the output of the data writers whose messages it does not get. When Spark rejects what create_writer returned, such as a state set more than once, it calls abort with the field rejected of SparkDsFinishWriteInfo set to 1, and an epoch ID of -1 also for a streaming write, before close_writer. A streaming write commits each micro-batch with its epoch ID, and the ID of the query stays the same when the query restarts from its checkpoint: together, they identify the output of a micro-batch, so that a library can skip an epoch that it already committed. A restarted query may also abort such an epoch, so an abort only removes what the data writers of its own writer wrote. A micro-batch streaming query creates a new stream writer for each micro-batch, and Spark closes it with close_writer once it committed or aborted the micro-batch. A stream writer that Spark neither commits nor aborts is closed before the query creates its next one, or, because Spark is not told when a streaming query stops, once it is no longer referenced.The library defines the format of the partitions, of the state of its readers and writers, and of the commit messages, which Spark sends to the executors and back as byte arrays. The library returns them, and the offsets, by passing them to a SparkDsSink callback, which copies them. A reader or a writer without a state does not call the set_state sink of partitions, stream_partitions or create_writer: read and create_data_writer then get an empty state. A function reports an error by returning a non-zero value, with the message returned by last_error, and Spark reports it as a NATIVE_DATA_SOURCE_ERROR. A C++ exception or a Rust panic must not cross the interface. Each handle that a create function returns is released exactly once, by the close function of its kind. The close functions cannot fail, and they can run after the query, so do the work that can fail, such as flushing data, in commit. After a function on a handle fails, Spark only closes the handle, except that it aborts a writer whose commit failed and a data writer whose write or commit failed, and it never aborts one whose commit succeeded. Spark may read a partition more than once, also at the same time, in several attempts of its task, such as after a failure or with speculative execution. The header lists the order of the calls that a library can rely on. Strings are UTF-8.
A library only sets the functions of the operations it supports, and leaves the others NULL. A data source must not declare a capability whose functions are NULL:
| Capability | Required functions | Optional functions |
|---|---|---|
| All | create, close | last_error, schema |
SPARK_DS_BATCH_READ | create_reader, partitions, read, close_reader | push_domains, push_limit, prune_columns |
SPARK_DS_MICRO_BATCH_READ | create_stream_reader, initial_offset, latest_offset, stream_partitions, read, close_stream_reader | commit_offset |
SPARK_DS_ADMISSION_CONTROL, with SPARK_DS_MICRO_BATCH_READ | default_read_limit | |
SPARK_DS_BATCH_WRITE, SPARK_DS_STREAMING_WRITE | create_writer, commit, abort, close_writer, create_data_writer, write, commit_data_writer, abort_data_writer, close_data_writer |
SPARK_DS_BATCH_TRUNCATE declares that a data source supports SPARK_DS_WRITE_TRUNCATE in batch writes, for the save mode overwrite, and SPARK_DS_STREAMING_TRUNCATE in streaming writes, for the output mode complete.
The comments of spark_datasource.h specify each function, including the ownership of the Arrow structs.
The arrays that a library exports must have an offset of 0: copy a sliced array, such as the result of RecordBatch::slice in arrow-rs, before exporting it. A batch must have no null rows, since Arrow Java ignores the validity of the struct of a batch: mark the values of its columns as null instead. Spark also checks that each array has at least as many values as its parent refers to, but it cannot check that the buffers of the arrays hold those values, because the C data interface does not carry the sizes of buffers: a library that exports a buffer that is too short makes Spark read past its end.
read and create_data_writer get a SparkDsTaskContext, with the IDs of the task, its attempt, and the partition: for read, the index of the partition in the list that the planning returned, which is unique within a scan but not across the scans that a task reads, and for create_data_writer, the partition of the task. It also has an is_interrupted function that returns whether the task was killed, for example because its query was canceled. Spark cannot interrupt native code: when a task is killed, Spark stops reading only once the get_next function of the stream returns. So a read or a write that runs for long, or blocks, for example on network I/O, should check is_interrupted regularly and use timeouts. The task reaper, spark.task.reaper.enabled and spark.task.reaper.killTimeout, kills an executor whose killed task does not end.
Spark does not pass the filters of a query to a library as expressions, which a library would have to evaluate with the exact semantics of Spark SQL. It passes the domains of the columns that the filters constrain instead: for each column, the values that it can have in the rows that match, which a library compares with the statistics of its data to skip what cannot match, such as files, row groups or blocks. With push_domains, a reader gets a SparkDsDomain for each column that the filters constrain, with:
path: the index of the column in the schema that create_reader got, and then the index of the field in each struct, for a field of a struct. Spark prunes the columns after it pushes the domains down, so the schema to read that read gets may have fewer columns, at other indices;null_allowed and nan_allowed: whether null and NaN are in the domain;ranges: the ranges of the other values in the domain, as a struct array whose fields are low and high, of the Arrow type of the column, with a null bound for no bound, and low_inclusive and high_inclusive. The ranges are sorted and do not overlap. A single value is a range whose bounds are equal and inclusive, and ranges is NULL when all the other values are in the domain.For example:
| Filter | Domain |
|---|---|
id > 5 AND id <= 10 | id: (5, 10] |
id IN (1, 3) OR id IS NULL | id: null, [1, 1], [3, 3] |
NOT (id = 5) | id: (-inf, 5), (5, +inf) |
value > 1.5, for a double column | value: NaN, (1.5, +inf) |
name LIKE 'ab%', which is STARTS_WITH(name, 'ab') | name: ['ab', 'ac') |
s.x = 1 AND name IS NOT NULL | s.x: [1, 1]; name: all the values but null |
id > 5 OR name = 'a' | none: each side only constrains one of the columns |
Spark derives the domains from the comparisons of a column with literals (=, <>, <=>, <, <=, >, >=), IN, IS NULL, IS NOT NULL and STARTS_WITH, combined with AND, OR and NOT, with the semantics of Spark SQL: comparisons with null are null, so that NOT (id IN (1, NULL)) matches no row, NaN is equal to NaN and greater than any other value, and -0.0 is equal to 0.0. A domain has all the values of its column in the rows for which the filters are true, and may have others: a column that only one side of an OR constrains gets no domain from it, and the other filters, such as id + 1 > 5, a = b or ENDS_WITH(name, 'x'), constrain no column. Columns of strings with a non-binary collation, such as UTF8_LCASE, and arrays, maps and structs, get no domain. A field of a struct is null in the rows where the struct, or a struct that contains it, is null, whatever the Arrow child array of the field holds there, so statistics computed from that array may not count these rows as nulls: Spark only passes domains without null for the fields of structs.
A library compares values with the order of Spark SQL: numbers, dates and timestamps by their values, with -0.0 equal to 0.0 (use IEEE 754 comparisons, not a total order such as f64::total_cmp), strings by their UTF-8 bytes, binary values as unsigned bytes, and false before true. It must never skip a row whose values are all in their domains, which Spark cannot detect, but it can return rows that are not: Spark evaluates all the filters again after reading. So a library only uses the domains to read less data, and Spark does not push a LIMIT down with a filter. The library sets used to whether it uses each domain, and the formatted plan of the query (EXPLAIN FORMATTED, or df.explain("formatted")) and the SQL tab of the Spark UI show the filters of the used domains as PushedFilters, and the domains as PushedDomains. Set spark.sql.dataSource.native.filterPushdown.enabled to false to read without passing the domains, for example to check whether a library skips data that matches.
The function table, SparkDsLibrary, and the structs that Spark passes to the library, SparkDsHost, SparkDsTaskContext, SparkDsReadLimit, SparkDsDomain and those whose names end with Info, such as SparkDsCreateInfo, start with their size, so that they can grow. push_domains was added this way, at the end of the table: Spark does not call it for a library built against a header without it. The version of the interface, 1, is SPARK_DS_ABI_VERSION, which the table and the manifest of a package declare. Within a version:
NULL. A library must only read the fields that fit in the size of a struct that Spark passes, and only set those of a struct that it fills, such as the SparkDsReadLimit of default_read_limit, so that it also runs on earlier releases of Spark that support its version;An incompatible change comes with a new version: spark_ds_init gets the newest version that Spark supports, in SparkDsHost, and returns a table of a version that the library implements. Spark supports version 1 at least in all the releases of Spark 4 from 4.4.0.
Version 1 of the interface does not cover these features of Data Source V2 yet. They can be added later as functions or fields appended to the structs of the interface:
spark.sql.defaultSizeInBytes, so it does not plan a native scan as the broadcast side of a join, unless the query has a broadcast hint. Adaptive query execution can still broadcast it at run time, from the size of a shuffle of the scan;Trigger.Continuous): a continuous query that reads from or writes to a native data source fails when it starts;ORDER BY with LIMIT), and reporting the partitioning and the sort order of the data, with which Spark avoids shuffles and sorts, as in storage-partitioned joins;UPDATE, DELETE and MERGE INTO, and metadata columns;Extensions of Spark outside of Data Source V2, such as table functions, SQL commands and optimizer rules, are out of the scope of this interface: see When to Use a Native Data Source.
Spark does not manage the memory of native data sources. The memory that a library allocates, and the Arrow buffers that it exports, which Spark reads without copying them, are not accounted by the memory manager of Spark, nor limited by spark.memory.offHeap.size. Neither are the Arrow batches that Spark writes, which end at spark.sql.execution.arrow.maxRecordsPerBatch rows or once they hold spark.sql.execution.arrow.maxBytesPerBatch bytes, or earlier when Spark copies the batches of a scan, while the size of the batches that a library reads is up to the library. This memory comes out of the memory overhead of the driver and the executors, spark.driver.memoryOverhead and spark.executor.memoryOverhead, which are 10% of their memory, and at least 384 MiB, by default. Increase it for libraries that use much memory, and bound the memory that a library uses, for example with the size of its batches: on YARN or Kubernetes, an executor that uses more memory than it requested is killed.
An engine that runs queries in native code within Spark, such as Apache Gluten or Apache DataFusion Comet, can read a native data source without Spark importing its data on the JVM. The reader factory of a batch or micro-batch scan of a native data source implements org.apache.spark.sql.connector.read.SupportsArrowStreamRead: in the task that reads a partition, openArrowStream initializes an Arrow C stream that the engine allocated with the stream that read returns. The engine reads the stream in the task that opened it, and owns and releases the stream, which it may do after the task ended: Spark frees the SparkDsTaskContext of the read once the library released its stream. Once the task ended, is_interrupted returns true, so the library may stop reading then. The callbacks of the stream can be called from any thread, but never concurrently. Each call goes through the JVM, which then calls the callback of the stream of the library: this adds tens of nanoseconds to the call, while the import of a batch on the JVM takes microseconds. The first call on a native thread also attaches the thread to the JVM, which takes tens of microseconds, and the thread stays attached until it exits. Since the callbacks call into the JVM, the engine releases the stream before the JVM exits. Spark does not check the arrays of such a stream, unlike the streams that it reads itself, so the engine checks what its import of the arrays requires.
When no Java or Python data source has the name that a query uses, Spark looks for a native data source with that name: in the packages of the session, and then in the installed libraries. A Java data source with the same name takes precedence over a native one, and so does a Python data source.
Like a Python data source installed in the Python path, a native data source library installed like other native libraries is found automatically, without any configuration. The library is named after the data source, in lower case, with the prefix spark_datasource_: the library of rust_range is libspark_datasource_rust_range.so on Linux, libspark_datasource_rust_range.dylib on macOS and spark_datasource_rust_range.dll on Windows. Spark passes the name of the data source to the create function of the library. A library that implements several data sources is installed under each of their names, for example with symbolic links.
Spark looks for the library in the native library path, which consists of, in order:
java.library.path, where the JVM finds native libraries. They include LD_LIBRARY_PATH on Linux, DYLD_LIBRARY_PATH on macOS and PATH on Windows, which also contain the directories set with spark.driver.extraLibraryPath and spark.executor.extraLibraryPath.lib directory of each installation prefix in PATH: <prefix>/lib for each directory <prefix>/bin in PATH.So Spark finds the libraries where native libraries are usually installed:
/usr/local/lib, the default directory of make install, CMake (cmake --install) and cargo-c (cargo cinstall), and of Homebrew on Intel macs;/opt/homebrew/lib, for Homebrew on Apple silicon;$CONDA_PREFIX/lib and $VIRTUAL_ENV/lib, in an activated conda environment or Python virtual environment;/usr/lib, for the packages of a Linux distribution.Cargo does not install libraries: copy the library that cargo build --release builds into one of these directories. A library in any other directory is found with LD_LIBRARY_PATH, or with spark.driver.extraLibraryPath and spark.executor.extraLibraryPath.
The executors load the library from the same path as the driver, or else find it in their own native library path, which they get from their own environment, such as the PATH of the executor processes. So install the library on every node, for example in the container image of the cluster. An executor only loads the same build of the library as the driver, which it checks with the SHA-256 checksum of the file, because the partitions and the state that the driver serializes are only meant for that build: a query fails with NATIVE_DATA_SOURCE_LIBRARY_NOT_FOUND on a node where a different build is installed.
A JVM loads a library from a path only once. So after a new build is installed at the same path, the queries of an application that loaded the previous build fail with INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED, on the driver or on the executors that loaded it, rather than running mixed builds: restart the application to use the new build.
A native data source package distributes a library with a session, for one or more platforms. It is a zip file with the extension .sparkpkg, which contains a manifest named spark-native-datasource.json and the libraries:
{ "abiVersion": 1, "dataSources": ["rust_range"], "libraries": { "linux-x86_64": "linux-x86_64/libspark_datasource_rust_range.so", "linux-aarch64": "linux-aarch64/libspark_datasource_rust_range.so", "osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib" } }
abiVersion is the oldest version of spark_datasource.h that the library implements, currently 1. Spark uses a package whose abiVersion it supports, or is older than the newest version that it supports, and spark_ds_init returns a table of a version that both support. Spark ignores the fields of the manifest that it does not know, so that later releases can add fields.dataSources lists the names of the data sources that the library implements. The names are case-insensitive, and Spark passes the name to create.libraries maps each platform, <os>-<arch> with an os of linux, osx or windows and an arch of x86_64 or aarch64, to the path of the library file in the package. linux is Linux with glibc: Linux with musl, as Alpine Linux, is linux-<arch>-musl, such as linux-x86_64-musl, on which native data sources do not run yet.A session finds the packages in two places, and the executors get them automatically:
Artifacts of a session: spark.addArtifact adds a package to a session, in Scala, Java and Python, and with Spark Connect:
spark.addArtifact("/path/to/rust_range.sparkpkg") spark.read.format("rust_range").option("end", 100).load().show()
Configured paths: the packages listed in spark.sql.dataSource.native.paths, or in the directories it lists, on the driver. A package can be replaced in place: the next queries use the new version.
The packages of a session take precedence over the installed libraries, and the package that a session added last over its other packages, also the configured ones: a session cannot remove an artifact, or add another file with the name of one that it added, so it upgrades a package by adding the new version under another file name, such as rust_range-1.1.sparkpkg. Distinct file names are also needed because the sessions that are not isolated share the files they add.
A query fails if more than one configured package provides its data source. Spark ignores, and logs, a configured package that it cannot read or whose version it does not support, unless its manifest lists the data source that a query uses and no other package or installed library provides it.
Unless spark.sql.dataSource.native.enabled is false, spark.addArtifact fails for a file that is not a valid package, or for a package whose version Spark does not support, or that has no library for the platform of the driver, which loads the library to plan a query, instead of adding a package that the session could not use: the session would ignore it and silently keep using the package or library that it used before, or fail every query on its data sources. After the failure, the session keeps the packages that it had, and a call that adds several artifacts in Scala or Java still adds the others. In Python, a rejected package stops the rest of the call: without Spark Connect, spark.addArtifacts adds the packages first, one at a time, so only the packages before it are added; with Spark Connect, it adds the paths one at a time, so the paths before the package are added. A file that does not exist or cannot be read stops the call, as for the other artifacts.
Spark does not load the library when it checks a package, so a library that fails to load fails the queries that use it. Nor does it check that the driver can run native data sources at all: before Java 22, a valid package is added, and the queries fail with NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION; on a platform where the C data interface of Arrow Java does not load, such as Linux with musl, only a package with a library for that platform is added, and the queries fail with NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM.
This data source reads the numbers [0, end) as a column id, in two partitions. It only depends on the arrow-array and arrow-schema crates of arrow-rs, which export schemas and streams with FFI_ArrowSchema and FFI_ArrowArrayStream. It declares SparkDsLibrary as a #[repr(C)] struct up to read, the last function that it implements, since Spark only reads the functions that fit in the size of the table, and leaves the functions that it does not implement None. The fields must follow the order of spark_datasource.h, which bindgen can also generate from the header. Cargo.toml, where the name of the library is the name that Spark finds when it is installed:
[package] name = "rust_range" version = "0.1.0" edition = "2021" [lib] name = "spark_datasource_rust_range" crate-type = ["cdylib"] [dependencies] arrow-array = { version = "58", features = ["ffi"] } arrow-schema = { version = "58", features = ["ffi"] }
src/lib.rs:
{% raw %}
//! A native data source for Apache Spark, written in Rust. It reads the numbers [0, end) as a //! column `id`, in two partitions. The option `end` defaults to 10. use std::cell::RefCell; use std::ffi::{c_char, c_int, c_void, CStr, CString}; use std::panic::{catch_unwind, AssertUnwindSafe}; use std::sync::Arc; use arrow_array::ffi_stream::FFI_ArrowArrayStream; use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator}; use arrow_schema::ffi::FFI_ArrowSchema; use arrow_schema::{DataType, Field, Schema, SchemaRef}; type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>; /// `SparkDsSink`: receives a byte array returned to Spark, and copies it. type Sink = unsafe extern "C" fn(sink: *mut c_void, data: *const u8, length: i64); /// A function of `SparkDsLibrary` that this data source does not implement, left NULL. type Unused = Option<unsafe extern "C" fn()>; /// `SparkDsCreateInfo` of `spark_datasource.h`: the name and the options of a data source. #[repr(C)] pub struct SparkDsCreateInfo { size: i64, name: *const c_char, option_keys: *const *const c_char, option_values: *const *const c_char, num_options: i64, } /// `SparkDsReadInfo` of `spark_datasource.h`: a partition to read, with the schema to read and /// the state of the reader. #[repr(C)] pub struct SparkDsReadInfo { size: i64, kind: i32, schema: *mut FFI_ArrowSchema, reader_state: *const u8, reader_state_length: i64, partition: *const u8, partition_length: i64, } /// `SparkDsLibrary` of `spark_datasource.h`, up to `read`, the last function that this data /// source implements: Spark only reads the functions that fit in `size`. #[repr(C)] pub struct SparkDsLibrary { size: i64, abi_version: i32, last_error: unsafe extern "C" fn() -> *const c_char, create: unsafe extern "C" fn( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int, schema: unsafe extern "C" fn(source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int, create_reader: unsafe extern "C" fn( source: *mut c_void, schema: *mut FFI_ArrowSchema, info: *const c_void, reader: *mut *mut c_void, ) -> c_int, create_stream_reader: Unused, create_writer: Unused, close: unsafe extern "C" fn(source: *mut c_void), push_limit: Unused, prune_columns: Unused, partitions: unsafe extern "C" fn( reader: *mut c_void, info: *const c_void, add_partition: Sink, set_state: Sink, sink: *mut c_void, ) -> c_int, close_reader: unsafe extern "C" fn(reader: *mut c_void), initial_offset: Unused, latest_offset: Unused, stream_partitions: Unused, commit_offset: Unused, close_stream_reader: Unused, read: unsafe extern "C" fn( task: *const c_void, info: *const SparkDsReadInfo, out: *mut FFI_ArrowArrayStream, ) -> c_int, } const SPARK_DS_BATCH_READ: u64 = 1; static LIBRARY: SparkDsLibrary = SparkDsLibrary { size: std::mem::size_of::<SparkDsLibrary>() as i64, abi_version: 1, last_error, create, schema, create_reader, create_stream_reader: None, create_writer: None, close, push_limit: None, prune_columns: None, partitions, close_reader, initial_offset: None, latest_offset: None, stream_partitions: None, commit_offset: None, close_stream_reader: None, read, }; /// The function that Spark calls when it loads the library. #[no_mangle] pub unsafe extern "C" fn spark_ds_init( _host: *const c_void, library: *mut *const SparkDsLibrary, ) -> c_int { *library = &LIBRARY; 0 } struct RangeSource { end: i64, } struct RangeReader { end: i64, } thread_local! { static LAST_ERROR: RefCell<CString> = RefCell::new(CString::default()); } fn arrow_schema() -> SchemaRef { Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])) } /// Runs `f`, and reports its error or panic to Spark: neither may cross into the JVM. fn call(f: impl FnOnce() -> Result<()>) -> c_int { let message = match catch_unwind(AssertUnwindSafe(f)) { Ok(Ok(())) => return 0, Ok(Err(e)) => e.to_string(), Err(_) => "the data source panicked".to_string(), }; LAST_ERROR.with(|error| *error.borrow_mut() = CString::new(message).unwrap_or_default()); 1 } unsafe extern "C" fn last_error() -> *const c_char { LAST_ERROR.with(|error| error.borrow().as_ptr()) } unsafe extern "C" fn create( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int { call(|| { let info = &*info; let mut range = RangeSource { end: 10 }; for i in 0..info.num_options as usize { if CStr::from_ptr(*info.option_keys.add(i)).to_str()? == "end" { range.end = CStr::from_ptr(*info.option_values.add(i)).to_str()?.parse()?; } } *capabilities = SPARK_DS_BATCH_READ; *source = Box::into_raw(Box::new(range)) as *mut c_void; Ok(()) }) } unsafe extern "C" fn schema(_source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int { call(|| { std::ptr::write(out, FFI_ArrowSchema::try_from(arrow_schema().as_ref())?); Ok(()) }) } unsafe extern "C" fn create_reader( source: *mut c_void, _schema: *mut FFI_ArrowSchema, _info: *const c_void, reader: *mut *mut c_void, ) -> c_int { call(|| { let end = (*(source as *const RangeSource)).end; *reader = Box::into_raw(Box::new(RangeReader { end })) as *mut c_void; Ok(()) }) } unsafe extern "C" fn close(source: *mut c_void) { drop(Box::from_raw(source as *mut RangeSource)); } /// Plans two partitions. It needs nothing from its `SparkDsPartitionsInfo`, and the reader has no /// state, so it does not call `set_state`. unsafe extern "C" fn partitions( reader: *mut c_void, _info: *const c_void, add_partition: Sink, _set_state: Sink, sink: *mut c_void, ) -> c_int { call(|| { let end = (*(reader as *const RangeReader)).end; for (start, end) in [(0, end / 2), (end / 2, end)] { let bytes = [start.to_le_bytes(), end.to_le_bytes()].concat(); add_partition(sink, bytes.as_ptr(), bytes.len() as i64); } Ok(()) }) } unsafe extern "C" fn close_reader(reader: *mut c_void) { drop(Box::from_raw(reader as *mut RangeReader)); } unsafe extern "C" fn read( _task: *const c_void, info: *const SparkDsReadInfo, out: *mut FFI_ArrowArrayStream, ) -> c_int { call(|| { let info = &*info; let bytes = std::slice::from_raw_parts(info.partition, info.partition_length as usize); let start = i64::from_le_bytes(bytes[0..8].try_into()?); let end = i64::from_le_bytes(bytes[8..16].try_into()?); let ids = Int64Array::from_iter_values(start..end); let batch = RecordBatch::try_new(arrow_schema(), vec![Arc::new(ids)])?; let batches = RecordBatchIterator::new(vec![Ok(batch)], arrow_schema()); std::ptr::write(out, FFI_ArrowArrayStream::new(Box::new(batches))); Ok(()) }) }
{% endraw %}
Build the library, here on macOS, and install it:
cargo build --release cp target/release/libspark_datasource_rust_range.dylib /usr/local/lib/
Spark then finds the data source rust_range automatically:
spark.read.format("rust_range").option("end", 100).load().show()
Or package it, to add it to a session with spark.addArtifact:
mkdir -p package/osx-aarch64 cp target/release/libspark_datasource_rust_range.dylib package/osx-aarch64/ cat > package/spark-native-datasource.json <<EOF { "abiVersion": 1, "dataSources": ["rust_range"], "libraries": {"osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib"} } EOF (cd package && zip -r ../rust_range.sparkpkg .)
The same data source in C++, with Arrow C++, which exports schemas and streams with arrow::ExportSchema and arrow::ExportRecordBatchReader. It includes spark_datasource.h, here from the include directory of a Spark distribution, and fills SparkDsLibrary with designated initializers, which leave the other functions NULL.
cpp_range.cc:
{% raw %}
// A native data source for Apache Spark, written in C++ with Arrow C++. It reads the numbers // [0, end) as a column `id`, in two partitions. The option `end` defaults to 10. #include <cstdint> #include <cstring> #include <memory> #include <stdexcept> #include <string> #include <arrow/api.h> #include <arrow/c/bridge.h> #include "spark_datasource.h" namespace { struct RangeSource { int64_t end = 10; }; struct RangeReader { int64_t end; }; thread_local std::string last_error; std::shared_ptr<arrow::Schema> Schema() { return arrow::schema({arrow::field("id", arrow::int64(), /*nullable=*/false)}); } // Runs `f`, and reports its error to Spark: no C++ exception may cross into the JVM. template <typename F> int Call(F&& f) { try { f(); return 0; } catch (const std::exception& e) { last_error = e.what(); return 1; } catch (...) { last_error = "unknown error"; return 1; } } void Check(const arrow::Status& status) { if (!status.ok()) throw std::runtime_error(status.ToString()); } const char* LastError() { return last_error.c_str(); } int Create(const SparkDsCreateInfo* info, uint64_t* capabilities, void** source) { return Call([&] { auto range = std::make_unique<RangeSource>(); for (int64_t i = 0; i < info->num_options; i++) { if (std::string(info->option_keys[i]) == "end") { range->end = std::stoll(info->option_values[i]); } } *capabilities = SPARK_DS_BATCH_READ; *source = range.release(); }); } int GetSchema(void*, struct ArrowSchema* out) { return Call([&] { Check(arrow::ExportSchema(*Schema(), out)); }); } int CreateReader(void* source, struct ArrowSchema*, const SparkDsReaderInfo*, void** reader) { return Call([&] { *reader = new RangeReader{static_cast<RangeSource*>(source)->end}; }); } void Close(void* source) { delete static_cast<RangeSource*>(source); } // Plans two partitions. It needs nothing from its info, and the reader has no state, so it does // not call set_state. int Partitions(void* reader, const SparkDsPartitionsInfo*, SparkDsSink add_partition, SparkDsSink, void* sink) { return Call([&] { int64_t end = static_cast<RangeReader*>(reader)->end; int64_t bounds[2][2] = {{0, end / 2}, {end / 2, end}}; for (const auto& partition : bounds) { add_partition(sink, reinterpret_cast<const uint8_t*>(partition), sizeof(partition)); } }); } void CloseReader(void* reader) { delete static_cast<RangeReader*>(reader); } int Read(const SparkDsTaskContext*, const SparkDsReadInfo* info, struct ArrowArrayStream* out) { return Call([&] { int64_t bounds[2]; std::memcpy(bounds, info->partition, sizeof(bounds)); arrow::Int64Builder ids; for (int64_t id = bounds[0]; id < bounds[1]; id++) Check(ids.Append(id)); std::shared_ptr<arrow::Array> array; Check(ids.Finish(&array)); auto batch = arrow::RecordBatch::Make(Schema(), array->length(), {array}); auto batches = arrow::RecordBatchReader::Make({batch}, Schema()).ValueOrDie(); Check(arrow::ExportRecordBatchReader(batches, out)); }); } // The functions of the data source. The ones that it does not implement are NULL. const SparkDsLibrary kLibrary = { .size = sizeof(SparkDsLibrary), .abi_version = SPARK_DS_ABI_VERSION, .last_error = LastError, .create = Create, .schema = GetSchema, .create_reader = CreateReader, .close = Close, .partitions = Partitions, .close_reader = CloseReader, .read = Read, }; } // namespace extern "C" SPARK_DS_EXPORT int spark_ds_init(const SparkDsHost*, const SparkDsLibrary** library) { *library = &kLibrary; return 0; }
{% endraw %}
Build the library, here on macOS with Arrow C++ 24, which requires C++20, and install it, or package it as above:
c++ -std=c++20 -O2 -shared -fPIC -I"$SPARK_HOME/include" -I"$ARROW_HOME/include" \ -o libspark_datasource_cpp_range.dylib cpp_range.cc -L"$ARROW_HOME/lib" -larrow cp libspark_datasource_cpp_range.dylib /usr/local/lib/
A library loaded by Spark must find its dependencies, such as libarrow, on every node: link them statically, or install them on the nodes.
| Property Name | Default | Meaning |
|---|---|---|
spark.sql.dataSource.native.enabled | true | Whether Spark loads native data sources. It is a static configuration, so it can only be set when the Spark application starts. When false, spark.addArtifact adds a .sparkpkg file as a file of the session, without checking it as a package. |
spark.sql.dataSource.native.paths | (none) | Comma-separated list of native data source packages, and of directories that contain them, on the local file system of the driver. They take precedence over the installed libraries, and the packages that a session adds with spark.addArtifact take precedence over them. |
spark.sql.dataSource.native.filterPushdown.enabled | true | Whether Spark passes the domains of the filtered columns of a batch scan to native data sources, so that they can skip data: see Filter Pushdown. Spark evaluates the filters after reading anyway. |
A native data source runs native code in the driver and executor processes, with their privileges, and is not sandboxed. Spark finds installed libraries by their file name, in the directories of the native library path, and loads one when a query uses its name and no Java or Python data source has that name, much like it imports the Python data sources installed as pyspark_* packages. So only install libraries and add packages that you trust, and do not let untrusted users write to the directories of the native library path, or to the directories in spark.sql.dataSource.native.paths. A user who can add artifacts to a session, for example a Spark Connect client, can also add a package, as they can add JAR files. Set spark.sql.dataSource.native.enabled to false to prevent loading native data sources, for example on a server shared by several users.