| --- | |
| layout: global | |
| title: Native Data Sources | |
| displayTitle: Native Data Sources | |
| license: | | |
| Licensed to the Apache Software Foundation (ASF) under one or more | |
| contributor license agreements. See the NOTICE file distributed with | |
| this work for additional information regarding copyright ownership. | |
| The ASF licenses this file to You under the Apache License, Version 2.0 | |
| (the "License"); you may not use this file except in compliance with | |
| the License. You may obtain a copy of the License at | |
| http://www.apache.org/licenses/LICENSE-2.0 | |
| Unless required by applicable law or agreed to in writing, software | |
| distributed under the License is distributed on an "AS IS" BASIS, | |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| See the License for the specific language governing permissions and | |
| limitations under the License. | |
| --- | |
| * Table of contents | |
| {:toc} | |
| ## Overview | |
| A native data source is a data source implemented in native code, such as Rust or C++, without | |
| any JVM code. It is a shared library that implements the C interface declared in the header | |
| [`spark_datasource.h`]({{site.SPARK_GITHUB_URL}}/blob/v{{site.SPARK_VERSION_SHORT}}/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h). Spark finds the library | |
| automatically when it is installed like other native libraries, or when a session adds it in a | |
| package, and loads it when a query uses the data source. Spark adapts it to | |
| [Data Source V2](sql-v2-data-sources.html), so it supports: | |
| - batch and micro-batch streaming reads, | |
| - batch and streaming writes, appending to or overwriting the existing data, | |
| - filter, limit and column pushdown. | |
| Data is exchanged only as columnar batches, through the | |
| [Arrow C data interface](https://arrow.apache.org/docs/format/CDataInterface.html), so Spark reads | |
| the data that a library returns without copying it. When a write gets the batches of a scan as | |
| they are, as in a plain copy from one native data source to another, Spark writes them without | |
| converting them to rows: it copies the buffers of their columns, since a library may keep the data | |
| that it writes until it closes its data writer, and converts value by value the columns of other | |
| Arrow types than those of the schema, such as `utf8_view` strings, and those that Arrow Java cannot | |
| slice, such as a column of a fixed width with 2 GiB of values or more. The copied values are written | |
| as the scan returned them, without the checks of a conversion, such as the precision of a decimal. | |
| The batches that Spark copies end with the batches of the scan, so a library may get smaller | |
| batches than `spark.sql.execution.arrow.maxRecordsPerBatch` rows then. The data of a query in which | |
| Spark also filters, projects or shuffles the rows is still converted to rows, and with adaptive | |
| query execution, so is the data of a query with a subquery. So is the data of every query when an | |
| extension of the session (`spark.sql.extensions`) has columnar rules, which may replace the | |
| conversions to rows with their own, such as those of an engine that runs queries in native code. | |
| Spark calls native data sources with the | |
| [Foreign Function and Memory API](https://openjdk.org/jeps/454) of Java, which is final from Java | |
| 22, so they require Java 22 or later on the driver and the executors. With an earlier version of | |
| Java, a query that uses a native data source fails with | |
| `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`. Spark exchanges the data with the C data interface | |
| of Arrow Java, whose JNI library is available for Linux with glibc and macOS on x86_64 and aarch64, | |
| and for Windows on x86_64. On other platforms, such as Linux on ppc64le or s390x, or Linux with | |
| musl, as Alpine Linux, a query that uses a native data source fails with | |
| `NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM`. | |
| ### When to Use a Native Data Source | |
| A native data source fits a source whose client exists only in native code, such as an in-house | |
| storage system written in C++ or the C SDK of a vendor, and that Spark needs to read and write. | |
| Such a source then needs no JVM connector: the same library works with all the releases of Spark | |
| that support its version of the interface, and with every Spark API, including Spark Connect | |
| clients in any language. | |
| A native data source provides what Data Source V2 offers to data sources, except for the features | |
| that version 1 of the interface does not cover yet, such as row-level operations: see | |
| [Limitations](#limitations). Extensions of Spark outside of Data Source V2, such as table | |
| functions, SQL commands and optimizer rules, are out of the scope of this interface. A source that | |
| needs them is better served by a JVM connector, which can still read and write through a native | |
| library with the Arrow C stream interface. | |
| ## How It Works | |
| A native data source library exports a single C function, `spark_ds_init`, which returns a table | |
| of function pointers, a `SparkDsLibrary`, that implements the data source: it needs no JVM headers | |
| or bindings. Each library has its own table, so any number of native data sources can be loaded | |
| side by side. When Spark loads a library, it calls `spark_ds_init` with a `SparkDsHost`, which | |
| gives the library the version of Spark, a function to log messages to the log of Spark, and a | |
| function that tells whether Spark logs the messages of a level, so that a library follows the log | |
| level of Spark. | |
| A Spark distribution contains the header in its `include` directory, as | |
| `$SPARK_HOME/include/spark_datasource.h`, and the `spark-sql` JAR contains it as | |
| `org/apache/spark/sql/datasource/spark_datasource.h`. The header only depends on the C standard | |
| library, so a library can also copy it into its source tree, to build without Spark, for example | |
| from the source of a release: | |
| ```bash | |
| curl -LO {{site.SPARK_GITHUB_URL}}/raw/v{{site.SPARK_VERSION_SHORT}}/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h | |
| ``` | |
| A library built against the header works with all the releases of Spark that support its version, | |
| `SPARK_DS_ABI_VERSION`: see [Compatibility](#compatibility). | |
| The schemas are Arrow schemas, with the types of version 1 that `spark_datasource.h` lists, such | |
| as `int64` for `bigint`, `utf8` for `string` and `timestamp[us]` for `timestamp`: Spark fails an | |
| operation whose schema has another type, such as an interval, before it calls the library. The data | |
| that a library returns must agree with the nullability of the schema to read. | |
| Spark calls the functions of the table as follows: | |
| - **Data sources**: for each schema inference, scan and write, Spark creates a data source on the | |
| driver with `create`, which receives the name of the data source and the options of the user in | |
| a `SparkDsCreateInfo`, and returns the capabilities of the data source, such as | |
| `SPARK_DS_BATCH_READ` or `SPARK_DS_STREAMING_WRITE`, and closes it right after with `close`. | |
| Spark reports the operations that a data source does not declare as unsupported. It also creates | |
| a data source with the options of the table while it analyzes a query, to reject a batch write | |
| that the data source does not support before it computes the data to write. When the user does | |
| not specify a schema, Spark asks for it with `schema`. | |
| - **Batch reads**: on the driver, Spark creates a reader with `create_reader`, pushes operations | |
| down with `push_domains` (see [Filter Pushdown](#filter-pushdown)), `push_limit` and | |
| `prune_columns`, and plans the partitions with `partitions`, which also returns the state of the | |
| reader. Then it calls `read` on the executors for each partition, in tasks that may each read | |
| several partitions, for example after a coalesce or a union, with a `SparkDsReadInfo` that has | |
| the partition, the schema to read and the state of the reader, and `read` exports an | |
| `ArrowArrayStream`. Its callbacks can be called from any thread, but never concurrently. The | |
| schema to read has only the columns that the query needs, which `prune_columns` got, and the | |
| arrays of the stream have exactly these columns, also when the library does not implement | |
| `prune_columns`, which only lets it plan to read them. | |
| - **Streaming reads**: on the driver, Spark creates a stream reader with `create_stream_reader`, | |
| which gets the columns to read, which may be only those that the query needs, and the location of | |
| the checkpoint of the source in a `SparkDsStreamReaderInfo`. When the checkpoint does not have the | |
| initial offset of the source yet, Spark asks for it with `initial_offset`, and keeps it in the | |
| checkpoint, so a query that restarts before its first micro-batch is committed starts from the | |
| same offset. For each micro-batch, it asks for the `latest_offset`, with the starting progress | |
| boundary of the micro-batch, plans the partitions with `stream_partitions`, which also returns the | |
| state of the stream reader, and calls `read` on the executors. `stream_partitions` gets the | |
| starting and ending progress boundaries of the micro-batch, and the library defines how offsets | |
| map to records: if an offset is the next row to read, `{"next":10}` to `{"next":13}` reads rows | |
| 10, 11, and 12. Equal boundaries describe no data, and adjacent ranges must neither repeat nor | |
| skip records. With `Trigger.AvailableNow`, a run | |
| reads the data that was available when it started: Spark asks for the `latest_offset` without a | |
| limit once, before the first micro-batch that it plans, and reads up to it. Offsets are defined | |
| by the library, as compact JSON on a single line: Spark stores each offset as a line of the | |
| offset log of the checkpoint, and rejects an offset that is not valid JSON or contains a line | |
| break, such as pretty-printed JSON. Spark finds new data by comparing the latest offset with the | |
| start as strings, so the same position must always have the same JSON. Spark keeps what it needs | |
| in the checkpoint of the source under the names that start with `_spark`, which the library must | |
| not use. A data source with the capability | |
| `SPARK_DS_ADMISSION_CONTROL` limits the data of each micro-batch, for example with an option for | |
| a maximum number of rows: Spark asks for the limit once with `default_read_limit`, which sets a | |
| `SparkDsReadLimit`, and passes it to the `latest_offset` that ends each micro-batch, in the | |
| `limit` of its `SparkDsLatestOffsetInfo`, which then returns the end of the data of the next | |
| micro-batch within the limit. When there is data after the start, the latest offset must be after | |
| the start, even if the next unit of data, such as a file, is larger than the limit: Spark sees no | |
| new data otherwise. With `Trigger.AvailableNow`, the latest offset without a limit that Spark asks | |
| for first is the end of the run: Spark passes it as the `run_end` that the limited latest offsets | |
| must not go beyond, so that the run reads the data that was available when it started in | |
| micro-batches within the limit. A query that restarts starts a new run, with a new end. So with a | |
| limit, Spark calls `latest_offset` twice for some micro-batches: with `Trigger.AvailableNow`, for | |
| the first one, first without a limit for the end of the run, and with the other triggers, for each | |
| one, first with `SPARK_DS_READ_ALL_AVAILABLE` only for the progress of the query. Spark does not | |
| ask again with the limit when the latest offset without a limit is the same as the start, such as | |
| while a query waits for new data: there is no data to limit then. So a library must not count the | |
| calls as micro-batches. `Trigger.Once` ignores the limit, and so does `Trigger.AvailableNow` when | |
| another source of the query does not support it and | |
| `spark.sql.streaming.triggerAvailableNowWrapper.enabled` is false, as by default, where Spark | |
| reads all the available data in a single micro-batch. Spark still asks for the limit then, and | |
| fails the query if it is not valid. | |
| - **Writes**: on the driver, right before the tasks of the write (or of a micro-batch) start, Spark | |
| creates a writer with `create_writer`, which gets a `SparkDsWriteInfo` with the mode of the write, | |
| `SPARK_DS_WRITE_APPEND` or `SPARK_DS_WRITE_TRUNCATE`, and the ID of the query of a streaming | |
| write, and returns the state of the writer. With adaptive query execution, it first runs the | |
| shuffle stages of the data to write, and creates no writer if they fail, but the last stage runs | |
| in the tasks of the write, so a writer can still be aborted because computing the data failed: do | |
| irreversible work, such as removing the existing data for an overwrite, in `commit`. On the | |
| executors, each task creates a data writer with `create_data_writer`, which gets the schema of the | |
| data and the state of the writer in a `SparkDsDataWriterInfo`, writes the batches of its data with | |
| `write`, and calls `commit_data_writer`, or `abort_data_writer` if the task or the commit fails | |
| before `commit_data_writer` succeeded. Then the driver calls `commit` with the messages of the | |
| tasks, or `abort` if the write failed. A data writer whose task failed after | |
| `commit_data_writer` succeeded is neither aborted nor passed to `commit`, so `commit` or `abort` | |
| cleans up the output of the data writers whose messages it does not get. | |
| When Spark rejects what `create_writer` returned, such as a state set more than once, it calls | |
| `abort` with the field `rejected` of `SparkDsFinishWriteInfo` set to 1, and an epoch ID of -1 also | |
| for a streaming write, before `close_writer`. A streaming write commits each micro-batch with its | |
| epoch ID, and the ID of the query stays the same when the query restarts from its checkpoint: | |
| together, they identify the output of a micro-batch, so that a library can skip an epoch that it | |
| already committed. A restarted query may also abort such an epoch, so an abort only removes what | |
| the data writers of its own writer wrote. A micro-batch streaming query creates a new stream | |
| writer for each micro-batch, and Spark closes it with `close_writer` once it committed or aborted | |
| the micro-batch. A stream writer that Spark neither commits nor aborts is closed before the | |
| query creates its next one, or, because Spark is not told when a streaming query stops, once it | |
| is no longer referenced. | |
| The library defines the format of the partitions, of the state of its readers and writers, and of | |
| the commit messages, which Spark sends to the executors and back as byte arrays. The library | |
| returns them, and the offsets, by passing them to a `SparkDsSink` callback, which copies them. A | |
| reader or a writer without a state does not call the `set_state` sink of `partitions`, | |
| `stream_partitions` or `create_writer`: `read` and `create_data_writer` then get an empty state. A | |
| function reports an error by returning a non-zero value, with the message returned by | |
| `last_error`, and Spark reports it as a `NATIVE_DATA_SOURCE_ERROR`. A C++ exception or a Rust panic | |
| must not cross the interface. Each handle that a `create` function returns is released exactly | |
| once, by the `close` function of its kind. The close functions cannot fail, and they can run after | |
| the query, so do the work that can fail, such as flushing data, in `commit`. After a function on | |
| a handle fails, Spark only closes the handle, except that it aborts a writer whose commit failed | |
| and a data writer whose write or commit failed, and it never aborts one whose commit succeeded. | |
| Spark may read a partition more than once, also at the same time, in several attempts of its task, | |
| such as after a failure or with speculative execution. The header lists the order of the calls | |
| that a library can rely on. Strings are UTF-8. | |
| A library only sets the functions of the operations it supports, and leaves the others `NULL`. A | |
| data source must not declare a capability whose functions are `NULL`: | |
| | Capability | Required functions | Optional functions | | |
| |-----------|--------------------|--------------------| | |
| | All | `create`, `close` | `last_error`, `schema` | | |
| | `SPARK_DS_BATCH_READ` | `create_reader`, `partitions`, `read`, `close_reader` | `push_domains`, `push_limit`, `prune_columns` | | |
| | `SPARK_DS_MICRO_BATCH_READ` | `create_stream_reader`, `initial_offset`, `latest_offset`, `stream_partitions`, `read`, `close_stream_reader` | `commit_offset` | | |
| | `SPARK_DS_ADMISSION_CONTROL`, with `SPARK_DS_MICRO_BATCH_READ` | `default_read_limit` | | | |
| | `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` | `create_writer`, `commit`, `abort`, `close_writer`, `create_data_writer`, `write`, `commit_data_writer`, `abort_data_writer`, `close_data_writer` | | | |
| `SPARK_DS_BATCH_TRUNCATE` declares that a data source supports `SPARK_DS_WRITE_TRUNCATE` in batch | |
| writes, for the save mode `overwrite`, and `SPARK_DS_STREAMING_TRUNCATE` in streaming writes, for | |
| the output mode `complete`. | |
| The comments of [`spark_datasource.h`]({{site.SPARK_GITHUB_URL}}/blob/v{{site.SPARK_VERSION_SHORT}}/sql/core/src/main/resources/org/apache/spark/sql/datasource/spark_datasource.h) | |
| specify each function, including the ownership of the Arrow structs. | |
| The arrays that a library exports must have an offset of 0: copy a sliced array, such as the | |
| result of `RecordBatch::slice` in arrow-rs, before exporting it. A batch must have no null rows, | |
| since Arrow Java ignores the validity of the struct of a batch: mark the values of its columns as | |
| null instead. Spark also checks that each array has at least as many values as its parent refers | |
| to, but it cannot check that the buffers of the arrays hold those values, because the C data | |
| interface does not carry the sizes of buffers: a library that exports a buffer that is too short | |
| makes Spark read past its end. | |
| `read` and `create_data_writer` get a `SparkDsTaskContext`, with the IDs of the task, its | |
| attempt, and the partition: for `read`, the index of the partition in the list that the planning | |
| returned, which is unique within a scan but not across the scans that a task reads, and for | |
| `create_data_writer`, the partition of the task. It also has an `is_interrupted` function that | |
| returns whether the task was killed, for example because its query was canceled. Spark cannot | |
| interrupt native code: when a task is killed, Spark stops reading only once the `get_next` function | |
| of the stream returns. So a read or a write that runs for long, or blocks, for example on network | |
| I/O, should check `is_interrupted` regularly and use timeouts. The task reaper, | |
| `spark.task.reaper.enabled` and `spark.task.reaper.killTimeout`, kills an executor whose killed | |
| task does not end. | |
| ### Filter Pushdown | |
| Spark does not pass the filters of a query to a library as expressions, which a library would have | |
| to evaluate with the exact semantics of Spark SQL. It passes the *domains* of the columns that the | |
| filters constrain instead: for each column, the values that it can have in the rows that match, | |
| which a library compares with the statistics of its data to skip what cannot match, such as files, | |
| row groups or blocks. With `push_domains`, a reader gets a `SparkDsDomain` for each column that the | |
| filters constrain, with: | |
| - `path`: the index of the column in the schema that `create_reader` got, and then the index of | |
| the field in each struct, for a field of a struct. Spark prunes the columns after it pushes the | |
| domains down, so the schema to read that `read` gets may have fewer columns, at other indices; | |
| - `null_allowed` and `nan_allowed`: whether null and NaN are in the domain; | |
| - `ranges`: the ranges of the other values in the domain, as a struct array whose fields are | |
| `low` and `high`, of the Arrow type of the column, with a null bound for no bound, and | |
| `low_inclusive` and `high_inclusive`. The ranges are sorted and do not overlap. A single value | |
| is a range whose bounds are equal and inclusive, and `ranges` is `NULL` when all the other | |
| values are in the domain. | |
| For example: | |
| | Filter | Domain | | |
| |--------|--------| | |
| | `id > 5 AND id <= 10` | `id`: `(5, 10]` | | |
| | `id IN (1, 3) OR id IS NULL` | `id`: null, `[1, 1]`, `[3, 3]` | | |
| | `NOT (id = 5)` | `id`: `(-inf, 5)`, `(5, +inf)` | | |
| | `value > 1.5`, for a `double` column | `value`: NaN, `(1.5, +inf)` | | |
| | `name LIKE 'ab%'`, which is `STARTS_WITH(name, 'ab')` | `name`: `['ab', 'ac')` | | |
| | `s.x = 1 AND name IS NOT NULL` | `s.x`: `[1, 1]`; `name`: all the values but null | | |
| | `id > 5 OR name = 'a'` | none: each side only constrains one of the columns | | |
| Spark derives the domains from the comparisons of a column with literals (`=`, `<>`, `<=>`, `<`, | |
| `<=`, `>`, `>=`), `IN`, `IS NULL`, `IS NOT NULL` and `STARTS_WITH`, combined with `AND`, `OR` and | |
| `NOT`, with the semantics of Spark SQL: comparisons with null are null, so that | |
| `NOT (id IN (1, NULL))` matches no row, NaN is equal to NaN and greater than any other value, and | |
| -0.0 is equal to 0.0. A domain has all the values of its column in the rows for which the filters | |
| are true, and may have others: a column that only one side of an `OR` constrains gets no domain | |
| from it, and the other filters, such as `id + 1 > 5`, `a = b` or `ENDS_WITH(name, 'x')`, constrain | |
| no column. Columns of strings with a non-binary collation, such as `UTF8_LCASE`, and arrays, maps | |
| and structs, get no domain. A field of a struct is null in the rows where the struct, or a struct | |
| that contains it, is null, whatever the Arrow child array of the field holds there, so statistics | |
| computed from that array may not count these rows as nulls: Spark only passes domains without null | |
| for the fields of structs. | |
| A library compares values with the order of Spark SQL: numbers, dates and timestamps by their | |
| values, with -0.0 equal to 0.0 (use IEEE 754 comparisons, not a total order such as | |
| `f64::total_cmp`), strings by their UTF-8 bytes, binary values as unsigned bytes, and false before | |
| true. It must never skip a row whose values are all in their domains, which Spark cannot detect, | |
| but it can return rows that are not: Spark evaluates all the filters again after reading. So a | |
| library only uses the domains to read less data, and Spark does not push a `LIMIT` down with a | |
| filter. The library sets `used` to whether it uses each domain, and the formatted plan of the query | |
| (`EXPLAIN FORMATTED`, or `df.explain("formatted")`) and the SQL tab of the Spark UI show the | |
| filters of the used domains as `PushedFilters`, and the domains as `PushedDomains`. Set | |
| `spark.sql.dataSource.native.filterPushdown.enabled` to false to read without passing the domains, | |
| for example to check whether a library skips data that matches. | |
| ### Compatibility | |
| The function table, `SparkDsLibrary`, and the structs that Spark passes to the library, | |
| `SparkDsHost`, `SparkDsTaskContext`, `SparkDsReadLimit`, `SparkDsDomain` and those whose names end | |
| with `Info`, such as `SparkDsCreateInfo`, start with their size, so that they can grow. | |
| `push_domains` was added this way, at the end of the table: Spark does not call it for a library | |
| built against a header without it. The version of the interface, 1, is `SPARK_DS_ABI_VERSION`, | |
| which the table and the manifest of a package declare. Within a version: | |
| - fields are only appended to these structs, and the existing fields and the semantics of the | |
| existing functions never change; | |
| - Spark only reads the functions that fit in the size of the table of a library, so a library built | |
| against an earlier header keeps working with later releases of Spark, with the functions that it | |
| does not know as `NULL`. A library must only read the fields that fit in the size of a struct | |
| that Spark passes, and only set those of a struct that it fills, such as the `SparkDsReadLimit` | |
| of `default_read_limit`, so that it also runs on earlier releases of Spark that support its | |
| version; | |
| - Spark only uses a new capability, write mode or kind of read with a data source that declares | |
| it. | |
| An incompatible change comes with a new version: `spark_ds_init` gets the newest version that Spark | |
| supports, in `SparkDsHost`, and returns a table of a version that the library implements. Spark | |
| supports version 1 at least in all the releases of Spark 4 from 4.4.0. | |
| ### Limitations | |
| Version 1 of the interface does not cover these features of Data Source V2 yet. They can be added | |
| later as functions or fields appended to the structs of the interface: | |
| - filters that a library applies itself: Spark evaluates the filters of a native scan after | |
| reading, and only passes the domains of the columns that they constrain, which a library can use | |
| to skip data (see [Filter Pushdown](#filter-pushdown)). So it does not push a limit down with a | |
| filter, and a library cannot evaluate filters that are not domains, such as functions of | |
| columns; | |
| - statistics: Spark estimates the size of a native scan with `spark.sql.defaultSizeInBytes`, so it | |
| does not plan a native scan as the broadcast side of a join, unless the query has a broadcast | |
| hint. Adaptive query execution can still broadcast it at run time, from the size of a shuffle of | |
| the scan; | |
| - custom metrics, shown in the Spark UI; | |
| - continuous processing (`Trigger.Continuous`): a continuous query that reads from or writes to a | |
| native data source fails when it starts; | |
| - the pushdown of aggregates and of top-N queries (`ORDER BY` with `LIMIT`), and reporting the | |
| partitioning and the sort order of the data, with which Spark avoids shuffles and sorts, as in | |
| storage-partitioned joins; | |
| - row-level operations, `UPDATE`, `DELETE` and `MERGE INTO`, and metadata columns; | |
| - memory accounting: see below. | |
| Extensions of Spark outside of Data Source V2, such as table functions, SQL commands and optimizer | |
| rules, are out of the scope of this interface: see | |
| [When to Use a Native Data Source](#when-to-use-a-native-data-source). | |
| ### Memory | |
| Spark does not manage the memory of native data sources. The memory that a library allocates, and | |
| the Arrow buffers that it exports, which Spark reads without copying them, are not accounted by the | |
| memory manager of Spark, nor limited by `spark.memory.offHeap.size`. Neither are the Arrow batches | |
| that Spark writes, which end at `spark.sql.execution.arrow.maxRecordsPerBatch` rows or once they | |
| hold `spark.sql.execution.arrow.maxBytesPerBatch` bytes, or earlier when Spark copies the batches of | |
| a scan, while the size of the batches that a library reads is up to the library. This memory comes | |
| out of the memory overhead of the driver and the executors, `spark.driver.memoryOverhead` and | |
| `spark.executor.memoryOverhead`, which are 10% of their memory, and at least 384 MiB, by default. | |
| Increase it for libraries that use much memory, and bound the memory that a library uses, for | |
| example with the size of its batches: on YARN or Kubernetes, an executor that uses more memory than | |
| it requested is killed. | |
| ### Native Execution Engines | |
| An engine that runs queries in native code within Spark, such as Apache Gluten or Apache DataFusion | |
| Comet, can read a native data source without Spark importing its data on the JVM. The reader factory | |
| of a batch or micro-batch scan of a native data source implements | |
| `org.apache.spark.sql.connector.read.SupportsArrowStreamRead`: in the task that reads a partition, | |
| `openArrowStream` initializes an Arrow C stream that the engine allocated with the stream that | |
| `read` returns. The engine reads the stream in the task that opened it, and owns and releases the | |
| stream, which it may do after the task ended: Spark frees the `SparkDsTaskContext` of the read once | |
| the library released its stream. Once the task ended, `is_interrupted` returns true, so the library | |
| may stop reading then. The callbacks of the stream can be called from any thread, but never | |
| concurrently. Each call goes through the JVM, which then calls the callback of the stream of the | |
| library: this adds tens of nanoseconds to the call, while the import of a batch on the JVM takes | |
| microseconds. The first call on a native thread also attaches the thread to the JVM, which takes | |
| tens of microseconds, and the thread stays attached until it exits. Since the callbacks call into | |
| the JVM, the engine releases the stream before the JVM exits. Spark does not check the arrays of | |
| such a stream, unlike the streams that it reads itself, so the engine checks what its import of the | |
| arrays requires. | |
| ## Finding Native Data Sources | |
| When no Java or Python data source has the name that a query uses, Spark looks for a native data | |
| source with that name: in the packages of the session, and then in the installed libraries. A Java | |
| data source with the same name takes precedence over a native one, and so does a Python data | |
| source. | |
| ### Installed Libraries | |
| Like a Python data source installed in the Python path, a native data source library installed | |
| like other native libraries is found automatically, without any configuration. The library is | |
| named after the data source, in lower case, with the prefix `spark_datasource_`: the library of | |
| `rust_range` is `libspark_datasource_rust_range.so` on Linux, | |
| `libspark_datasource_rust_range.dylib` on macOS and `spark_datasource_rust_range.dll` on Windows. | |
| Spark passes the name of the data source to the `create` function of the library. A library that | |
| implements several data sources is installed under each of their names, for example with symbolic | |
| links. | |
| Spark looks for the library in the native library path, which consists of, in order: | |
| 1. the directories of `java.library.path`, where the JVM finds native libraries. They include | |
| `LD_LIBRARY_PATH` on Linux, `DYLD_LIBRARY_PATH` on macOS and `PATH` on Windows, which also | |
| contain the directories set with `spark.driver.extraLibraryPath` and | |
| `spark.executor.extraLibraryPath`. | |
| 2. the `lib` directory of each installation prefix in `PATH`: `<prefix>/lib` for each directory | |
| `<prefix>/bin` in `PATH`. | |
| So Spark finds the libraries where native libraries are usually installed: | |
| - `/usr/local/lib`, the default directory of `make install`, CMake (`cmake --install`) and | |
| [cargo-c](https://github.com/lu-zero/cargo-c) (`cargo cinstall`), and of Homebrew on Intel macs; | |
| - `/opt/homebrew/lib`, for Homebrew on Apple silicon; | |
| - `$CONDA_PREFIX/lib` and `$VIRTUAL_ENV/lib`, in an activated conda environment or Python virtual | |
| environment; | |
| - `/usr/lib`, for the packages of a Linux distribution. | |
| Cargo does not install libraries: copy the library that `cargo build --release` builds into one of | |
| these directories. A library in any other directory is found with `LD_LIBRARY_PATH`, or with | |
| `spark.driver.extraLibraryPath` and `spark.executor.extraLibraryPath`. | |
| The executors load the library from the same path as the driver, or else find it in their own | |
| native library path, which they get from their own environment, such as the `PATH` of the | |
| executor processes. So install the library on every node, for example in the container image of | |
| the cluster. An executor only loads the same build of the library as the driver, which it checks | |
| with the SHA-256 checksum of the file, because the partitions and the state that the driver | |
| serializes are only meant for that build: a query fails with | |
| `NATIVE_DATA_SOURCE_LIBRARY_NOT_FOUND` on a node where a different build is installed. | |
| A JVM loads a library from a path only once. So after a new build is installed at the same path, | |
| the queries of an application that loaded the previous build fail with | |
| `INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED`, on the driver or on the executors that loaded it, | |
| rather than running mixed builds: restart the application to use the new build. | |
| ### Packages | |
| A native data source package distributes a library with a session, for one or more platforms. It | |
| is a zip file with the extension `.sparkpkg`, which contains a manifest named | |
| `spark-native-datasource.json` and the libraries: | |
| ```json | |
| { | |
| "abiVersion": 1, | |
| "dataSources": ["rust_range"], | |
| "libraries": { | |
| "linux-x86_64": "linux-x86_64/libspark_datasource_rust_range.so", | |
| "linux-aarch64": "linux-aarch64/libspark_datasource_rust_range.so", | |
| "osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib" | |
| } | |
| } | |
| ``` | |
| - `abiVersion` is the oldest version of `spark_datasource.h` that the library implements, | |
| currently 1. Spark uses a package whose `abiVersion` it supports, or is older than the newest | |
| version that it supports, and `spark_ds_init` returns a table of a version that both support. | |
| Spark ignores the fields of the manifest that it does not know, so that later releases can add | |
| fields. | |
| - `dataSources` lists the names of the data sources that the library implements. The names are | |
| case-insensitive, and Spark passes the name to `create`. | |
| - `libraries` maps each platform, `<os>-<arch>` with an `os` of `linux`, `osx` or `windows` and an | |
| `arch` of `x86_64` or `aarch64`, to the path of the library file in the package. `linux` is Linux | |
| with glibc: Linux with musl, as Alpine Linux, is `linux-<arch>-musl`, such as `linux-x86_64-musl`, | |
| on which native data sources do not run yet. | |
| A session finds the packages in two places, and the executors get them automatically: | |
| - **Artifacts of a session**: `spark.addArtifact` adds a package to a session, in Scala, Java and | |
| Python, and with Spark Connect: | |
| ```python | |
| spark.addArtifact("/path/to/rust_range.sparkpkg") | |
| spark.read.format("rust_range").option("end", 100).load().show() | |
| ``` | |
| - **Configured paths**: the packages listed in `spark.sql.dataSource.native.paths`, or in the | |
| directories it lists, on the driver. A package can be replaced in place: the next queries use | |
| the new version. | |
| The packages of a session take precedence over the installed libraries, and the package that a | |
| session added last over its other packages, also the configured ones: a session cannot remove an | |
| artifact, or add another file with the name of one that it added, so it upgrades a package by adding | |
| the new version under another file name, such as `rust_range-1.1.sparkpkg`. Distinct file names are | |
| also needed because the sessions that are not isolated share the files they add. | |
| A query fails if more than one configured package provides its data source. Spark ignores, and logs, | |
| a configured package that it cannot read or whose version it does not support, unless its manifest | |
| lists the data source that a query uses and no other package or installed library provides it. | |
| Unless `spark.sql.dataSource.native.enabled` is false, `spark.addArtifact` fails for a file that is | |
| not a valid package, or for a package whose version Spark does not support, or that has no library | |
| for the platform of the driver, which loads the library to plan a query, instead of adding a package | |
| that the session could not use: the session would ignore it and silently keep using the package or | |
| library that it used before, or fail every query on its data sources. After the failure, the session | |
| keeps the packages that it had, and a call that adds several artifacts in Scala or Java still adds | |
| the others. In Python, a rejected package stops the rest of the call: without Spark Connect, | |
| `spark.addArtifacts` adds the packages first, one at a time, so only the packages before it are | |
| added; with Spark Connect, it adds the paths one at a time, so the paths before the package are | |
| added. A file that does not exist or cannot be read stops the call, as for the other artifacts. | |
| Spark does not load the library when it checks a package, so a library that fails to load fails the | |
| queries that use it. Nor does it check that the driver can run native data sources at all: before | |
| Java 22, a valid package is added, and the queries fail with | |
| `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`; on a platform where the C data interface of Arrow | |
| Java does not load, such as Linux with musl, only a package with a library for that platform is | |
| added, and the queries fail with `NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM`. | |
| ## Example in Rust | |
| This data source reads the numbers `[0, end)` as a column `id`, in two partitions. It only depends | |
| on the `arrow-array` and `arrow-schema` crates of [arrow-rs](https://github.com/apache/arrow-rs), | |
| which export schemas and streams with `FFI_ArrowSchema` and `FFI_ArrowArrayStream`. It declares | |
| `SparkDsLibrary` as a `#[repr(C)]` struct up to `read`, the last function that it implements, since | |
| Spark only reads the functions that fit in the size of the table, and leaves the functions that it | |
| does not implement `None`. The fields must follow the order of `spark_datasource.h`, which | |
| [bindgen](https://github.com/rust-lang/rust-bindgen) can also generate from the header. | |
| `Cargo.toml`, where the name of the library is the name that Spark finds when it is installed: | |
| ```toml | |
| [package] | |
| name = "rust_range" | |
| version = "0.1.0" | |
| edition = "2021" | |
| [lib] | |
| name = "spark_datasource_rust_range" | |
| crate-type = ["cdylib"] | |
| [dependencies] | |
| arrow-array = { version = "58", features = ["ffi"] } | |
| arrow-schema = { version = "58", features = ["ffi"] } | |
| ``` | |
| `src/lib.rs`: | |
| {% raw %} | |
| ```rust | |
| //! A native data source for Apache Spark, written in Rust. It reads the numbers [0, end) as a | |
| //! column `id`, in two partitions. The option `end` defaults to 10. | |
| use std::cell::RefCell; | |
| use std::ffi::{c_char, c_int, c_void, CStr, CString}; | |
| use std::panic::{catch_unwind, AssertUnwindSafe}; | |
| use std::sync::Arc; | |
| use arrow_array::ffi_stream::FFI_ArrowArrayStream; | |
| use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator}; | |
| use arrow_schema::ffi::FFI_ArrowSchema; | |
| use arrow_schema::{DataType, Field, Schema, SchemaRef}; | |
| type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>; | |
| /// `SparkDsSink`: receives a byte array returned to Spark, and copies it. | |
| type Sink = unsafe extern "C" fn(sink: *mut c_void, data: *const u8, length: i64); | |
| /// A function of `SparkDsLibrary` that this data source does not implement, left NULL. | |
| type Unused = Option<unsafe extern "C" fn()>; | |
| /// `SparkDsCreateInfo` of `spark_datasource.h`: the name and the options of a data source. | |
| #[repr(C)] | |
| pub struct SparkDsCreateInfo { | |
| size: i64, | |
| name: *const c_char, | |
| option_keys: *const *const c_char, | |
| option_values: *const *const c_char, | |
| num_options: i64, | |
| } | |
| /// `SparkDsReadInfo` of `spark_datasource.h`: a partition to read, with the schema to read and | |
| /// the state of the reader. | |
| #[repr(C)] | |
| pub struct SparkDsReadInfo { | |
| size: i64, | |
| kind: i32, | |
| schema: *mut FFI_ArrowSchema, | |
| reader_state: *const u8, | |
| reader_state_length: i64, | |
| partition: *const u8, | |
| partition_length: i64, | |
| } | |
| /// `SparkDsLibrary` of `spark_datasource.h`, up to `read`, the last function that this data | |
| /// source implements: Spark only reads the functions that fit in `size`. | |
| #[repr(C)] | |
| pub struct SparkDsLibrary { | |
| size: i64, | |
| abi_version: i32, | |
| last_error: unsafe extern "C" fn() -> *const c_char, | |
| create: unsafe extern "C" fn( | |
| info: *const SparkDsCreateInfo, | |
| capabilities: *mut u64, | |
| source: *mut *mut c_void, | |
| ) -> c_int, | |
| schema: unsafe extern "C" fn(source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int, | |
| create_reader: unsafe extern "C" fn( | |
| source: *mut c_void, | |
| schema: *mut FFI_ArrowSchema, | |
| info: *const c_void, | |
| reader: *mut *mut c_void, | |
| ) -> c_int, | |
| create_stream_reader: Unused, | |
| create_writer: Unused, | |
| close: unsafe extern "C" fn(source: *mut c_void), | |
| push_limit: Unused, | |
| prune_columns: Unused, | |
| partitions: unsafe extern "C" fn( | |
| reader: *mut c_void, | |
| info: *const c_void, | |
| add_partition: Sink, | |
| set_state: Sink, | |
| sink: *mut c_void, | |
| ) -> c_int, | |
| close_reader: unsafe extern "C" fn(reader: *mut c_void), | |
| initial_offset: Unused, | |
| latest_offset: Unused, | |
| stream_partitions: Unused, | |
| commit_offset: Unused, | |
| close_stream_reader: Unused, | |
| read: unsafe extern "C" fn( | |
| task: *const c_void, | |
| info: *const SparkDsReadInfo, | |
| out: *mut FFI_ArrowArrayStream, | |
| ) -> c_int, | |
| } | |
| const SPARK_DS_BATCH_READ: u64 = 1; | |
| static LIBRARY: SparkDsLibrary = SparkDsLibrary { | |
| size: std::mem::size_of::<SparkDsLibrary>() as i64, | |
| abi_version: 1, | |
| last_error, | |
| create, | |
| schema, | |
| create_reader, | |
| create_stream_reader: None, | |
| create_writer: None, | |
| close, | |
| push_limit: None, | |
| prune_columns: None, | |
| partitions, | |
| close_reader, | |
| initial_offset: None, | |
| latest_offset: None, | |
| stream_partitions: None, | |
| commit_offset: None, | |
| close_stream_reader: None, | |
| read, | |
| }; | |
| /// The function that Spark calls when it loads the library. | |
| #[no_mangle] | |
| pub unsafe extern "C" fn spark_ds_init( | |
| _host: *const c_void, | |
| library: *mut *const SparkDsLibrary, | |
| ) -> c_int { | |
| *library = &LIBRARY; | |
| 0 | |
| } | |
| struct RangeSource { | |
| end: i64, | |
| } | |
| struct RangeReader { | |
| end: i64, | |
| } | |
| thread_local! { | |
| static LAST_ERROR: RefCell<CString> = RefCell::new(CString::default()); | |
| } | |
| fn arrow_schema() -> SchemaRef { | |
| Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])) | |
| } | |
| /// Runs `f`, and reports its error or panic to Spark: neither may cross into the JVM. | |
| fn call(f: impl FnOnce() -> Result<()>) -> c_int { | |
| let message = match catch_unwind(AssertUnwindSafe(f)) { | |
| Ok(Ok(())) => return 0, | |
| Ok(Err(e)) => e.to_string(), | |
| Err(_) => "the data source panicked".to_string(), | |
| }; | |
| LAST_ERROR.with(|error| *error.borrow_mut() = CString::new(message).unwrap_or_default()); | |
| 1 | |
| } | |
| unsafe extern "C" fn last_error() -> *const c_char { | |
| LAST_ERROR.with(|error| error.borrow().as_ptr()) | |
| } | |
| unsafe extern "C" fn create( | |
| info: *const SparkDsCreateInfo, | |
| capabilities: *mut u64, | |
| source: *mut *mut c_void, | |
| ) -> c_int { | |
| call(|| { | |
| let info = &*info; | |
| let mut range = RangeSource { end: 10 }; | |
| for i in 0..info.num_options as usize { | |
| if CStr::from_ptr(*info.option_keys.add(i)).to_str()? == "end" { | |
| range.end = CStr::from_ptr(*info.option_values.add(i)).to_str()?.parse()?; | |
| } | |
| } | |
| *capabilities = SPARK_DS_BATCH_READ; | |
| *source = Box::into_raw(Box::new(range)) as *mut c_void; | |
| Ok(()) | |
| }) | |
| } | |
| unsafe extern "C" fn schema(_source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int { | |
| call(|| { | |
| std::ptr::write(out, FFI_ArrowSchema::try_from(arrow_schema().as_ref())?); | |
| Ok(()) | |
| }) | |
| } | |
| unsafe extern "C" fn create_reader( | |
| source: *mut c_void, | |
| _schema: *mut FFI_ArrowSchema, | |
| _info: *const c_void, | |
| reader: *mut *mut c_void, | |
| ) -> c_int { | |
| call(|| { | |
| let end = (*(source as *const RangeSource)).end; | |
| *reader = Box::into_raw(Box::new(RangeReader { end })) as *mut c_void; | |
| Ok(()) | |
| }) | |
| } | |
| unsafe extern "C" fn close(source: *mut c_void) { | |
| drop(Box::from_raw(source as *mut RangeSource)); | |
| } | |
| /// Plans two partitions. It needs nothing from its `SparkDsPartitionsInfo`, and the reader has no | |
| /// state, so it does not call `set_state`. | |
| unsafe extern "C" fn partitions( | |
| reader: *mut c_void, | |
| _info: *const c_void, | |
| add_partition: Sink, | |
| _set_state: Sink, | |
| sink: *mut c_void, | |
| ) -> c_int { | |
| call(|| { | |
| let end = (*(reader as *const RangeReader)).end; | |
| for (start, end) in [(0, end / 2), (end / 2, end)] { | |
| let bytes = [start.to_le_bytes(), end.to_le_bytes()].concat(); | |
| add_partition(sink, bytes.as_ptr(), bytes.len() as i64); | |
| } | |
| Ok(()) | |
| }) | |
| } | |
| unsafe extern "C" fn close_reader(reader: *mut c_void) { | |
| drop(Box::from_raw(reader as *mut RangeReader)); | |
| } | |
| unsafe extern "C" fn read( | |
| _task: *const c_void, | |
| info: *const SparkDsReadInfo, | |
| out: *mut FFI_ArrowArrayStream, | |
| ) -> c_int { | |
| call(|| { | |
| let info = &*info; | |
| let bytes = std::slice::from_raw_parts(info.partition, info.partition_length as usize); | |
| let start = i64::from_le_bytes(bytes[0..8].try_into()?); | |
| let end = i64::from_le_bytes(bytes[8..16].try_into()?); | |
| let ids = Int64Array::from_iter_values(start..end); | |
| let batch = RecordBatch::try_new(arrow_schema(), vec![Arc::new(ids)])?; | |
| let batches = RecordBatchIterator::new(vec![Ok(batch)], arrow_schema()); | |
| std::ptr::write(out, FFI_ArrowArrayStream::new(Box::new(batches))); | |
| Ok(()) | |
| }) | |
| } | |
| ``` | |
| {% endraw %} | |
| Build the library, here on macOS, and install it: | |
| ```bash | |
| cargo build --release | |
| cp target/release/libspark_datasource_rust_range.dylib /usr/local/lib/ | |
| ``` | |
| Spark then finds the data source `rust_range` automatically: | |
| ```python | |
| spark.read.format("rust_range").option("end", 100).load().show() | |
| ``` | |
| Or package it, to add it to a session with `spark.addArtifact`: | |
| ```bash | |
| mkdir -p package/osx-aarch64 | |
| cp target/release/libspark_datasource_rust_range.dylib package/osx-aarch64/ | |
| cat > package/spark-native-datasource.json <<EOF | |
| { | |
| "abiVersion": 1, | |
| "dataSources": ["rust_range"], | |
| "libraries": {"osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib"} | |
| } | |
| EOF | |
| (cd package && zip -r ../rust_range.sparkpkg .) | |
| ``` | |
| ## Example in C++ | |
| The same data source in C++, with [Arrow C++](https://arrow.apache.org/docs/cpp/), which exports | |
| schemas and streams with `arrow::ExportSchema` and `arrow::ExportRecordBatchReader`. It includes | |
| `spark_datasource.h`, here from the `include` directory of a Spark distribution, and fills | |
| `SparkDsLibrary` with designated initializers, which leave the other functions `NULL`. | |
| `cpp_range.cc`: | |
| {% raw %} | |
| ```cpp | |
| // A native data source for Apache Spark, written in C++ with Arrow C++. It reads the numbers | |
| // [0, end) as a column `id`, in two partitions. The option `end` defaults to 10. | |
| #include <cstdint> | |
| #include <cstring> | |
| #include <memory> | |
| #include <stdexcept> | |
| #include <string> | |
| #include <arrow/api.h> | |
| #include <arrow/c/bridge.h> | |
| #include "spark_datasource.h" | |
| namespace { | |
| struct RangeSource { | |
| int64_t end = 10; | |
| }; | |
| struct RangeReader { | |
| int64_t end; | |
| }; | |
| thread_local std::string last_error; | |
| std::shared_ptr<arrow::Schema> Schema() { | |
| return arrow::schema({arrow::field("id", arrow::int64(), /*nullable=*/false)}); | |
| } | |
| // Runs `f`, and reports its error to Spark: no C++ exception may cross into the JVM. | |
| template <typename F> | |
| int Call(F&& f) { | |
| try { | |
| f(); | |
| return 0; | |
| } catch (const std::exception& e) { | |
| last_error = e.what(); | |
| return 1; | |
| } catch (...) { | |
| last_error = "unknown error"; | |
| return 1; | |
| } | |
| } | |
| void Check(const arrow::Status& status) { | |
| if (!status.ok()) throw std::runtime_error(status.ToString()); | |
| } | |
| const char* LastError() { return last_error.c_str(); } | |
| int Create(const SparkDsCreateInfo* info, uint64_t* capabilities, void** source) { | |
| return Call([&] { | |
| auto range = std::make_unique<RangeSource>(); | |
| for (int64_t i = 0; i < info->num_options; i++) { | |
| if (std::string(info->option_keys[i]) == "end") { | |
| range->end = std::stoll(info->option_values[i]); | |
| } | |
| } | |
| *capabilities = SPARK_DS_BATCH_READ; | |
| *source = range.release(); | |
| }); | |
| } | |
| int GetSchema(void*, struct ArrowSchema* out) { | |
| return Call([&] { Check(arrow::ExportSchema(*Schema(), out)); }); | |
| } | |
| int CreateReader(void* source, struct ArrowSchema*, const SparkDsReaderInfo*, void** reader) { | |
| return Call([&] { *reader = new RangeReader{static_cast<RangeSource*>(source)->end}; }); | |
| } | |
| void Close(void* source) { delete static_cast<RangeSource*>(source); } | |
| // Plans two partitions. It needs nothing from its info, and the reader has no state, so it does | |
| // not call set_state. | |
| int Partitions(void* reader, const SparkDsPartitionsInfo*, SparkDsSink add_partition, SparkDsSink, | |
| void* sink) { | |
| return Call([&] { | |
| int64_t end = static_cast<RangeReader*>(reader)->end; | |
| int64_t bounds[2][2] = {{0, end / 2}, {end / 2, end}}; | |
| for (const auto& partition : bounds) { | |
| add_partition(sink, reinterpret_cast<const uint8_t*>(partition), sizeof(partition)); | |
| } | |
| }); | |
| } | |
| void CloseReader(void* reader) { delete static_cast<RangeReader*>(reader); } | |
| int Read(const SparkDsTaskContext*, const SparkDsReadInfo* info, struct ArrowArrayStream* out) { | |
| return Call([&] { | |
| int64_t bounds[2]; | |
| std::memcpy(bounds, info->partition, sizeof(bounds)); | |
| arrow::Int64Builder ids; | |
| for (int64_t id = bounds[0]; id < bounds[1]; id++) Check(ids.Append(id)); | |
| std::shared_ptr<arrow::Array> array; | |
| Check(ids.Finish(&array)); | |
| auto batch = arrow::RecordBatch::Make(Schema(), array->length(), {array}); | |
| auto batches = arrow::RecordBatchReader::Make({batch}, Schema()).ValueOrDie(); | |
| Check(arrow::ExportRecordBatchReader(batches, out)); | |
| }); | |
| } | |
| // The functions of the data source. The ones that it does not implement are NULL. | |
| const SparkDsLibrary kLibrary = { | |
| .size = sizeof(SparkDsLibrary), | |
| .abi_version = SPARK_DS_ABI_VERSION, | |
| .last_error = LastError, | |
| .create = Create, | |
| .schema = GetSchema, | |
| .create_reader = CreateReader, | |
| .close = Close, | |
| .partitions = Partitions, | |
| .close_reader = CloseReader, | |
| .read = Read, | |
| }; | |
| } // namespace | |
| extern "C" SPARK_DS_EXPORT int spark_ds_init(const SparkDsHost*, const SparkDsLibrary** library) { | |
| *library = &kLibrary; | |
| return 0; | |
| } | |
| ``` | |
| {% endraw %} | |
| Build the library, here on macOS with Arrow C++ 24, which requires C++20, and install it, or | |
| package it as above: | |
| ```bash | |
| c++ -std=c++20 -O2 -shared -fPIC -I"$SPARK_HOME/include" -I"$ARROW_HOME/include" \ | |
| -o libspark_datasource_cpp_range.dylib cpp_range.cc -L"$ARROW_HOME/lib" -larrow | |
| cp libspark_datasource_cpp_range.dylib /usr/local/lib/ | |
| ``` | |
| A library loaded by Spark must find its dependencies, such as `libarrow`, on every node: link them | |
| statically, or install them on the nodes. | |
| ## Configuration | |
| | Property Name | Default | Meaning | | |
| |---------------|---------|---------| | |
| | `spark.sql.dataSource.native.enabled` | true | Whether Spark loads native data sources. It is a static configuration, so it can only be set when the Spark application starts. When false, `spark.addArtifact` adds a `.sparkpkg` file as a file of the session, without checking it as a package. | | |
| | `spark.sql.dataSource.native.paths` | (none) | Comma-separated list of native data source packages, and of directories that contain them, on the local file system of the driver. They take precedence over the installed libraries, and the packages that a session adds with `spark.addArtifact` take precedence over them. | | |
| | `spark.sql.dataSource.native.filterPushdown.enabled` | true | Whether Spark passes the domains of the filtered columns of a batch scan to native data sources, so that they can skip data: see [Filter Pushdown](#filter-pushdown). Spark evaluates the filters after reading anyway. | | |
| ## Security | |
| A native data source runs native code in the driver and executor processes, with their | |
| privileges, and is not sandboxed. Spark finds installed libraries by their file name, in the | |
| directories of the native library path, and loads one when a query uses its name and no Java or | |
| Python data source has that name, much like it imports the Python data sources installed as | |
| `pyspark_*` packages. So only install libraries and add packages that you trust, and do not let | |
| untrusted users write to the directories of the native library path, or to the directories in | |
| `spark.sql.dataSource.native.paths`. A user who can add artifacts to a session, for example a Spark | |
| Connect client, can also add a package, as they can add JAR files. Set | |
| `spark.sql.dataSource.native.enabled` to false to prevent loading native data sources, for example | |
| on a server shared by several users. |