(io_custom_table_provider)=

Custom Table Provider

If you have a custom data source that you want to integrate with DataFusion, you can do so by implementing the TableProvider interface in Rust and then exposing it in Python. To do so, you must use DataFusion 43.0.0 or later and expose a FFI_TableProvider via PyCapsule.

A complete example can be found in the examples folder.

The method takes the SessionContext it is being registered on. Take whatever the FFI constructor needs from that session — here the logical extension codec — rather than building one inside your library. See the {ref}ffi guide for the full capsule protocol.

#[pymethods]
impl MyTableProvider {

    fn __datafusion_table_provider__<'py>(
        &self,
        py: Python<'py>,
        session: Bound<'py, PyAny>,
    ) -> PyResult<Bound<'py, PyCapsule>> {
        let provider = Arc::new(self.clone());
        let codec = ffi_logical_codec_from_pycapsule(session, None)?;
        let provider = FFI_TableProvider::new_with_ffi_codec(provider, false, None, codec);

        PyCapsule::new_with_value(py, provider, cr"datafusion_table_provider")
    }
}

Once you have this library available, you can construct a {py:class}~datafusion.Table in Python and register it with the SessionContext.

from datafusion import SessionContext, Table

ctx = SessionContext()
provider = MyTableProvider()

ctx.register_table("capsule_table", provider)

ctx.table("capsule_table").show()