This crate is the provider library in the three-library query-planning example. It exports table providers, functions, and the logical and physical codecs needed to serialize objects owned by this library. The companion planner is in ../datafusion-ffi-query-planner-example.
The example intentionally uses separate cdylib crates for these roles:
datafusion-python: owns the SessionContext and executes the result.Separate shared libraries guarantee distinct DataFusion library markers. This catches type-identity mistakes that a planner and provider compiled into one shared library would hide.
MyLogicalExtensionCodec serializes this example's in-memory table providers, and MyPhysicalExtensionCodec serializes provider-owned memory scans and opaque FFI wrappers around them. Both use documented, process-local, one-shot token registries. The registries make ownership and callback routing visible without pretending to be a portable format. They assume trusted in-process payloads and consume each token during decoding. A production provider should instead encode durable metadata from which its provider and plans can be reconstructed.
Both codec getters take the SessionContext they are being installed on and pull the TaskContextProvider off it, so decode callbacks resolve session configuration and registered functions against the session that is running the query. Passing require_udf_on_decode to either constructor makes every decode call resolve a named scalar function out of that context, which is how the tests check where the registry came from.
Extension codecs compose: each with_logical_extension_codec / with_physical_extension_codec call appends the codec to the session‘s codec chain, and DataFusion’s default codec handles whatever no installed codec claims. Every payload a codec writes is wrapped in an envelope naming that codec, and decoding consults exactly the codec that encoded it — so several independent plugin libraries can install codecs on the same session without any of them having to recognise or reject the others' payloads. The codecs here are written as ordinary LogicalExtensionCodec / PhysicalExtensionCodec implementations; the envelope is applied and stripped by datafusion-python and never reaches them.
MyLogicalExtensionCodec takes an optional provider_prefix argument (MyLogicalExtensionCodec(provider_prefix="TOKENAAA")) that overrides the byte prefix it stamps on encoded table providers. It exists so the tests can install two instances that own disjoint slices of the wire format, which is what makes install ordering observable from Python. Two instances of one class derive the same id, so those tests also pass codec_id= to tell them apart. Real plugin libraries should hard-code a prefix unique to the library rather than accept one from the caller.
NameOnlyUdfCodec is the opposite shape: it owns functions that are fully described by their names, so it encodes no bytes at all and rebuilds each function from the name on decode. It exists to pin the by-name path, which is the one place a payload has no id to dispatch on.
Register both provider codecs before installing the planner:
ctx = ctx.with_logical_extension_codec(provider_logical_codec) ctx = ctx.with_physical_extension_codec(provider_physical_codec) ctx.set_query_planner(planner)
Installing a codec after the planner rebuilds the planner against it, so this order is a recommendation rather than a requirement. Planner-last states the ownership flow more clearly. The exception is a planner that wraps a fallback: the rebuild reaches the installed planner only, not the fallback inside it, so codecs-first is a requirement there. See Rebinding a planner's codecs is one level deep, which also covers why re-installing a planner rebinds the session to the codecs of whichever handle it was installed on.
For the limits behind that choice — how the codec chain dispatches, which node kinds survive the boundary, and what a derived context shares with the context it came from — see Query Planners Across Multiple Libraries in the contributor guide.