This crate is an independent query-planner Python extension. Together with ../datafusion-ffi-example it demonstrates a real three-library plan exchange:
datafusion-python: owns the session and final execution.datafusion-ffi-example: owns a table provider, UDF, and provider codecs.Two extension crates are used rather than placing the planner in the provider crate. Loading distinct cdylib images gives each library a distinct DataFusion marker and proves that foreign sessions, providers, and plans survive the actual ABI boundary.
From the repository root, build and install all three extensions, then run the integration tests:
maturin develop --uv uv run maturin develop --manifest-path examples/datafusion-ffi-example/Cargo.toml uv run maturin develop \ --manifest-path examples/datafusion-ffi-query-planner-example/Cargo.toml uv run pytest \ examples/datafusion-ffi-query-planner-example/python/tests/_test*.py
The integration test follows this setup:
config = SessionConfig().with_extension(MyPlannerConfig(max_rows=3)) ctx = SessionContext(config) ctx = ctx.with_logical_extension_codec(provider_logical_codec) ctx = ctx.with_physical_extension_codec(provider_physical_codec) ctx.register_table("numbers", provider) ctx.register_udf(provider_udf) ctx.set_query_planner(MyQueryPlanner())
MyPlannerConfig is transferred through the foreign session. MyQueryPlanner reads ffi_query_planner.max_rows, creates the plan with DefaultPhysicalPlanner, and adds a built-in GlobalLimitExec. The test changes the setting with SET and verifies the new row limit.
The provider‘s codec chain is attached to the planner when it is installed and is also used to decode the returned physical plan in datafusion-python. Extension codecs compose: each with_logical_extension_codec / with_physical_extension_codec call appends to the session’s codec chain, and each payload records which codec wrote it, so several libraries can install codecs on the same session and the order between them does not affect decoding. This planner owns no serializable types of its own and deliberately uses only built-in physical nodes. Install the codecs before the planner where possible; installing a codec afterwards rebuilds the planner against the new chain, but planner-last order is easier to audit. That rebuild is one level deep — a planner constructed with fallback= keeps the codecs its fallback was imported with — so codecs-first is a requirement rather than a preference once planners are layered. See Rebinding a planner's codecs is one level deep.
For the limits behind that choice — how the codec chain dispatches, which node kinds survive the boundary, and what a derived context shares with the context it came from — see Query Planners Across Multiple Libraries in the contributor guide.