)]}' { "commit": "c0ac93b68fff606aed570aed09d72d749ba79bae", "tree": "46d37d40b06b7436b4a3cfc0ce41a86b2fba0e73", "parents": [ "fd15b03ecc227957b752429bf5b7c56c0f139f1e" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Mon Jun 15 16:23:05 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jun 15 16:23:05 2026 -0400" }, "message": "feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata (#1568)\n\n* feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata\n\nAdds Python bindings for five scalar functions from\ndatafusion::functions::expr_fn that were not previously surfaced:\n\n- arrow_field: returns a struct describing an expression\u0027s Arrow field\n (name, data_type, nullable, metadata).\n- arrow_try_cast: like arrow_cast but yields NULL on cast failure.\n- cast_to_type / try_cast_to_type: casts a value to the type of a\n reference expression. These are exposed as a single Python entry\n point cast_to_type(value, type_ref, *, try_cast\u003dFalse); the kwarg\n switches between the strict and try variants.\n- with_metadata: attach Arrow field metadata; the inverse of\n arrow_metadata. Accepts a dict[str, str] for ergonomics.\n\nUpdates skills/datafusion_python/SKILL.md to list the new functions\nand documents the cast_to_type kwarg behavior.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: collapse try_cast_to_type into cast_to_type kwarg\n\nThe previous commit exposed cast_to_type and try_cast_to_type as two\nseparate pyo3 bindings and unified them in the Python wrapper via a\ntry_cast kwarg. That left try_cast_to_type in datafusion._internal\nwithout a matching public Python name, breaking\ntest_datafusion_missing_exports.\n\nMove the dispatch into the rust binding: cast_to_type now takes a\ntry_cast kwarg and selects between functions::expr_fn::cast_to_type\nand try_cast_to_type internally. Only one pyo3 binding is registered,\nso the wrapper-coverage check passes and the Python entrypoint is\nunchanged.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: accept pyarrow DataType in arrow_try_cast\n\nMirrors arrow_cast: arrow_try_cast now accepts `pa.DataType` in addition\nto `str` and `Expr`. Adds `Expr.try_cast(pa.DataType)` PyO3 binding for\nthe pyarrow-type routing path.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: guard with_metadata against empty dict and empty keys\n\nEmpty `metadata` dict now returns the input expression unchanged\n(previously bubbled an opaque DataFusion error about minimum arg\ncount). Empty keys raise `ValueError` to match the docstring contract.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: assert full struct shape in arrow_field doctest\n\nPrevious doctest set metadata on the input field but only checked the\nname — the metadata setup was dead. Now the example asserts the full\nreturned struct (name, data_type, nullable, metadata) so the demo\nshows what the function actually produces.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: add unit tests for arrow_try_cast, arrow_field, cast_to_type, with_metadata\n\nMirrors the existing test_arrow_cast pattern. Covers:\n- arrow_try_cast: string-syntax, pa.DataType, and null-on-failure paths\n- arrow_field: full returned struct shape (name, data_type, nullable, metadata)\n- cast_to_type: type-from-expr happy path and try_cast\u003dTrue null behavior\n- with_metadata: round-trip through arrow_metadata, empty-dict no-op, and\n empty-key ValueError\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: parameterize arrow cast / try_cast tests\n\nFolds the previous four cast tests (arrow_cast + arrow_try_cast × str\n+ pyarrow target type) into a single parameterized test that runs both\nfunctions across all five target-type variants. Collapses the two\ncast_to_type tests (happy path + try_cast\u003dTrue) into one parameterized\ntest, and parameterizes arrow_try_cast null-on-failure over both\ntarget-type syntaxes. 7 test functions, 19 cases — net less code, same\ncoverage.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: point cast_to_type at arrow_cast for static target types\n\nAdds a one-line cross-reference so users with a known target type\nreach for arrow_cast / arrow_try_cast instead of building a sentinel\nexpression to feed cast_to_type.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: split cast_to_type into cast_to_type and try_cast_to_type\n\nReplace the try_cast bool flag with separate cast_to_type and\ntry_cast_to_type functions, matching upstream DataFusion and the\narrow_cast / arrow_try_cast pair. Also drop the redundant data_type\nparametrization on test_arrow_try_cast_null_on_failure, since the\nstr-vs-pyarrow distinction is already covered by test_arrow_cast_variants.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e", "tree_diff": [ { "type": "modify", "old_id": "eac571a11b034fabb9eb7fc7ff2c73905cdeecda", "old_mode": 33188, "old_path": "crates/core/src/expr.rs", "new_id": "432c4cd2317deacd1a0e3e132b68346735c951de", "new_mode": 33188, "new_path": "crates/core/src/expr.rs" }, { "type": "modify", "old_id": "a56873c582dc00aafe2dd10afb0c9ec07f035071", "old_mode": 33188, "old_path": "crates/core/src/functions.rs", "new_id": "69d10559b6e0f626b779c400ccaf8bea62ebc9a8", "new_mode": 33188, "new_path": "crates/core/src/functions.rs" }, { "type": "modify", "old_id": "4fdbdc5d449dfce735f83461c1bafc5159b86691", "old_mode": 33188, "old_path": "python/datafusion/expr.py", "new_id": "be3a141232affa77347706a70e4c158d55c10169", "new_mode": 33188, "new_path": "python/datafusion/expr.py" }, { "type": "modify", "old_id": "4e08fa1d95efe29255dcc72497804a8821b4df2b", "old_mode": 33188, "old_path": "python/datafusion/functions.py", "new_id": "f4eef763a36547fb19b4d2e0efee590f9f44ba1c", "new_mode": 33188, "new_path": "python/datafusion/functions.py" }, { "type": "modify", "old_id": "b02cdf6dbaeafd80000e37347bb80810235890fe", "old_mode": 33188, "old_path": "python/tests/test_functions.py", "new_id": "73a7ad727a87564a516bebb92b456b58df436a4c", "new_mode": 33188, "new_path": "python/tests/test_functions.py" }, { "type": "modify", "old_id": "1aeb78777fecc4b10de269a4a079f54b3f79868c", "old_mode": 33188, "old_path": "skills/datafusion_python/SKILL.md", "new_id": "4ddf54e99dee8781173830e3ae19e2a9c81aee2e", "new_mode": 33188, "new_path": "skills/datafusion_python/SKILL.md" } ] }