)]}' { "commit": "52932128d353e417ddae2c5ff3f14135cb806f7e", "tree": "b88ea32d565a6b1708831f95a42034c7028a50e4", "parents": [ "898d73de20346bba7241907bb18cba47da53e9a9" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Tue Apr 07 14:58:09 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Apr 07 14:58:09 2026 -0400" }, "message": "Add missing Dataframe functions (#1472)\n\n* Add missing DataFrame methods for set operations and query\n\nExpose upstream DataFusion DataFrame methods that were not yet\navailable in the Python API. Closes #1455.\n\nSet operations:\n- except_distinct: set difference with deduplication\n- intersect_distinct: set intersection with deduplication\n- union_by_name: union matching columns by name instead of position\n- union_by_name_distinct: union by name with deduplication\n\nQuery:\n- distinct_on: deduplicate rows based on specific columns\n- sort_by: sort by expressions with ascending order and nulls last\n\nNote: show_limit is already covered by the existing show(num) method.\nexplain_with_options and with_param_values are deferred as they require\nexposing additional types (ExplainOption, ParamValues).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add ExplainFormat enum and format option to DataFrame.explain()\n\nExtend the existing explain() method with an optional format parameter\ninstead of adding a separate explain_with_options() method. This keeps\nthe API simple while exposing all upstream ExplainOption functionality.\n\nAvailable formats: indent (default), tree, pgjson, graphviz.\n\nThe ExplainFormat enum is exported from the top-level datafusion module.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add DataFrame.window() and unnest recursion options\n\nExpose remaining DataFrame methods from upstream DataFusion.\nCloses #1456.\n\n- window(*exprs): apply window function expressions and append results\n as new columns\n- unnest_column/unnest_columns: add optional recursions parameter for\n controlling unnest depth via (input_column, output_column, depth)\n tuples\n\nNote: drop_columns is already exposed as the existing drop() method.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Update docstring\n\nCo-authored-by: Copilot \u003c175728472+Copilot@users.noreply.github.com\u003e\n\n* Improve docstrings and test robustness for new DataFrame methods\n\nClarify except_distinct/intersect_distinct docstrings, add deterministic\nsort to test_window, add sort_by ascending verification test, and add\nsmoke tests for PGJSON and GRAPHVIZ explain formats.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate new DataFrame tests into parametrized tests\n\nCombine set operation tests (except_distinct, intersect_distinct,\nunion_by_name, union_by_name_distinct) into a single parametrized\ntest_set_operations_distinct. Merge sort_by tests and convert\nexplain format tests to parametrized form.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add doctest examples to new DataFrame method docstrings\n\nAdd \u003e\u003e\u003e style usage examples for window, explain, except_distinct,\nintersect_distinct, union_by_name, union_by_name_distinct, distinct_on,\nsort_by, and unnest_columns to match existing docstring conventions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Improve error messages, tests, and API hygiene from PR review\n\n- Provide actionable error message for invalid explain format strings\n- Remove recursions param from deprecated unnest_column (use unnest_columns)\n- Add null-handling test case for sort_by to verify nulls-last behavior\n- Add format-specific assertions to explain tests (TREE, PGJSON, GRAPHVIZ)\n- Add deep recursion test for unnest_columns with depth \u003e 1\n- Add multi-expression window test to verify variadic *exprs\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Consolidate window and unnest tests into parametrized tests\n\nCombine test_window and test_window_multiple_expressions into a single\nparametrized test. Merge unnest recursion tests into one parametrized\ntest covering basic, explicit depth 1, and deep recursion cases.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Address PR review feedback for DataFrame operations\n\n- Use upstream parse error for explain format instead of hardcoded options\n- Fix sort_by to use column name resolution consistent with sort()\n- Use ExplainFormat enum members directly in tests instead of string lookup\n- Merge union_by_name_distinct into union_by_name(distinct\u003dFalse) for a\n more Pythonic API\n- Update check-upstream skill to note union_by_name_distinct coverage\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Add DataFrame.column(), col(), and find_qualified_columns() methods\n\nExpose upstream find_qualified_columns to resolve unqualified column\nnames into fully qualified column expressions. This is especially\nuseful for disambiguating columns after joins.\n\n- find_qualified_columns(*names) on Rust side calls upstream directly\n- DataFrame.column(name) and col(name) alias on Python side\n- Update join and join_on docstrings to reference DataFrame.col()\n- Add \"Disambiguating Columns with DataFrame.col()\" section to joins docs\n- Add tests for qualified column resolution, ambiguity, and join usage\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* Merge union_by_name and union_by_name_distinct into a single method with distinct flag\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* converting into a python dict loses a column when the names are identical\n\n* Consolidate except_all/except_distinct and intersect/intersect_distinct into single methods with distinct flag\n\nFollows the same pattern as union(distinct\u003d) and union_by_name(distinct\u003d).\nAlso deprecates union_distinct() in favor of union(distinct\u003dTrue).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.6 (1M context) \u003cnoreply@anthropic.com\u003e\nCo-authored-by: Copilot \u003c175728472+Copilot@users.noreply.github.com\u003e", "tree_diff": [ { "type": "modify", "old_id": "f77210371bc74511c445c007ce7b492de5c0d368", "old_mode": 33188, "old_path": ".ai/skills/check-upstream/SKILL.md", "new_id": "ac4835a4e699e55bbbcdb8f596ce5d9f4f5718ce", "new_mode": 33188, "new_path": ".ai/skills/check-upstream/SKILL.md" }, { "type": "modify", "old_id": "72595ba81cb1989e3736d8aac851092cb3cdab03", "old_mode": 33188, "old_path": "crates/core/src/dataframe.rs", "new_id": "fff5118d5ad6921656636cf9b669c9619a926c6f", "new_mode": 33188, "new_path": "crates/core/src/dataframe.rs" }, { "type": "modify", "old_id": "1d9d7038526aa4baf9fbc1963bef7fae2e72cfca", "old_mode": 33188, "old_path": "docs/source/user-guide/common-operations/joins.rst", "new_id": "a289c93773965121b02d71dedf24f0d10c45f047", "new_mode": 33188, "new_path": "docs/source/user-guide/common-operations/joins.rst" }, { "type": "modify", "old_id": "2e6f8116645a4db1e941ac22e90f4107fb855b95", "old_mode": 33188, "old_path": "python/datafusion/__init__.py", "new_id": "a736c3966501a7e4434f9b5c6206a87719c03e94", "new_mode": 33188, "new_path": "python/datafusion/__init__.py" }, { "type": "modify", "old_id": "9907eae8bb0d82ad443f256130104acb77967c3c", "old_mode": 33188, "old_path": "python/datafusion/dataframe.py", "new_id": "9dc5f0e7de0d92080731b09d43b5875b19cef007", "new_mode": 33188, "new_path": "python/datafusion/dataframe.py" }, { "type": "modify", "old_id": "759d6278c06b49d466e4894607c5b0514cff82df", "old_mode": 33188, "old_path": "python/tests/test_dataframe.py", "new_id": "bb8e9685ccc119c161c94d76f0db71c46adfa850", "new_mode": 33188, "new_path": "python/tests/test_dataframe.py" } ] }