)]}'
{
  "commit": "dac9ec6230dba8717d8a0d27de19a141600486b1",
  "tree": "2c4c4327e94248d28aeb205cfb44d8596b0849d7",
  "parents": [
    "afaeccbf27623e983bba594f466024d1f52c5a0f"
  ],
  "author": {
    "name": "Tim Saucer",
    "email": "timsaucer@gmail.com",
    "time": "Wed May 20 18:58:32 2026 -0400"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Wed May 20 18:58:32 2026 -0400"
  },
  "message": "feat: enable pickling for Python aggregate and window UDFs (#1545)\n\n* feat: inline encoding for Python aggregate and window UDFs\n\nExtends the PythonLogicalCodec / PythonPhysicalCodec inline encoding\nintroduced for scalar UDFs to also cover Python-defined aggregate and\nwindow UDFs. The cloudpickle tuple shape per family is:\n\n  DFPYUDA  (agg)     (name, accumulator_factory, input_schema_bytes,\n                      return_schema_bytes, state_schema_bytes,\n                      volatility_str)\n  DFPYUDW  (window)  (name, evaluator_factory, input_schema_bytes,\n                      return_schema_bytes, volatility_str)\n\nSame wire-framing as scalar (family magic + version byte + cloudpickle\nblob), same schema serde (arrow-rs native IPC), same cached cloudpickle\nhandle. The agg state schema is encoded as a full IPC schema so the\npost-decode UDF reports the same names + nullability + metadata as the\nsender — relevant for accumulators whose StateFieldsArgs consumers key\noff names rather than positional DataType.\n\nRequired restructuring two existing UDF impls so the codec can grab\nthe Python callable directly:\n\n* udaf.rs: replaces create_udaf + AccumulatorFactoryFunction closure\n  with a named PythonFunctionAggregateUDF that stores the Py\u003cPyAny\u003e\n  accumulator factory. Synthesizes state_{i} field names when the\n  Python constructor passes only Vec\u003cDataType\u003e; from_parts preserves\n  the full state schema on the decode side.\n* udwf.rs: renames MultiColumnWindowUDF -\u003e PythonFunctionWindowUDF,\n  drops the PartitionEvaluatorFactory PtrEq wrapper, stores the\n  Py\u003cPyAny\u003e evaluator directly. PartialEq and Hash get the same\n  pointer-identity fast path + debug-log exception handling already\n  on PythonFunctionScalarUDF.\n\nUser-facing surface:\n\n* AggregateUDF.name and WindowUDF.name properties (parallel to the\n  ScalarUDF.name shipped in PR1).\n* Existing UDAF/UDWF construction paths are unchanged.\n\nThe per-session with_python_udf_inlining toggle, sender-side context,\nstrict refusal, and user-guide docs land in PRs 3-4 of this series.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat: restore pub UDAF/UDWF helpers and document inline encoding\n\nRe-export `to_rust_accumulator`, `to_rust_partition_evaluator`, and\n`PythonFunctionWindowUDF` (with a `MultiColumnWindowUDF` alias) by\npromoting `udaf` and `udwf` to `pub mod` so prior downstream Rust\nconsumers keep their API surface after the inline-encoding refactor.\n\nAdds an end-to-end window UDF pickle round-trip test that runs the\ndecoded evaluator over a real session, mirroring the aggregate test.\n\nDocuments the cloudpickle-based shipping behavior of Python aggregate\nand window UDFs in the user-guide aggregations and windows pages.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: address PR #1545 review feedback\n\n- Fix CountAcc.merge in pickle test: sum over states[0] (partition\n  counts), not over the list of state fields. The prior implementation\n  only added partition 0\u0027s count when merging across partitions.\n- Drive test_agg_udf_evaluates_after_roundtrip with a two-batch\n  DataFrame so merge actually runs and the round-tripped state-field\n  schema is exercised end-to-end.\n- Correct PY_AGG_UDF_FAMILY / PY_WINDOW_UDF_FAMILY doc comments and the\n  aggregate block comment to reference \"return schema bytes\" rather\n  than \"return type\" / \"return_type_bytes\" so the docs match the actual\n  on-wire layout.\n- Keep `udaf` and `udwf` modules private (matching `udf`) and\n  selectively re-export the helpers downstream Rust consumers rely on\n  (`to_rust_accumulator`, `to_rust_partition_evaluator`,\n  `PythonFunctionWindowUDF`, `MultiColumnWindowUDF`) instead of\n  exposing the whole module surface.\n- Rename codec helpers `*_agg_udf` -\u003e `*_udaf` and `*_window_udf` -\u003e\n  `*_udwf` for naming consistency with the Python public aliases.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "cc038edc959ac8691fdbdc3bf8db8b3936f44b5b",
      "old_mode": 33188,
      "old_path": "crates/core/src/codec.rs",
      "new_id": "363ee82b8c89b9467f4acb52af1c8e7ad84ae341",
      "new_mode": 33188,
      "new_path": "crates/core/src/codec.rs"
    },
    {
      "type": "modify",
      "old_id": "e3551c937e1fc9aa7e7e64a003256cff850e29e7",
      "old_mode": 33188,
      "old_path": "crates/core/src/lib.rs",
      "new_id": "8b622d344ee6c2aeaffb26fb74b3c24e07443615",
      "new_mode": 33188,
      "new_path": "crates/core/src/lib.rs"
    },
    {
      "type": "modify",
      "old_id": "80ef5171650fcffea144ae573e58dc4c75dcf9f3",
      "old_mode": 33188,
      "old_path": "crates/core/src/udaf.rs",
      "new_id": "caf7b97bcc7d6062ad7c8807d67b16ba133032df",
      "new_mode": 33188,
      "new_path": "crates/core/src/udaf.rs"
    },
    {
      "type": "modify",
      "old_id": "40e6208c4fedbede29ce72abdb8682b1127a93f4",
      "old_mode": 33188,
      "old_path": "crates/core/src/udwf.rs",
      "new_id": "ebec8f3bd4b1922096c274cd764560175e2f49c8",
      "new_mode": 33188,
      "new_path": "crates/core/src/udwf.rs"
    },
    {
      "type": "modify",
      "old_id": "f59b62ab4185a7c2a2df70cb5157643d105aa43c",
      "old_mode": 33188,
      "old_path": "docs/source/user-guide/common-operations/aggregations.rst",
      "new_id": "8f218abd8a31b010a9a886d57faa41c8064aa3c7",
      "new_mode": 33188,
      "new_path": "docs/source/user-guide/common-operations/aggregations.rst"
    },
    {
      "type": "modify",
      "old_id": "d77881bcfd876f54342f14c118747b762cba1672",
      "old_mode": 33188,
      "old_path": "docs/source/user-guide/common-operations/windows.rst",
      "new_id": "127f691b561c1bd05969b8e4aacb0520b365733c",
      "new_mode": 33188,
      "new_path": "docs/source/user-guide/common-operations/windows.rst"
    },
    {
      "type": "modify",
      "old_id": "645bd9c185c2e5a2066714d51d5159cc8f0f8f06",
      "old_mode": 33188,
      "old_path": "python/datafusion/expr.py",
      "new_id": "7e95bc127f7a559db6a4da15d121dde02cf7d479",
      "new_mode": 33188,
      "new_path": "python/datafusion/expr.py"
    },
    {
      "type": "modify",
      "old_id": "78b6873f7a5853221685f1cd7b81bae76750d9da",
      "old_mode": 33188,
      "old_path": "python/datafusion/ipc.py",
      "new_id": "8dd7fc4633d8c08db926cc4b81bd385d52b6eb5b",
      "new_mode": 33188,
      "new_path": "python/datafusion/ipc.py"
    },
    {
      "type": "modify",
      "old_id": "d79cf22e8e851ec4ab35cd91361da349ca7cee9c",
      "old_mode": 33188,
      "old_path": "python/datafusion/user_defined.py",
      "new_id": "3eb50a09490230de783f9a381197000600ce5fe9",
      "new_mode": 33188,
      "new_path": "python/datafusion/user_defined.py"
    },
    {
      "type": "modify",
      "old_id": "5d8d9285f6943f1e166a015b7898925e84dd0b86",
      "old_mode": 33188,
      "old_path": "python/tests/test_pickle_expr.py",
      "new_id": "eb0441c498e5fca2f8dcbbacb7e616abff20c68b",
      "new_mode": 33188,
      "new_path": "python/tests/test_pickle_expr.py"
    }
  ]
}