)]}'
{
  "commit": "afaeccbf27623e983bba594f466024d1f52c5a0f",
  "tree": "88bff2377b13d0acef907ac81871225f7e27fcf1",
  "parents": [
    "8ba06e4147122962f356e11b73175379473f75a7"
  ],
  "author": {
    "name": "Tim Saucer",
    "email": "timsaucer@gmail.com",
    "time": "Tue May 19 09:31:30 2026 -0400"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Tue May 19 09:31:30 2026 -0400"
  },
  "message": "feat: enable pickling of most Expr except udaf and udwf (#1544)\n\n* feat: pickle support for Expr via inline scalar UDF encoding\n\nAdds Python-aware encoding to PythonLogicalCodec/PythonPhysicalCodec\nso a ScalarUDF defined in Python travels inside the serialized\nexpression (cloudpickled into fun_definition) instead of needing a\nmatching registration on the receiver. With that in place, Expr gains\n__reduce__ + classmethod from_bytes(buf, ctx\u003dNone) so pickle.dumps /\npickle.loads work end-to-end on expressions built from col, lit,\nbuilt-in functions, and Python scalar UDFs.\n\nWire format is framed as \u003cDFPYUDF magic, version byte, cloudpickle\ntuple\u003e; the version byte lets a too-new/too-old payload surface a\nclean Execution error instead of an opaque cloudpickle unpack\nfailure. Schema serde is via arrow-rs\u0027s native IPC (no pyarrow\nround-trip). Cloudpickle module handle is cached per-interpreter\nthrough PyOnceLock.\n\nWorker-side context resolution lives in a new datafusion.ipc module:\nset_worker_ctx / get_worker_ctx / clear_worker_ctx plus a private\n_resolve_ctx helper consulted by Expr.from_bytes. Priority is\nexplicit ctx \u003e worker ctx \u003e global SessionContext. FFI UDFs still\ntravel by name and require the matching registration on the\nreceiver\u0027s context.\n\nAggregate and window UDF inline encoding, the per-session\nwith_python_udf_inlining toggle, sender-side context, and the\nuser-guide docs land in follow-on PRs.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(pickle): add cloudpickle security warnings, docstring examples, edge-case tests\n\nInline `.. warning::` blocks on `Expr.to_bytes`, `Expr.from_bytes`, and\n`Expr.__reduce__` so the cloudpickle / arbitrary-code-execution caveat is\nvisible at the public API surface in advance of the user-guide page that\nlands in PR 4.\n\nAdd doctest-style `Examples:` blocks to `datafusion.ipc` functions\n(`set_worker_ctx`, `clear_worker_ctx`, `get_worker_ctx`, `_resolve_ctx`),\n`ScalarUDF.name`, and the new `Expr` pickle methods, per CLAUDE.md.\n\nTighten `Expr.__reduce__` return annotation to\n`tuple[Callable[[bytes], Expr], tuple[bytes]]`.\n\nTests: multi-arg UDF round-trip (covers synthetic `arg_{i}` schema-field\nloop in the codec) plus malformed-bytes paths through `Expr.from_bytes`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* as_any no longer in api\n\n* feat(pickle): stamp Python (major, minor) in UDF wire header\n\ncloudpickle bytecode is not portable across Python minor versions —\na payload produced on 3.11 fails to load on 3.12 with an opaque\nmarshal/unpickle error. Embed the sender\u0027s (major, minor) in the\nDFPYUDF wire header and reject mismatches at decode time with an\nactionable error that names both versions, instead of letting the\nfailure surface from inside cloudpickle.loads.\n\nHeader layout becomes:\n  DFPYUDF (7) | version (1) | py_major (1) | py_minor (1) | cloudpickle\n\nExtend the Security warnings on Expr.to_bytes / from_bytes /\n__reduce__ with a Portability section covering the cross-version\nconstraint and cloudpickle\u0027s by-value/by-reference behavior (the\ncallable inlines bytecode and closure cells, but imported names\ntravel by reference and must be importable on the receiver). Add\na matching Serialization model note to the datafusion.ipc module\ndocstring.\n\nNew tests:\n  - codec::wire_header_tests: py-major/minor mismatch, truncated\n    py-version bytes, round-trip with py-version\n  - test_pickle_expr::test_cross_version_error_message: patches the\n    py_minor byte inside an emitted payload and asserts the error\n    message identifies the version mismatch\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "088532df2b39f8dfc9eeb787c271ddd76afc3f7b",
      "old_mode": 33188,
      "old_path": "crates/core/src/codec.rs",
      "new_id": "cc038edc959ac8691fdbdc3bf8db8b3936f44b5b",
      "new_mode": 33188,
      "new_path": "crates/core/src/codec.rs"
    },
    {
      "type": "modify",
      "old_id": "d48bc729c258a06cb1edb816338e8174cab12da7",
      "old_mode": 33188,
      "old_path": "crates/core/src/udf.rs",
      "new_id": "2006401db33e64d7124c1ee17b028b8485fbe55b",
      "new_mode": 33188,
      "new_path": "crates/core/src/udf.rs"
    },
    {
      "type": "modify",
      "old_id": "951f7adc3415b348d92ff2831110271904742151",
      "old_mode": 33188,
      "old_path": "pyproject.toml",
      "new_id": "a02f4608abc4cb9d1b789455cb63c65712c6d28a",
      "new_mode": 33188,
      "new_path": "pyproject.toml"
    },
    {
      "type": "modify",
      "old_id": "601419fab021c7805d7f658f949003b6f49bb4d8",
      "old_mode": 33188,
      "old_path": "python/datafusion/__init__.py",
      "new_id": "f8d523e0d5cca1896f73bc19c89bdae3532276b0",
      "new_mode": 33188,
      "new_path": "python/datafusion/__init__.py"
    },
    {
      "type": "modify",
      "old_id": "55cc2e52a7669a029535be8a85bb577ec7b71b80",
      "old_mode": 33188,
      "old_path": "python/datafusion/expr.py",
      "new_id": "645bd9c185c2e5a2066714d51d5159cc8f0f8f06",
      "new_mode": 33188,
      "new_path": "python/datafusion/expr.py"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "78b6873f7a5853221685f1cd7b81bae76750d9da",
      "new_mode": 33188,
      "new_path": "python/datafusion/ipc.py"
    },
    {
      "type": "modify",
      "old_id": "848ab4cee8cfa428fffff6b4b3a2262a5f4eb089",
      "old_mode": 33188,
      "old_path": "python/datafusion/user_defined.py",
      "new_id": "d79cf22e8e851ec4ab35cd91361da349ca7cee9c",
      "new_mode": 33188,
      "new_path": "python/datafusion/user_defined.py"
    },
    {
      "type": "modify",
      "old_id": "cf1fe43e87a72f736d44c32219c72b03e51f1628",
      "old_mode": 33188,
      "old_path": "python/tests/test_expr.py",
      "new_id": "485d69624c0abd5386c987d43a30866fcf4136a9",
      "new_mode": 33188,
      "new_path": "python/tests/test_expr.py"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "5d8d9285f6943f1e166a015b7898925e84dd0b86",
      "new_mode": 33188,
      "new_path": "python/tests/test_pickle_expr.py"
    },
    {
      "type": "modify",
      "old_id": "3b7135e32c561fbd771e911f080a43bc2a1cd003",
      "old_mode": 33188,
      "old_path": "uv.lock",
      "new_id": "3fd3eec4b87fffa7e45de1ff3fecd8a65d881ae5",
      "new_mode": 33188,
      "new_path": "uv.lock"
    }
  ]
}