)]}' { "commit": "2d074713d1505256ed92a07e77c18ee43713a303", "tree": "7e54074583f8eed5c34e746a9c7d88a645fe2889", "parents": [ "55fd2c925c95711ad7222a5a48b6691d482d6056" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Wed Jun 17 13:03:04 2026 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Jun 17 13:03:04 2026 -0400" }, "message": "feat: expose spark-compatible functions (#1564)\n\n* feat: expose Spark-compatible functions (#1482)\n\nAdd `datafusion.functions.spark` module exposing the upstream\n`datafusion-spark` crate\u0027s UDF/UDAF library (~87 functions across string,\nmath, datetime, hash, array, aggregate, bitwise, bitmap, conditional,\ncollection, conversion, json, map, url categories).\n\nFor DataFrame use, import the typed Python wrappers from\n`datafusion.functions.spark`. For SQL use, call\n`SessionContext.enable_spark_functions()` to register the Spark UDFs by\nname (overriding DataFusion built-ins of the same name with their Spark\nsemantics — NULL-propagating `concat`, 1-indexed `substring`, HALF_UP\n`round`, etc.).\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* chore: drop unused borrow_deref_ref allows\n\nSeven `#[allow(clippy::borrow_deref_ref)]` attributes on module\ndeclarations in `crates/core/src/lib.rs` had become stale — the only\nremaining lint hit was a redundant `\u0026*x.as_str()` pattern in\n`parse_file_compression_type`. Rewriting that call to\n`\u0026x.unwrap_or_default()` lets every allow come off, removing noise that\nnew modules were copying without need.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor: tighten spark_functions macros via expr_fn\n\nSwitch most spark wrappers from UDF-direct path (which forced\n`spark_udf_fixed!(name, fn_category::name, args...)` repetition) to a\n`spark_expr_fn!` macro that mirrors the existing `expr_fn!` macro in\n`functions.rs`, so calls collapse to `spark_expr_fn!(sha2, arg1\nbit_length);`.\n\nUDF-direct retained for genuinely variadic functions whose upstream\n`expr_fn` wrappers were generated with a single-`Expr` arm by\n`export_functions!` (concat, array, xxhash64, parse_url family, etc.) so\nthat the Python side keeps its `*args` ergonomics.\n\nAggregates collapse the same way via `spark_aggregate!` mirroring\n`aggregate_function!`. Net 173 lines removed.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: clarify spark functions cover DataFrame API too\n\nThe intro wording implied \"SQL functions\" only; the same wrappers are the\nprimary entry point for the DataFrame API as well.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: rewrite spark DataFrame intro for users\n\nReplace API-speak (\"Import the submodule\", \"Returned values are Expr\ninstances that compose\") with a concrete description of where users can\nactually drop these calls.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs: defer spark function list to API reference\n\nHand-maintained category list would drift from the actual module as\nupstream `datafusion-spark` adds/removes functions. Replace with a\npointer to the AutoAPI-generated reference, which renders from the\nmodule itself.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* test: replace spark doctest skips with verified examples\n\n38 wrappers carried `# doctest: +SKIP` because outputs weren\u0027t verified at\nauthoring time. Run each with concrete inputs, capture actual outputs, and\ninline the values so the doctests execute and stay correct.\n\nCovers datetime (20), URL (5), bitmap (3), map (3), and remaining hash,\nJSON, math, string, conversion, and format_string cases. Net new doctest\ncoverage: 65 examples now run that were skipped before; total skipped\nacross the suite drops from 53 to 12.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(spark): rename function params to match pyspark\n\nAlign positional parameter names in `functions.spark` with pyspark.sql.functions:\n- aggregate first positional → `col` (avg, try_sum, collect_list, collect_set)\n- unary `arg` → `col` across math/string/byte/datetime helpers\n- multi-arg renames: array_contains (col, value), array (*cols), shuffle (col),\n array_repeat (col, count), slice (x, start, length), shiftleft/right/rightunsigned\n (col, numBits), add_months (start, months), date_add/sub (start, days),\n date_diff (end, start), date_trunc (format, timestamp), time_trunc (unit, time),\n trunc (date, format), next_day (date, dayOfWeek), from/to_utc_timestamp\n (timestamp, tz), sha2 (col, numBits), xxhash64 (*cols), map_from_arrays\n (col1, col2), width_bucket (v, min, max, numBucket), substring (str, pos, len),\n concat (*cols), elt (*inputs), is_valid_utf8/make_valid_utf8 (str)\n\nBodies updated to reference the new names; positional callers unaffected.\nThis finishes Category 1 / Category 4 (spark-side BOTH-bucket) renames from\nPYSPARK_ALIGNMENT_PLAN.md PR 1.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): make pyspark-optional params optional\n\nMatch pyspark\u0027s optional-parameter surface in the spark namespace:\n- make_dt_interval, make_interval: all parts default to zero (int32 0 / lit 0.0)\n- str_to_map: pair_delim defaults to \u0027,\u0027, key_value_delim defaults to \u0027:\u0027\n- round: scale defaults to 0 (HALF_UP rounding to nearest integer)\n- shuffle: accepts `seed` kwarg for pyspark parity; raises NotImplementedError\n for non-None values until the Rust binding supports it\n- like, ilike: accept `escapeChar` for pyspark parity; same NotImplementedError\n guard; first positional renamed `string` → `str` to match pyspark\n\nceil/floor `scale\u003d` deferred — the underlying Rust expr_fn is single-arg.\n\nAdded a module-level `_ZERO_I32` literal to avoid rebuilding the pyarrow\nint32 zero scalar on every call.\n\nTests: positional-compat coverage for aggregates (`spark.avg(col)` etc.),\ndefaults-omitted cases for the optional-arg functions, and\nNotImplementedError cases for `shuffle(seed\u003d)` and `like/ilike(escapeChar\u003d)`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* refactor(spark): reshape varargs to match pyspark signatures\n\nReplace generic ``*args`` with explicit pyspark-style signatures:\n- json_tuple(col, *fields) — first positional is the JSON expr\n- format_string(format, *cols) — `format` is the printf template; a plain\n ``str`` is auto-promoted to a literal\n- parse_url(url, partToExtract, key\u003dNone) — `key` is optional and only\n meaningful with ``partToExtract\u003d\u0027QUERY\u0027``\n- try_parse_url(url, partToExtract, key\u003dNone) — same shape\n- url_decode(str), try_url_decode(str), url_encode(str) — single-argument\n forms (multi-arg calls were always semantically wrong)\n\nTests cover the three-arg parse_url path and the plain-str format_string\nauto-promotion.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): cover the new spark function namespace\n\n`functions.spark` mirrors `pyspark.sql.functions` and now ships on this\nbranch. Update every skill that references the function surface:\n\n- skills/datafusion_python/SKILL.md (user-facing): add an import\n reference, a Core Abstractions row, and a \"Spark-Compatible Functions\"\n subsection listing coverage by category, the SQL-vs-DataFrame usage\n (`enable_spark_functions`), and the divergent-semantics table\n (concat NULL, round HALF_UP, trunc) so callers know which namespace\n to pick.\n- .ai/skills/check-upstream/SKILL.md: new area for the `datafusion-spark`\n crate with the coverage policy (parity with pyspark, extras allowed\n when positional pyspark calls still work). Hygiene check also now\n spans `functions/spark.py`\u0027s `__all__`.\n- .ai/skills/audit-skill-md/SKILL.md: add `functions.spark` to the\n surface table and a `spark-functions` scope so this audit also\n validates the new subsection and divergent-semantics table.\n- .ai/skills/make-pythonic/SKILL.md: explicit scope note that the\n spark namespace is a deliberate pyspark mirror — generic native-type\n coercion does not apply there. Path references updated to the new\n `functions/__init__.py` module layout.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): drop references to PYSPARK_ALIGNMENT_PLAN.md\n\nThe plan file is a working document, not a committed artifact, so skills\nmust not point at it. Inline the one substantive reference (the\n\"deferred to follow-up PRs\" callout in make-pythonic) and drop the\ncross-cutting pointer from check-upstream.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skills): make-pythonic also targets functions.spark\n\nPrevious guidance said to skip the spark namespace entirely. That was\nwrong: the spark namespace should also feel pythonic — it just carries\nthe extra constraint that every signature must remain compatible with\npyspark.sql.functions (parameter names, positional order, accepted input\ntypes). Pythonic widenings like `Expr → Expr | int` are on-brand there\nbecause pyspark itself accepts the int form.\n\nRewrite the scope section to spell out the compatibility rules (keep\nparameter names/order; widen input types, never narrow; extra kwargs\ndefault to None) and extend \"How to Identify Candidates\" to include\n`functions/spark.py`.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(skill): point at spark __all__ instead of enumerating\n\nEnumerating spark functions in the user-facing skill duplicates the\n__all__ list in python/datafusion/functions/spark.py and will drift the\nmoment a new function lands or is renamed. Replace the per-function\nlisting with a category summary and a discovery snippet that queries\nthe actual __all__ at runtime, which is the authoritative source.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): use isoformat in spark_cast doctest\n\npyarrow tzinfo repr differs across versions (\u003cUTC\u003e vs\nzoneinfo.ZoneInfo(key\u003d\u0027UTC\u0027)), breaking the doctest on some platforms.\nisoformat is stable across versions.\n\nCo-Authored-By: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): fix map_from_entries doctest to use the right function\n\nThe example called map_from_arrays, so it never exercised\nmap_from_entries. Build an array-of-struct input and call the\ndocumented function.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): list Spark-compatible aggregates in aggregations guide\n\nAdd avg, try_sum, collect_list, and collect_set under a dedicated\nSpark-Compatible Functions entry, with a note that the\ndatafusion.functions.spark namespace mirrors Spark semantics and may\ndiffer from the like-named built-ins. Adds a (spark-functions) anchor\nfor the cross-reference.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* docs(spark): align if_ doctest with the single-value accessor style\n\nUse the same lit-based single-row select and [0].as_py() accessor as\nthe other wrappers instead of the lone to_pylist() call.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): accept native Python literals for literal-friendly args\n\nAudit the functions.spark namespace against pyspark.sql.functions and\nwiden arguments that pyspark types as a non-column literal so callers can\npass bare int/float/str instead of wrapping in lit():\n\n- int args: array_repeat count, slice start/length, shiftleft/shiftright/\n shiftrightunsigned numBits, sha2 numBits, round scale, substring pos/len,\n width_bucket numBucket\n- int32-coerced args (binding requires int32): add_months months,\n date_add/date_sub days, space n, make_dt_interval/make_interval parts\n- float args: modulus/pmod operands; make_*_interval secs\n- str args: next_day dayOfWeek, date_trunc/trunc format, date_part field,\n from_utc_timestamp/to_utc_timestamp tz, spark_cast type_str,\n json_tuple *fields\n- Any: array_contains value, if_ if_true/if_false\n\nArguments that pyspark types as ColumnOrName (str means column name, not a\nliteral) are left as Expr to avoid diverging from pyspark semantics:\nilike/like pattern, parse_url partToExtract/key, str_to_map delimiters,\nbit_get pos, time_trunc unit.\n\nAlso rename str_to_map\u0027s delimiter params to pairDelim/keyValueDelim to\nmatch pyspark exactly (they were pair_delim/key_value_delim).\n\nAdd a coercion test matrix and update docstring examples to show the\nnative-literal calling convention.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* feat(spark): accept column-name str for ColumnOrName args\n\nFor arguments that pyspark types as ColumnOrName, a bare str means a\ncolumn name (not a literal). Widen these to Expr | str and resolve a str\nto a column reference via _to_raw_expr, matching pyspark semantics:\n\n- ilike/like pattern\n- parse_url/try_parse_url partToExtract and key\n- str_to_map pairDelim/keyValueDelim\n- bit_get pos\n- time_trunc unit\n\nDocument the column-name behavior in each docstring and add a test\nconfirming a bare str resolves to a per-row column value.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n* fix: correct grammar in file_compression_type error message\n\n\"must one of\" → \"must be one of\".\n\nCo-Authored-By: Claude Opus 4.8 (1M context) \u003cnoreply@anthropic.com\u003e\n\n---------\n\nCo-authored-by: Claude Opus 4.7 (1M context) \u003cnoreply@anthropic.com\u003e", "tree_diff": [ { "type": "modify", "old_id": "30e1a90fdb452464a2aa9ec49b4dae06487b8c70", "old_mode": 33188, "old_path": ".ai/skills/audit-skill-md/SKILL.md", "new_id": "ba5255a5975a8fe7746502c517aaa441a6eb05e8", "new_mode": 33188, "new_path": ".ai/skills/audit-skill-md/SKILL.md" }, { "type": "modify", "old_id": "24b4e1bb1d4adc63890b7988b90db791bb086b2b", "old_mode": 33188, "old_path": ".ai/skills/check-upstream/SKILL.md", "new_id": "a3d82a6706cee60a7b2eda1ed803d2f75e854cc0", "new_mode": 33188, "new_path": ".ai/skills/check-upstream/SKILL.md" }, { "type": "modify", "old_id": "57145ac6c9b1d36848c9cb74fc85f7921f2492c8", "old_mode": 33188, "old_path": ".ai/skills/make-pythonic/SKILL.md", "new_id": "7d490ec03fe9a52de7c8f490252db37d355743cf", "new_mode": 33188, "new_path": ".ai/skills/make-pythonic/SKILL.md" }, { "type": "modify", "old_id": "84f5719b7f48d7b40a81d00b0dface9715a16c74", "old_mode": 33188, "old_path": "Cargo.lock", "new_id": "583950ecb4bf51efee7b8bb4a2e29c84f80eb808", "new_mode": 33188, "new_path": "Cargo.lock" }, { "type": "modify", "old_id": "6a3ad3ab49a40bd9bedf661fa3f52447c127a8fa", "old_mode": 33188, "old_path": "Cargo.toml", "new_id": "0f3236ecf33bae388f12ccd8eb7933baa376955f", "new_mode": 33188, "new_path": "Cargo.toml" }, { "type": "modify", "old_id": "1f5b4e305bbf9ffdaa3d3f1a33d76873d94fd442", "old_mode": 33188, "old_path": "crates/core/Cargo.toml", "new_id": "2e8cf6c92c78ffdb67675067b7a9c002f56e0dcf", "new_mode": 33188, "new_path": "crates/core/Cargo.toml" }, { "type": "modify", "old_id": "0db49625f60c91c9519b8e72abd3a3e27f3a442c", "old_mode": 33188, "old_path": "crates/core/src/context.rs", "new_id": "0613a96dcceb82fe21489ceb2a62429353f24395", "new_mode": 33188, "new_path": "crates/core/src/context.rs" }, { "type": "modify", "old_id": "69d10559b6e0f626b779c400ccaf8bea62ebc9a8", "old_mode": 33188, "old_path": "crates/core/src/functions.rs", "new_id": "e57c7702dc6ed3521837d07f45ff71bbe109b29a", "new_mode": 33188, "new_path": "crates/core/src/functions.rs" }, { "type": "modify", "old_id": "48abcedc94a49a5500dae49a0092d1cb19fd8d41", "old_mode": 33188, "old_path": "crates/core/src/lib.rs", "new_id": "7f0f9cb39cb7436bed7630c39ed53f21c52d4846", "new_mode": 33188, "new_path": "crates/core/src/lib.rs" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "e7cb94f8cdad882ae3ebabc59caf99dd7500e670", "new_mode": 33188, "new_path": "crates/core/src/spark_functions.rs" }, { "type": "modify", "old_id": "32b8ce468af72e4aa678e90addbfc5ccbe3741a3", "old_mode": 33188, "old_path": "docs/source/user-guide/common-operations/aggregations.md", "new_id": "9c2d58e3cc439e9c0133120f22512d06e3018a72", "new_mode": 33188, "new_path": "docs/source/user-guide/common-operations/aggregations.md" }, { "type": "modify", "old_id": "8f4a05f527a204337ac8197ceb2123d91fb91a34", "old_mode": 33188, "old_path": "docs/source/user-guide/common-operations/functions.md", "new_id": "50411825d7312af0466d1010418dece72a2f61d0", "new_mode": 33188, "new_path": "docs/source/user-guide/common-operations/functions.md" }, { "type": "modify", "old_id": "cf1559e8c21a990bddd2fe6f6081f2fed1846b89", "old_mode": 33188, "old_path": "docs/source/user-guide/common-operations/index.md", "new_id": "d7b0cccd4662164b78b9b11ef40a2087daf69edf", "new_mode": 33188, "new_path": "docs/source/user-guide/common-operations/index.md" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "dbe478a4cfadf34cff970b3f05bf4f5f1c68bc83", "new_mode": 33188, "new_path": "docs/source/user-guide/common-operations/spark-functions.md" }, { "type": "modify", "old_id": "ff39f959a4e8189782008e6b6c29848edc1a75e4", "old_mode": 33188, "old_path": "pyproject.toml", "new_id": "e71098799c9fdc868c0d608815531e37549c29b4", "new_mode": 33188, "new_path": "pyproject.toml" }, { "type": "modify", "old_id": "5dfeed719d5b3ef8f5975130ed23c724fea05a80", "old_mode": 33188, "old_path": "python/datafusion/context.py", "new_id": "0bfc59bfee5d80924f5c2cc47e121be6b770bbcc", "new_mode": 33188, "new_path": "python/datafusion/context.py" }, { "type": "rename", "old_id": "f4eef763a36547fb19b4d2e0efee590f9f44ba1c", "old_mode": 33188, "old_path": "python/datafusion/functions.py", "new_id": "9bffa3590cac910d5c4ac8d209ca735da62aff02", "new_mode": 33188, "new_path": "python/datafusion/functions/__init__.py", "score": 99 }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "5163a6701ee046f0c7e754f8449b44dfc8e44813", "new_mode": 33188, "new_path": "python/datafusion/functions/spark.py" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "aed46d7fa04d9d83e3381d4450493cb660e60abb", "new_mode": 33188, "new_path": "python/tests/test_spark_functions.py" }, { "type": "modify", "old_id": "4ddf54e99dee8781173830e3ae19e2a9c81aee2e", "old_mode": 33188, "old_path": "skills/datafusion_python/SKILL.md", "new_id": "b6c548a83a7e821161446e45f627b000ec47ad82", "new_mode": 33188, "new_path": "skills/datafusion_python/SKILL.md" } ] }