)]}'
{
  "commit": "ab7c8a56de71e41cd674273d392cadf2d7d0b2c1",
  "tree": "0a9e7c403bda5cba6d5ee9417a41ed9b2cfbfd9c",
  "parents": [
    "2ef1374a8995ceb5fa814d65de47215780430350"
  ],
  "author": {
    "name": "Dongjoon Hyun",
    "email": "dongjoon@apache.org",
    "time": "Wed Nov 18 15:29:36 2026 -0800"
  },
  "committer": {
    "name": "Dongjoon Hyun",
    "email": "dongjoon@apache.org",
    "time": "Wed Nov 18 15:29:36 2026 -0800"
  },
  "message": "[SPARK-61935][SQL] Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input\n\n### What changes were proposed in this pull request?\n\nThis PR fixes DSv2 writes into a NOT NULL column whose value is a typed Scala UDF with a primitive parameter over a nullable input, e.g. `spark.udf.register(\"plus1\", (x: Int) \u003d\u003e x + 1)`. This is a correctness issue: UPDATE and MERGE silently write NULL (or `-1` with whole-stage codegen) into the NOT NULL column, while INSERT INTO and insert-only MERGE fail with `INTERNAL_ERROR` even when no NULL is produced.\n\nThe analyzer makes such a UDF return NULL for a NULL input by wrapping it as `If(IsNull(x), null, udf(KnownNotNull(x)))` in `HandleNullInputsForUDF`, which runs only after the `Resolution` batch. Until then, `ScalaUDF.nullable` is the declared nullability of the function result, which is `false` for `Int \u003d\u003e Int`. So the rules in the `Resolution` batch decide on stale nullability:\n- `TableOutputResolver` (INSERT) and `AssignmentUtils` (UPDATE and MERGE assignments) add no `AssertNotNull`.\n- For INSERT and insert-only MERGE, `UpdateAttributeNullability` later makes the query output nullable, so `V2WriteCommand.outputResolved` becomes false and `CheckAnalysis` reports `Found the unresolved operator: \u0027AppendData` (or `\u0027InsertOnlyMerge`).\n- For UPDATE and MERGE with other clauses, nothing checks the value and NULL is written. The row projections of `ReplaceData` and `WriteDelta` also keep the stale nullability, so their output stays resolved.\n\nThe implicit cast of an input to the parameter type has the same timing issue: UPDATE and MERGE assignments are aligned before the type coercion rules insert it, and with `spark.sql.ansi.enabled\u003dfalse` it returns NULL for a malformed string.\n\nThis PR makes `ScalaUDF.nullable` also true when the input of a primitive parameter can be null, in the same way as `deterministic` combines `udfDeterministic` with the children:\n- The constructor parameter `nullable` is renamed to `udfNullable`. It keeps the declared nullability, which the code generation still uses to decide whether the result must be unwrapped from an `Option`, so the generated code does not change.\n- `nullable` is `udfNullable || \u003ca primitive parameter has an input that is nullable, or whose cast to the parameter type can return NULL\u003e`, computed once per instance.\n\nOnce the inputs are cast and `HandleNullInputsForUDF` wraps the UDF, its primitive inputs are `KnownNotNull`, so `nullable` equals `udfNullable` again. Neither the cast nor the wrapping then changes the nullability that the rules in the `Resolution` batch saw. This PR also updates the comment of the `ScalaUDF Null Handling` batch, as the wrapping no longer changes the nullability; the batch still runs `UpdateAttributeNullability` and `HandleNullInputsForUDF` to a fixed point for attributes that become nullable after resolution.\n\n### Why are the changes needed?\n\nWith a DSv2 table `t (pk INT NOT NULL, salary INT NOT NULL, dep STRING)` that keeps NOT NULL and supports row-level operations, like the test `InMemoryRowLevelOperationTableCatalog`:\n\n```scala\nspark.udf.register(\"plus1\", (x: Int) \u003d\u003e x + 1)\nSeq((1, None), (2, Some(201)), (3, Some(301))).toDF(\"pk\", \"salary\").createOrReplaceTempView(\"source\")\n```\n\n```sql\n-- t has (1, 100, \u0027hr\u0027) and (2, 200, \u0027software\u0027)\n\n-- Issue 1: It should fail with NOT_NULL_ASSERT_VIOLATION, but it writes NULL (or `-1` with\n-- whole-stage codegen in MERGE) into the NOT NULL column `salary` of the row 1.\nMERGE INTO t USING source s ON t.pk \u003d s.pk WHEN MATCHED THEN UPDATE SET salary \u003d plus1(s.salary);\nUPDATE t SET salary \u003d plus1(IF(pk \u003d 1, NULL, salary)) WHERE pk \u003d 1;\n-- With spark.sql.ansi.enabled\u003dfalse, the same happens for a non-NULL malformed string input.\nMERGE INTO t USING source s ON t.pk \u003d s.pk WHEN MATCHED THEN UPDATE SET salary \u003d plus1(\u0027x\u0027);\n\n-- Issue 2: It should insert (3, 302, \u0027new\u0027), but it fails with\n-- [INTERNAL_ERROR] Found the unresolved operator: \u0027InsertOnlyMerge\nMERGE INTO t USING source s ON t.pk \u003d s.pk\nWHEN NOT MATCHED THEN INSERT (pk, salary, dep) VALUES (s.pk, plus1(s.salary), \u0027new\u0027);\n\n-- Issue 3: It should insert (3, 302, \u0027new\u0027), but it fails with\n-- [INTERNAL_ERROR] Found the unresolved operator: \u0027AppendData\nINSERT INTO t SELECT pk, plus1(salary), \u0027new\u0027 FROM source WHERE pk \u003d 3;\n```\n\nThe same stale nullability is behind the wrong results fixed by SPARK-61934 (DSv2 MERGE into a nullable column writes `-1` with whole-stage codegen), and this PR also fixes them at the source. Other wrong results that this PR fixes, reproduced on Apache Spark 4.3.0 and 3.5.9:\n\n```sql\n-- Issue 4: It should return [NULL, 2], [202, 1], but it returns [-1, 2], [202, 1] (`0` without codegen).\nSELECT plus1(salary) AS k, count(*) FROM gs GROUP BY GROUPING SETS ((plus1(salary))) ORDER BY 1;\n\n-- Issue 5: For a CTE that is not inlined, it should return [1, NULL, true] and [3, NULL, true],\n-- but it returns [1, 0, false] and [3, 0, false].\nWITH c AS (SELECT pk, plus1(salary) AS k, rand() AS r FROM gs)\nSELECT a.pk, a.k, a.k IS NULL FROM c a JOIN c b ON a.pk \u003d b.pk ORDER BY a.pk;\n```\n\nwhere `gs` has the rows `(1, NULL)`, `(2, 201)` and `(3, NULL)`.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes. DSv2 writes of such a UDF into a NOT NULL column fail with `NOT_NULL_ASSERT_VIOLATION` only for the rows that produce NULL and succeed otherwise, instead of writing NULL or failing with `INTERNAL_ERROR`. The queries above return NULL instead of `-1`, `0` or `false`.\n\nThe schema that Spark reports can also change: a column that is derived from such a UDF and whose nullability is recorded during analysis, for example the output of `explode(array(plus1(a)))` over a nullable `a`, is now nullable. As a result, a streaming query started before Spark 4.3 that keeps such a column in its state, for example as a grouping key of a streaming aggregation, fails to restart with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE`. A query started on Spark 4.3 or later restarts, because its state schema is already nullable, but its state can hold the same wrong keys, and new NULL inputs go under a separate NULL key. The Structured Streaming migration guide describes both with the recovery.\n\n### How was this patch tested?\n\nPass the CIs with the newly added test cases. The new tests fail without this fix, except the `UDFSuite` test for an outer join, which guards the existing fixed point of the `ScalaUDF Null Handling` batch.\n- `DataSourceV2SQLSuite`: INSERT INTO a NOT NULL column (it fails with `INTERNAL_ERROR` without this fix).\n- `MergeIntoTableSuiteBase` (7 suites): a NOT MATCHED INSERT succeeds, and a MATCHED UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input and, with ANSI off, for a malformed string input (it fails with `INTERNAL_ERROR` or writes NULL without this fix).\n- `UpdateTableSuiteBase`: UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input (it writes `0` without this fix).\n- `ScalaUDFSuite`: the nullability of `ScalaUDF` with a non-primitive and a primitive parameter, including an input that needs a cast.\n- `UDFSuite`:\n  - Issues 4 and 5, and the `explode` output, which is now nullable and NULL for a NULL input (it is non-nullable and `-1` without this fix).\n  - A UDF over a column that the outer join makes nullable after resolution returns NULL. This keeps testing the fixed point of the `ScalaUDF Null Handling` batch, which the existing SPARK-47927 tests no longer need with this fix. It passes without this fix too, and fails only when the batch runs once.\n\nI also checked the streaming case manually: a checkpoint written by Spark 4.2.1 for `explode(array(plus1($\"a\"), plus1($\"b\"))).as(\"v\")` grouped by `v` restarts on Spark 4.3.0 and fails with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE` with this PR, while a checkpoint written by Spark 4.3.0 restarts with this PR.\n\n### Was this patch authored or co-authored using generative AI tooling?\n\nGenerated-by: Claude Opus 5.5\n\nCloses #60651 from dongjoon-hyun/SPARK-61935.\n\nAuthored-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\nSigned-off-by: Dongjoon Hyun \u003cdongjoon@apache.org\u003e\n\n\nskip-checks: true\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "6f2dd1c6367a9470636d1172161119b9955eb6ba",
      "old_mode": 33188,
      "old_path": "docs/streaming/ss-migration-guide.md",
      "new_id": "61d91e01951bba9b4c88d8eb5efda75581909861",
      "new_mode": 33188,
      "new_path": "docs/streaming/ss-migration-guide.md"
    },
    {
      "type": "modify",
      "old_id": "0c27f18460fc8efa37bb16159d3e1d911d545a04",
      "old_mode": 33188,
      "old_path": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala",
      "new_id": "a86f4f4e179bd989c6e820987c405b4772d63b12",
      "new_mode": 33188,
      "new_path": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala"
    },
    {
      "type": "modify",
      "old_id": "6571a00d0b51f47600d2ff41d248e061d1a51eff",
      "old_mode": 33188,
      "old_path": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala",
      "new_id": "8539423c9117231754b47e798815475c74216a2a",
      "new_mode": 33188,
      "new_path": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala"
    },
    {
      "type": "modify",
      "old_id": "1be5876a790059fda348769b3ed65a68b9b96d8c",
      "old_mode": 33188,
      "old_path": "sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala",
      "new_id": "be2d96c3ee7c887fd2f3e1e92c98ef9d5d55e909",
      "new_mode": 33188,
      "new_path": "sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala"
    },
    {
      "type": "modify",
      "old_id": "178eb2ffc5ef60b119294bf500ccb7a9f31e9407",
      "old_mode": 33188,
      "old_path": "sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala",
      "new_id": "f8ea0c0f0cba2eb3a603596298f019d37592a4f0",
      "new_mode": 33188,
      "new_path": "sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala"
    },
    {
      "type": "modify",
      "old_id": "124372057c2c90acef0791ace91c377c972242ec",
      "old_mode": 33188,
      "old_path": "sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala",
      "new_id": "d808c15cedee8e70787748c1eedc184bf6a765ff",
      "new_mode": 33188,
      "new_path": "sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala"
    },
    {
      "type": "modify",
      "old_id": "e7e8564568f6167c8d1d6568dd26df29ee94de80",
      "old_mode": 33188,
      "old_path": "sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala",
      "new_id": "64df7f86dcba67d6de130125e32292410b14ffcd",
      "new_mode": 33188,
      "new_path": "sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala"
    },
    {
      "type": "modify",
      "old_id": "9aef714a0b35a4ad05b8f30f3f74eab16909cab5",
      "old_mode": 33188,
      "old_path": "sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala",
      "new_id": "f302b1dd123f6c08d0887fb1258869d29ac9550b",
      "new_mode": 33188,
      "new_path": "sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala"
    },
    {
      "type": "modify",
      "old_id": "76c6ecbdea7cf19ad572e1312c8aaecd7316b78e",
      "old_mode": 33188,
      "old_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala",
      "new_id": "107a45c8c5b530bf3a50ca1fe41c2d4c201b7ede",
      "new_mode": 33188,
      "new_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala"
    },
    {
      "type": "modify",
      "old_id": "e338e61c2a84e68da8685957f84a2831b7cb80e2",
      "old_mode": 33188,
      "old_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala",
      "new_id": "966eed5fd80b8872e6e2d48f598bfebd14b779fd",
      "new_mode": 33188,
      "new_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala"
    },
    {
      "type": "modify",
      "old_id": "951bebfde959aa282e845270accf2a2134cf44f3",
      "old_mode": 33188,
      "old_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala",
      "new_id": "396a3509d8002e5ba238fa8ef98852df4261d6f9",
      "new_mode": 33188,
      "new_path": "sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala"
    }
  ]
}
