[SPARK-61935][SQL] Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input

### What changes were proposed in this pull request?

This PR fixes DSv2 writes into a NOT NULL column whose value is a typed Scala UDF with a primitive parameter over a nullable input, e.g. `spark.udf.register("plus1", (x: Int) => x + 1)`. This is a correctness issue: UPDATE and MERGE silently write NULL (or `-1` with whole-stage codegen) into the NOT NULL column, while INSERT INTO and insert-only MERGE fail with `INTERNAL_ERROR` even when no NULL is produced.

The analyzer makes such a UDF return NULL for a NULL input by wrapping it as `If(IsNull(x), null, udf(KnownNotNull(x)))` in `HandleNullInputsForUDF`, which runs only after the `Resolution` batch. Until then, `ScalaUDF.nullable` is the declared nullability of the function result, which is `false` for `Int => Int`. So the rules in the `Resolution` batch decide on stale nullability:
- `TableOutputResolver` (INSERT) and `AssignmentUtils` (UPDATE and MERGE assignments) add no `AssertNotNull`.
- For INSERT and insert-only MERGE, `UpdateAttributeNullability` later makes the query output nullable, so `V2WriteCommand.outputResolved` becomes false and `CheckAnalysis` reports `Found the unresolved operator: 'AppendData` (or `'InsertOnlyMerge`).
- For UPDATE and MERGE with other clauses, nothing checks the value and NULL is written. The row projections of `ReplaceData` and `WriteDelta` also keep the stale nullability, so their output stays resolved.

The implicit cast of an input to the parameter type has the same timing issue: UPDATE and MERGE assignments are aligned before the type coercion rules insert it, and with `spark.sql.ansi.enabled=false` it returns NULL for a malformed string.

This PR makes `ScalaUDF.nullable` also true when the input of a primitive parameter can be null, in the same way as `deterministic` combines `udfDeterministic` with the children:
- The constructor parameter `nullable` is renamed to `udfNullable`. It keeps the declared nullability, which the code generation still uses to decide whether the result must be unwrapped from an `Option`, so the generated code does not change.
- `nullable` is `udfNullable || <a primitive parameter has an input that is nullable, or whose cast to the parameter type can return NULL>`, computed once per instance.

Once the inputs are cast and `HandleNullInputsForUDF` wraps the UDF, its primitive inputs are `KnownNotNull`, so `nullable` equals `udfNullable` again. Neither the cast nor the wrapping then changes the nullability that the rules in the `Resolution` batch saw. This PR also updates the comment of the `ScalaUDF Null Handling` batch, as the wrapping no longer changes the nullability; the batch still runs `UpdateAttributeNullability` and `HandleNullInputsForUDF` to a fixed point for attributes that become nullable after resolution.

### Why are the changes needed?

With a DSv2 table `t (pk INT NOT NULL, salary INT NOT NULL, dep STRING)` that keeps NOT NULL and supports row-level operations, like the test `InMemoryRowLevelOperationTableCatalog`:

```scala
spark.udf.register("plus1", (x: Int) => x + 1)
Seq((1, None), (2, Some(201)), (3, Some(301))).toDF("pk", "salary").createOrReplaceTempView("source")
```

```sql
-- t has (1, 100, 'hr') and (2, 200, 'software')

-- Issue 1: It should fail with NOT_NULL_ASSERT_VIOLATION, but it writes NULL (or `-1` with
-- whole-stage codegen in MERGE) into the NOT NULL column `salary` of the row 1.
MERGE INTO t USING source s ON t.pk = s.pk WHEN MATCHED THEN UPDATE SET salary = plus1(s.salary);
UPDATE t SET salary = plus1(IF(pk = 1, NULL, salary)) WHERE pk = 1;
-- With spark.sql.ansi.enabled=false, the same happens for a non-NULL malformed string input.
MERGE INTO t USING source s ON t.pk = s.pk WHEN MATCHED THEN UPDATE SET salary = plus1('x');

-- Issue 2: It should insert (3, 302, 'new'), but it fails with
-- [INTERNAL_ERROR] Found the unresolved operator: 'InsertOnlyMerge
MERGE INTO t USING source s ON t.pk = s.pk
WHEN NOT MATCHED THEN INSERT (pk, salary, dep) VALUES (s.pk, plus1(s.salary), 'new');

-- Issue 3: It should insert (3, 302, 'new'), but it fails with
-- [INTERNAL_ERROR] Found the unresolved operator: 'AppendData
INSERT INTO t SELECT pk, plus1(salary), 'new' FROM source WHERE pk = 3;
```

The same stale nullability is behind the wrong results fixed by SPARK-61934 (DSv2 MERGE into a nullable column writes `-1` with whole-stage codegen), and this PR also fixes them at the source. Other wrong results that this PR fixes, reproduced on Apache Spark 4.3.0 and 3.5.9:

```sql
-- Issue 4: It should return [NULL, 2], [202, 1], but it returns [-1, 2], [202, 1] (`0` without codegen).
SELECT plus1(salary) AS k, count(*) FROM gs GROUP BY GROUPING SETS ((plus1(salary))) ORDER BY 1;

-- Issue 5: For a CTE that is not inlined, it should return [1, NULL, true] and [3, NULL, true],
-- but it returns [1, 0, false] and [3, 0, false].
WITH c AS (SELECT pk, plus1(salary) AS k, rand() AS r FROM gs)
SELECT a.pk, a.k, a.k IS NULL FROM c a JOIN c b ON a.pk = b.pk ORDER BY a.pk;
```

where `gs` has the rows `(1, NULL)`, `(2, 201)` and `(3, NULL)`.

### Does this PR introduce _any_ user-facing change?

Yes. DSv2 writes of such a UDF into a NOT NULL column fail with `NOT_NULL_ASSERT_VIOLATION` only for the rows that produce NULL and succeed otherwise, instead of writing NULL or failing with `INTERNAL_ERROR`. The queries above return NULL instead of `-1`, `0` or `false`.

The schema that Spark reports can also change: a column that is derived from such a UDF and whose nullability is recorded during analysis, for example the output of `explode(array(plus1(a)))` over a nullable `a`, is now nullable. As a result, a streaming query started before Spark 4.3 that keeps such a column in its state, for example as a grouping key of a streaming aggregation, fails to restart with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE`. A query started on Spark 4.3 or later restarts, because its state schema is already nullable, but its state can hold the same wrong keys, and new NULL inputs go under a separate NULL key. The Structured Streaming migration guide describes both with the recovery.

### How was this patch tested?

Pass the CIs with the newly added test cases. The new tests fail without this fix, except the `UDFSuite` test for an outer join, which guards the existing fixed point of the `ScalaUDF Null Handling` batch.
- `DataSourceV2SQLSuite`: INSERT INTO a NOT NULL column (it fails with `INTERNAL_ERROR` without this fix).
- `MergeIntoTableSuiteBase` (7 suites): a NOT MATCHED INSERT succeeds, and a MATCHED UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input and, with ANSI off, for a malformed string input (it fails with `INTERNAL_ERROR` or writes NULL without this fix).
- `UpdateTableSuiteBase`: UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input (it writes `0` without this fix).
- `ScalaUDFSuite`: the nullability of `ScalaUDF` with a non-primitive and a primitive parameter, including an input that needs a cast.
- `UDFSuite`:
  - Issues 4 and 5, and the `explode` output, which is now nullable and NULL for a NULL input (it is non-nullable and `-1` without this fix).
  - A UDF over a column that the outer join makes nullable after resolution returns NULL. This keeps testing the fixed point of the `ScalaUDF Null Handling` batch, which the existing SPARK-47927 tests no longer need with this fix. It passes without this fix too, and fails only when the batch runs once.

I also checked the streaming case manually: a checkpoint written by Spark 4.2.1 for `explode(array(plus1($"a"), plus1($"b"))).as("v")` grouped by `v` restarts on Spark 4.3.0 and fails with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE` with this PR, while a checkpoint written by Spark 4.3.0 restarts with this PR.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

Closes #60651 from dongjoon-hyun/SPARK-61935.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>


skip-checks: true
diff --git a/docs/streaming/ss-migration-guide.md b/docs/streaming/ss-migration-guide.md
index 6f2dd1c..61d91e0 100644
--- a/docs/streaming/ss-migration-guide.md
+++ b/docs/streaming/ss-migration-guide.md
@@ -38,6 +38,17 @@
   To recover, write to a new table location with a new checkpoint location, or rewrite the
   committed files with the partition columns of the table.
 
+- Since Spark 4.4, a column derived from a typed Scala UDF with a primitive parameter is nullable
+  when the input of the parameter can be null, as the UDF returns NULL for a NULL input.
+  Previously, some of these columns, for example the output of a generator such as `explode` over
+  an array of the UDF results, were non-nullable. As a result, an existing query started before
+  Spark 4.3 that keeps such a column in its state, for example as a grouping key of a streaming
+  aggregation, fails to restart with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE`. To recover, use a new
+  checkpoint location, or set `spark.sql.streaming.stateStore.stateSchemaCheck` to `false`. Note
+  that the state of such a query can hold wrong keys, such as `-1` or `0`, for NULL inputs. The
+  same holds for a query started on Spark 4.3 or later, which restarts and then puts new NULL
+  inputs under a separate NULL key. To drop such keys, use a new checkpoint location.
+
 ## Upgrading from Structured Streaming 4.2 to 4.3
 
 - Since Spark 4.3, in Spark Connect the DataFrame passed to a Python `foreachBatch` function is bound to the streaming query's own session, a clone of the session that started the query, matching classic `foreachBatch`. This runs the batch under the configuration the streaming engine pins on the clone rather than the root session's. As a result, the batch DataFrame can no longer be combined with the root session inside the function: a temporary view created from it is not visible to the root session, referencing it from `spark.sql` on the root session fails with `DATAFRAME_NOT_FOUND`, and joining it with a DataFrame captured from the root session fails with `SESSION_NOT_SAME`. Run such work through the batch DataFrame's own session (`batch_df.sparkSession`) instead, and pass results out through a table or a global temporary view rather than a session-local temporary view. To restore the previous behavior, set `spark.sql.connect.streaming.foreachBatch.useClonedSession` to `false`.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 0c27f18..a86f4f4 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -821,11 +821,12 @@
     Batch("Nondeterministic", Once,
       PullOutNondeterministic),
     Batch("ScalaUDF Null Handling", fixedPoint,
-      // `HandleNullInputsForUDF` may wrap the `ScalaUDF` with `If` expression to return null for
-      // null inputs, so the result can be null even if `ScalaUDF#nullable` is false. We need to
-      // run `UpdateAttributeNullability` to update nullability of the UDF output attribute in
-      // downstream operators. After updating attribute nullability, `ScalaUDF`s in downstream
-      // operators may need null handling as well, so we should run these two rules repeatedly.
+      // `HandleNullInputsForUDF` wraps the `ScalaUDF` with `If` expression to return null for null
+      // inputs of primitive parameters. An attribute can become nullable after resolution, e.g. a
+      // resolved column of the null-supplying side of an outer join, so we need to run
+      // `UpdateAttributeNullability` to update nullability of the attributes. After updating
+      // attribute nullability, `ScalaUDF`s reading them may need null handling as well, so we
+      // should run these two rules repeatedly.
       HandleNullInputsForUDF,
       UpdateAttributeNullability),
     Batch("UDF", Once,
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
index 6571a00..8539423 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
@@ -41,7 +41,8 @@
  * @param outputEncoder ExpressionEncoder for the return type of function. It's only defined when
  *                      this is a typed Scala UDF.
  * @param udfName  The user-specified name of this UDF.
- * @param nullable  True if the UDF can return null value.
+ * @param udfNullable  True if the function itself can return null. `nullable` is also true when
+ *                     the input of a primitive parameter can be null.
  * @param udfDeterministic  True if the UDF is deterministic. Deterministic UDF returns same result
  *                          each time it is invoked with a particular input.
  */
@@ -52,12 +53,23 @@
     inputEncoders: Seq[Option[ExpressionEncoder[_]]] = Nil,
     outputEncoder: Option[ExpressionEncoder[_]] = None,
     udfName: Option[String] = None,
-    nullable: Boolean = true,
+    udfNullable: Boolean = true,
     udfDeterministic: Boolean = true)
   extends Expression with NonSQLExpression with UserDefinedExpression {
 
   override lazy val deterministic: Boolean = udfDeterministic && children.forall(_.deterministic)
 
+  // The analyzer casts the inputs to the parameter types (see `ImplicitTypeCasts`) and makes the
+  // UDF return null for a null input of a primitive parameter (see `HandleNullInputsForUDF`) only
+  // after the UDF is resolved, so the nullability covers both for the rules that run before, e.g.
+  // the ones adding NOT NULL checks to table writes.
+  override lazy val nullable: Boolean = udfNullable ||
+    inputPrimitives.zip(children).zip(inputTypes).exists {
+      case ((true, child), inputType: DataType) =>
+        child.nullable || Cast.forceNullable(child.dataType, inputType)
+      case _ => false
+    }
+
   // `ScalaUDF` uses `ExpressionEncoder` to convert the function result to Catalyst internal format.
   // `ExpressionEncoder` is stateful as it reuses the `UnsafeRow` instance, thus `ScalaUDF` is
   // stateful as well.
@@ -1158,7 +1170,7 @@
 
     val funcInvocation = if (isPrimitive(dataType)
         // If the output is nullable, the returned value must be unwrapped from the Option
-        && !nullable) {
+        && !udfNullable) {
       s"$resultTerm = ($boxedType)$getFuncResult"
     } else {
       s"$resultTerm = ($boxedType)$resultConverter.apply($getFuncResult)"
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
index 1be5876..be2d96c 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
@@ -2203,7 +2203,7 @@
       function = (i: Int) => i + 1,
       dataType = IntegerType,
       children = $"a" :: Nil,
-      nullable = false,
+      udfNullable = false,
       inputEncoders = Seq(Some(ExpressionEncoder[Int]().resolveAndBind())))
     val plan = testRelation.select(udf.as("u")).select($"u").analyze
     assert(plan.output.head.nullable)
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala
index 178eb2f..f8ea0c0 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDFSuite.scala
@@ -139,4 +139,15 @@
     val code = udf.genCode(new CodegenContext).code.toString
     assert(!code.contains("fieldName") && !code.contains("udfName") && !code.contains("outputName"))
   }
+
+  test("SPARK-61935: nullable when the input of a primitive parameter can be null") {
+    def stringIntUdf(s: Expression, i: Expression): ScalaUDF = ScalaUDF(
+      (str: String, n: Int) => str.length + n, IntegerType, s :: i :: Nil,
+      Option(resolvedEncoder[String]()) :: Option(resolvedEncoder[Int]()) :: Nil,
+      udfNullable = false)
+    assert(!stringIntUdf(Literal.create(null, StringType), Literal(1)).nullable)
+    assert(stringIntUdf(Literal("a"), Literal.create(null, IntegerType)).nullable)
+    // the implicit cast of a string to the primitive parameter can return null
+    assert(stringIntUdf(Literal("a"), Literal("1")).nullable)
+  }
 }
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala
index 1243720..d808c15 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/classic/UserDefinedFunctionUtils.scala
@@ -38,7 +38,7 @@
       }),
       udf.outputEncoder.map(encoderFor(_)),
       udfName = udf.givenName,
-      nullable = udf.nullable,
+      udfNullable = udf.nullable,
       udfDeterministic = udf.deterministic)
   }
 }
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
index e7e8564..64df7f8 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
@@ -1276,6 +1276,44 @@
     checkAnswer(df, Seq(Row(2, 3), Row(null, null)))
   }
 
+  test("SPARK-61935: ScalaUDF null handling of a column made nullable by an outer join") {
+    val f = udf[Int, Int](_ + 1)
+    val ds1 = Seq(1, 2).toDS()
+    val ds2 = Seq(2).toDS()
+    // `ds2("value")` is resolved as non-nullable, and becomes nullable after resolution
+    checkAnswer(
+      ds1.join(ds2, ds1("value") === ds2("value"), "left_outer").select(f(ds2("value"))),
+      Row(null) :: Row(3) :: Nil)
+  }
+
+  test("SPARK-61935: grouping sets, CTE and generator over a ScalaUDF with a primitive input") {
+    withTempView("gs") {
+      withUserDefinedFunction("plus1" -> true) {
+        spark.udf.register("plus1", (x: Int) => x + 1)
+        Seq((1, None), (2, Some(201)), (3, None)).toDF("pk", "salary")
+          .createOrReplaceTempView("gs")
+
+        checkAnswer(
+          sql("SELECT plus1(salary) AS k, count(*) FROM gs " +
+            "GROUP BY GROUPING SETS ((plus1(salary)))"),
+          Row(null, 2) :: Row(202, 1) :: Nil)
+
+        // a CTE that is not inlined
+        checkAnswer(
+          sql(
+            """WITH c AS (SELECT pk, plus1(salary) AS k, rand() AS r FROM gs)
+              |SELECT a.pk, a.k, a.k IS NULL FROM c a JOIN c b ON a.pk = b.pk
+              |""".stripMargin),
+          Row(1, null, true) :: Row(2, 202, false) :: Row(3, null, true) :: Nil)
+
+        val df = sql("SELECT explode(array(plus1(a), plus1(b))) AS v " +
+          "FROM VALUES (1, 2), (NULL, 3) AS t(a, b)")
+        assert(df.schema("v").nullable)
+        checkAnswer(df, Row(2) :: Row(3) :: Row(null) :: Row(4) :: Nil)
+      }
+    }
+  }
+
   test("SPARK-52819: Support using Kryo to encode BUF in Aggregator") {
     val kryoBufUDAF = udaf(KryoBufAggregator())
     val input = Seq(1L, 2L, 3L).toDF("value")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala
index 9aef714..f302b1d 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/classic/ColumnNodeToExpressionConverterSuite.scala
@@ -386,7 +386,7 @@
         inputEncoders = Seq(None, Option(encoderFor(StringEncoder))),
         outputEncoder = Option(encoderFor(StringEncoder)),
         udfName = None,
-        nullable = false,
+        udfNullable = false,
         udfDeterministic = false))
   }
 
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index 76c6ecb..107a45c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -1581,6 +1581,29 @@
     }
   }
 
+  test("SPARK-61935: insert checks NOT NULL for a Scala UDF with a primitive input") {
+    import testImplicits._
+    val t = "testcat.tbl"
+    withTable(t) {
+      withTempView("source") {
+        withUserDefinedFunction("plus1" -> true) {
+          // the UDF is not nullable, but it returns NULL for NULL input as its input is primitive
+          spark.udf.register("plus1", (x: Int) => x + 1)
+          sql(s"CREATE TABLE $t (id INT NOT NULL, data INT NOT NULL) USING foo")
+          Seq((1, Some(10)), (2, None)).toDF("id", "data").createOrReplaceTempView("source")
+
+          sql(s"INSERT INTO $t SELECT id, plus1(data) FROM source WHERE id = 1")
+          checkAnswer(spark.table(t), Row(1, 11))
+
+          val e = intercept[SparkRuntimeException] {
+            sql(s"INSERT INTO $t SELECT id, plus1(data) FROM source")
+          }
+          assert(e.getCondition == "NOT_NULL_ASSERT_VIOLATION")
+        }
+      }
+    }
+  }
+
   test("ShowViews: using v1 catalog, db name with multipartIdentifier ('a.b') is not allowed.") {
     checkError(
       exception = analysisException("SHOW VIEWS FROM a.b"),
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
index e338e61..966eed5 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
@@ -2484,6 +2484,60 @@
     }
   }
 
+  test("SPARK-61935: merge checks NOT NULL for a Scala UDF with a primitive input") {
+    withTempView("source") {
+      withUserDefinedFunction("plus1" -> true) {
+        // the UDF is not nullable, but it returns NULL for NULL input as its input is primitive
+        spark.udf.register("plus1", (x: Int) => x + 1)
+        createAndInitTable("pk INT NOT NULL, salary INT NOT NULL, dep STRING",
+          """{ "pk": 1, "salary": 100, "dep": "hr" }
+            |{ "pk": 2, "salary": 200, "dep": "software" }
+            |""".stripMargin)
+
+        Seq((1, None), (2, Some(201)), (3, Some(301))).toDF("pk", "salary")
+          .createOrReplaceTempView("source")
+
+        // the only inserted row has a non-NULL input
+        sql(
+          s"""MERGE INTO $tableNameAsString t
+             |USING source s
+             |ON t.pk = s.pk
+             |WHEN NOT MATCHED THEN
+             | INSERT (pk, salary, dep) VALUES (s.pk, plus1(s.salary), 'new')
+             |""".stripMargin)
+
+        checkAnswer(
+          sql(s"SELECT * FROM $tableNameAsString"),
+          Row(1, 100, "hr") :: Row(2, 200, "software") :: Row(3, 302, "new") :: Nil)
+
+        val e = intercept[SparkRuntimeException] {
+          sql(
+            s"""MERGE INTO $tableNameAsString t
+               |USING source s
+               |ON t.pk = s.pk
+               |WHEN MATCHED THEN
+               | UPDATE SET salary = plus1(s.salary)
+               |""".stripMargin)
+        }
+        assert(e.getCondition == "NOT_NULL_ASSERT_VIOLATION")
+
+        // with ANSI off, the implicit cast of a malformed string to the primitive input is NULL
+        withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+          val e2 = intercept[SparkRuntimeException] {
+            sql(
+              s"""MERGE INTO $tableNameAsString t
+                 |USING source s
+                 |ON t.pk = s.pk
+                 |WHEN MATCHED THEN
+                 | UPDATE SET salary = plus1('x')
+                 |""".stripMargin)
+          }
+          assert(e2.getCondition == "NOT_NULL_ASSERT_VIOLATION")
+        }
+      }
+    }
+  }
+
   test("merge with NOT NULL checks") {
     withTempView("source") {
       createAndInitTable("pk INT NOT NULL, s STRUCT<n_i: INT NOT NULL, n_l: LONG>, dep STRING",
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
index 951bebf..396a350 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
@@ -1185,6 +1185,20 @@
     }
   }
 
+  test("SPARK-61935: update checks NOT NULL for a Scala UDF with a primitive input") {
+    withUserDefinedFunction("plus1" -> true) {
+      spark.udf.register("plus1", (x: Int) => x + 1)
+      createAndInitTable("pk INT NOT NULL, salary INT NOT NULL, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "software" }
+          |""".stripMargin)
+      val e = intercept[SparkRuntimeException] {
+        sql(s"UPDATE $tableNameAsString SET salary = plus1(IF(pk = 1, NULL, salary)) WHERE pk = 1")
+      }
+      assert(e.getCondition == "NOT_NULL_ASSERT_VIOLATION")
+    }
+  }
+
   test("update with NOT NULL checks") {
     createAndInitTable("pk INT NOT NULL, s STRUCT<n_i: INT NOT NULL, n_l: LONG>, dep STRING",
       """{ "pk": 1, "s": { "n_i": 1, "n_l": 11 }, "dep": "hr" }