[SPARK-61935][SQL] Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input

### What changes were proposed in this pull request?

This PR fixes DSv2 writes into a NOT NULL column whose value is a typed Scala UDF with a primitive parameter over a nullable input, e.g. `spark.udf.register("plus1", (x: Int) => x + 1)`. This is a correctness issue: UPDATE and MERGE silently write NULL (or `-1` with whole-stage codegen) into the NOT NULL column, while INSERT INTO and insert-only MERGE fail with `INTERNAL_ERROR` even when no NULL is produced.

The analyzer makes such a UDF return NULL for a NULL input by wrapping it as `If(IsNull(x), null, udf(KnownNotNull(x)))` in `HandleNullInputsForUDF`, which runs only after the `Resolution` batch. Until then, `ScalaUDF.nullable` is the declared nullability of the function result, which is `false` for `Int => Int`. So the rules in the `Resolution` batch decide on stale nullability:
- `TableOutputResolver` (INSERT) and `AssignmentUtils` (UPDATE and MERGE assignments) add no `AssertNotNull`.
- For INSERT and insert-only MERGE, `UpdateAttributeNullability` later makes the query output nullable, so `V2WriteCommand.outputResolved` becomes false and `CheckAnalysis` reports `Found the unresolved operator: 'AppendData` (or `'InsertOnlyMerge`).
- For UPDATE and MERGE with other clauses, nothing checks the value and NULL is written. The row projections of `ReplaceData` and `WriteDelta` also keep the stale nullability, so their output stays resolved.

The implicit cast of an input to the parameter type has the same timing issue: UPDATE and MERGE assignments are aligned before the type coercion rules insert it, and with `spark.sql.ansi.enabled=false` it returns NULL for a malformed string.

This PR makes `ScalaUDF.nullable` also true when the input of a primitive parameter can be null, in the same way as `deterministic` combines `udfDeterministic` with the children:
- The constructor parameter `nullable` is renamed to `udfNullable`. It keeps the declared nullability, which the code generation still uses to decide whether the result must be unwrapped from an `Option`, so the generated code does not change.
- `nullable` is `udfNullable || <a primitive parameter has an input that is nullable, or whose cast to the parameter type can return NULL>`, computed once per instance.

Once the inputs are cast and `HandleNullInputsForUDF` wraps the UDF, its primitive inputs are `KnownNotNull`, so `nullable` equals `udfNullable` again. Neither the cast nor the wrapping then changes the nullability that the rules in the `Resolution` batch saw. This PR also updates the comment of the `ScalaUDF Null Handling` batch, as the wrapping no longer changes the nullability; the batch still runs `UpdateAttributeNullability` and `HandleNullInputsForUDF` to a fixed point for attributes that become nullable after resolution.

### Why are the changes needed?

With a DSv2 table `t (pk INT NOT NULL, salary INT NOT NULL, dep STRING)` that keeps NOT NULL and supports row-level operations, like the test `InMemoryRowLevelOperationTableCatalog`:

```scala
spark.udf.register("plus1", (x: Int) => x + 1)
Seq((1, None), (2, Some(201)), (3, Some(301))).toDF("pk", "salary").createOrReplaceTempView("source")
```

```sql
-- t has (1, 100, 'hr') and (2, 200, 'software')

-- Issue 1: It should fail with NOT_NULL_ASSERT_VIOLATION, but it writes NULL (or `-1` with
-- whole-stage codegen in MERGE) into the NOT NULL column `salary` of the row 1.
MERGE INTO t USING source s ON t.pk = s.pk WHEN MATCHED THEN UPDATE SET salary = plus1(s.salary);
UPDATE t SET salary = plus1(IF(pk = 1, NULL, salary)) WHERE pk = 1;
-- With spark.sql.ansi.enabled=false, the same happens for a non-NULL malformed string input.
MERGE INTO t USING source s ON t.pk = s.pk WHEN MATCHED THEN UPDATE SET salary = plus1('x');

-- Issue 2: It should insert (3, 302, 'new'), but it fails with
-- [INTERNAL_ERROR] Found the unresolved operator: 'InsertOnlyMerge
MERGE INTO t USING source s ON t.pk = s.pk
WHEN NOT MATCHED THEN INSERT (pk, salary, dep) VALUES (s.pk, plus1(s.salary), 'new');

-- Issue 3: It should insert (3, 302, 'new'), but it fails with
-- [INTERNAL_ERROR] Found the unresolved operator: 'AppendData
INSERT INTO t SELECT pk, plus1(salary), 'new' FROM source WHERE pk = 3;
```

The same stale nullability is behind the wrong results fixed by SPARK-61934 (DSv2 MERGE into a nullable column writes `-1` with whole-stage codegen), and this PR also fixes them at the source. Other wrong results that this PR fixes, reproduced on Apache Spark 4.3.0 and 3.5.9:

```sql
-- Issue 4: It should return [NULL, 2], [202, 1], but it returns [-1, 2], [202, 1] (`0` without codegen).
SELECT plus1(salary) AS k, count(*) FROM gs GROUP BY GROUPING SETS ((plus1(salary))) ORDER BY 1;

-- Issue 5: For a CTE that is not inlined, it should return [1, NULL, true] and [3, NULL, true],
-- but it returns [1, 0, false] and [3, 0, false].
WITH c AS (SELECT pk, plus1(salary) AS k, rand() AS r FROM gs)
SELECT a.pk, a.k, a.k IS NULL FROM c a JOIN c b ON a.pk = b.pk ORDER BY a.pk;
```

where `gs` has the rows `(1, NULL)`, `(2, 201)` and `(3, NULL)`.

### Does this PR introduce _any_ user-facing change?

Yes. DSv2 writes of such a UDF into a NOT NULL column fail with `NOT_NULL_ASSERT_VIOLATION` only for the rows that produce NULL and succeed otherwise, instead of writing NULL or failing with `INTERNAL_ERROR`. The queries above return NULL instead of `-1`, `0` or `false`.

The schema that Spark reports can also change: a column that is derived from such a UDF and whose nullability is recorded during analysis, for example the output of `explode(array(plus1(a)))` over a nullable `a`, is now nullable. As a result, a streaming query started before Spark 4.3 that keeps such a column in its state, for example as a grouping key of a streaming aggregation, fails to restart with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE`. A query started on Spark 4.3 or later restarts, because its state schema is already nullable, but its state can hold the same wrong keys, and new NULL inputs go under a separate NULL key. The Structured Streaming migration guide describes both with the recovery.

### How was this patch tested?

Pass the CIs with the newly added test cases. The new tests fail without this fix, except the `UDFSuite` test for an outer join, which guards the existing fixed point of the `ScalaUDF Null Handling` batch.
- `DataSourceV2SQLSuite`: INSERT INTO a NOT NULL column (it fails with `INTERNAL_ERROR` without this fix).
- `MergeIntoTableSuiteBase` (7 suites): a NOT MATCHED INSERT succeeds, and a MATCHED UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input and, with ANSI off, for a malformed string input (it fails with `INTERNAL_ERROR` or writes NULL without this fix).
- `UpdateTableSuiteBase`: UPDATE fails with `NOT_NULL_ASSERT_VIOLATION` for a NULL input (it writes `0` without this fix).
- `ScalaUDFSuite`: the nullability of `ScalaUDF` with a non-primitive and a primitive parameter, including an input that needs a cast.
- `UDFSuite`:
  - Issues 4 and 5, and the `explode` output, which is now nullable and NULL for a NULL input (it is non-nullable and `-1` without this fix).
  - A UDF over a column that the outer join makes nullable after resolution returns NULL. This keeps testing the fixed point of the `ScalaUDF Null Handling` batch, which the existing SPARK-47927 tests no longer need with this fix. It passes without this fix too, and fails only when the batch runs once.

I also checked the streaming case manually: a checkpoint written by Spark 4.2.1 for `explode(array(plus1($"a"), plus1($"b"))).as("v")` grouped by `v` restarts on Spark 4.3.0 and fails with `STATE_STORE_KEY_SCHEMA_NOT_COMPATIBLE` with this PR, while a checkpoint written by Spark 4.3.0 restarts with this PR.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

Closes #60651 from dongjoon-hyun/SPARK-61935.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>


skip-checks: true
11 files changed
tree: 0a9e7c403bda5cba6d5ee9417a41ed9b2cfbfd9c
  1. .agents/
  2. .github/
  3. .mvn/
  4. assembly/
  5. bin/
  6. binder/
  7. build/
  8. common/
  9. conf/
  10. connector/
  11. core/
  12. data/
  13. dev/
  14. docs/
  15. examples/
  16. graphx/
  17. hadoop-cloud/
  18. launcher/
  19. licenses/
  20. licenses-binary/
  21. mllib/
  22. mllib-local/
  23. project/
  24. python/
  25. R/
  26. repl/
  27. resource-managers/
  28. sbin/
  29. sql/
  30. streaming/
  31. tools/
  32. udf/
  33. ui-test/
  34. .asf.yaml
  35. .gitattributes
  36. .gitignore
  37. .ignore
  38. .nojekyll
  39. .pre-commit-config.yaml
  40. .sbtopts
  41. AGENTS.md
  42. CONTRIBUTING.md
  43. LICENSE
  44. LICENSE-binary
  45. NOTICE
  46. NOTICE-binary
  47. pom.xml
  48. pyproject.toml
  49. README.md
  50. scalastyle-config.xml
  51. SECURITY.md
README.md

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R (Deprecated), and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

License Maven Central Java GitHub Actions Build PySpark Coverage PyPI Downloads

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project web page. This README file only contains basic setup instructions.

Build Pipeline Status

See the Apache CI dashboard for an overview of Spark's CI usage.

BranchStatus
masterGitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.xGitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.4GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.3GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.2GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.1GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-4.0GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
GitHub Actions Build
branch-3.5GitHub Actions Build
GitHub Actions Build
GitHub Actions Build

Building Spark

Spark is built using Apache Maven. To build Spark and its example programs, run:

./build/mvn -DskipTests clean package

(You do not need to do this if you downloaded a pre-built package.)

More detailed documentation is available from the project site, at “Building Spark”.

For general development tips, including info on developing Spark using an IDE, see “Useful Developer Tools”.

Interactive Scala Shell

The easiest way to start using Spark is through the Scala shell:

./bin/spark-shell

Try the following command, which should return 1,000,000,000:

scala> spark.range(1000 * 1000 * 1000).count()

Interactive Python Shell

Alternatively, if you prefer Python, you can use the Python shell:

./bin/pyspark

And run the following command, which should also return 1,000,000,000:

>>> spark.range(1000 * 1000 * 1000).count()

Example Programs

Spark also comes with several sample programs in the examples directory. To run one of them, use ./bin/run-example <class> [params]. For example:

./bin/run-example SparkPi

will run the Pi example locally.

You can set the MASTER environment variable when running examples to submit examples to a cluster. This can be spark:// URL, “yarn” to run on YARN, and “local” to run locally with one thread, or “local[N]” to run locally with N threads. You can also use an abbreviated class name if the class is in the examples package. For instance:

MASTER=spark://host:7077 ./bin/run-example SparkPi

Many of the example programs print usage help if no params are given.

Running Tests

Testing first requires building Spark. Once Spark is built, tests can be run using:

./dev/run-tests

Please see the guidance on how to run tests for a module, or individual tests.

There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md

A Note About Hadoop Versions

Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported storage systems. Because the protocols have changed in different versions of Hadoop, you must build Spark against the same version that your cluster runs.

Please refer to the build documentation at “Specifying the Hadoop Version and Enabling YARN” for detailed guidance on building for a particular distribution of Hadoop, including building for particular Hive and Hive Thriftserver distributions.

Configuration

Please refer to the Configuration Guide in the online documentation for an overview on how to configure Spark.

Contributing

Please review the Contribution to Spark guide for information on how to get started contributing to the project.