Fix struct casts to align fields by name (prevent positional mis-casts) (#19674)

## Which issue does this PR close?

* Closes #17285.

## Rationale for this change

DataFusion’s struct casting and some coercion paths were effectively
positional: when two structs had the same field types but different
field *orders*, casting could silently swap values. This is surprising
to users and can lead to silent data corruption (e.g. `{b: 3, a:
4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`).

The goal of this PR is to make struct casting behavior match user
expectations by matching fields by **name** (case-sensitive) and
recursively applying the same logic to nested structs, while keeping a
compatible fallback for structs with **no** shared field names.

## What changes are included in this PR?

* **Name-based struct casting implementation** in
`datafusion_common::nested_struct`:

* Match struct fields by **name**, reorder to match target schema,
recursively cast nested structs.
  * Fill **missing target fields** with null arrays.
  * Ignore **extra source fields**.
* **Positional mapping fallback** when there is *no name overlap*
**and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a
INT, b VARCHAR)` style casts).
* Improved handling for **NULL / all-null struct inputs** by producing a
correctly typed null struct array.
* Centralized validation via `validate_field_compatibility` and helper
`fields_have_name_overlap`.

* **Ensure struct casting paths use the name-based logic**:

* `ScalarValue::cast_to_with_options`: route `Struct` casts through
`nested_struct::cast_column`.
* `ColumnarValue::cast_to`: for `Struct` targets, cast via
`nested_struct::cast_column`; non-struct casts still use Arrow’s
standard casting.

* **Type coercion improvements for structs in binary operators / CASE**:

* When two structs have at least one overlapping name, coerce **by
name**.
  * Otherwise, preserve prior behavior by coercing **positionally**.

* **Planning-time cast validation for struct-to-struct**:

* `physical-expr` CAST planning now validates struct compatibility using
the same rules as runtime (`validate_struct_compatibility`) to fail
fast.
* `ExprSchemable` allows struct-to-struct casts to pass type checking;
detailed compatibility is enforced by the runtime / planning-time
validator.

* **Optimizer safety**:

  * Avoid const-folding struct casts when field counts differ.
* Avoid const-folding casts of **0-row** struct literals due to
evaluation batch dimension mismatches.

* **Tests and SQL logic tests**:

  * New unit tests covering:

    * name-based reordering
    * missing fields (nullable vs non-nullable)
    * null struct fields and nested nulls
    * positional fallback with no overlap
    * coercion behavior and simplifier behavior
* Updated/added `.slt` cases to reflect the new semantics and to add
coverage for struct casts and nested struct reordering.

* **Minor docs/maintenance**:

* Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t
break when the `parquet` feature is disabled.

## Are these changes tested?

Yes.

* Added/updated Rust unit tests in:

  * `datafusion/common/src/nested_struct.rs`
  * `datafusion/expr-common/src/columnar_value.rs`
  * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs`
* Added/updated SQL logic tests in:

  * `datafusion/sqllogictest/test_files/case.slt`
  * `datafusion/sqllogictest/test_files/struct.slt`

These tests cover:

* correct value mapping when struct field order differs
* nested struct reordering
* insertion of nulls for missing nullable fields
* erroring on missing non-nullable target fields
* positional mapping fallback when there is no name overlap
* planning-time validation vs runtime behavior alignment

## Are there any user-facing changes?

Yes.

* **Struct casts are now name-based** (case-sensitive): fields are
matched by name, reordered to the target schema, missing fields are
null-filled (if nullable), and extra fields are ignored.
* **Fallback behavior**: if there is *no* name overlap and field counts
match, casting proceeds **positionally**.
* **Potential behavior change** in queries relying on the prior
positional behavior when structs shared names but were out of order
(previously could yield swapped values). This PR changes that to the
safer, expected behavior.

No public API changes are introduced, but this is a semantic change in
struct casting.

## LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed and tested.
12 files changed
tree: 88c6e50ca92db52158b1bfdbabd9419a87fe0fd4
  1. .devcontainer/
  2. .github/
  3. benchmarks/
  4. ci/
  5. datafusion/
  6. datafusion-cli/
  7. datafusion-examples/
  8. dev/
  9. docs/
  10. python/
  11. test-utils/
  12. .asf.yaml
  13. .dockerignore
  14. .editorconfig
  15. .gitattributes
  16. .gitignore
  17. .gitmodules
  18. Cargo.lock
  19. Cargo.toml
  20. CHANGELOG.md
  21. clippy.toml
  22. CODE_OF_CONDUCT.md
  23. CONTRIBUTING.md
  24. doap.rdf
  25. header
  26. LICENSE.txt
  27. licenserc.toml
  28. NOTICE.txt
  29. pre-commit.sh
  30. README.md
  31. rust-toolchain.toml
  32. rustfmt.toml
  33. taplo.toml
  34. typos.toml
README.md

Apache DataFusion

Crates.io Apache licensed Build Status Commit Activity Open Issues Pending PRs Discord chat Linkedin Crates.io MSRV

Website | API Docs | Chat

DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format.

This crate provides libraries and binaries for developers building fast and feature rich database and analytic systems, customized to particular workloads. See use cases for examples. The following related subprojects target end users:

“Out of the box,” DataFusion offers SQL and Dataframe APIs, excellent performance, built-in support for CSV, Parquet, JSON, and Avro, extensive customization, and a great community.

DataFusion features a full query planner, a columnar, streaming, multi-threaded, vectorized execution engine, and partitioned data sources. You can customize DataFusion at almost all points including additional data sources, query languages, functions, custom operators and more. See the Architecture section for more details.

Here are links to some important information

What can you do with this crate?

DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more. It lets you start quickly from a fully working engine, and then customize those features specific to your use. Click Here to see a list known users.

Contributing to DataFusion

Please see the contributor guide and communication pages for more information.

Crate features

This crate has several features which can be specified in your Cargo.toml.

Default features:

  • nested_expressions: functions for working with nested type function such as array_to_string
  • compression: reading files compressed with xz2, bzip2, flate2, and zstd
  • crypto_expressions: cryptographic functions such as md5 and sha256
  • datetime_expressions: date and time functions such as to_timestamp
  • encoding_expressions: encode and decode functions
  • parquet: support for reading the Apache Parquet format
  • sql: Support for sql parsing / planning
  • regex_expressions: regular expression functions, such as regexp_match
  • unicode_expressions: Include unicode aware functions such as character_length
  • unparser: enables support to reverse LogicalPlans back into SQL
  • recursive_protection: uses recursive for stack overflow protection.

Optional features:

  • avro: support for reading the Apache Avro format
  • backtrace: include backtrace information in error messages
  • parquet_encryption: support for using Parquet Modular Encryption
  • serde: enable arrow-schema's serde feature

DataFusion API Evolution and Deprecation Guidelines

Public methods in Apache DataFusion evolve over time: while we try to maintain a stable API, we also improve the API over time. As a result, we typically deprecate methods before removing them, according to the deprecation guidelines.

Dependencies and Cargo.lock

Following the guidance on committing Cargo.lock files, this project commits its Cargo.lock file.

CI uses the committed Cargo.lock file, and dependencies are updated regularly using Dependabot PRs.