fix(io): keep null lists of structs when projecting Arrow data (#4065)

The projection visitor rebuilds a list of structs from its offsets alone,
which cannot represent a null list, so every null list<struct<...>> was
turned into an empty list. On the write path this is persisted into the
Parquet file and the null is lost for every reader.

Rebuild the list from the original validity and offsets buffers instead,
which also keeps the original array offset so sliced batches stay correct.

Closes #3833
3 files changed
tree: 7779d592ee6624519f35570f3e45cdd64b79a37b
  1. .github/
  2. dev/
  3. mkdocs/
  4. notebooks/
  5. pyiceberg/
  6. tests/
  7. vendor/
  8. .asf.yaml
  9. .gitignore
  10. .markdownlint.yaml
  11. .pre-commit-config.yaml
  12. AGENTS.md
  13. LICENSE
  14. Makefile
  15. MANIFEST.in
  16. NOTICE
  17. pyproject.toml
  18. README.md
  19. SECURITY-THREAT-MODEL.md
  20. setup.py
  21. uv.lock
README.md

Iceberg Python

PyIceberg is a Python library for programmatic access to Iceberg table metadata as well as to table data in Iceberg format. It is a Python implementation of the Iceberg table spec.

The documentation is available at https://py.iceberg.apache.org/.

Get in Touch