Make partition expression evaluation stateless (#3664)

*I used CODEX to analyze this problem and create this PR. I've reviewed
the code and tests and stand by them. This summary is written completely
by a human (me) other than very light copy editing by an LLM.*

Currently, because `_ExpressionEvaluator` can't be safely shared one is
created per file. That's expensive relative to actually evaluating the
expression. This change makes `_ExpressionEvaluator` shareable by moving
the mutable parts into a new `_ExpressionEvaluationVisitor` which stays
local to the per-file work of evaluation.

These benchmarks are showing improvements in sub-second times, but in
the production workloads I tested this on it did translate to real wall
clock improvement. But irrespective of performance gains the new
`_ExpressionEvaluator` is safer since nothing was preventing its
accidental concurrent reuse in the future.

CODEX-generated summary follows:
-------
Partition pruning currently binds the same projected expression for
every data file because `_ExpressionEvaluator` stores the current record
as mutable instance state. That repeated preparation is expensive, but
sharing the existing evaluator across workers would allow concurrent
calls to mix records.

This separates preparation from per-record evaluation.
`_ExpressionEvaluator` binds the expression once, while every call
creates a private `_ExpressionEvaluationVisitor` containing that call's
record. Manifest planning can therefore prepare one evaluator per
partition spec and safely share it across manifests and workers. This
provides the construction-reuse benefit targeted by #3656 without
depending on files within a manifest remaining sequential, and it also
benefits one-file manifests.

## Summary

- Split the prepared expression evaluator from the call-local mutable
visitor.
- Reuse one prepared partition evaluator per partition spec during
manifest planning.
- Add deterministic concurrent-use, prepared-state, and planner-sharing
coverage.
- Add a benchmark using a realistic 15-leaf predicate across dense and
one-file manifests.

## Performance

I compared this branch with `main` using the partition-evaluator
workload in this PR. It evaluates 1,000 files with two identity
partition fields and a 15-leaf predicate modeling five `(event_day range
AND region_id)` branches. Timings are medians from seven samples of five
iterations.

| Manifest layout | `main` | This PR | Speedup |
|---|---:|---:|---:|
| 1 manifest × 1,000 files | 187.928 ms | 14.413 ms | 13.04× |
| 1,000 manifests × 1 file | 188.000 ms | 14.584 ms | 12.89× |

The improvement comes from binding the projected partition expression
once per partition spec instead of once per file. This is an isolated
partition-pruning benchmark rather than an end-to-end scan-planning
measurement.

## Testing

- `pytest tests/expressions/test_visitors.py
tests/table/test_partition_evaluator_planning.py
tests/table/test_init.py` (200 passed)
- `pytest tests/benchmark/test_partition_evaluator_benchmark.py -m
benchmark` (2 passed)
5 files changed
tree: f6681522a074321c19d82a2c45f4ae550619af85
  1. .github/
  2. dev/
  3. mkdocs/
  4. notebooks/
  5. pyiceberg/
  6. tests/
  7. vendor/
  8. .asf.yaml
  9. .codespellrc
  10. .gitignore
  11. .markdownlint.yaml
  12. .pre-commit-config.yaml
  13. AGENTS.md
  14. LICENSE
  15. Makefile
  16. MANIFEST.in
  17. NOTICE
  18. pyproject.toml
  19. README.md
  20. ruff.toml
  21. SECURITY-THREAT-MODEL.md
  22. setup.py
  23. uv.lock
README.md

Iceberg Python

PyIceberg is a Python library for programmatic access to Iceberg table metadata as well as to table data in Iceberg format. It is a Python implementation of the Iceberg table spec.

The documentation is available at https://py.iceberg.apache.org/.

Get in Touch