Make partition expression evaluation stateless (#3664) *I used CODEX to analyze this problem and create this PR. I've reviewed the code and tests and stand by them. This summary is written completely by a human (me) other than very light copy editing by an LLM.* Currently, because `_ExpressionEvaluator` can't be safely shared one is created per file. That's expensive relative to actually evaluating the expression. This change makes `_ExpressionEvaluator` shareable by moving the mutable parts into a new `_ExpressionEvaluationVisitor` which stays local to the per-file work of evaluation. These benchmarks are showing improvements in sub-second times, but in the production workloads I tested this on it did translate to real wall clock improvement. But irrespective of performance gains the new `_ExpressionEvaluator` is safer since nothing was preventing its accidental concurrent reuse in the future. CODEX-generated summary follows: ------- Partition pruning currently binds the same projected expression for every data file because `_ExpressionEvaluator` stores the current record as mutable instance state. That repeated preparation is expensive, but sharing the existing evaluator across workers would allow concurrent calls to mix records. This separates preparation from per-record evaluation. `_ExpressionEvaluator` binds the expression once, while every call creates a private `_ExpressionEvaluationVisitor` containing that call's record. Manifest planning can therefore prepare one evaluator per partition spec and safely share it across manifests and workers. This provides the construction-reuse benefit targeted by #3656 without depending on files within a manifest remaining sequential, and it also benefits one-file manifests. ## Summary - Split the prepared expression evaluator from the call-local mutable visitor. - Reuse one prepared partition evaluator per partition spec during manifest planning. - Add deterministic concurrent-use, prepared-state, and planner-sharing coverage. - Add a benchmark using a realistic 15-leaf predicate across dense and one-file manifests. ## Performance I compared this branch with `main` using the partition-evaluator workload in this PR. It evaluates 1,000 files with two identity partition fields and a 15-leaf predicate modeling five `(event_day range AND region_id)` branches. Timings are medians from seven samples of five iterations. | Manifest layout | `main` | This PR | Speedup | |---|---:|---:|---:| | 1 manifest × 1,000 files | 187.928 ms | 14.413 ms | 13.04× | | 1,000 manifests × 1 file | 188.000 ms | 14.584 ms | 12.89× | The improvement comes from binding the projected partition expression once per partition spec instead of once per file. This is an isolated partition-pruning benchmark rather than an end-to-end scan-planning measurement. ## Testing - `pytest tests/expressions/test_visitors.py tests/table/test_partition_evaluator_planning.py tests/table/test_init.py` (200 passed) - `pytest tests/benchmark/test_partition_evaluator_benchmark.py -m benchmark` (2 passed)
PyIceberg is a Python library for programmatic access to Iceberg table metadata as well as to table data in Iceberg format. It is a Python implementation of the Iceberg table spec.
The documentation is available at https://py.iceberg.apache.org/.