AVRO-4241: [Java] Bound zero-byte collection elements per datum, not per collection (#3927) * AVRO-4241: [Java] Bound zero-byte collection elements per datum, not per collection The heap-aware zero-byte-element allocation cap (null, a zero-length fixed, an all-zero-byte record, or a recursive schema broken with a 0 minimum) was enforced per collection: readArray/readCollection and the skip/fast-reader paths each started counting from zero. Because a container file carries its own schema, an attacker can declare a record with many array<null> fields, each block individually under the limit but jointly unbounded, so a tiny payload still drives a huge aggregate allocation (e.g. ~16 array<null> fields near the per-array cap exhaust the heap; a handful burn tens of seconds of CPU). Track the cumulative zero-byte allocation per decode on a per-thread scope in SystemLimitException. GenericDatumReader.read and the static skip open the scope (scopes nest, so a delegated fast reader or a skipped writer field accumulates into the enclosing datum budget instead of resetting it); only the outermost scope resets the running total. All zero-byte call sites (GenericDatumReader read/skip, FastReaderBuilder, ReflectDatumReader) now use the cumulative checkMaxCollectionAllocation(long). Outside any scope the check falls back to the previous per-collection behaviour, so no existing caller becomes stricter. Positive-size elements are unchanged: they remain bounded per collection by the bytes-remaining check, which consumes input as it advances. Adds regression tests for a multi-field record rejected cumulatively and a within-limit record that still decodes (and confirms the budget resets between datums), on both the fast and classic reader paths. * AVRO-4241: [Java] Scope fast array reader so zero-byte cap is cumulative standalone Open a collection-allocation scope around the fast array reader's block-reading loop in a try/finally. When the fast reader is used standalone via createDatumReader(...), without GenericDatumReader.read opening the outer datum scope, the zero-byte element cap is now cumulative across all array blocks instead of degrading to a per-block stateless check, so a large array<null>-style array split across many blocks cannot bypass the cap. The scope nests into the outer datum scope on the normal path, and the finally guarantees it is always closed so ThreadLocal state cannot leak into later decodes.
[!IMPORTANT]
The Rust SDK is moving to https://github.com/apache/avro-rs. Please use it for new issues and pull requests!
Apache Avro™ is a data serialization system.
Learn more about Avro, please visit our website at:
To contribute to Avro, please read:
https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
You can use devcontainers to develop Avro:
Apache®, Apache Avro and the Apache Avro airplane logo are trademarks of The Apache Software Foundation.
The Apache Avro airplane logo on this page has been designed by Emma Kellam for use by this project.