AVRO-4241: [Java] Bound zero-byte collection elements per datum, not per collection (#3927)

* AVRO-4241: [Java] Bound zero-byte collection elements per datum, not per collection

The heap-aware zero-byte-element allocation cap (null, a zero-length fixed, an
all-zero-byte record, or a recursive schema broken with a 0 minimum) was enforced
per collection: readArray/readCollection and the skip/fast-reader paths each
started counting from zero. Because a container file carries its own schema, an
attacker can declare a record with many array<null> fields, each block
individually under the limit but jointly unbounded, so a tiny payload still drives
a huge aggregate allocation (e.g. ~16 array<null> fields near the per-array cap
exhaust the heap; a handful burn tens of seconds of CPU).

Track the cumulative zero-byte allocation per decode on a per-thread scope in
SystemLimitException. GenericDatumReader.read and the static skip open the scope
(scopes nest, so a delegated fast reader or a skipped writer field accumulates
into the enclosing datum budget instead of resetting it); only the outermost
scope resets the running total. All zero-byte call sites (GenericDatumReader
read/skip, FastReaderBuilder, ReflectDatumReader) now use the cumulative
checkMaxCollectionAllocation(long). Outside any scope the check falls back to the
previous per-collection behaviour, so no existing caller becomes stricter.
Positive-size elements are unchanged: they remain bounded per collection by the
bytes-remaining check, which consumes input as it advances.

Adds regression tests for a multi-field record rejected cumulatively and a
within-limit record that still decodes (and confirms the budget resets between
datums), on both the fast and classic reader paths.

* AVRO-4241: [Java] Scope fast array reader so zero-byte cap is cumulative standalone

Open a collection-allocation scope around the fast array reader's
block-reading loop in a try/finally. When the fast reader is used
standalone via createDatumReader(...), without GenericDatumReader.read
opening the outer datum scope, the zero-byte element cap is now
cumulative across all array blocks instead of degrading to a per-block
stateless check, so a large array<null>-style array split across many
blocks cannot bypass the cap. The scope nests into the outer datum
scope on the normal path, and the finally guarantees it is always
closed so ThreadLocal state cannot leak into later decodes.
5 files changed
tree: b171573f635488279549004b11c27798d28a781f
  1. .devcontainer/
  2. .github/
  3. .mvn/
  4. doc/
  5. lang/
  6. share/
  7. .asf.yaml
  8. .editorconfig
  9. .gitattributes
  10. .gitignore
  11. .yamllint.yml
  12. BUILD.md
  13. build.sh
  14. composer.json
  15. DIST_README.txt
  16. docker-compose.yml
  17. LICENSE.txt
  18. NOTICE.txt
  19. pom.xml
  20. README.md
README.md

[!IMPORTANT]
The Rust SDK is moving to https://github.com/apache/avro-rs. Please use it for new issues and pull requests!

Apache Avro™

Current CI status (Github servers)

test c test c# test c++ test java test javascript test perl test ruby test python test php

Current CodeQL status

codeql c# codeql java codeql javascript codeql python


Apache Avro™ is a data serialization system.

Learn more about Avro, please visit our website at:

https://avro.apache.org/

To contribute to Avro, please read:

https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute

You can use devcontainers to develop Avro:

  • Open in Visual Studio Code
  • Open in Github Codespaces

Trademark & logos

Apache®, Apache Avro and the Apache Avro airplane logo are trademarks of The Apache Software Foundation.

The Apache Avro airplane logo on this page has been designed by Emma Kellam for use by this project.