GH-3719: Fix vectored read allocation limits and fallback safety (#3726)

## Why are the changes needed?

The Hadoop vectored-read path can fall back to ordinary reads after filesystem
requests have been submitted or results partially consumed. Replaying reads into
the same chunk builder can duplicate selected page data and return incorrect
filtered rows. Outstanding requests also need to retain ownership of their stream
and buffers during failure cleanup.

The current path requests a single buffer for an entire contiguous projected
range, without splitting it at `parquet.read.allocation.size`.

## What changes were proposed in this PR?

- Split requested ranges at the configured allocation size while preserving the
  logical page/column plan. Verify dictionary and V1/V2 page checksums across split
  buffers without another contiguous checksum allocation.
- Cache file length lazily. A non-interruption metadata lookup failure before
  submission disables vectored IO for that reader and permits ordinary reads.
- Fail the read after filesystem submission is attempted, including synchronous
  filesystem rejection; the interface cannot reliably distinguish rejection from
  partial submission.
- Share bounded vectored-read workers across readers: 64 by default, configurable
  once through the JVM system property `parquet.hadoop.vectored.io.threads`.
  Admission, submission, and range waits share one 300-second deadline. Admission
  failure leaves the stream with its reader. Accepted operations retain capacity
  until submission exits, the caller transfers or abandons buffer ownership, and
  failure cleanup finishes. Cancellation cannot free capacity prematurely.
  Idle workers in the default pool expire after 60 seconds.
- Allow engines to supply and share their own pool through
  `withVectoredReadExecutor(new VectoredReadExecutor(executor, maximumConcurrentReads))`
  on read options or the `ParquetReader` builder. Reuse the wrapper to share an
  admission limit across readers; both base and Hadoop options preserve it when
  copied. The shared JVM pool remains the default. Parquet never shuts down a
  caller-owned pool. Reject inline execution before filesystem submission and
  preserve the reader for retry.
- Track original allocator buffers, including merged and checksum allocations,
  and transfer them to the row-group releaser on success. Preserve automatic
  row-group cleanup on sequential reads and reader closure.
- Publish available Hadoop futures even when submission fails. After an operation
  is accepted, invalidate failed readers and defer stream cleanup until submission
  exits; release buffers only when completion can be established. Guarantee abort
  with `finally` even when draining sibling reads throws an `Error`, and retain the
  original failure as suppressed on the escaping error. Remove the unused CRC allocator.

The allocation setting bounds requested range lengths, not every filesystem
allocation: backends may merge ranges or align them for checksums and allocate
larger buffers. It does not bound footer/decoder allocations or total memory.
The worker limit does not bound threads created by the filesystem. A backend
that ignores interruption, or a blocked stream close, can retain capacity
indefinitely; when all workers are occupied, further reads time out at admission.
Missing, cancelled, or never-completing result futures can prevent safe buffer
reclamation. Caller-supplied executors must dispatch asynchronously, reject instead
of silently dropping work, and drain accepted tasks during shutdown. Reading
threads must not exhaust that same pool while waiting for their vectored work.

## How was this PR tested?

Local validation used Java 17.0.20.1 with Maven 3.9.16 and Thrift 0.24.0:

- Full `parquet-hadoop` verification with default Hadoop 3.3.0: **877 tests,
  zero failures/errors, 26 Hadoop-capability skips**, plus **17 integration tests
  with zero failures/errors/skips**. Packaging, API compatibility, dependency
  analysis, Spotless, and RAT (274 approved licenses) passed.
- Focused Hadoop 3.4.1 run with `-Dparquet.hadoop.vectored.io.threads=2`:
  **118 tests, zero failures/errors/skips**, including the real local-filesystem
  vectored and file-range bridge tests.
- `git diff --check` passed.

The focused run selected
`TestParquetFileReaderVectoredIO,TestParquetFileReaderVectoredOwnership,TestVectoredReadBufferAllocator,TestVectoredReadOperation,TestVectoredReadExecutor,TestDataPageChecksums,TestVectorIoBridge,TestFileRangeBridge,TestParquetFileReaderBufferLeak`.

Coverage includes shared admission across repeated retries, capacity retained
through blocked submission and cleanup, original-buffer ownership, interrupted
admission and reader reuse, queued aborts, and sibling errors with pending reads.
New regressions exercise caller-owned pools through both options builders and
copies, successful reads and error cleanup without pool shutdown, inline and
`CallerRunsPolicy` rejection, permit recovery, and retrying the same reader after
inline rejection.

The full unit suite used a RAM-backed temporary directory for its large-file tests.
Local setup also used a test-JVM hosts file for the machine hostname and interop
fixtures fetched at the commits pinned in the tests. No live S3 validation or
performance benchmark was performed.

GitHub CI for head `effbc384` also passed:

- Hadoop JDK 17 verification ran on merge revision `ad7b9934`, which includes this
  head: **898 `parquet-hadoop` tests, zero failures/errors, 26 skips**. With default
  Hadoop 3.3.0, all 22 `TestVectorIoBridge` tests were skipped; real local-filesystem
  bridge coverage comes from the separate Hadoop 3.4.1 run above.
- Vector-plugin verification on JDK 17, 21, and 25, and zizmor analysis passed.

Closes #3719.
16 files changed
tree: 2dae917c5364357e46d6e10f2a65af4fbe938217
  1. .github/
  2. .mvn/
  3. agents/
  4. dev/
  5. doc/
  6. parquet-arrow/
  7. parquet-avro/
  8. parquet-benchmarks/
  9. parquet-cli/
  10. parquet-column/
  11. parquet-common/
  12. parquet-encoding/
  13. parquet-format-structures/
  14. parquet-generator/
  15. parquet-hadoop/
  16. parquet-hadoop-bundle/
  17. parquet-jackson/
  18. parquet-plugins/
  19. parquet-protobuf/
  20. parquet-thrift/
  21. parquet-variant/
  22. release/
  23. src/
  24. .asf.yaml
  25. .editorconfig
  26. .gitattributes
  27. .gitignore
  28. AGENTS.md
  29. changelog.sh
  30. CHANGES.md
  31. LICENSE
  32. mvnw
  33. mvnw.cmd
  34. NOTICE
  35. pom.xml
  36. PoweredBy.md
  37. README.md
  38. SECURITY.md
README.md

Parquet Java (formerly Parquet MR) CI Hadoop 3

This repository contains a Java implementation of Apache Parquet

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.

The parquet-format repository contains the file format specification.

Parquet uses the record shredding and assembly algorithm described in the Dremel paper to represent nested structures. You can find additional details about the format and intended use cases in our Hadoop Summit 2013 presentation

Building

Parquet-Java requires Java 17 or higher and uses Maven to build. It also depends on the thrift compiler (protoc is now managed by maven plugin).

Install Thrift

To build and install the thrift compiler, run:

wget -nv https://archive.apache.org/dist/thrift/0.25.0/thrift-0.25.0.tar.gz
tar xzf thrift-0.25.0.tar.gz
cd thrift-0.25.0
chmod +x ./configure
./configure --disable-libs
sudo make install -j

Note: if you wish to verify the signature and checksum of a release:

  1. The GPG and sha checksums can be found under https://archive.apache.org/dist/thrift/0.25.0/
  2. Validate the signature of the artifact against the Thrift committer KEYS.

If you're on OSX and use homebrew, you can instead install Thrift 0.25.0 with brew and ensure that it comes first in your PATH.

brew install thrift
export PATH="$(brew --prefix thrift)/bin:$PATH"

Update Parquet Thrift Definitions

Parquet-Java uses an inlined version of the parquet.thrift IDL, located at parquet-format-structures/src/main/thrift. To update this definition to include changes from the canonical definition in parquet-format, you can use:

# Update to a parquet-format release
> ./dev/update-parquet-thrift.sh apache-parquet-format-2.13.0

# Update to an arbitrary parquet-format commit (can be unreleased)
> ./dev/update-parquet-thrift.sh <commit hash>

# Update to a commit from a parquet-format fork (e.g. for a reference implementation of an in-progress spec change)
> PARQUET_FORMAT_REPO="https://github.com/divjotarora/parquet-format" ./dev/update-parquet-thrift.sh <commit hash>

Build Parquet with Maven

Once protobuf and thrift are available in your path, you can build the project by running:

LC_ALL=C ./mvnw clean install

Features

Parquet is an active project, and new features are being added quickly. Here are a few features:

  • Type-specific encoding
  • Hive integration (deprecated)
  • Pig integration (deprecated)
  • Cascading integration (deprecated)
  • Crunch integration
  • Apache Arrow integration
  • Scrooge integration (deprecated)
  • Impala integration (non-nested)
  • Java Map/Reduce API
  • Native Avro support
  • Native Thrift support
  • Native Protocol Buffers support
  • Complex structure support
  • Run-length encoding (RLE)
  • Bit Packing
  • Adaptive dictionary encoding
  • Predicate pushdown
  • Column stats
  • Delta encoding
  • Index pages
  • Scala DSL (deprecated)
  • Java Vector API support (experimental)

Java Vector API support

The feature is experimental and is currently not part of the parquet distribution.

Parquet-Java has supported Java Vector API to speed up reading, to enable this feature:

  • Java 17+, 64-bit
  • Requiring the CPU to support instruction sets:
    • avx512vbmi
    • avx512_vbmi2
  • To build the jars: ./mvnw clean package -P vector-plugins
  • For Apache Spark to enable this feature:
    • Build parquet and replace the parquet-encoding-{VERSION}.jar on the spark jars folder
    • Build parquet-encoding-vector and copy parquet-encoding-vector-{VERSION}.jar to the spark jars folder
    • Edit spark class#VectorizedRleValuesReader, function#readNextGroup refer to parquet class#ParquetReadRouter, function#readBatchUsing512Vector
    • Build spark with maven and replace spark-sql_2.12-{VERSION}.jar on the spark jars folder

Map/Reduce integration

Input and Output formats. Note that to use an Input or Output format, you need to implement a WriteSupport or ReadSupport class, which will implement the conversion of your object to and from a Parquet schema.

We've implemented this for 2 popular data formats to provide a clean migration path as well:

Thrift

Thrift integration is provided by the parquet-thrift sub-project.

Avro

Avro conversion is implemented via the parquet-avro sub-project.

Protobuf

Protobuf conversion is implemented via the parquet-protobuf sub-project.

Create your own objects

  • The ParquetOutputFormat can be provided a WriteSupport to write your own objects to an event based RecordConsumer.
  • The ParquetInputFormat can be provided a ReadSupport to materialize your own objects by implementing a RecordMaterializer

See the APIs:

Hive integration

Hive integration is now deprecated within the Parquet project. It is now maintained by Apache Hive.

Build

To run the unit tests: ./mvnw test

To build the jars: ./mvnw package

The build runs in GitHub Actions: Build Status

Add Parquet as a dependency in Maven

The current release is version 1.18.1.

  <dependencies>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-common</artifactId>
      <version>1.18.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-encoding</artifactId>
      <version>1.18.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-column</artifactId>
      <version>1.18.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-hadoop</artifactId>
      <version>1.18.1</version>
    </dependency>
  </dependencies>

How To Contribute

We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the parquet-java Git repository. If you've previously forked Parquet from its old location, you will need to add a remote or update your origin remote to https://github.com/apache/parquet-java.git.

If you are looking for some ideas on what to contribute, check out GitHub issues for labeled Good first issue. Comment on the issue and/or contact dev@parquet.apache.org with your questions and ideas.

If you’d like to report a bug but don’t have time to fix it, you can still raise an issue on GitHub, or email the mailing list dev@parquet.apache.org.

To contribute a patch:

  1. Break your work into small, single-purpose patches if possible. It’s much harder to merge in a large change with a lot of disjoint features.
  2. Create an issue for your patch on the GitHub issues.
  3. Submit the patch as a GitHub pull request against the master branch. For a tutorial, see the GitHub guides on forking a repo and sending a pull request. Prefix your pull request name with the issue (ex: https://github.com/apache/parquet-java/pull/3260).
  4. Make sure that your code passes the unit tests. You can run the tests with ./mvnw test in the root directory.
  5. Add new unit tests for your code.

We tend to do fairly close readings of pull requests, and you may get a lot of comments. Some common issues that are not code structure related, but still important:

  • Use 2 spaces for whitespace. Not tabs, not 4 spaces. The number of the spacing shall be 2.
  • Give your operators some room. Not a+b but a + b and not foo(int a,int b) but foo(int a, int b).
  • Generally speaking, stick to the Sun Java Code Conventions
  • Make sure tests pass!

Thank you for getting involved!

Authors and contributors

Code of Conduct

We hold ourselves and the Parquet developer community to two codes of conduct:

  1. The Apache Software Foundation Code of Conduct
  2. The Twitter OSS Code of Conduct

Discussions

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0