GH-3719: Fix vectored read allocation limits and fallback safety (#3726) ## Why are the changes needed? The Hadoop vectored-read path can fall back to ordinary reads after filesystem requests have been submitted or results partially consumed. Replaying reads into the same chunk builder can duplicate selected page data and return incorrect filtered rows. Outstanding requests also need to retain ownership of their stream and buffers during failure cleanup. The current path requests a single buffer for an entire contiguous projected range, without splitting it at `parquet.read.allocation.size`. ## What changes were proposed in this PR? - Split requested ranges at the configured allocation size while preserving the logical page/column plan. Verify dictionary and V1/V2 page checksums across split buffers without another contiguous checksum allocation. - Cache file length lazily. A non-interruption metadata lookup failure before submission disables vectored IO for that reader and permits ordinary reads. - Fail the read after filesystem submission is attempted, including synchronous filesystem rejection; the interface cannot reliably distinguish rejection from partial submission. - Share bounded vectored-read workers across readers: 64 by default, configurable once through the JVM system property `parquet.hadoop.vectored.io.threads`. Admission, submission, and range waits share one 300-second deadline. Admission failure leaves the stream with its reader. Accepted operations retain capacity until submission exits, the caller transfers or abandons buffer ownership, and failure cleanup finishes. Cancellation cannot free capacity prematurely. Idle workers in the default pool expire after 60 seconds. - Allow engines to supply and share their own pool through `withVectoredReadExecutor(new VectoredReadExecutor(executor, maximumConcurrentReads))` on read options or the `ParquetReader` builder. Reuse the wrapper to share an admission limit across readers; both base and Hadoop options preserve it when copied. The shared JVM pool remains the default. Parquet never shuts down a caller-owned pool. Reject inline execution before filesystem submission and preserve the reader for retry. - Track original allocator buffers, including merged and checksum allocations, and transfer them to the row-group releaser on success. Preserve automatic row-group cleanup on sequential reads and reader closure. - Publish available Hadoop futures even when submission fails. After an operation is accepted, invalidate failed readers and defer stream cleanup until submission exits; release buffers only when completion can be established. Guarantee abort with `finally` even when draining sibling reads throws an `Error`, and retain the original failure as suppressed on the escaping error. Remove the unused CRC allocator. The allocation setting bounds requested range lengths, not every filesystem allocation: backends may merge ranges or align them for checksums and allocate larger buffers. It does not bound footer/decoder allocations or total memory. The worker limit does not bound threads created by the filesystem. A backend that ignores interruption, or a blocked stream close, can retain capacity indefinitely; when all workers are occupied, further reads time out at admission. Missing, cancelled, or never-completing result futures can prevent safe buffer reclamation. Caller-supplied executors must dispatch asynchronously, reject instead of silently dropping work, and drain accepted tasks during shutdown. Reading threads must not exhaust that same pool while waiting for their vectored work. ## How was this PR tested? Local validation used Java 17.0.20.1 with Maven 3.9.16 and Thrift 0.24.0: - Full `parquet-hadoop` verification with default Hadoop 3.3.0: **877 tests, zero failures/errors, 26 Hadoop-capability skips**, plus **17 integration tests with zero failures/errors/skips**. Packaging, API compatibility, dependency analysis, Spotless, and RAT (274 approved licenses) passed. - Focused Hadoop 3.4.1 run with `-Dparquet.hadoop.vectored.io.threads=2`: **118 tests, zero failures/errors/skips**, including the real local-filesystem vectored and file-range bridge tests. - `git diff --check` passed. The focused run selected `TestParquetFileReaderVectoredIO,TestParquetFileReaderVectoredOwnership,TestVectoredReadBufferAllocator,TestVectoredReadOperation,TestVectoredReadExecutor,TestDataPageChecksums,TestVectorIoBridge,TestFileRangeBridge,TestParquetFileReaderBufferLeak`. Coverage includes shared admission across repeated retries, capacity retained through blocked submission and cleanup, original-buffer ownership, interrupted admission and reader reuse, queued aborts, and sibling errors with pending reads. New regressions exercise caller-owned pools through both options builders and copies, successful reads and error cleanup without pool shutdown, inline and `CallerRunsPolicy` rejection, permit recovery, and retrying the same reader after inline rejection. The full unit suite used a RAM-backed temporary directory for its large-file tests. Local setup also used a test-JVM hosts file for the machine hostname and interop fixtures fetched at the commits pinned in the tests. No live S3 validation or performance benchmark was performed. GitHub CI for head `effbc384` also passed: - Hadoop JDK 17 verification ran on merge revision `ad7b9934`, which includes this head: **898 `parquet-hadoop` tests, zero failures/errors, 26 skips**. With default Hadoop 3.3.0, all 22 `TestVectorIoBridge` tests were skipped; real local-filesystem bridge coverage comes from the separate Hadoop 3.4.1 run above. - Vector-plugin verification on JDK 17, 21, and 25, and zizmor analysis passed. Closes #3719.
This repository contains a Java implementation of Apache Parquet
Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.
The parquet-format repository contains the file format specification.
Parquet uses the record shredding and assembly algorithm described in the Dremel paper to represent nested structures. You can find additional details about the format and intended use cases in our Hadoop Summit 2013 presentation
Parquet-Java requires Java 17 or higher and uses Maven to build. It also depends on the thrift compiler (protoc is now managed by maven plugin).
To build and install the thrift compiler, run:
wget -nv https://archive.apache.org/dist/thrift/0.25.0/thrift-0.25.0.tar.gz tar xzf thrift-0.25.0.tar.gz cd thrift-0.25.0 chmod +x ./configure ./configure --disable-libs sudo make install -j
Note: if you wish to verify the signature and checksum of a release:
If you're on OSX and use homebrew, you can instead install Thrift 0.25.0 with brew and ensure that it comes first in your PATH.
brew install thrift export PATH="$(brew --prefix thrift)/bin:$PATH"
Parquet-Java uses an inlined version of the parquet.thrift IDL, located at parquet-format-structures/src/main/thrift. To update this definition to include changes from the canonical definition in parquet-format, you can use:
# Update to a parquet-format release > ./dev/update-parquet-thrift.sh apache-parquet-format-2.13.0 # Update to an arbitrary parquet-format commit (can be unreleased) > ./dev/update-parquet-thrift.sh <commit hash> # Update to a commit from a parquet-format fork (e.g. for a reference implementation of an in-progress spec change) > PARQUET_FORMAT_REPO="https://github.com/divjotarora/parquet-format" ./dev/update-parquet-thrift.sh <commit hash>
Once protobuf and thrift are available in your path, you can build the project by running:
LC_ALL=C ./mvnw clean install
Parquet is an active project, and new features are being added quickly. Here are a few features:
The feature is experimental and is currently not part of the parquet distribution.
Parquet-Java has supported Java Vector API to speed up reading, to enable this feature:
./mvnw clean package -P vector-pluginsInput and Output formats. Note that to use an Input or Output format, you need to implement a WriteSupport or ReadSupport class, which will implement the conversion of your object to and from a Parquet schema.
We've implemented this for 2 popular data formats to provide a clean migration path as well:
Thrift integration is provided by the parquet-thrift sub-project.
Avro conversion is implemented via the parquet-avro sub-project.
Protobuf conversion is implemented via the parquet-protobuf sub-project.
See the APIs:
Hive integration is now deprecated within the Parquet project. It is now maintained by Apache Hive.
To run the unit tests: ./mvnw test
To build the jars: ./mvnw package
The build runs in GitHub Actions:
The current release is version 1.18.1.
<dependencies> <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-common</artifactId> <version>1.18.1</version> </dependency> <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-encoding</artifactId> <version>1.18.1</version> </dependency> <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-column</artifactId> <version>1.18.1</version> </dependency> <dependency> <groupId>org.apache.parquet</groupId> <artifactId>parquet-hadoop</artifactId> <version>1.18.1</version> </dependency> </dependencies>
We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the parquet-java Git repository. If you've previously forked Parquet from its old location, you will need to add a remote or update your origin remote to https://github.com/apache/parquet-java.git.
If you are looking for some ideas on what to contribute, check out GitHub issues for labeled Good first issue. Comment on the issue and/or contact dev@parquet.apache.org with your questions and ideas.
If you’d like to report a bug but don’t have time to fix it, you can still raise an issue on GitHub, or email the mailing list dev@parquet.apache.org.
To contribute a patch:
./mvnw test in the root directory.We tend to do fairly close readings of pull requests, and you may get a lot of comments. Some common issues that are not code structure related, but still important:
a+b but a + b and not foo(int a,int b) but foo(int a, int b).Thank you for getting involved!
We hold ourselves and the Parquet developer community to two codes of conduct:
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0