)]}'
{
  "commit": "c972bd040f5448ba551877e87941ff20fe96cb9f",
  "tree": "2dae917c5364357e46d6e10f2a65af4fbe938217",
  "parents": [
    "cc4e2bd7b2755178174f267827d3a4e2d28007d0"
  ],
  "author": {
    "name": "Chao Sun",
    "email": "chao@openai.com",
    "time": "Wed Oct 21 01:54:09 2026 -0700"
  },
  "committer": {
    "name": "GitHub",
    "email": "noreply@github.com",
    "time": "Wed Oct 21 16:54:09 2026 +0800"
  },
  "message": "GH-3719: Fix vectored read allocation limits and fallback safety (#3726)\n\n## Why are the changes needed?\n\nThe Hadoop vectored-read path can fall back to ordinary reads after filesystem\nrequests have been submitted or results partially consumed. Replaying reads into\nthe same chunk builder can duplicate selected page data and return incorrect\nfiltered rows. Outstanding requests also need to retain ownership of their stream\nand buffers during failure cleanup.\n\nThe current path requests a single buffer for an entire contiguous projected\nrange, without splitting it at `parquet.read.allocation.size`.\n\n## What changes were proposed in this PR?\n\n- Split requested ranges at the configured allocation size while preserving the\n  logical page/column plan. Verify dictionary and V1/V2 page checksums across split\n  buffers without another contiguous checksum allocation.\n- Cache file length lazily. A non-interruption metadata lookup failure before\n  submission disables vectored IO for that reader and permits ordinary reads.\n- Fail the read after filesystem submission is attempted, including synchronous\n  filesystem rejection; the interface cannot reliably distinguish rejection from\n  partial submission.\n- Share bounded vectored-read workers across readers: 64 by default, configurable\n  once through the JVM system property `parquet.hadoop.vectored.io.threads`.\n  Admission, submission, and range waits share one 300-second deadline. Admission\n  failure leaves the stream with its reader. Accepted operations retain capacity\n  until submission exits, the caller transfers or abandons buffer ownership, and\n  failure cleanup finishes. Cancellation cannot free capacity prematurely.\n  Idle workers in the default pool expire after 60 seconds.\n- Allow engines to supply and share their own pool through\n  `withVectoredReadExecutor(new VectoredReadExecutor(executor, maximumConcurrentReads))`\n  on read options or the `ParquetReader` builder. Reuse the wrapper to share an\n  admission limit across readers; both base and Hadoop options preserve it when\n  copied. The shared JVM pool remains the default. Parquet never shuts down a\n  caller-owned pool. Reject inline execution before filesystem submission and\n  preserve the reader for retry.\n- Track original allocator buffers, including merged and checksum allocations,\n  and transfer them to the row-group releaser on success. Preserve automatic\n  row-group cleanup on sequential reads and reader closure.\n- Publish available Hadoop futures even when submission fails. After an operation\n  is accepted, invalidate failed readers and defer stream cleanup until submission\n  exits; release buffers only when completion can be established. Guarantee abort\n  with `finally` even when draining sibling reads throws an `Error`, and retain the\n  original failure as suppressed on the escaping error. Remove the unused CRC allocator.\n\nThe allocation setting bounds requested range lengths, not every filesystem\nallocation: backends may merge ranges or align them for checksums and allocate\nlarger buffers. It does not bound footer/decoder allocations or total memory.\nThe worker limit does not bound threads created by the filesystem. A backend\nthat ignores interruption, or a blocked stream close, can retain capacity\nindefinitely; when all workers are occupied, further reads time out at admission.\nMissing, cancelled, or never-completing result futures can prevent safe buffer\nreclamation. Caller-supplied executors must dispatch asynchronously, reject instead\nof silently dropping work, and drain accepted tasks during shutdown. Reading\nthreads must not exhaust that same pool while waiting for their vectored work.\n\n## How was this PR tested?\n\nLocal validation used Java 17.0.20.1 with Maven 3.9.16 and Thrift 0.24.0:\n\n- Full `parquet-hadoop` verification with default Hadoop 3.3.0: **877 tests,\n  zero failures/errors, 26 Hadoop-capability skips**, plus **17 integration tests\n  with zero failures/errors/skips**. Packaging, API compatibility, dependency\n  analysis, Spotless, and RAT (274 approved licenses) passed.\n- Focused Hadoop 3.4.1 run with `-Dparquet.hadoop.vectored.io.threads\u003d2`:\n  **118 tests, zero failures/errors/skips**, including the real local-filesystem\n  vectored and file-range bridge tests.\n- `git diff --check` passed.\n\nThe focused run selected\n`TestParquetFileReaderVectoredIO,TestParquetFileReaderVectoredOwnership,TestVectoredReadBufferAllocator,TestVectoredReadOperation,TestVectoredReadExecutor,TestDataPageChecksums,TestVectorIoBridge,TestFileRangeBridge,TestParquetFileReaderBufferLeak`.\n\nCoverage includes shared admission across repeated retries, capacity retained\nthrough blocked submission and cleanup, original-buffer ownership, interrupted\nadmission and reader reuse, queued aborts, and sibling errors with pending reads.\nNew regressions exercise caller-owned pools through both options builders and\ncopies, successful reads and error cleanup without pool shutdown, inline and\n`CallerRunsPolicy` rejection, permit recovery, and retrying the same reader after\ninline rejection.\n\nThe full unit suite used a RAM-backed temporary directory for its large-file tests.\nLocal setup also used a test-JVM hosts file for the machine hostname and interop\nfixtures fetched at the commits pinned in the tests. No live S3 validation or\nperformance benchmark was performed.\n\nGitHub CI for head `effbc384` also passed:\n\n- Hadoop JDK 17 verification ran on merge revision `ad7b9934`, which includes this\n  head: **898 `parquet-hadoop` tests, zero failures/errors, 26 skips**. With default\n  Hadoop 3.3.0, all 22 `TestVectorIoBridge` tests were skipped; real local-filesystem\n  bridge coverage comes from the separate Hadoop 3.4.1 run above.\n- Vector-plugin verification on JDK 17, 21, and 25, and zizmor analysis passed.\n\nCloses #3719.",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "13b513270182e8ddb541931715b23109f59e60b4",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/README.md",
      "new_id": "2025243143610660a80123a0d3865915103495fc",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/README.md"
    },
    {
      "type": "modify",
      "old_id": "e277fcde5f8d0d5f0b5bc96f3b4c4b042117458c",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/main/java/org/apache/parquet/HadoopReadOptions.java",
      "new_id": "69491abcc92ba4e33c367b9431a415f887da8f84",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/HadoopReadOptions.java"
    },
    {
      "type": "modify",
      "old_id": "895d0670fa640775074145256b93de1406a1ccd4",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/main/java/org/apache/parquet/ParquetReadOptions.java",
      "new_id": "b627efbde0460d18ffff7e70c785a99e6595fc70",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/ParquetReadOptions.java"
    },
    {
      "type": "modify",
      "old_id": "ebb1208dc0604729441027251692e304c6609bd8",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java",
      "new_id": "d29738a6d8c7e66488bbbe6918eb95a4cceb051c",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java"
    },
    {
      "type": "modify",
      "old_id": "01ac69b330b6143eb1f7d6bf772da06374c4adc4",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetReader.java",
      "new_id": "ae068bd04a3c222cf4b2698c12e725fd2369d72c",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetReader.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "af93d0506d8acee0b9fe8b8cb770593c7c4adba0",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/VectoredReadBufferAllocator.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "7a25e8dfab80bea9b0a6abcecb0fc72921d698d6",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/VectoredReadExecutor.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "366ff35a4f55c960acc0e91db430ca2f609b685a",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/VectoredReadOperation.java"
    },
    {
      "type": "modify",
      "old_id": "7720f7fe94cc3715eb7e8492888042b15ddc3eb8",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/wrapped/io/VectorIoBridge.java",
      "new_id": "ce26477e78071794dccafbd79d0ef2e323caf8cd",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/wrapped/io/VectorIoBridge.java"
    },
    {
      "type": "modify",
      "old_id": "da3f9248b2ecf48ce9ca1d560905976feb45cd59",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestDataPageChecksums.java",
      "new_id": "faa2bb967b7015621babeeafd9d1e196e1c35f52",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestDataPageChecksums.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "41d65591c91c9683b1145fc63f7dec151387a1fc",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetFileReaderVectoredIO.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "bb6e0ff0dcb59c94344e25e678d4fa35d92b367d",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetFileReaderVectoredOwnership.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "0f3dfcb8155b541e4eb4c7094ce1f90bf97f2ae0",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestVectoredReadBufferAllocator.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "93410df1f7bfbc3ed42091962d1bf1c20e46e8c0",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestVectoredReadExecutor.java"
    },
    {
      "type": "add",
      "old_id": "0000000000000000000000000000000000000000",
      "old_mode": 0,
      "old_path": "/dev/null",
      "new_id": "7734aee200d98bf2fdcebe8fb0e315ca31002996",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestVectoredReadOperation.java"
    },
    {
      "type": "modify",
      "old_id": "8a710d2ee2e6c75d474db0bad1ba44b88319df9e",
      "old_mode": 33188,
      "old_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/util/wrapped/io/TestVectorIoBridge.java",
      "new_id": "4840db73d27285c89c451e66fdd592fbdcf02846",
      "new_mode": 33188,
      "new_path": "parquet-hadoop/src/test/java/org/apache/parquet/hadoop/util/wrapped/io/TestVectorIoBridge.java"
    }
  ]
}