HBASE-30111 Fix for flapper TestBlockBytesScannedQuota by AI (#8117) AI tool: Claude Code AI Model: Sonnet 4.5 (reverified using Opus 4.6 (1M context)) Prompt: You need to diagnose the HBase test TestBlockBytesScannedQuota that is flapping. It had multiple failures and errors in the PR validation build, but none when I ran locally. The source code is at @../../src/apache/hbase/ and the logs for good run is at @Good-Run and bad run is at @bad-run. Can you compare the logs and try to understand why the run failed and look at the code for a potential fix? Summary by AI: The test is flapping due to a **timing/race condition** in the quota system: 1. **5-second timeout too short**: The `testTraffic` method only waited 5 seconds for quotas to take effect 2. **Quota cache not fully propagated**: On slower systems (like CI), the quota cache refresh can be asynchronous and may not fully propagate in time 3. **Quotas bypassed**: When cache isn't refreshed, the logs show `"bypass expected false, actual true"`, meaning all requests succeed instead of being throttled 4. **Insufficient retries**: Each iteration takes ~1.3 seconds, so only 3-4 retries fit in 5 seconds, not enough for the quota system to stabilize **Bad run pattern:** - Test expects 1 successful request but gets 5 (all succeed because quotas not enforced) - Retries every ~1.3 seconds for 4 attempts - Times out after 5 seconds with "Waiting timed out after [5,000] msec" **Good run pattern:** - Quotas enforced immediately - Tests pass quickly (36.97s total vs 63.14s for failed run) Increased the timeout in `testTraffic()` from **5,000ms to 30,000ms** (line 263). This gives the quota system sufficient time to: - Complete cache refresh - Propagate quota settings across all components - Handle slower CI environments This is a conservative fix that maintains the retry logic while allowing adequate time for the distributed quota system to stabilize. The 30-second timeout is still reasonable for a test and should handle the asynchronous nature of quota enforcement. Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit de30a2bdb08326ec69195bf0da52e4fa2d18311a)
Apache HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Apache Hadoop.
To get started using HBase, visit the project home page. The HBase Reference Guide has a ‘Quick Start’ section and is where you should begin your exploration of the HBase project.
The latest HBase can be downloaded from the download page.
We use mailing lists to send notice and discuss. The mailing lists and archives are listed here
We use the #hbase channel on the official ASF Slack Workspace for real time questions and discussions. Please mail dev@hbase.apache.org to request an invite.
The source code can be found at https://hbase.apache.org/source-repository.html
The HBase issue tracker is at https://hbase.apache.org/issue-tracking.html
Notice that, the public registration for https://issues.apache.org/ has been disabled due to spam. If you want to contribute to HBase, please visit the Request a jira account page to submit your request. Please make sure to select hbase as the ‘ASF project you want to file a ticket’ so we can receive your request and process it.
NOTE: we need to process the requests manually so it may take sometime, for example, up to a week, for us to respond to your request.
Apache HBase is made available under the Apache License, version 2.0
The HBase distribution includes cryptographic software. See the export control notice here.