)]}' { "commit": "e575776bf0e8c6e3e19cbbbec6236c9fccff8062", "tree": "70cd91b0fb29e66207ac38eaf71cf0e561ec6078", "parents": [ "6ffb42301a9d073e6e999e7319b699232806cfdf" ], "author": { "name": "Gianluca Graziadei", "email": "48211703+GGraziadei@users.noreply.github.com", "time": "Sat Sep 19 11:08:27 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 19 11:08:27 2026 +0200" }, "message": "#2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols (#2135)\n\n* #2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols\n\nThe bolt-level fetch timeout ran the protocol call on a single-thread\nexecutor owned by each FetcherThread and abandoned it with\nfuture.cancel(true) on expiry. okhttp does not honour the interrupt while\nconnecting or reading, so the helper stayed blocked on the socket and the\nFetcherThread queued behind it at its next fetch: a host that dribbles\nbytes could take a thread out of service for as long as http.timeout.\nIt also doubled the thread count of the bolt whenever the option was on,\nand the robots.txt lookup was not covered by the timeout at all.\n\nokhttp: fetcher.thread.timeout is applied as a deadline on the call\n(Call.timeout()), enforced by okio\u0027s shared watchdog: on expiry the call\nis cancelled, the socket closed and the fetching thread gets a\nFetchTimeoutException at once. The hops of a redirect chain share one\ndeadline, each getting the time left. The robots.txt lookup and each\nredirect it follows are calls of their own with their own deadline, which\nis documented. With http.content.partial.as.trimmed the content received\nbefore the deadline is kept and flagged as trimmed for \"time\", as it\nalready was for the call timeout. Protocol gains a default\nsupportsFetchTimeout(url, metadata) (false); okhttp returns true when\nconfigured, DelegatorProtocol resolves the delegate per URL and requires\nthe one the robots.txt lookup is routed to to support it as well.\n\nBolts: the timeout machinery moves to a new package-private\nFetchTimeoutHelpers owned by FetcherBolt and SimpleFetcherBolt. Its\ncall() runs both the robots.txt lookup and the fetch on the calling\nthread when the protocol enforces the timeout itself, otherwise on a\nhelper thread from one bounded pool per bolt (fetcher.thread.timeout.helpers,\ndefault 2 x fetcher.threads.number, 2 for SimpleFetcherBolt, threads\ncreated on demand and released after a minute idle). With the default\nprotocol no helper thread is ever created. A deadline on a helper throws\nthe same FetchTimeoutException as okhttp. A robots.txt lookup which times\nout on a helper lets the page be fetched without rules, as\nHttpRobotRulesParser already does for a failed lookup with okhttp, so\nboth paths behave the same. A full pool rejects at once: the URL never\nreached the network, so it is acked without a status, like a URL which\nwaited too long in the queue, rather than reported as FETCH_ERROR and\ncounted towards max.fetch.errors.\n\nFetchTimeout.secs(conf) reads and clamps the deadline to\ntopology.message.timeout.secs for both the protocol and the helpers.\nFetchTimeoutException extends InterruptedIOException so that it is\nclassified as a timeout; the new fetch.deadline counter counts only the\ndeadline, fetch.timeout keeps counting every timeout, fetch.helper.rejected\nthe saturations, robots.timeout the abandoned lookups; fetchhelpers gauge.\n\nThe fetcher.thread.timeout key is defined in Constants; the alias\nFetcherBolt.FETCH_TIMEOUT_PARAM_KEY shipped in 3.6.0 is kept, deprecated\nfor removal. Documentation for the parameter, which was missing, is\nadded to configuration.adoc.\n\nTests: HttpProtocolFetchTimeoutTest (cancellation at the deadline with\nthe typed exception, socket timeout not reported as the deadline,\nredirect chain sharing the deadline, clamp, partial content),\nFetchTimeoutHelpersTest (every branch of call(), clamp), fetcher bolt\ntests with a protocol that hangs and ignores interruption (stuck fetch\nnot blocking the following ones, bounded pool with the rejected URL acked\nwithout status, hanging robots.txt not failing the URL), slow robots.txt\nbounded with okhttp, no helper threads with okhttp, delegator capability\nper URL and for the robots.txt route.\n\nFixes #2134.\n\n* Fix open comments\n\n* Update deprecation comment for FETCH_TIMEOUT_PARAM_KEY\n\n* Configure HTTPRobot Cache as read first. Handle sequence: fetcher signals EMPTY handler ends after\n\n* Handle nit on ROBOTS_HUNG, ROBOTS_TIMED_OUT (STARTED as well) on StuckProtocol\n\n* Handle race: FutureTask can be canceled in the cs between run() and started.set(true) in FetchTimeoutHelpers\n\n* Refactors the ERRORCACHE invalidation mechanism to properly handle the fetcher vs. helper sequence and expands test coverage\n\n* Saturation backoff: handle negative fetcher.max.crawl.delay, keep helper 5xx robots rules\n\n- A negative fetcher.max.crawl.delay (any crawl delay accepted) made the\n backoff cap negative, so ThreadLocalRandom.nextLong threw in the finally\n block of the FetcherThread loop: the thread died without acking the tuple.\n Fall back to the default cap of 30 s, and never back a queue off for less\n than its own delay.\n- cacheLookupFailure uses putIfAbsent: a lookup completing on its helper with\n an error of its own (e.g. 5xx, forbid all) between the check and the put\n kept its rules instead of being replaced by allow all.\n- Restore permitsAreNeverReleasedTwice.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e", "tree_diff": [ { "type": "modify", "old_id": "882f2d9652fda080310b4171870802474edcb0ae", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/Constants.java", "new_id": "f925e79d9c9e1c36d9bc1fc41b13f6ce888e1f0d", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/Constants.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "27dc6df87f7f361e5c5ccf107a9c94cec6b0969f", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/bolt/FetchTimeoutHelpers.java" }, { "type": "modify", "old_id": "67ae0acfce1d13dfab10d0c4341644af2427f42c", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java", "new_id": "6f6b0c9df98ffc8dd3f1cf47b05de801945d77c0", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java" }, { "type": "modify", "old_id": "612d2ede0c82000416d4eefe931e86fdb1819246", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/bolt/SimpleFetcherBolt.java", "new_id": "a7eda57b82e66ec5410c53d7a3bd88458ca6094d", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/bolt/SimpleFetcherBolt.java" }, { "type": "modify", "old_id": "a2f37b70b5a14b5ae9da01c583cf851e27917b45", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/AbstractHttpProtocol.java", "new_id": "6f9b091f71351f30f7c269c87333298306328335", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/AbstractHttpProtocol.java" }, { "type": "modify", "old_id": "b19760b93142698c9406fd29586c287283634e18", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/DelegatorProtocol.java", "new_id": "9a602efa7bc3359288be01fbc18b486edf28da81", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/DelegatorProtocol.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "e4b1d195d6599c32997e5d3c714bec7e2a2bf888", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/FetchTimeout.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "eb75c66a17afdf7de4fbaef7f3c85cbde408628a", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/FetchTimeoutException.java" }, { "type": "modify", "old_id": "808982140e602c7036e1c6f07ab1187e421e0650", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java", "new_id": "531e744aaba22a5690c91d1cb3ea438bdfa454a2", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java" }, { "type": "modify", "old_id": "242395235b51b07df0cdfd7e7f8e68c6e1fcadae", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/Protocol.java", "new_id": "642153c1922578374c40734a8aecdba4dbfa9457", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/Protocol.java" }, { "type": "modify", "old_id": "121dae615d4e497be2d922f5b7ea0b6aae0b21c2", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/RobotRulesParser.java", "new_id": "24220b474d7ad830345cbf5e286550ae45f0802d", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/RobotRulesParser.java" }, { "type": "modify", "old_id": "614dc90912e89cbc7ba670e9964b1cb2e83779fc", "old_mode": 33188, "old_path": "core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java", "new_id": "5c8c7f550eabb35a3345447b5e035868d3102986", "new_mode": 33188, "new_path": "core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java" }, { "type": "modify", "old_id": "581669fedb6d3680169c6b79798f833ecbe2473b", "old_mode": 33188, "old_path": "core/src/main/resources/crawler-default.yaml", "new_id": "7847873757baede89b150ffba8fbfbebe0cb3354", "new_mode": 33188, "new_path": "core/src/main/resources/crawler-default.yaml" }, { "type": "modify", "old_id": "35cd7b5f52849da6355e89af1cee5a59cf3e8e94", "old_mode": 33188, "old_path": "core/src/test/java/org/apache/stormcrawler/bolt/AbstractFetcherBoltTest.java", "new_id": "8614b5824620d7599405a76acaa20188b3c9293a", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/bolt/AbstractFetcherBoltTest.java" }, { "type": "modify", "old_id": "6fe7670c2cd8b921fc35d72c24d2a2e7778e59c6", "old_mode": 33188, "old_path": "core/src/test/java/org/apache/stormcrawler/bolt/FetchItemQueuesTest.java", "new_id": "e697740b9107b91d3c0681ce5f05eed5f1bf5a55", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/bolt/FetchItemQueuesTest.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "55a5317b89063b6f905aa6770066cdbaff863b8b", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/bolt/FetchTimeoutHelpersTest.java" }, { "type": "modify", "old_id": "e7f592e141c84b248fcff413968c067ae5368627", "old_mode": 33188, "old_path": "core/src/test/java/org/apache/stormcrawler/bolt/FetcherBoltTest.java", "new_id": "9d8ddd719bae2fbb60b3c0fefbcd608ff7987941", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/bolt/FetcherBoltTest.java" }, { "type": "modify", "old_id": "9b6ba8bcee87ded015f0b0854903c4eee8db7042", "old_mode": 33188, "old_path": "core/src/test/java/org/apache/stormcrawler/protocol/DelegationProtocolTest.java", "new_id": "079d82df42f37ebf040717058d94831d7734c9d2", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/protocol/DelegationProtocolTest.java" }, { "type": "modify", "old_id": "135c3df52ac556758904819f5afa93e4fd5ccdc3", "old_mode": 33188, "old_path": "core/src/test/java/org/apache/stormcrawler/protocol/HttpRobotRulesParserTest.java", "new_id": "e900ebc704b69f08e41075f90acb085d26cfac97", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/protocol/HttpRobotRulesParserTest.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "b83f7c0297d108389939180e70f8f1c8e739f078", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/protocol/StuckProtocol.java" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "9fa6483f9b13a77516daf2c8c5a7bda7e68cf624", "new_mode": 33188, "new_path": "core/src/test/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocolFetchTimeoutTest.java" }, { "type": "modify", "old_id": "23aeba9e1e3fa2c7542a92c1db9a780d16d1ba08", "old_mode": 33188, "old_path": "docs/src/main/asciidoc/configuration.adoc", "new_id": "157a9b218d59b9733bd1667c737cddf378d745cc", "new_mode": 33188, "new_path": "docs/src/main/asciidoc/configuration.adoc" } ] }