)]}' { "log": [ { "commit": "68b9ecbf17574b46584b003c36fa93cf8ac2ba3b", "tree": "49b931e74bc0639bf0eac53bfd931d22499d08bf", "parents": [ "40b59eb183ff87f4d37636397bad97263875d879" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sun Sep 13 12:23:40 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 08:53:40 2026 +0200" }, "message": "Run redirect hops through the URL filters (#2128)\n\n* Run redirect hops through the URL filters (#2088)\n\nRedirects are followed in HttpProtocol instead of by the client, so\nevery target passes through the URL filter chain before the hop is\ntaken. Credential headers are stripped when the host changes or the\ntarget is not authenticated, reusing the credential policy from #2126.\nThe number of hops is limited by http.allow.redirects.max and the final\nURL is recorded as _redirectedTo.\n\nRebased onto main after #2126 was merged.\n\n* Strip credentials on any origin change during redirect hops (#2088)\n\nCompare scheme, host and port instead of the host only, like OkHttp\u0027s own\nredirect follower. Load the URL filters only when redirects are followed and\ndocument http.allow.redirects.max.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e" }, { "commit": "40b59eb183ff87f4d37636397bad97263875d879", "tree": "a992c83449393ad9681bf9feaf05f2aecb503887", "parents": [ "7ef7d923d483dad96b01506e96d2ae479b3b10f6" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sun Sep 13 12:06:57 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 08:36:57 2026 +0200" }, "message": "Spouts check the scheme of stored URLs before emitting (#2123)\n\n* Spouts check the scheme of stored URLs before emitting (#2085)\n\nThe frontier is the crawl instruction set: whatever ends up in the\nstore is fetched, with no scheme check and no filtering, since URL\nfiltering only runs on the discovery path. A row whose URL uses a\nscheme the operator never intended to crawl was emitted as long as a\nprotocol implementation was registered for it.\n\nAbstractQueryingSpout.nextTuple now checks the scheme of each buffered\nURL against the configured protocols list before emitting, skips rows\nwith other schemes at WARN with a skipped.scheme counter, and keeps\ndraining the buffer so one bad row does not block the ones behind it.\n\n* Report unemittable rows to the status stream (#2085)\n\nReview feedback on #2123:\n\n- a row whose URL the spout refuses to emit is reported to the status\n updater as Status.ERROR on the status stream: it is removed from the\n store instead of staying there and coming back with every query,\n which bounded the skipped.scheme loop and gives the operator a\n signal other than one warn per row per query\n- the scheme comparison lowercases with Locale.ROOT; the previous\n default-locale lowercase would reject every URL on a Turkish worker\n- a null from SimpleURLBuffer.next() is handled instead of throwing\n- the throttleQueries javadoc formatting fixed, the redundant\n buffer.hasNext() check dropped\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Split the shipped protocols string into schemes; wire spout status stream (#2085)\n\nReview feedback on #2123:\n\n- the protocols key was parsed with loadListFromConf alone, which\n returns a comma-separated string as one entry: the shipped\n \"http,https,file\" became a single unmatchable scheme and every\n stored URL was rejected. The key is now split on commas with\n surrounding whitespace tolerated, exactly like ProtocolFactory\n parses it; tests cover the shipped string, an explicit list and\n whitespace forms\n- declaring the status stream routes nothing by itself: the archetype\n crawler.flux files (urlfrontier, opensearch, solr) now connect the\n spout to the status updater on Constants.StatusStreamName with the\n same url-field grouping the other status producers use, so rejected\n rows actually reach the store and are removed. internals.adoc\n documents the required wiring for hand-built topologies\n- SpoutStatusStreamWiringTest parses the archetype topology and fails\n if the spout-to-status-updater connection is missing\n\n* Format the wiring test with the project formatter\n\nThe CI format check would reject the javadoc line wrapping; the file is\nnow in the exact form the git-code-format plugin produces with the\nglob from the README note for developers.\n\n* Keep stored metadata on rejected rows, check every archetype flux\n\nemitStatus passes the buffered metadata on instead of an empty set, so\nthe status updater does not overwrite the stored row. The wiring test\nnow checks each archetype crawler.flux instead of taking a parameter it\nnever used.\n\n* Record the rejection cause and correct the status-stream wording (#2085)\n\nReview follow-ups on #2123 (@dpol1):\n\n- the rejected-row ERROR now carries STATUS_ERROR_CAUSE, like the\n fetcher and the updater set it, so the store says why the row is ERROR\n- a rejected row whose status stream reaches no subscriber (a 3.x\n crawler.flux without the spout-to-updater edge) is logged once, so\n the endless re-query is visible rather than silent\n- the three archetype flux comments and internals.adoc now say the\n updater marks the row ERROR and notifies the deletion stream, rather\n than removing it\n\n* Align remaining comments with the status updater marking rejected rows ERROR (#2085)\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e" }, { "commit": "7ef7d923d483dad96b01506e96d2ae479b3b10f6", "tree": "483d2865a5078ea5e1e782758bef09f55ffd5666", "parents": [ "1d455c2c96d3b4757aa140fe890b439ed20a9182" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sun Sep 13 11:40:47 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 08:10:47 2026 +0200" }, "message": "SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure (#2125)\n\n* SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure (#2083)\n\n- the parser is now built in strict mode (sitemap.strict, default\n true): crawler-commons then discards URLs a sitemap lists on hosts\n other than its own, so a sitemap cannot enrol URLs on hosts it has\n nothing to do with, and an HTML page mentioning the sitemap namespace\n is not parsed leniently into half a sitemap\n- content sniffing moves behind sitemap.sniffContent (default false,\n like feed.sniffContent for feeds): a page carrying the namespace\n string in its first bytes was reclassified as a sitemap, never\n reached the parser bolt and was never indexed. The key the existing\n test already set but the bolt ignored is now honoured; a content type\n which rules a sitemap out (a page served as HTML) stops the sniffing\n- a document marked as a sitemap through persisted metadata whose body\n does not parse is emitted as FETCH_ERROR with the isSitemap key\n dropped, instead of a terminal ERROR: with the archetype\u0027s\n fetchInterval.error of -1, an ERROR removed the URL from the crawl\n for good, letting whoever controls the content decide what stays in\n the corpus. On its next fetch the document goes to the parser bolt\n like any other page\n\nThe image and all-extensions test sitemaps listed their first URL on\nwww.example.com, which strict mode now correctly excludes; that entry\nmoved under the sitemap\u0027s own host.\n\n* Sitemap strict checking defaults to off; sniffing confirms declared sitemaps (#2083)\n\nReview feedback on #2125:\n\n- the strict flag of crawler-commons is strict URL checking (a sitemap\n only yields URLs below its own host and path), not the namespace\n check the javadoc and yaml comment described. Fixed the wording and\n flipped the default to false: a sitemap living at example.com while\n listing URLs under www.example.com violates the sitemap spec but is\n common, and switching strict checking on by default silently shrinks\n such crawls. Recommended for open crawls, documented in the yaml\n- the sniff override no longer happens only for unmarked documents:\n a document already marked isSitemap\u003dtrue has the namespace sniffed\n to confirm its type and gets ct forced to application/xml, so a\n sitemap served as text/html parses again instead of failing with\n UnknownFormatException - this was a regression against main\n- a test asserts the default keeps parsing cross-host sitemaps, and\n one asserts the wrong-content-type sitemap still parses\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Replace the default-locale toLowerCase in the test\n\nThe forbiddenapis check rejects String.toLowerCase() without a locale;\nthe metadata key is a literal, so it is written out directly.\n\n* Parse the media type separately from its parameters when sniffing (#2083)\n\nReview feedback on #2125:\n\n- sniffsAsSitemap checked whether the whole Content-Type header\n contains xml, so application/xhtml+xml and text/html; profile\u003dxml\n both passed: ordinary HTML or XHTML mentioning the sitemap namespace\n was routed into sitemap parsing and emitted as FETCH_ERROR instead\n of reaching the page parser. The media type is now compared\n separately from its parameters, and HTML/XHTML is excluded\n explicitly from promotion\n- sniffingRequiresSitemapCompatibleContentType never enabled\n sitemap.sniffContent despite its name; it now does, and XHTML,\n parameterised HTML and a positive XML case are covered\n- the PR description no longer claims sitemap.strict defaults to true\n\n* Apply strict URL checking to sitemap indexes; carry isSitemap across redirects (#2083)\n\nReview follow-ups on #2125 (@dpol1):\n\n- crawler-commons applies its strict host/path check to \u003curlset\u003e entries\n only, and this bolt marks every sitemap-index \u003cloc\u003e as a sitemap, so an\n index could still pull in another host\u0027s sitemap. The index loop now\n applies SiteMapParser.urlIsValid against the directory of the index when strict\n is on; a test pins that a cross-host sub-sitemap is not discovered\n- a sitemap which redirects (e.g. /sitemap.xml to /sitemap_index.xml)\n arrived at the bolt without isSitemap (the key is persisted, not\n transferred to outlinks), so with sniffing off it was stored as a page.\n Both fetchers now carry isSitemap onto the redirect target\u0027s outlink\n- the sitemap.strict yaml comment notes it covers index \u003cloc\u003e entries too" }, { "commit": "1d455c2c96d3b4757aa140fe890b439ed20a9182", "tree": "9543a4a46330e9cab73c09bb95c90c590d5c4c20", "parents": [ "765cb11e4b491bb221564cbe4a7c85a4b07e18e2" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sun Sep 13 11:37:28 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 08:07:28 2026 +0200" }, "message": "Canonicalise hosts for politeness queues and the robots.txt cache (#2122)\n\n* Canonicalise hosts for politeness queues and the robots.txt cache (#2082)\n\nokhttp percent-decodes and lowercases the host when it parses the URL,\nbut FetchItem.create, SimpleFetcherBolt.getPolitenessKey and\nHttpRobotRulesParser.getCacheKey keyed on the raw host string, so\nhttp://exampl%65.org/ and http://example.org/ were one origin at\nconnect time and two queue ids and two robots.txt cache entries in the\nbolt. One server could end up with several delay clocks running in\nparallel and its robots.txt fetched once per spelling.\n\nURLUtil.getCanonicalHost percent-decodes the host, strips a trailing\ndot and lowercases it, and all three call sites now use it.\n\n* Never-throwing percent-decoder, one canonical host for every queue mode (#2082)\n\nReview feedback on #2122:\n\n- URLDecoder.decode throws on a malformed escape and maps + to a\n space, both reachable from crawled content which the security model\n treats as hostile. getCanonicalHost now uses a hand-rolled\n percent-decoder that leaves + alone and keeps malformed escapes as\n literal characters; it never throws\n- the javadoc no longer claims to return \u0027the host okhttp connects\n to\u0027: okhttp keeps a trailing dot, this method normalises it away.\n It returns the form used to key politeness queues and the robots.txt\n cache\n- FetcherBolt and SimpleFetcherBolt canonicalise the host once and use\n it in all three queue modes: byIP and byDomain still resolved and\n split the raw escaped host, so http://%65xample.org/ kept its own\n queue in those modes\n- IDN hosts are not collapsed yet (unicode vs punycode spellings);\n recorded as #2145\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Reformat FetcherBolt and SimpleFetcherBolt with the project formatter\n\nThe hand-edited import reorder in the previous commit left one code\nline indented differently than google-java-format requires, which the\nCI format check rejects. Files are now formatted with the exact\ngoogle-java-format version the git-code-format plugin pins (1.35.0,\nAOSP), followed by the import-block normalisation the checkstyle\nconfig demands. No code changes.\n\n* Canonicalise host-derived partition keys in the partitioners (#2082)\n\nThe fetcher canonicalised the host, but the shipped topology derives\nthe politeness key one bolt earlier: URLPartitionerBolt (and the\nURLPartitioner the URL buffer uses) keyed on the raw hostname, and\nFetchItem.create accepts an explicitly provided key without touching\nit, so http://example.org/ and http://exampl%65.org/ kept separate\npoliteness queues and could even be routed to different fetcher tasks.\n\nBoth partitioners now derive host, domain and IP keys from\nURLUtil.getCanonicalHost in every mode; an explicitly supplied\nip metadata value is used unchanged. Regression tests cover the\nutility for all three modes and the partitioner bolt end to end,\nincluding the unchanged-URL guarantee.\n\n* Format the partitioner tests with the project formatter\n\nThe CI format check rejected the javadoc line wrapping and one unused\nimport in the new test files; they are now in the exact form the\ngit-code-format plugin produces.\n\n* Normalise the host of discovered URLs in the status updater (#2082)\n\nDesign change from review (@jnioche, @rzo1): collapse host aliases at\nthe one write path into the store instead of only at the read sites.\n\n- URLUtil.normaliseHost replaces just the host component with the\n canonical form (percent-decoded, lowercased, trailing dot removed),\n never throws on a malformed escape, and leaves hostless, IPv6 and\n non-hierarchical URLs byte for byte unchanged\n- AbstractStatusUpdaterBolt.execute normalises the host of DISCOVERED\n URLs before the dedup cache lookup and the as-is early return, so the\n cache, scheduler and stored record all see one form; other statuses\n carry URLs read back from the store and are left alone\n- behind status.updater.normalise.hosts, default false: getDocumentID\n hashes the URL, so enabling it re-discovers aliases already stored\n raw as a second, bounded record; documented as a release note\n- the politeness-queue and robots-cache canonicalisation on the read\n side is kept (it must hold up while the flag is off)\n\nRegression tests cover normalised storage, dedup-cache collapse, the\ndefault-off passthrough and the only-DISCOVERED guarantee. Full core\nsuite green (449). Docs and crawler-default.yaml updated.\n\n* Decode percent-escaped host octets as UTF-8 (#2082)\n\nReview follow-up on #2122 (@dpol1): percentDecodeHost turned each %XX\ninto one char, so a multi-byte sequence like %C3%BC became two chars\n(the mojibake A-umlaut + 1/4) instead of the single character it\nencodes. The octets are now collected and decoded as UTF-8, so\nhttp://\\%C3\\%BCnicode.example.org/ canonicalises to the same host as its\nliteral-unicode spelling. A test pins the multi-byte case." }, { "commit": "765cb11e4b491bb221564cbe4a7c85a4b07e18e2", "tree": "1b06244d1d60088765bb4508a4186a403479cbc6", "parents": [ "28a467e078f6ff2d28300cdb13132f76a1418d44" ], "author": { "name": "Sebastian Nagel", "email": "snagel@apache.org", "time": "Sun Sep 13 08:06:46 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 13 08:06:46 2026 +0200" }, "message": "#1998 WARC writer: WARC-Protocol header to follow WARC field proposals (#2034)\n\n* WARC writer: WARC-Protocol header to follow WARC field proposals\n(fixes #1998)\n\nOkHttp protocol: add protocol response header key `_cipher_suites_`\nto hold the SSL/TLS Cipher suite separate from `_protocol_versions_`.\n\nWARC writer:\n1. add WARC header `WARC-Cipher-Suite`\n2. split multiple values in `WARC-Protocol` header and\n repeat header\n\n* WARC writer: WARC-Protocol header to follow WARC field proposals\n(fixes #1998)\n\nAddress PR review comments:\n- `_cipher_suites_` -\u003e `_cipher_suite_` because it holds only a single\n value, also adapt Javadoc accordingly\n- do not add Cipher Suite to protocol metadata if it does not exist\n (http://, no TLS)\n- normalize OkHttp internal SSL/TLS version strings following the\n WARC-Protocol field proposal\n- add unit test for the OkHttp protocol response interceptor\n (contributed by @rzo1)\n- WARC writer: trim values of the WARC-Protocol field\n- updata WARC module unit tests" }, { "commit": "28a467e078f6ff2d28300cdb13132f76a1418d44", "tree": "640e8995cb111d78d3577548d9133490360c0107", "parents": [ "b4df3e329426b655211c5b8cd6f331af440a6c42" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Sep 12 18:55:57 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Sep 12 17:55:57 2026 +0100" }, "message": "URLFrontier StatusUpdaterBolt: make lock objects transient so the bolt can be serialized (#2149)\n\n#2117 replaced the ReentrantLock with plain Object monitors held in final\nfields. java.lang.Object is not serializable, so submitting a topology with\nthe bolt fails with NotSerializableException. The monitors are now transient\nand created in prepare()." }, { "commit": "b4df3e329426b655211c5b8cd6f331af440a6c42", "tree": "d0d73d3823b9586d857946d261e0b73b8689539a", "parents": [ "151e568eafa9fe2585f4f443cf55d1c56969d413" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sat Sep 12 00:32:35 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Sep 11 21:02:35 2026 +0200" }, "message": "Keep the fetcher off private address space by default (#2127)\n\n* Keep the fetcher off private address space by default (#2080)\n\nThe IP address filter was the only check that looked at the address the\nfetcher actually connects to, and both of its keys shipped commented\nout, so the interceptor was never installed at library defaults. The\nregex exclusion list in the archetypes matched literal localhost and\nfour dotted-quad ranges and nothing else: link-local, CGNAT, IPv6\nunique-local and the abbreviated and integer IPv4 forms the resolver\nmaps to 127.0.0.1 all passed it.\n\n- crawler-default.yaml ships http.filter.ipaddress.exclude enabled\n (loopback, RFC1918, link-local, CGNAT, IPv6 unique-local); a fetched\n page decides which hosts the fetcher connects to, and these ranges\n host unauthenticated services a public index must not leak into.\n Crawling an intranet needs an explicit opt-out, documented in the\n comment next to the key\n- the archetype default-regex-filters.txt (and its opensearch/solr\n copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8,\n IPv6 fc00::/7 and fe80::/10, the abbreviated loopback forms (127.1)\n and integer IPv4 hosts (2130706433), and says in its comment that a\n regex list judges the bytes of the URL only, so the IP filter stays\n the authoritative check\n\n* Fix the CGNAT regex, close case and IPv6-literal gaps (#2080)\n\nReview feedback on #2127:\n\n- the CGNAT alternation blocked 100.60-100.63 (public space) and missed\n 100.65-100.69 entirely: the second octet runs 64 to 127, so the first\n branch is 6[4-9]. Tests pin 100.65.0.1 as blocked and 100.63.0.1 as\n accepted, both ends of the mistake\n- RegexURLFilter compiles case-sensitively, so LOCALHOST, [FD00::1] and\n [FE80::1] slipped past the rules: the private-range rules carry (?i)\n- ::ffff:127.0.0.1 and 0:0:0:0:0:0:0:1 resolve to loopback through the\n JVM resolver and matched nothing: a rule catches the IPv4-mapped and\n fully expanded loopback literals\n- the yaml opt-out advice was wrong: exclude rules win over include\n rules in IPFilterRules.accept, so an operator has to override\n http.filter.ipaddress.exclude explicitly; the comment says so, and\n the shipped exclude list covers the full ULA range (fc00::/7)\n- core/src/test/resources/default-regex-filters.txt is back to its\n upstream content: RegexFilterTest and URLFilterBoltTest load it, and\n overwriting it here was an unrelated change. The archetype rules\n ship as default-regex-filters-archetype.txt for the new test instead\n- the description documents that the resolver rejects hex (0x7f000001)\n and does not read leading zeros as octal, so the \\d{1,10} rule\n closes the only numeric form that actually works\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Close the 0/8 and IPv6-unspecified gaps in the shipped IP policy (#2080)\n\nReview feedback on #2127:\n\n- the shipped http.filter.ipaddress.exclude omitted 0.0.0.0/8 and the\n IPv6 unspecified address: IPFilterRules.accept returned true for\n 0.0.0.0, 0.1.2.3 and ::, and blocking the literal URLs in the regex\n does not cover a hostname that resolves to them. Both are added\n (0.0.0.0/8 and ::/128); ::/128 rather than an ::/0 range so no real\n address is excluded\n- core/src/test/resources/default-regex-filters.txt restored to its\n pre-PR content; this PR\u0027s fixture is the separate\n default-regex-filters-archetype.txt, so RegexFilterTest and\n URLFilterBoltTest are untouched as before\n- the shipped-default test now walks every excluded range with its\n boundary addresses and the public neighbours, plus the documented\n explicit opt-out\n- HttpProtocolIPFilterTest proves the policy is enforced by the actual\n HTTP protocol, not just the standalone predicates: a fetch of\n http://localhost (resolves to loopback) throws the interceptor\u0027s\n Forbidden IOException under the shipped list, and the same fetch\n succeeds once the exclude list is overridden to empty\n\n* Do not apply the IP address filter to fetches through a proxy\n\nBehind a proxy the socket is connected to the proxy, which resolves the\ntarget host itself, so the filter only ever saw the proxy\u0027s address. With\nthe new default exclude list, an HTTP proxy on a private range failed\nevery fetch. The interceptor now lets proxied connections through, a log\nline at configure time says so, and the yaml and configuration.adoc\ndocument that the proxy\u0027s own egress rules apply there.\n\nconfiguration.adoc also shows the new default of\nhttp.filter.ipaddress.exclude and no longer describes the filter as\noptional.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e" }, { "commit": "151e568eafa9fe2585f4f443cf55d1c56969d413", "tree": "8ef5e955081cc45f3ba3beab26751412d8a7c33d", "parents": [ "dcf4e9d884a53fd6c62c4584784d2de73ea1df6a" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Thu Sep 10 23:27:39 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 19:57:39 2026 +0200" }, "message": "OKHttp: authenticated TLS by default, credentials withheld from unauthenticated servers (#2126)\n\n* OKHttp: authenticated TLS by default, credentials withheld from unauthenticated servers (#2079)\n\nhttp.trust.everything shipped enabled with a static SSLContext(\u0027SSL\u0027),\na no-op trust manager and a hostname verifier that accepted any name:\nanybody able to answer for a host name received the Authorization\nheader built from http.basicauth.*, credential headers from\nhttp.custom.headers and the replayed cookies.\n\n- http.trust.everything now defaults to false: certificate chains are\n validated and the servers are authenticated, as a normal TLS client\n would do. Self-signed intranet hosts need the operator to opt in\n- the trust-all context is built from \u0027TLS\u0027 instead of \u0027SSL\u0027\n- hostname verification is a separate decision\n (http.verify.hostnames, default true) and is no longer disabled as a\n side effect of trusting any certificate\n- credentials are withheld over connections whose server certificate\n was not validated, unless http.credentials.allow.insecure is set:\n basic auth, credential headers (authorization, proxy-authorization,\n cookie, x-api-key) in http.custom.headers, headers set by request\n and cookies are all covered\n- every insecure state logs a WARN naming the keys, and the keys are\n surfaced in crawler-default.yaml and the archetype crawler-conf.yaml\n files\n\n* OkHttpTrustEverythingTest: serve TLS with WireMock\n\nThe JDK\u0027s com.sun.net.httpserver classes are rejected by the\nforbiddenapis check (jdk-non-portable). WireMock is already a test\ndependency and serves HTTPS from a PKCS12 keystore; the request\njournal replaces the hand-written header recorder. The custom headers\nuse the Name\u003dvalue syntax that http.custom.headers parses and the\ncode is google-java-format clean.\n\n* Withhold credentials from every unauthenticated server (#2079)\n\nReview feedback on #2126:\n\n- the decision to send credentials is now made per request: a server is\n only authenticated when the connection is https:// and the certificate\n chain is validated. Cleartext http:// requests never authenticate the\n server and no longer receive the Authorization header, credential\n headers or cookies, closing the cleartext case the first commit\n missed\n- the credential header denylist is configurable with\n http.credentials.headers (default: authorization, proxy-authorization,\n cookie, x-api-key), so site-specific names like x-auth-token can be\n covered without dropping every custom header\n- the withheld-cookies warning only fires when a cookie was actually\n going to be sent, and the withheld-request-headers warning is logged\n once per protocol instance instead of once per header per request\n- the proxy authenticator keeps its credentials by design and says so\n in a comment: the proxy is not the crawled server\n- the keytool commands that generated the test keystores are recorded\n in the test class, and the TLS test server moved to WireMock so the\n forbiddenapis check passes\n\n* Enforce credential policy on redirect hops and hostname verification (#2079)\n\n- credentialsAllowed now requires http.verify.hostnames: a trusted cert for the wrong host with verification off no longer receives credentials\n- network interceptor strips credential headers on every hop whose URL is not authenticated, covering automatic HTTPS-\u003eHTTP redirects\n- regression tests for hostname-off, per-request headers, decision matrix, and HTTPS-\u003eHTTP redirect stripping/forwarding\n\n* http.credentials.headers adds to the built-in names\n\nSetting the key replaced the whole list, so the example in\ncrawler-default.yaml (\"x-api-key\", \"x-auth-token\") dropped\nauthorization, proxy-authorization and cookie from it. A header\nconfigured through http.custom.headers under one of those names was then\nclassified as an ordinary header and sent to servers which were not\nauthenticated, and the redirect interceptor stopped stripping it on the\nnext hop. The configured names are merged into the defaults instead, and\nthe yaml and configuration.adoc say so.\n\nAlso decide the policy on the HttpUrl the interceptor already holds\nrather than re-parsing the string, and trim the comments added by this\nbranch.\n\n* Strip credentials before recording headers, withhold static Proxy-Authorization on hops\n\nThe credential interceptor now runs before HTTPHeadersInterceptor, so the\nrequest headers stored for a hop (and written to WARC request records) are\nthe ones actually sent, not the ones stripped from a cleartext hop.\n\nProxy-Authorization is only kept on a hop when the proxy authenticator set\nit and the hop is cleartext, i.e. when the proxy reads it. A static\nProxy-Authorization from http.custom.headers is treated like any other\ncredential on every hop, as the configuration documents.\n\nDocument that http.verify.hostnames: false withholds credentials too.\n\n* Apply code formatting to okhttp HttpProtocol\n\nCo-Authored-By: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e\nClaude-Session: https://claude.ai/code/session_014U6Zqh1d6Z9WF6TAUgk3AE\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e\nCo-authored-by: Claude Opus 5 (1M context) \u003cnoreply@anthropic.com\u003e" }, { "commit": "dcf4e9d884a53fd6c62c4584784d2de73ea1df6a", "tree": "0ad0cbbf7cda91f8448ca145c90a870312dd9ac3", "parents": [ "c6bcf446c621c904ea4347b096976a3418a2d9a8" ], "author": { "name": "Riddhish Bonde", "email": "bonderiddhish@gmail.com", "time": "Thu Sep 10 16:49:20 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 13:19:20 2026 +0200" }, "message": "#2104 feat: implement MIME type whitelist filtering in ParserBolt for Tika (#2116)\n\n* feat: implement MIME type whitelist filtering in ParserBolt for Tika-based parsing\n\n* format code\n\n* feat: add MIME type whitelist support to ParserBolt with detection logic and corresponding unit test\n\n* Address review: fix comments, strengthen tests, document whitelist behaviour in configuration.adoc\n\n* Adapt the MIME detection block to Tika 4 after the rebase onto main\n\n`org.apache.tika.metadata.Metadata` no longer carries the HTTP header\nconstants and the parser reads from a `TikaInputStream` since #2118; the\nwhitelist detection block uses the same two APIs.\n\nCo-Authored-By: Claude Fable 5.1 \u003cnoreply@anthropic.com\u003e\nClaude-Session: https://claude.ai/code/session_01FsGvRM2W3aHeMyVtSbBWGb\n\n---------\n\nCo-authored-by: Davide Polato \u003cdavidepolato05@gmail.com\u003e\nCo-authored-by: Claude Fable 5.1 \u003cnoreply@anthropic.com\u003e" }, { "commit": "c6bcf446c621c904ea4347b096976a3418a2d9a8", "tree": "f3914c58306ccf251658002b532cd0acbf3fead3", "parents": [ "44f39f83ecfaaa79305c40aa093bb47bec96837d" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Sep 10 11:34:07 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 11:34:07 2026 +0200" }, "message": "Minor: Fix formatting in ColumnNameValidationTest (#2148)" }, { "commit": "44f39f83ecfaaa79305c40aa093bb47bec96837d", "tree": "42d5a382f2168dc69d332d2d52d992ba74c22281", "parents": [ "1c6f1ae750f812ac085063a6cac5624ef8269f39" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Thu Sep 10 10:31:36 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:31:36 2026 +0200" }, "message": "Minor: Regenerated License File for 1c6f1ae750f812ac085063a6cac5624ef8269f39 (#2147)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "1c6f1ae750f812ac085063a6cac5624ef8269f39", "tree": "4001309cba350b5e00f3a1fdfe92e2febff20d35", "parents": [ "3c5fcd64539d7d7c74eab968b65e05c0fe5bf11d" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Thu Sep 10 13:30:26 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 10:00:26 2026 +0200" }, "message": "Use batch endpoint in URLFrontier module for DISCOVERED urls (#2117)\n\n* Use batched PutDiscovered endpoint for DISCOVERED URLs in URLFrontier StatusUpdaterBolt\n\nDiscovered URLs, which are the bulk of what a crawl writes, are now grouped\ninto batches and sent on the PutDiscovered endpoint introduced in URLFrontier\n2.6, amortising the per-message cost that limits the ingestion rate. Known\nURLs keep using the streaming PutURLs endpoint.\n\n- new config key urlfrontier.batch.size (default 100, 0 disables batching)\n- partially filled batches are flushed after 1s by a scheduled flusher, and\n whenever a known URL arrives (natural end-of-page boundary) or permits run\n short\n- batches are acked as a whole through BatchAck: one status per URL in the\n order they were sent, each resolved against the waitAck cache\n- frontiers without PutDiscovered (pre-2.6) are detected via UNIMPLEMENTED\n and the bolt falls back to sending discovered URLs individually\n- flow control follows the PutURLs client shipped with URLFrontier: throttled\n sends wait on a monitor woken by acks and by the transports\u0027 on-ready\n notifications instead of polling\n- requires urlfrontier-API 2.6 (wire compatible with 2.5 servers)\n\nFixes #2062\n\n* Address review: no tuple stranding, honest throttling metric, docs and formatting\n\n- fail the tuples of a batch that was never sent (interrupted flush) or\n whose stream died right away instead of leaving them to the waitAck\n cache expiry, which would outlive Storm\u0027s tuple timeouts\n- measure the time actually spent waiting in the throttle loop: an ack\n can wake the wait well before the timeout, and the\n timeSpentThrottling metric must keep meaning something\n- document that the permit tryAcquire is deliberately outside the flow\n monitor (a racing notifyAll costs one backstop interval, never a\n deadlock) and the batchLock then sendLock acquisition order\n- document urlfrontier.batch.size in configuration.adoc\n- reformat with the google-java-format version the\n git-code-format-maven-plugin embeds" }, { "commit": "3c5fcd64539d7d7c74eab968b65e05c0fe5bf11d", "tree": "56b6decaf21c4b5d2b499df2d859dbdbaf8219ee", "parents": [ "ead30fc90a77047342ed6114a996b4e890364403" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Sep 10 09:50:29 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 09:50:29 2026 +0200" }, "message": "SQL IndexerBolt: only plain identifiers may be used as column names, fixes #2096 (#2146)\n\n* SQL IndexerBolt: only plain identifiers may be used as column names, fixes #2096\n\nbuildQuery() joined the metadata labels straight into the column list and\ninto the ON DUPLICATE KEY UPDATE clause. Values are bound, but identifiers\ncannot be, and with a glob mapping (an indexer.md.mapping entry ending in *)\nthe label is the raw metadata key - which crawled content mints: the Tika\nParserBolt copies every \u003cmeta name\u003d\"...\"\u003e of a page to parse.\u003cname\u003e, and\nresponse header names are stored the same way.\n\nLabels are now checked against ^[A-Za-z0-9_]+$ and dropped when they do not\nmatch, with a warning logged once per distinct label and an\nunusable_column_name metric. Dropping loses nothing that worked before: any\nlabel outside that pattern already produced a broken unquoted identifier and\na guaranteed SQLException. That also ends a replay loop, since execute()\nfails the tuple without emitting to the status stream, so an ordinary dotted\nlabel such as parse.title - which MySQL reads as table.column - previously\nfailed the same tuple for the life of the topology.\n\nA dropped label is arbitrary crawled content and is written to a log line,\nso it is rendered as printable ASCII first. A \\p{Cntrl} blacklist would not\ndo: it is ASCII-only and leaves U+0085, U+00A0, U+2028 and U+2029 in place,\nthree of which break the line for a log reader.\n\nAlias mappings (parse.title\u003dtitle) remain the supported way to index a\ndotted key. The table name and the URL field are operator-owned topology\nconfig and are left as they are.\n\n* SQL IndexerBolt: only check labels from glob mappings\n\nAliases and plain mapping keys are written by the operator and are\ntrusted like the table name and the URL field name, so a configured\ncolumn such as col$1 or résumé, which MySQL accepts unquoted, is indexed\nagain. Labels produced by a glob mapping are metadata keys that crawled\ncontent can mint, and must still be plain identifiers.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003czowalla@colorful-bytes.com\u003e" }, { "commit": "ead30fc90a77047342ed6114a996b4e890364403", "tree": "afbc04a6168e31a81f3964e343dba66efb2b855b", "parents": [ "5dd6d99cba62e849cd857edccb260d052b749f29" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Thu Sep 10 13:18:01 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 10 09:48:01 2026 +0200" }, "message": "Upgrade to Tika 4.0.0 (#2118)\n\n* Upgrade to Tika 4.0.0\n\nCloses #2063\n\nTika 4.0.0 requires several API and configuration changes:\n\n- core: TikaConfig has been removed - JSoupParserBolt now uses\n DefaultDetector for mime type detection; Detector.detect takes a\n TikaInputStream and a ParseContext; the CONTENT_TYPE and\n CONTENT_LENGTH constants moved from Metadata to HttpHeaders.\n- external/tika: ParserBolt loads configurations with TikaLoader\n (tika-serialization) since XML configurations are no longer\n supported, and the bundled tika-config.xml has been migrated to\n JSON. Config resources bundled in a jar are copied to a temporary\n file as TikaLoader reads configs from the filesystem only.\n- external/tika: tika-parsers-standard-package is now a POM\n aggregating the individual parser modules and is consumed as such.\n- the default value of parser.tika.config.file is now\n tika-config.json\n- THIRD-PARTY.txt regenerated\n\n* Address review feedback on the Tika 4 upgrade\n\n- seed each parse with the ParseContext from the \u0027parse-context\u0027\n section of the Tika configuration instead of ignoring it\n- fail fast when the Tika configuration is missing or cannot be\n loaded: silently degrading to the default configuration would\n activate parsers the configuration excluded, e.g. OCR\n- delete the temporary copy of the configuration right after\n TikaLoader has read it instead of keeping it for the lifetime\n of the worker\n- bind EmptyParser in the ParseContext unless parser.extract.embedded\n is set: the AutoDetectParser sets itself as the embedded parser\n otherwise, so embedded documents were parsed even though the\n configuration said not to\n- use org.apache.tika.metadata.HttpHeaders constants for Tika\n metadata calls\n- fix the tika config key in the archetype crawler-conf.yaml files\n (parser.tika.config.file, not parse.tika.config.file) and point\n the README at a working Tika 4 documentation URL\n- new TikaConfigTest covering the exclusions from the bundled\n configuration, the parse-context seeding and the fail-fast\n behaviour, plus a regression test for the embedded documents\n\n* Tika 4 docs: document parser.extract.embedded and parse.* key renames" }, { "commit": "5dd6d99cba62e849cd857edccb260d052b749f29", "tree": "b4e0ab61ea70dd798370df8556b128ef39e7b838", "parents": [ "a12bd13cd47cbece293b53a65542e35aefed4730" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Wed Sep 09 22:57:45 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 19:27:45 2026 +0200" }, "message": "File scheme off by default; reads confined to file.protocol.root (#2124)\n\n* File scheme off by default; reads confined to file.protocol.root (#2081)\n\ncrawler-default.yaml enabled the file scheme in every topology built on\nthe library defaults, and FileProtocol read whatever path the worker\nuser could read: a fetched page can put a file:// URL into the frontier\nand the response body - including the topology configuration - went on\nto be parsed and indexed.\n\nThe shipped protocols list is now http,https, and the file scheme has\nto be enabled deliberately. FileProtocol gains a file.protocol.root\nkey: when it is set, only paths canonicalising below that directory are\nserved (symlinks included), everything else returns 403; when it is\nnot set, the file scheme serves nothing. An operator crawling a local\ncorpus sets both keys.\n\n* Refuse file URLs with a host, note the check-vs-read window (#2081)\n\nReview feedback on #2124:\n\n- a file URL carrying a host component (file://evil.example.com/etc/passwd)\n is refused outright instead of the host being silently ignored: the\n host is meaningless for a local read, and accepting it would only\n invite spellings that look remote\n- a comment documents the accepted check-vs-read window: canonicalise\n and check are two operations, so a symlink swapped in between would\n be followed; that is fine now that the file scheme is opt-in and\n root-confined, the operator who enables it accepts the worker user\u0027s\n read rights as the boundary\n- confirmed nothing in the archetypes or the ProtocolFactory assumes\n file is in the protocols list\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Pin the host check and document file.protocol.root (#2081)\n\nReview follow-ups on #2124 (@rzo1, @dpol1):\n\n- fileProtocolRejectsHostComponents now points at a real file inside the\n configured root: without the host check the request would resolve to\n that readable file and return 200, so the assertion (403) is tied to\n the host check specifically\n- the symlink escape test asserts 403 instead of not-200, which\n distinguishes the root-confinement rejection from the 300 the\n canonical-path redirect gives for a symlink inside the root\n- configuration.adoc: protocols now shows http,https as the shipped\n default and documents file.protocol.root" }, { "commit": "a12bd13cd47cbece293b53a65542e35aefed4730", "tree": "cb4624ba1f0afff356fc4410c254c004aca3465a", "parents": [ "e5b05d3c1caa0869d33c18e67d01e4f88e212525" ], "author": { "name": "Gianluca Graziadei", "email": "48211703+GGraziadei@users.noreply.github.com", "time": "Wed Sep 09 11:51:54 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 11:51:54 2026 +0200" }, "message": "#2129 FetcherBolt: lock-free fetch queues ordered by next fetch time (#2132)\n\n* #2129 FetcherBolt: lock-free fetch queues ordered by next fetch time\n\nFetchItemQueues used a single monitor for adding, taking and finishing\nitems, and getFetchItem() rotated a LinkedHashMap linearly over all the\nqueues that were not ready yet. With many hosts the fetcher threads held\nthat lock most of the time and the executor thread calling execute()\nstalled on every incoming tuple.\n\nQueues are now kept in a ConcurrentHashMap and the ones that may have an\nitem ready are referenced from a DelayQueue of tickets ordered by their\nnext fetch time, so taking an item is O(log n) and adding never waits for\nthe fetcher threads. Per-queue state (size bound, in-progress count, next\nfetch time, crawl delays) is handled with atomics and a per-queue monitor\nonly for the add/reap race. Empty queues are removed from the map as\nsoon as they drain.\n\nFixes #2129.\n\nBehaviour preserved: politeness per queue, asap release, max threads per\nqueue, max queue size, crawl delay overrides from metadata and\nrobots.txt, queue modes, metrics and the debug dump. Covered by the new\nFetchItemQueuesTest, including a concurrent producers/consumers test.\n\nBenchmark (50 fetcher threads, 1 producer, 20 URLs per host, delay 1s):\n 20000 hosts getFetchItem avg 1.58 ms -\u003e 24 us\n addFetchItem p99 458 ms -\u003e 0.11 ms, max 571 ms -\u003e 0.8 ms\n throughput unchanged (bounded by politeness)\n\n* #2129 FetchItemQueues: do not lose the wakeup when a fetch finishes during the free-slot check\n\nIn getFetchItem(), a fetch on the same queue could finish between\nhasFreeSlot() returning false and the clearing of the scheduled flag.\nfinishFetchItem() then found the flag still set and did not issue a\nticket, after which the flag was cleared: the queue was left with items\nand a free slot but no ticket, and stalled until an unrelated URL for\nthe same host arrived.\n\nClear the flag first and re-check, as the other branches already do.\nCovered by a deterministic test that pauses the poller inside the\nfree-slot check while the in-progress fetch finishes.\n\n* #2129 FetchItemQueues: document the transient over-report in offer() and the smeared debug dump\n\noffer() increments the size before checking the bound and decrements on\noverflow, so getQueueSize() can transiently over-report by one while an\noffer is rejected; the value only feeds metrics and the debug dump.\n\nlogQueuesContent() is no longer synchronized: it iterates the queues and\ntheir items with weakly consistent iterators while the fetcher threads\nkeep working, so the dump is a smear rather than a point-in-time\nsnapshot. Both are now stated in the javadoc.\n\n* #2129 FetchItemQueue: reserve the slot before dequeuing so a concurrent finish cannot reap the queue\n\npoll() ran queue.poll() before inProgress.incrementAndGet(). With more\nthan one thread per queue, a fetch finishing in that window saw an empty\nqueue and zero in progress and removed the queue while the just-polled\nitem was being dispatched; that item then finished on an unknown or\nreplacement queue, driving the replacement\u0027s counter negative and\nletting it exceed maxThreads.\n\nThe slot is now reserved with a CAS before the dequeue and released if\nthe queue turned out to be empty, so inProgress is never zero while an\nitem is in flight and the bound is exact when several threads race for\nthe last slot. getFetchItem re-issues the ticket on a null poll only if\na slot is free; a fetch finishing later schedules the queue itself.\n\nA package-private afterDequeue() hook lets the regression test pause the\npoll right after the last item has been taken and finish the other fetch\nin the meantime.\n\n* #2129 FetchItemQueues: atomic max on crawl delays, size reserved before dequeue, stale tickets do not stall the poll\n\nThree places where the removed global monitor was still doing work:\n\n- getFetchItemQueue raised the per-queue delays with a read-check-write\n on volatile fields. Two URLs for the same host arriving together with\n different delays could settle on the smaller one. The delays are now\n AtomicLongs raised with accumulateAndGet(Math::max); the fetcher\n thread keeps setting them outright from robots.txt as before.\n Reproduced by a spin-barrier test: 2284 lost rounds out of 300000.\n\n- poll() decremented size after the dequeue while offer() bounds on it\n without the poll holding anything. At the bound, an add racing a poll\n was refused for a queue that had room and the tuple failed. The size\n is now reserved before the dequeue and given back on an empty queue,\n mirroring the slot reservation. getQueueSize() clamps at zero for the\n metrics.\n\n- A stale head ticket (delay extended after it was issued) made\n getFetchItem return null, and the fetcher thread slept 100 ms with\n other queues due. The ticket is re-issued at its new time and the loop\n moves on to the next one." }, { "commit": "e5b05d3c1caa0869d33c18e67d01e4f88e212525", "tree": "3d748c4807dd183de6bd9df16e77699e5c9ddf90", "parents": [ "c58fd218b3ea756bc20dac09bc1714caf198b9ad" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Wed Sep 09 15:21:04 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 11:51:04 2026 +0200" }, "message": "URLFilters: treat an exception from a filter as a rejection (#2120)\n\n* URLFilters: treat an exception from a filter as a rejection (#2084)\n\nThe try/catch wrapped the whole chain: when a filter threw, the catch\nlogged and fell through to the value the last successful filter had\nproduced, so callers saw the URL as accepted and every filter after the\none that threw was skipped - including the regex exclusions placed last\nin the archetype chain.\n\nThe try/catch now sits inside the loop, logs which filter threw and\nreturns null: a chain which throws must not widen what the crawl\naccepts. main() applies the same verdict, and a counter records how\noften it happens.\n\nFastURLFilter.Rule no longer accepts a rule line without a recognised\ntype; it failed at evaluation time with an NPE and now fails at load\ntime instead.\n\n* Rename the exception counter, drop the duplicate CLI line (#2084)\n\nReview feedback on #2120:\n\n- the field is renamed to exceptionsCount to match what\n getExceptionsCount reports\n- main() no longer prints the rejection twice: the timing line\u0027s\n \u0027\u003d\u003e null\u0027 output already says it, the System.err line is gone\n- rejectionShortensTheChain now proves the chain stops: a counting\n filter sits after the thrower and asserts it is never evaluated\n- getExceptionsCount stays public for operators embedding the chain\n (bolts with a TopologyContext can expose it as a metric later; the\n filter itself has no context to register one)\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Drop narrative comments\n\nRemove the commentary added in this PR; the code and the exception\nmessages already say what happens.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e" }, { "commit": "c58fd218b3ea756bc20dac09bc1714caf198b9ad", "tree": "f270e73fdd43eb3a4dfcdf0edf2af346404ca4ae", "parents": [ "c3fa48bbbea14ad843df2c10937ab8e41a161383" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Wed Sep 09 14:38:52 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Sep 09 10:08:52 2026 +0100" }, "message": "Robots.txt fetch no longer replaces the configured content limit (#2121)\n\n* Robots.txt fetch no longer removes the configured content limit (#2087)\n\nHttpRobotRulesParser wrote http.robots.content.limit (-1 by default)\ninto the fetch metadata as http.content.limit, which HttpProtocol reads\nunconditionally: a robots.txt fetch ran with no limit at all, even when\nthe operator had configured a finite http.content.limit, despite the\nshipped comment saying the default was \u0027same as http.content.limit\u0027.\n\n- a http.robots.content.limit of -1 now means \u0027inherit the global\n limit\u0027 and the key is only written into the fetch metadata when a\n robots specific limit is configured\n- the shipped default of http.robots.content.limit is 524288 (512 kiB),\n matching the minimum the robots.txt RFC draft asks crawlers to parse\n- HttpProtocol ignores a metadata http.content.limit of -1 when the\n global limit is finite, so per-URL metadata can tighten the limit but\n not remove it\n\n* Ignore invalid metadata content limits, reword the widening rationale (#2087)\n\nReview feedback on #2121:\n\n- a http.content.limit in the metadata below -1 is a configuration\n error and is ignored instead of being applied as a nonsensical limit\n- the comment on the -1 special case now says what the shipped default\n of http.robots.content.limit actually does: the robots.txt fetch is\n deliberately allowed a larger read than pages when\n http.content.limit is below the 512 kiB RFC floor, because the floor\n is the reason the robots specific key exists\n- confirmed no other protocol reads the http.content.limit metadata\n key: only the okhttp HttpProtocol does, WARCSpout reads the config\n value directly and is unaffected\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes.\n\n* Shorten the content limit comments\n\n* Document the robots.txt content limit default, cover the OkHttp path\n\nconfiguration.adoc still listed -1 as the default of\nhttp.robots.content.limit and did not mention that the limit may be\nlarger than http.content.limit.\n\nThe existing tests use a recording Protocol stub, so they never reach\nthe limit handling in HttpProtocol. HttpProtocolContentLimitTest fetches\nfrom Jetty and covers a metadata limit of -1 against a finite global\nlimit, values below -1, a non-numeric value, widening and tightening,\n-1 with no global limit, and gzip responses trimmed on their\ndecompressed length. RobotsContentLimitTest pins the shipped default\nand the inheritance of an explicit -1.\n\n---------\n\nCo-authored-by: Richard Zowalla \u003crzo1@apache.org\u003e" }, { "commit": "c3fa48bbbea14ad843df2c10937ab8e41a161383", "tree": "5ef7a659ece109652d0056b989f943765f8159e2", "parents": [ "2cb93d1c012ce6a94aaf675c4ec353208a359255" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Wed Sep 09 00:37:36 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 08 21:07:36 2026 +0200" }, "message": "IPFilterRules: add anylocal and multicast keywords, fail on unparseable rules (#2119)\n\n* IPFilterRules: add anylocal and multicast keywords, fail on unparseable rules (#2086)\n\nA rule that is neither a keyword, a CIDR block nor a single address was\nlogged and dropped, silently widening what the crawler will connect to.\nConfiguration now fails with an IllegalArgumentException instead, so a\ntypo in an exclude list stops the topology rather than removing a rule\nthe operator believed was in force.\n\nAdds the anylocal and multicast keywords, documents linklocal (already\npresent but missing from the javadoc), and updates the example in\ncrawler-default.yaml to also cover CGNAT and IPv6 unique-local space.\n\n* Example covers all keywords and the full ULA range (#2086)\n\nThe commented example in crawler-default.yaml now exercises anylocal\nand multicast and uses fc00::/7 (the whole unique-local range), not\njust the locally-assigned fd00::/8 half. The documented-example test\nfollows.\n\n* Normalize imports and formatting to satisfy CI\n\nThe import groups are collapsed into the single ASCII-sorted group the\nproject checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the\nconstructors in URLFilters are grouped as checkstyle requires, and the\ntouched files are reformatted with the google-java-format version the\ngit-code-format plugin pins (1.35.0, AOSP). No code changes." }, { "commit": "2cb93d1c012ce6a94aaf675c4ec353208a359255", "tree": "964ab725ff52660fbd67853ad7e9a088c9e5d866", "parents": [ "096136432d5ce06e94d0b230f402e8db88155e06" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:42:19 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:42:19 2026 +0200" }, "message": "Minor: Regenerated License File for 096136432d5ce06e94d0b230f402e8db88155e06 (#2144)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "096136432d5ce06e94d0b230f402e8db88155e06", "tree": "b2e33239621a5f00eea86e2973739aedcdcc9c69", "parents": [ "3a49d077b35c48ad6ac135918d829ffd33e23ec3" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:37:20 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:37:20 2026 +0200" }, "message": "Minor: Regenerated License File for db885724d5bc2fd730fdd5f0b7ceff3dcbc2f90e (#2143)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "3a49d077b35c48ad6ac135918d829ffd33e23ec3", "tree": "caabfc4fa40a4b4c9251e32f03f68c3136af7821", "parents": [ "0661651f11612d4d09be2f3d5def740fcc36fac4" ], "author": { "name": "Gianluca Graziadei", "email": "48211703+GGraziadei@users.noreply.github.com", "time": "Mon Sep 07 19:10:46 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:10:46 2026 +0200" }, "message": "#2130 MetadataTransfer: single pass over the metadata for wildcard keys (#2133)\n\nPre-compile the keys to transfer into exact keys and wildcard prefixes\nand copy the matching entries in one pass over the metadata, instead of\nbuilding an intermediate key set per wildcard for every outlink.\nMicro-benchmark on a 12-key metadata with 3 wildcards: ~780 ns to\n~265 ns per outlink.\n\nThe compiled form is built lazily and keyed on a snapshot of the key\nset, so a subclass editing mdToTransfer / mdToPersistOnly (in an\noverridden configure() or later, even without changing the size) is\nalways honoured. Null value arrays in a caller-supplied map are skipped\nas before.\n\nFixes #2130." }, { "commit": "0661651f11612d4d09be2f3d5def740fcc36fac4", "tree": "5d1001053c02db3ff36520e5e58fd4f7fa9ce7ff", "parents": [ "8f9923718fb04cd7525a0734e9391937a76e789a" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:21 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:21 2026 +0200" }, "message": "build(deps): bump org.netpreserve:jwarc from 0.36.0 to 0.37.0 (#2142)\n\nBumps [org.netpreserve:jwarc](https://github.com/iipc/jwarc) from 0.36.0 to 0.37.0.\n- [Release notes](https://github.com/iipc/jwarc/releases)\n- [Changelog](https://github.com/iipc/jwarc/blob/master/CHANGELOG.md)\n- [Commits](https://github.com/iipc/jwarc/compare/v0.36.0...v0.37.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.netpreserve:jwarc\n dependency-version: 0.37.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "8f9923718fb04cd7525a0734e9391937a76e789a", "tree": "ebb28c37cc7f7e5c974112e8efbf0e198e962ed6", "parents": [ "db885724d5bc2fd730fdd5f0b7ceff3dcbc2f90e" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:18 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:18 2026 +0200" }, "message": "build(deps): bump org.apache.maven.plugins:maven-surefire-plugin (#2141)\n\nBumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.5.6 to 3.6.0.\n- [Release notes](https://github.com/apache/maven-surefire/releases)\n- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.5.6...surefire-3.6.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.maven.plugins:maven-surefire-plugin\n dependency-version: 3.6.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "db885724d5bc2fd730fdd5f0b7ceff3dcbc2f90e", "tree": "6917b2c895a4e8a99feea50088b4faaddf881028", "parents": [ "80ae6324319811c06fd9be989172ff1208a8be56" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:14 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:14 2026 +0200" }, "message": "build(deps): bump org.opensearch.client:opensearch-java (#2140)\n\nBumps [org.opensearch.client:opensearch-java](https://github.com/opensearch-project/opensearch-java) from 3.9.0 to 3.10.0.\n- [Release notes](https://github.com/opensearch-project/opensearch-java/releases)\n- [Changelog](https://github.com/opensearch-project/opensearch-java/blob/v3.10.0/CHANGELOG.md)\n- [Commits](https://github.com/opensearch-project/opensearch-java/compare/v3.9.0...v3.10.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.opensearch.client:opensearch-java\n dependency-version: 3.10.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "80ae6324319811c06fd9be989172ff1208a8be56", "tree": "440736bdfd68860941b992717249b0214836fa3c", "parents": [ "f062642c4180a1092f23cc5c8c65cfd927b6709c" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:11 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:11 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.54.6 to 2.54.12 (#2139)\n\nBumps software.amazon.awssdk:bom from 2.54.6 to 2.54.12.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.54.12\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "f062642c4180a1092f23cc5c8c65cfd927b6709c", "tree": "678338f8bb1d395f4046f9b60b3589e70d1279a5", "parents": [ "770389afdf710dd360654083fee8844018a6f405" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:08 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:08 2026 +0200" }, "message": "build(deps): bump com.puppycrawl.tools:checkstyle from 14.0.0 to 14.1.0 (#2138)\n\nBumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 14.0.0 to 14.1.0.\n- [Release notes](https://github.com/checkstyle/checkstyle/releases)\n- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-14.0.0...checkstyle-14.1.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.puppycrawl.tools:checkstyle\n dependency-version: 14.1.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "770389afdf710dd360654083fee8844018a6f405", "tree": "7d378212a5b777808d1b85c5deff5fceef5ce15b", "parents": [ "d8d55819ba2e69f4df06f7dbbfc9e268174c0174" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:05 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:05 2026 +0200" }, "message": "build(deps): bump org.yaml:snakeyaml from 2.6 to 2.7 (#2137)\n\nBumps org.yaml:snakeyaml from 2.6 to 2.7.\n\n---\nupdated-dependencies:\n- dependency-name: org.yaml:snakeyaml\n dependency-version: \u00272.7\u0027\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "d8d55819ba2e69f4df06f7dbbfc9e268174c0174", "tree": "159ca5343a22c7e719a1c38a0baeed2684ad950b", "parents": [ "fb139100f4939e98f42023dde4ae09d0412768a1" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Sep 07 19:09:01 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Sep 07 19:09:01 2026 +0200" }, "message": "build(deps): bump org.apache.maven.plugins:maven-compiler-plugin (#2136)\n\nBumps [org.apache.maven.plugins:maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) from 3.15.0 to 3.16.0.\n- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)\n- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.15.0...maven-compiler-plugin-3.16.0)\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.maven.plugins:maven-compiler-plugin\n dependency-version: 3.16.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "fb139100f4939e98f42023dde4ae09d0412768a1", "tree": "776803645e50a5c1a1aec53fe18d5f5eb9c25b1f", "parents": [ "3c4f5e7632696c475f091b130ec626ce68ca9c6a" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Sun Sep 06 19:09:54 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Sep 06 15:39:54 2026 +0200" }, "message": "Make the WARC digest algorithm configurable (#2110)\n\n* Make the WARC digest algorithm configurable\n\nWARCRecordFormat hard coded the algorithm for the WARC-Payload-Digest\nand WARC-Block-Digest fields to SHA-1 with no way for an operator to\nchoose another algorithm. Add the configuration key\nwarc.digest.algorithm which accepts sha1 (the default, kept for\ncompatibility with CDX indexes and other downstream tooling) and\nsha256, and thread it through WARCHdfsBolt so that response, request,\nresource and metadata records all use it. The static getDigestSha1\nhelpers are kept and deprecated in favour of the new getDigest\ninstance methods.\n\nhttps://github.com/apache/stormcrawler/issues/2106\n\n* Omit Base32 padding in WARC digest values\n\nThe WARC digest fields define the digest value as a token, which does\nnot allow the character \"\u003d\" used for Base32 padding (ISO 28500 WARC\n1.1). Strip the trailing padding so that e.g. SHA-256 digests conform;\nSHA-1 digests are 32 characters without padding and are unaffected.\nSuggested in review of #2110.\n\n* Apply google-java-format\n\nThe validate-code-format check requires google-java-format (AOSP style);\nrun git-code-format:format-code on the touched files. No functional\nchanges.\n\n* Address review feedback on the configurable digest algorithm\n\n- remove the deprecated static getDigestSha1 methods: the next release\n is a major one (4.0.0), so the deprecated API can go straight away\n- validate the inputs of the getDigest methods (requireNonNull)\n- treat a null or blank warc.digest.algorithm value as the default\n (SHA-1), like an unset one\n- add javadoc to the constructors of the record formats and document\n the parameters, including the null/blank behaviour\n\nSuggested in review of #2110 by rzo1." }, { "commit": "3c4f5e7632696c475f091b130ec626ce68ca9c6a", "tree": "ad69f2c41da4313f458d7a030429b79f5fb33e83", "parents": [ "04ee3514fc33211b9bb7780d817b01ff15bc65e3" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Thu Sep 03 12:47:42 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Sep 03 08:17:42 2026 +0100" }, "message": "ParseResult: non-mutating lookup, explicit getOrCreate() and no empty documents emitted (#2112)\n\n* ParseResult: read-only getIfPresent(), explicit getOrCreate(), skip empty entries\n\nParseResult.get(url) created and stored an empty ParseData on a miss, so\na lookup for a URL that was never parsed inserted an entry which the\nparser bolts emitted as a content-less document (issue #2108).\n\n- add ParseResult.getIfPresent(url), a read-only lookup returning null\n on a miss without modifying the ParseResult\n- add ParseResult.getOrCreate(url) with the creating behaviour, used by\n put(), set() and the parser bolts; deprecate get() as a delegate so\n existing parse filters keep working while the intent is explicit at\n the call site; migrate all in-tree callers\n- JSoupParserBolt and the Tika ParserBolt skip entries with no content,\n no text and no metadata when emitting documents\n- ParseData no-arg and metadata-only constructors set content to an\n empty byte array, consistent with the (text, metadata) constructor\n- new ParseResultTest and bolt-level tests verifying that empty entries\n are not emitted; SubDocumentsParseFilter fixture attaches metadata to\n the sub-documents it creates\n\n* Skip only empty sub-documents, count skips, document for filter authors\n\nReview feedback on #2112:\n\n- the parser bolts now skip empty entries only for URLs other than the\n one being parsed: the document itself is always emitted, even when a\n filter emptied it, so that its status keeps being updated downstream\n- skips are no longer silent: each one increments a\n skipped_empty_documents counter and logs at debug level\n- document the emission semantics for filter authors in internals.adoc\n- new tests assert that an emptied parent document is still emitted\n\n* Remove ParseResult.get() and trim comments\n\nReview feedback on #2112:\n\n- since the next release is a major one (4.0.0), remove the ambiguous\n get(url) method outright instead of deprecating it; filters must use\n getIfPresent(url) for a read-only lookup or getOrCreate(url) to create\n an entry\n- shorten the Javadoc of the new ParseResult accessors and the comments\n above the emission loops in the parser bolts\n\n* Fix remaining ParseResult.get() callers in playwright tests\n\nJsRenderingDetectorTest holds the ParseResult in a variable named p,\nwhich the mechanical migration of parse.get() to getOrCreate() missed.\nReplace get() with getOrCreate() there as well; verified by a full\nreactor clean build." }, { "commit": "04ee3514fc33211b9bb7780d817b01ff15bc65e3", "tree": "2c3c37b0360f4dfc30154d1b6ee10a97793916cb", "parents": [ "0663b8c7f49140daaeba35572e4488b3766d9e8c" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Tue Sep 01 22:01:50 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 17:31:50 2026 +0100" }, "message": "CommaSeparatedToMultivaluedMetadata: replace quadratic token append with a single bulk write (#2111)\n\n* Replace quadratic metadata append in CommaSeparatedToMultivaluedMetadata\n\nMetadata.addValue copies the whole array on every call, so splitting a\nvalue into n tokens and appending them one at a time in\nCommaSeparatedToMultivaluedMetadata.filter costs n^2/2 element copies.\nThe key is removed just before the loop, so the tokens can be stored\nwith a single setValues call instead. The number of tokens taken from\none value is capped (maxTokens, 65536 by default) and a warning is\nlogged when the cap trims, so that the amount of metadata a page\nproduces does not grow unbounded with the page size.\n\nMetadata.addValues(String, String[]) had the same shape when the key\nwas already present: it looped over addValue. It now appends with a\nsingle array copy; as a side effect it no longer skips blank values in\nthat case, which matches what it already did when the key was absent.\n\n* CommaSeparatedToMultivaluedMetadata: cap after blank filtering, default 128, document maxTokens\n\n- apply the maxTokens cap to the values kept after blank tokens are\n dropped, so that blank tokens do not consume the budget and a low cap\n keeps the first real tokens\n- lower the default cap from 65536 to 128: a page cannot generate an\n unbounded amount of metadata with the default configuration\n- drop the confusing reference to http.content.limit\n- document maxTokens in the parse filters docs and the filter javadoc\n- the scale tests configure maxTokens explicitly since the bulk write\n path they exercise is independent of the cap" }, { "commit": "0663b8c7f49140daaeba35572e4488b3766d9e8c", "tree": "faef42b1a6aa3a33215f746b5075904080156660", "parents": [ "f7cf183ab59e27804cfdc00be3d74bedf5915f98" ], "author": { "name": "Abhinav", "email": "alpha9coder@gmail.com", "time": "Tue Sep 01 21:38:28 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 17:08:28 2026 +0100" }, "message": "Escape CR and LF in WARC metadata records and resource Content-Type (#2109)\n\n* Escape CR and LF in WARC metadata records and resource Content-Type\n\nMetadataRecordFormat.format() wrote one line per metadata value into the\napplication/warc-fields payload without checking for CR or LF. A value\ncontaining CR LF (e.g. feed.description set by FeedParserBolt, or values\nof parse.* filters such as the XPath, LDJson and Tika filters) therefore\nbecame additional field lines that look exactly like fields written by\nthe crawler - for example a fabricated hopsFromSeed or via. Framing\nstayed valid because Content-Length is computed from the finished\npayload, so WARC readers had no way to detect the injected fields.\n\n- replace CR and LF by spaces in metadata values written into the\n warc-fields payload, and drop metadata keys that are not valid WARC\n field names (printable ASCII without colon, RFC 5322 section 2.2)\n- sanitise the server-supplied Content-Type used for resource records in\n WARCRecordFormat.format(), which was appended verbatim into the WARC\n header block\n- log MetadataRecordFormat messages under MetadataRecordFormat instead\n of WARCRequestRecordFormat\n\nFixes #2105\n\n* Validate warc.metadata.keys once on instantiation\n\nThe configured metadata keys are fixed topology configuration: check\nthem for valid WARC field names in the constructor and drop invalid\nkeys with a single warning, instead of repeating the check and warning\nfor every record written. Suggested in review.\n\n* Make null check in isValidWarcFieldName explicit\n\nThe conjunction already returned false for a null name through\nshort-circuit evaluation, but the intent was easy to miss. Return false\nexplicitly and cover the field name and value sanitisation helpers with\nunit tests. Suggested in review." }, { "commit": "f7cf183ab59e27804cfdc00be3d74bedf5915f98", "tree": "49f17374dd513b8e09d3cab2f911f03d89459cec", "parents": [ "a9157028783c711ec72ef1154400186f201a7302" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Sep 01 09:18:48 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Sep 01 09:18:48 2026 +0200" }, "message": "Regenerated License file after dependency upgrades (#2115)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "a9157028783c711ec72ef1154400186f201a7302", "tree": "dedbdbe5b489bec8792ed8e0919650a22de046e1", "parents": [ "05e1d4a9c6299f6d188551ed0dabe98f04b2348a" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 31 21:32:32 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 21:32:32 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.54.1 to 2.54.6 (#2113)\n\nBumps software.amazon.awssdk:bom from 2.54.1 to 2.54.6.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.54.6\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "05e1d4a9c6299f6d188551ed0dabe98f04b2348a", "tree": "6f7476b4718e24b4ed52eda4c577d2a2c4093a93", "parents": [ "93b6e24d016caad06cb05546a042b143ef3a5dad" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 31 21:32:15 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 31 21:32:15 2026 +0200" }, "message": "build(deps): bump org.jsoup:jsoup from 1.23.1 to 1.23.2 (#2114)\n\nBumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.23.1 to 1.23.2.\n- [Release notes](https://github.com/jhy/jsoup/releases)\n- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)\n- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.23.1...jsoup-1.23.2)\n\n---\nupdated-dependencies:\n- dependency-name: org.jsoup:jsoup\n dependency-version: 1.23.2\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "93b6e24d016caad06cb05546a042b143ef3a5dad", "tree": "68f96af88fdb85e38f51969d5ab4d58532e98dbf", "parents": [ "f520b4e4d3ef15df84eff84d80921c327768d7e7" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 29 13:22:20 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 13:22:20 2026 +0200" }, "message": "Only follow robots.txt redirects within the same origin (#2077)\n\nHttpRobotRulesParser followed the Location header of a robots.txt response\nfor up to 5 hops, honouring an absolute target verbatim and re-fetching it\nwith no restriction, so the crawled site chose which host was requested.\nA redirect is now only followed when the target shares scheme, host and\nport with the URL it was reached from, unless\nhttp.robots.redirect.crossorigin.allow is set.\n\nA redirect which only replaces http with https while staying on the same\nhost and port is always followed. getCacheKey derives the default port\nfrom the scheme, so such a target has a different key although no other\nhost is involved, and refusing it left the very common apex redirect\nwithout any rules. Only that direction is exempt: https to http would send\nthe headers configured for the fetch unencrypted and would let rules read\nover a plain text connection replace the ones the redirect was reached\nfrom.\n\nA redirect which is not followed no longer ends as allow-all in the\nsuccess cache. It yields no rules for the host, cached in the error cache\nlike a 429 or a 5xx, so nothing is crawled there until the redirect is\nresolved. http.robots.redirect.refused.allow: true restores the previous\nbehaviour and crawls the host without any rules. The refusal is logged at\nwarn level, naming the target, why it was not followed and which setting\nchanges it.\n\nFollowing a redirect to another origin re-fetches the robots.txt with the\nheaders configured through http.basicauth.* and http.custom.headers, as\nany other request does. That is now stated where the setting is defined\nand logged once at configuration time when either is set, and\nhttp.filter.ipaddress.exclude gained a linklocal rule so that the\naddresses those fetches may reach can be restricted: link local addresses\nare matched by neither loopback nor sitelocal, so the documented example\ndid not cover them." }, { "commit": "f520b4e4d3ef15df84eff84d80921c327768d7e7", "tree": "076c8bb810b57897f857bc2c40a27513a143491e", "parents": [ "88fcf3bb2c6919d79937e851ca9e13493acfdcf3" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 29 13:22:10 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 13:22:10 2026 +0200" }, "message": "Escape the metadata column of the SQL status store (#2076)\n\nThe SQL status store flattened the metadata into one column as tab\nseparated key\u003dvalue pairs without escaping, so a value containing a\ntabulation came back from SQLSpout as extra keys and any legitimate\nvalue with a tabulation was corrupted. Keys and values are now escaped\nwhen the column is written and unescaped when it is read, which makes\nthe round trip lossless rather than dropping characters silently.\n\nColumns written by the previous code start with a tabulation, columns\nwritten now start with a format marker, so existing rows keep being\ndecoded exactly as before. Rows written from now on are not readable by\nearlier versions, which see the marker as an extra key. SQLSpout no\nlonger decodes the column with StringTabScheme, whose format cannot\nrepresent a tabulation in a value.\n\nAbstractStatusUpdaterBolt also parsed status.store.as.is.with.nextfetchdate\noutside its try/catch, so a stored value that is not a valid instant threw\nout of execute. It is now logged and the URL scheduled normally, including\na value which parses as an instant but does not fit a java.util.Date.\nEncoding a metadata no longer fails on a key which holds no readable\nvalue, such as the empty key a column written before the escaping can\nproduce." }, { "commit": "88fcf3bb2c6919d79937e851ca9e13493acfdcf3", "tree": "139bb1b23eecd3894c6b1446bf83d3b832049768", "parents": [ "2223fe8f586e5a388a9c139107d5e4f30cb9fc72" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 29 13:21:56 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Aug 29 13:21:56 2026 +0200" }, "message": "Scope cookies to the host that set them (#2075)\n\nCookieConverter only checked the domain when the cookie carried a Domain\nattribute, so a cookie without one was sent to any target URL, a Domain\nwhich covers every host under it such as \"com\" or \"co.uk\" was accepted as\na scope, and checkDomainMatchToUrl returned true when it threw. getCookies\nnow takes the URL whose response set the cookies, keeps a cookie without a\nDomain attribute for that host only, and fails closed on error.\n\nA Domain attribute is normalised with IDN.toASCII and validated before it\nis used, so the unicode and punycode forms of a name are interchangeable\nand a malformed value such as \"com..\" is rejected rather than matching\nevery host under \"com\". The public suffix list shipped with\ncrawler-commons, its private section included, decides whether a domain\nmay cover subdomains at all; a domain the list does not know about, e.g.\nan internal name under \".local\", falls back to the plain suffix match so\nthat unlisted domains keep working. A domain which can not own subdomains\nis ignored and the cookie is bound to the host, as RFC 6265 5.3 requires,\nso a single label intranet domain keeps its cookies while \"com\" no longer\nscopes anything. An address is only matched by itself, per RFC 6265 5.1.3,\ninstead of covering the hosts under it as \"2.3.4\" covered \"9.2.3.4\".\n\nThe protocols now record which host set the cookies. When a response\ncarries Set-Cookie, okhttp and playwright write its url next to the header\nas set-cookie-origin, dropping any key of that name from the response\nfirst so that a response can not choose where its cookies are sent. The\nkey has to travel with protocol.set-cookie for host-only cookies to be\nsent back, so it is added to the metadata.persist and metadata.transfer\nexamples, and a warning naming both keys is logged once when cookies are\npresent without it. The metadata.transfer example in internals.adoc named\nset-cookie instead of protocol.set-cookie and is corrected too.\n\nA Set-Cookie header whose first token has no \"\u003d\" threw\nStringIndexOutOfBoundsException out of getCookies, failing the whole\nfetch, and is now skipped. Cookies dropped on the path, secure and expiry\nchecks are logged like the other ones." }, { "commit": "2223fe8f586e5a388a9c139107d5e4f30cb9fc72", "tree": "9774d1b8f5c607d85c9520549404551e341e871a", "parents": [ "e73aaebab261e429ae530923a3f3dc510a08f1d0" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Fri Aug 28 12:52:13 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 11:52:13 2026 +0100" }, "message": "Honour ignoreOutsideDomain when set on its own in HostURLFilter (#2074)\n\nHostURLFilter.configure() read the ignoreOutsideDomain parameter but tested\nthe ignoreOutsideHost node for null. A configuration naming only\nignoreOutsideDomain was therefore discarded and the filter let every URL\nthrough, while one naming only ignoreOutsideHost: false failed with a\nNullPointerException while the topology was starting. ignoreOutsideHost: true\nwas never affected, as it short-circuits before the domain parameter is read.\nNow each parameter is tested independently. Topologies which set only\nignoreOutsideDomain: true start restricting the crawl to the source domain,\nas configured." }, { "commit": "e73aaebab261e429ae530923a3f3dc510a08f1d0", "tree": "f44a9fed42423f31598fe67d5c2a677876d9ccc7", "parents": [ "07d43927c8052b5bebe5d020f203ececc6c3c4d0" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Fri Aug 28 09:57:46 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 28 09:57:46 2026 +0200" }, "message": "build(deps): bump actions/setup-java from 5.7.0 to 6.0.0 (#2078)\n\n* build(deps): bump actions/setup-java from 5.7.0 to 6.0.0\n\nBumps [actions/setup-java](https://github.com/actions/setup-java) from 5.7.0 to 6.0.0.\n- [Release notes](https://github.com/actions/setup-java/releases)\n- [Commits](https://github.com/actions/setup-java/compare/b6effb05e454b25005698d916606bdc6ffcbf961...dd06d9cba3e5552c54d9f8ea23572deb30010f7c)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-java\n dependency-version: 6.0.0\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\n\n* Change JDK distribution from adopt to temurin\n\n* Change JDK distribution from adopt to temurin\n\n---------\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e\nCo-authored-by: Richard Zowalla \u003c13417392+rzo1@users.noreply.github.com\u003e" }, { "commit": "07d43927c8052b5bebe5d020f203ececc6c3c4d0", "tree": "f24c4c2cab1a6b6ac8dd1652c33c7f177f2fab4b", "parents": [ "191e5babcaeb9b448607dd10f024cfa7cdae713c" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Aug 25 13:14:09 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 13:14:09 2026 +0100" }, "message": "Minor: Regenerated License File for 191e5babcaeb9b448607dd10f024cfa7cdae713c (#2072)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: jnioche \u003cjnioche@users.noreply.github.com\u003e" }, { "commit": "191e5babcaeb9b448607dd10f024cfa7cdae713c", "tree": "bfddc22d3a8a79649af7f7fb41c7eb8ab7d6239e", "parents": [ "4d49466df34dbd094d09021d1bc3bdc4293f01d0" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Tue Aug 25 13:08:59 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 13:08:59 2026 +0100" }, "message": "build(deps): bump langchain4j.version from 1.18.1 to 1.19.0 (#2064)\n\nBumps `langchain4j.version` from 1.18.1 to 1.19.0.\n\nUpdates `dev.langchain4j:langchain4j` from 1.18.1 to 1.19.0\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.18.1...1.19.0)\n\nUpdates `dev.langchain4j:langchain4j-open-ai` from 1.18.1 to 1.19.0\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.18.1...1.19.0)\n\n---\nupdated-dependencies:\n- dependency-name: dev.langchain4j:langchain4j\n dependency-version: 1.19.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n- dependency-name: dev.langchain4j:langchain4j-open-ai\n dependency-version: 1.19.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "4d49466df34dbd094d09021d1bc3bdc4293f01d0", "tree": "6b3f22cc834f80c518146a1be903702cf42427a6", "parents": [ "13b1679313c0cd796d62f2f58e8e253590093cad" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Tue Aug 25 13:08:23 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 13:08:23 2026 +0100" }, "message": "Removed apache http protocol implementation. Closes #1995 (#2071)\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "13b1679313c0cd796d62f2f58e8e253590093cad", "tree": "acfe8efc3ebf5d51c0433fd87c4f35b9281b203c", "parents": [ "e46d752dcdb33ab756ccf678c2aaad9781c4577a" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Aug 25 07:52:59 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Aug 25 07:52:59 2026 +0100" }, "message": "Minor: Regenerated License File for e46d752dcdb33ab756ccf678c2aaad9781c4577a (#2070)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: jnioche \u003cjnioche@users.noreply.github.com\u003e" }, { "commit": "e46d752dcdb33ab756ccf678c2aaad9781c4577a", "tree": "f18f5ebd43ca1052d88fe66d327bd12dea7581f0", "parents": [ "6ccccba6c3752c6ccf4c81e4328c26654c30a927" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 24 07:33:36 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 24 07:33:36 2026 +0100" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.53.0 to 2.54.1 (#2068)\n\nBumps software.amazon.awssdk:bom from 2.53.0 to 2.54.1.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.54.1\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "6ccccba6c3752c6ccf4c81e4328c26654c30a927", "tree": "c31ccf40d107dae7c36060af8b1284020ec97052", "parents": [ "e62aac7bc22461d39b51d8e6ae855e02e52fd22f" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 24 07:32:38 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 24 07:32:38 2026 +0100" }, "message": "build(deps): bump com.puppycrawl.tools:checkstyle from 12.3.1 to 14.0.0 (#2069)\n\nBumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 12.3.1 to 14.0.0.\n- [Release notes](https://github.com/checkstyle/checkstyle/releases)\n- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-12.3.1...checkstyle-14.0.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.puppycrawl.tools:checkstyle\n dependency-version: 14.0.0\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "e62aac7bc22461d39b51d8e6ae855e02e52fd22f", "tree": "04861120568b15808138772ff765a2f0de21ce12", "parents": [ "8f7c17d9c6973cab0ea963c9c5b98429846d648a" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 24 07:32:00 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 24 07:32:00 2026 +0100" }, "message": "build(deps): bump com.google.guava:guava from 33.6.0-jre to 33.7.1-jre (#2067)\n\nBumps [com.google.guava:guava](https://github.com/google/guava) from 33.6.0-jre to 33.7.1-jre.\n- [Release notes](https://github.com/google/guava/releases)\n- [Commits](https://github.com/google/guava/commits)\n\n---\nupdated-dependencies:\n- dependency-name: com.google.guava:guava\n dependency-version: 33.7.1-jre\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "8f7c17d9c6973cab0ea963c9c5b98429846d648a", "tree": "a3c27fec2eb217cc0e1c76711be022a84a003a77", "parents": [ "358d761e502fbf27a2e22efd8cc9d3a081cd54cd" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 24 07:31:26 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 24 07:31:26 2026 +0100" }, "message": "build(deps): bump okhttp.version from 5.4.0 to 5.5.0 (#2065)\n\nBumps `okhttp.version` from 5.4.0 to 5.5.0.\n\nUpdates `com.squareup.okhttp3:okhttp` from 5.4.0 to 5.5.0\n- [Changelog](https://github.com/lysine-dev/okhttp/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/lysine-dev/okhttp/compare/parent-5.4.0...parent-5.5.0)\n\nUpdates `com.squareup.okhttp3:okhttp-brotli` from 5.4.0 to 5.5.0\n- [Changelog](https://github.com/lysine-dev/okhttp/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/lysine-dev/okhttp/compare/parent-5.4.0...parent-5.5.0)\n\nUpdates `com.squareup.okhttp3:okhttp-zstd` from 5.4.0 to 5.5.0\n- [Changelog](https://github.com/lysine-dev/okhttp/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/lysine-dev/okhttp/compare/parent-5.4.0...parent-5.5.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.squareup.okhttp3:okhttp\n dependency-version: 5.5.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n- dependency-name: com.squareup.okhttp3:okhttp-brotli\n dependency-version: 5.5.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n- dependency-name: com.squareup.okhttp3:okhttp-zstd\n dependency-version: 5.5.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "358d761e502fbf27a2e22efd8cc9d3a081cd54cd", "tree": "0a31a094719947c276b10d4583655f7865dcf6c1", "parents": [ "b7b0f842d578c220f6f51f3e3e0e87ccd63cd711" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Aug 17 20:22:44 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 20:22:44 2026 +0200" }, "message": "Minor: Regenerated License File for 00bca0d0a904d3a3bb7a6d84c31fc3ffc9c9e35b (#2060)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "b7b0f842d578c220f6f51f3e3e0e87ccd63cd711", "tree": "aaba494cb9b12cbfb326eb8c8d927185652a86e5", "parents": [ "00bca0d0a904d3a3bb7a6d84c31fc3ffc9c9e35b" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 17 20:22:22 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 20:22:22 2026 +0200" }, "message": "build(deps): bump git-code-format-maven-plugin.version from 5.4 to 6.0 (#1825)\n\nBumps `git-code-format-maven-plugin.version` from 5.4 to 6.0.\n\nUpdates `com.cosium.code:google-java-format` from 5.4 to 6.0\n- [Release notes](https://github.com/Cosium/git-code-format-maven-plugin/releases)\n- [Commits](https://github.com/Cosium/git-code-format-maven-plugin/compare/5.4...6.0)\n\nUpdates `com.cosium.code:git-code-format-maven-plugin` from 5.4 to 6.0\n- [Release notes](https://github.com/Cosium/git-code-format-maven-plugin/releases)\n- [Commits](https://github.com/Cosium/git-code-format-maven-plugin/compare/5.4...6.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.cosium.code:git-code-format-maven-plugin\n dependency-version: \u00276.0\u0027\n dependency-type: direct:development\n update-type: version-update:semver-major\n- dependency-name: com.cosium.code:google-java-format\n dependency-version: \u00276.0\u0027\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "00bca0d0a904d3a3bb7a6d84c31fc3ffc9c9e35b", "tree": "28d9688a2f40cc822bae3a146f6930b5a81d7c3e", "parents": [ "ca7196295d21c50e182c0c63fefa2bb368a4aeb6" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 17 20:20:54 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 20:20:54 2026 +0200" }, "message": "build(deps): bump junit.version from 6.1.2 to 6.1.3 (#2054)\n\nBumps `junit.version` from 6.1.2 to 6.1.3.\n\nUpdates `org.junit.jupiter:junit-jupiter-api` from 6.1.2 to 6.1.3\n- [Release notes](https://github.com/junit-team/junit-framework/releases)\n- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.2...r6.1.3)\n\nUpdates `org.junit.jupiter:junit-jupiter` from 6.1.2 to 6.1.3\n- [Release notes](https://github.com/junit-team/junit-framework/releases)\n- [Commits](https://github.com/junit-team/junit-framework/compare/r6.1.2...r6.1.3)\n\n---\nupdated-dependencies:\n- dependency-name: org.junit.jupiter:junit-jupiter-api\n dependency-version: 6.1.3\n dependency-type: direct:production\n update-type: version-update:semver-patch\n- dependency-name: org.junit.jupiter:junit-jupiter\n dependency-version: 6.1.3\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "ca7196295d21c50e182c0c63fefa2bb368a4aeb6", "tree": "0a046efdcd8d3157d165663efb6d2ff1599f68f4", "parents": [ "735160efe864ddf9b47bf21991c53f585a04adb1" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 17 20:20:36 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 20:20:36 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.51.2 to 2.53.0 (#2055)\n\nBumps software.amazon.awssdk:bom from 2.51.2 to 2.53.0.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.53.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "735160efe864ddf9b47bf21991c53f585a04adb1", "tree": "cf8286d6658bb3cee0e696e0e74a87ab63f2d58c", "parents": [ "42b88e0031c11d8d9a05eff63a2fe7e488ef0ced" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Mon Aug 17 19:20:06 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 20:20:06 2026 +0200" }, "message": "Remove Selenium module, fixes #1994 (#2059)\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "42b88e0031c11d8d9a05eff63a2fe7e488ef0ced", "tree": "709b0bd701819e05f68174cbfaf4b64750b73533", "parents": [ "a0838c86861958adffd8c6be15e82ea8975a4926" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Mon Aug 17 11:33:55 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Aug 17 12:33:55 2026 +0200" }, "message": "bugfix set Java 25 for snapshots workflow (#2056)" }, { "commit": "a0838c86861958adffd8c6be15e82ea8975a4926", "tree": "9c592d1a824d93d372d76fc6cd6507ef4ee2c76e", "parents": [ "e6c4f7b2f85e6e970367cbecc3e5fd496aed7765" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Wed Aug 12 17:12:54 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 12 17:12:54 2026 +0200" }, "message": "Regenerated License file after dependency upgrades (#2051)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e" }, { "commit": "e6c4f7b2f85e6e970367cbecc3e5fd496aed7765", "tree": "aa344c0ccaaf47000a51d257cf090470d29fa570", "parents": [ "bbdebac96142f91a7cbaed07610bcd1f2131cfcc" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Wed Aug 12 16:43:41 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 12 16:43:41 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.50.0 to 2.51.2 (#2049)\n\nBumps software.amazon.awssdk:bom from 2.50.0 to 2.51.2.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.51.2\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "bbdebac96142f91a7cbaed07610bcd1f2131cfcc", "tree": "cf20a7450f25a7775a1cad8b722493695c17fb18", "parents": [ "e1d54a6572438a5518d772cb88ac51e8dada57fb" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Wed Aug 12 16:43:21 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Wed Aug 12 16:43:21 2026 +0200" }, "message": "build(deps): bump com.microsoft.playwright:playwright from 1.61.0 to 1.62.0 (#2048)\n\nBumps [com.microsoft.playwright:playwright](https://github.com/microsoft/playwright-java) from 1.61.0 to 1.62.0.\n- [Release notes](https://github.com/microsoft/playwright-java/releases)\n- [Commits](https://github.com/microsoft/playwright-java/compare/v1.61.0...v1.62.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.microsoft.playwright:playwright\n dependency-version: 1.62.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "e1d54a6572438a5518d772cb88ac51e8dada57fb", "tree": "889956d4b19340f1f09e68c30e511036e0be27e5", "parents": [ "e2fccebff0130e7785bd91ac429a7f691661a801" ], "author": { "name": "Davide Polato", "email": "dpol1@apache.org", "time": "Fri Aug 07 17:19:27 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Aug 07 17:19:27 2026 +0200" }, "message": "refresh NOTICE copyright year to 2026 (#2047)" }, { "commit": "e2fccebff0130e7785bd91ac429a7f691661a801", "tree": "727df4a006beff8911b0b4615982a721e040a64d", "parents": [ "54fcf5b58d6e070b2d0fe89df130b66eafbf0449" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Thu Aug 06 12:31:44 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 06 12:31:44 2026 +0200" }, "message": "Minor: Regenerated License File for 54fcf5b58d6e070b2d0fe89df130b66eafbf0449 (#2046)" }, { "commit": "54fcf5b58d6e070b2d0fe89df130b66eafbf0449", "tree": "bc0d9e0a3ad047edd32eb5182c0ef02c4400243a", "parents": [ "00b5be02764ade4645d60a552c6cc5dcc1b7899e" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 12:24:50 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 06 12:24:50 2026 +0200" }, "message": "Upgrade to Storm 3 (#2045)" }, { "commit": "00b5be02764ade4645d60a552c6cc5dcc1b7899e", "tree": "22ed1a0541e93457f012e7afe9afed5ef2e1e7c0", "parents": [ "738c3024793d609bbb0a5b742c6ec5f50566038b" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Thu Aug 06 12:00:24 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 06 12:00:24 2026 +0200" }, "message": "Minor: Regenerated License File for 738c3024793d609bbb0a5b742c6ec5f50566038b (#2044)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "738c3024793d609bbb0a5b742c6ec5f50566038b", "tree": "7bc40da23945c71a39bd7b8fe32b6a36f17c8d6f", "parents": [ "e4d30f71bd496bc86f5c1cdd7030967c57956a9c" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:57:24 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Aug 06 11:57:24 2026 +0200" }, "message": "Upgrade to Apache Storm 3.0.0 and Java 25 (#2017)\n\n* Upgrade to Apache Storm 3.0.0 and Java 25\n\n- bump storm-client.version to 3.0.0 and raise the compiler/javadoc level to Java 25 (Storm 3.0.0 requires Java 25)\n- bump storm.version and compiler level in the archetypes, update CI workflows to JDK 25\n- WARC module: drop the obsolete jdk.tools/hive/hbase-server exclusions on storm-hdfs; the 3.0.0 artifact only pulls in the shaded hadoop-client-api/runtime. Exclude hbase-client, which is only needed by the AutoHBase credentials plugin\n- WARC module: replace the removed org.apache.storm.utils.Utils.gzip with a local GZIP helper in GzipHdfsBolt\n- update READMEs, quick-start docs and the regenerated THIRD-PARTY.txt\n\nCloses #2015\n\n* Move to StormCrawler 4\n\n* Fix SOLR readme" }, { "commit": "e4d30f71bd496bc86f5c1cdd7030967c57956a9c", "tree": "ffe3591a1971355a955dc131a89db6a33f403a32", "parents": [ "9b72b5a58dbda17d6f2ff2880f3bdbbc7be72a7c" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 04:03:53 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:55:52 2026 +0200" }, "message": "build(deps): bump org.jsoup:jsoup from 1.22.2 to 1.23.1\n\nBumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.22.2 to 1.23.1.\n- [Release notes](https://github.com/jhy/jsoup/releases)\n- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)\n- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.22.2...jsoup-1.23.1)\n\n---\nupdated-dependencies:\n- dependency-name: org.jsoup:jsoup\n dependency-version: 1.23.1\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "9b72b5a58dbda17d6f2ff2880f3bdbbc7be72a7c", "tree": "7cc2913d07e8be0600eca688b22b151c470580cc", "parents": [ "de6977df3342603f06f2b69164aea1cf5ba1254a" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 04:03:20 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:55:34 2026 +0200" }, "message": "build(deps): bump com.mysql:mysql-connector-j from 9.7.0 to 26.7.0\n\nBumps [com.mysql:mysql-connector-j](https://github.com/mysql/mysql-connector-j) from 9.7.0 to 26.7.0.\n- [Changelog](https://github.com/mysql/mysql-connector-j/blob/trunk/CHANGES)\n- [Commits](https://github.com/mysql/mysql-connector-j/compare/9.7.0...26.7.0)\n\n---\nupdated-dependencies:\n- dependency-name: com.mysql:mysql-connector-j\n dependency-version: 26.7.0\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "de6977df3342603f06f2b69164aea1cf5ba1254a", "tree": "e6e82fc4f1d9d3ebb7047b1096ea237170768e23", "parents": [ "caec3bcf28173252b0fe59d17afb284a60ef9af5" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 04:04:00 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:55:18 2026 +0200" }, "message": "build(deps): bump langchain4j.version from 1.18.0 to 1.18.1\n\nBumps `langchain4j.version` from 1.18.0 to 1.18.1.\n\nUpdates `dev.langchain4j:langchain4j` from 1.18.0 to 1.18.1\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.18.0...1.18.1)\n\nUpdates `dev.langchain4j:langchain4j-open-ai` from 1.18.0 to 1.18.1\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.18.0...1.18.1)\n\n---\nupdated-dependencies:\n- dependency-name: dev.langchain4j:langchain4j\n dependency-version: 1.18.1\n dependency-type: direct:production\n update-type: version-update:semver-patch\n- dependency-name: dev.langchain4j:langchain4j-open-ai\n dependency-version: 1.18.1\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "caec3bcf28173252b0fe59d17afb284a60ef9af5", "tree": "c1f9fa51fdc02339cbf23c6e315f6921804ef6cf", "parents": [ "306474dab26714299ee0dfd46a7c37778860fbd5" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 04:04:09 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:55:04 2026 +0200" }, "message": "build(deps): bump commons-codec:commons-codec from 1.22.0 to 1.22.1\n\nBumps [commons-codec:commons-codec](https://github.com/apache/commons-codec) from 1.22.0 to 1.22.1.\n- [Changelog](https://github.com/apache/commons-codec/blob/master/RELEASE-NOTES.txt)\n- [Commits](https://github.com/apache/commons-codec/compare/rel/commons-codec-1.22.0...rel/commons-codec-1.22.1)\n\n---\nupdated-dependencies:\n- dependency-name: commons-codec:commons-codec\n dependency-version: 1.22.1\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "306474dab26714299ee0dfd46a7c37778860fbd5", "tree": "e60629c616c8b4a9f53f63b2ee342edfe39c9b28", "parents": [ "2dddb1da6c16e8f8a178ced63b010991eeef1fe4" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 14:48:59 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:54:48 2026 +0200" }, "message": "build(deps): bump actions/setup-java from 5.6.0 to 5.7.0\n\nBumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 5.7.0.\n- [Release notes](https://github.com/actions/setup-java/releases)\n- [Commits](https://github.com/actions/setup-java/compare/03ad4de0992f5dab5e18fcb136590ce7c4a0ac95...b6effb05e454b25005698d916606bdc6ffcbf961)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-java\n dependency-version: 5.7.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "2dddb1da6c16e8f8a178ced63b010991eeef1fe4", "tree": "cb27b710a17571a3c77d940e415563931b27684f", "parents": [ "8f94a75fbf356e3b6563ac8d21f21d475324e8b4" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Aug 03 04:03:46 2026 +0000" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Thu Aug 06 11:54:32 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.49.3 to 2.50.0\n\nBumps software.amazon.awssdk:bom from 2.49.3 to 2.50.0.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.50.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e" }, { "commit": "8f94a75fbf356e3b6563ac8d21f21d475324e8b4", "tree": "62b58fdc3569766b569b180f02dae84b86a1b653", "parents": [ "747f5d0a8663335359edcfb446c87bb00ee5f1c7" ], "author": { "name": "Sebastian Nagel", "email": "snagel@apache.org", "time": "Fri Jul 31 15:43:33 2026 +0200" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 01 18:27:42 2026 +0200" }, "message": "Fix broken anchor in Asciidoc\n" }, { "commit": "747f5d0a8663335359edcfb446c87bb00ee5f1c7", "tree": "4860d12fd81cfe25cdef51f50018028f8abd0a1b", "parents": [ "214b23e16717c32add54c035762ce4ec176697fe" ], "author": { "name": "Sebastian Nagel", "email": "sebastian@commoncrawl.org", "time": "Wed Nov 26 11:43:45 2025 +0100" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 01 18:27:31 2026 +0200" }, "message": "Fix broken link to Guava\u0027s CacheBuilderSpec\n" }, { "commit": "214b23e16717c32add54c035762ce4ec176697fe", "tree": "38eb595c1952b864b553d8bd62a95faa2175ccb5", "parents": [ "883b8546b3ec773218a550fdebec24391612f8ff" ], "author": { "name": "Sebastian Nagel", "email": "sebastian@commoncrawl.org", "time": "Thu May 07 12:12:01 2026 +0200" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 01 18:27:31 2026 +0200" }, "message": "Fix typo in OpenSearch configuration comment\n" }, { "commit": "883b8546b3ec773218a550fdebec24391612f8ff", "tree": "c843bbcfb4ccbf0b7a90e669352e632793e844cf", "parents": [ "0dc2ab5e2b717fe4730ba910f96f8d57a178947c" ], "author": { "name": "Davide Polato", "email": "davide@polato.dev", "time": "Wed Jul 29 14:08:59 2026 +0200" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 01 18:26:31 2026 +0200" }, "message": "[maven-release-plugin] prepare for next development iteration\n" }, { "commit": "0dc2ab5e2b717fe4730ba910f96f8d57a178947c", "tree": "29439852d8e20d2e9065de04b15863157d082faa", "parents": [ "e3297e4892353e468ab1d026b4605d00b0c13c57" ], "author": { "name": "Davide Polato", "email": "davide@polato.dev", "time": "Wed Jul 29 14:08:44 2026 +0200" }, "committer": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Sat Aug 01 18:26:31 2026 +0200" }, "message": "[maven-release-plugin] prepare release stormcrawler-3.7.0\n" }, { "commit": "e3297e4892353e468ab1d026b4605d00b0c13c57", "tree": "1e382a63252e5bbbc008147e4b172a39d2eb318d", "parents": [ "6e690830d6e062435881fa6eaacfa8e08d17f231" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Jul 28 14:21:28 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 14:21:28 2026 +0200" }, "message": "Regenerated License file after dependency upgrades (#2033)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "6e690830d6e062435881fa6eaacfa8e08d17f231", "tree": "76ce5e0c4698bd557841a1ecb2aab6df5fb9d4e1", "parents": [ "9304adac117ae16e94fa797b629944cedeabb982" ], "author": { "name": "Davide Polato", "email": "dpol1@apache.org", "time": "Tue Jul 28 13:37:42 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 13:37:42 2026 +0200" }, "message": "Upgrade to Storm 2.8.9 (#2032)" }, { "commit": "9304adac117ae16e94fa797b629944cedeabb982", "tree": "812c02ff138a94bc9bc96eb3eac94ed989bd4e2c", "parents": [ "dc1f70bb88c29050497e9962dc1607816e73683f" ], "author": { "name": "Davide Polato", "email": "dpol1@apache.org", "time": "Tue Jul 28 09:19:03 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 09:19:03 2026 +0200" }, "message": "Enforce robots.txt Crawl-delay in URLFrontier queues (#2000)\n\n* feat: report the forced robots crawl-delay in the status metadata\n\nWhen a robots.txt Crawl-delay exceeds fetcher.max.crawl.delay and the\nforce flag makes the fetcher keep going anyway, FetcherBolt now writes\nthe requested delay in seconds to the new robots.crawl.delay metadata\nkey. A frontier-side consumer can then enforce the pace at the source\ninstead of losing it, which is the gap #1979 describes for #867.\n\nThe unforced path is untouched and keeps emitting Status.ERROR with\nerror.cause\u003dcrawl_delay, pinned by a regression test.\n\n* feat: report the forced robots crawl-delay from SimpleFetcherBolt too\n\nSame signal as in FetcherBolt: when maxCrawlDelayForce caps a long\nCrawl-delay, the original robots value lands in robots.crawl.delay so\nthe frontier side of #1979 sees both fetchers alike. The shared test\nharness moves up to AbstractFetcherBoltTest rather than being copied.\n\n* feat: add CrawlDelayPolicy for forwarding robots delays to the frontier\n\nPackage-private decision logic for the queue-stream consumer, same\nsplit as HostBackoff: reads robots.crawl.delay from the tuple metadata,\ncaps it with urlfrontier.backoff.max.secs since sites do request absurd\nvalues (#1979), and deduplicates per host. An unchanged value is\nre-sent at most once per urlfrontier.backoff.decay.secs window, which\nalso lets a lost fire-and-forget RPC converge. The shared _DEFAULT_\nqueue is skipped, as one host must not slow down the catch-all bucket.\n\n* feat: pace host queues on robots crawl-delay via setDelay\n\nQueueRegulatorBolt now lives up to its name: besides blocking queues on\nrate limits, it forwards the robots.crawl.delay carried by the queue\nstream to URLFrontier through setDelay, closing the loop of #1979. The\ncall is fire-and-forget with the same short deadline as the blocks, and\nsetDelay(key, 0) is deliberately never sent because it would erase a\nserver-side default for the queue: a site that drops its Crawl-delay\nkeeps the old pace, which costs throughput on that host but never\npoliteness.\n\nThe startup metadata.persist probe now checks robots.crawl.delay too,\nsince the signal dies silently if the filter strips it.\n\n* test: prove the frontier actually paces a queue on robots crawl-delay\n\nTestcontainers regression against a real frontier. The delay does not\nshrink a batch, it gates the queue between hand-outs (lastProduced +\ndelay, first hand-out always served): so the test takes one URL out,\nthen asserts a later getURLs comes back empty although a second URL is\nneither served nor in-flight. The pause between the two calls also pins\nthe unit of setDelayRequestable as seconds, a millisecond window would\nhave expired. Seeding is generalised to seedUrls(paths...) with the old\nseedOneUrl() delegating, existing tests untouched.\n\n* fix: preserve robots crawl-delay metadata semantics\n\nRound fractional delays up so frontier pacing never becomes more aggressive than robots.txt. Rebuild the control signal from the locally parsed rules after protocol metadata is merged, preventing stale metadata or a colliding response header from changing it.\n\n* feat: enforce safe robots crawl-delay pacing\n\nKeep frontier pacing opt-in and fail fast unless queues are partitioned by host, hand out one URL at a time, and carry robots.crawl.delay as persist-only metadata. Serialize setDelay calls per queue so asynchronous completions cannot violate the last-value-wins contract.\n\n* docs: document URLFrontier robots crawl-delay pacing\n\nDescribe the opt-in wiring and the host partitioning, single-URL hand-out, persist-only metadata, and delay-cap requirements for #1979.\n\n* fix: robots pacing coalesces to the max observed delay, single endpoint only\n\nThe delay signal is now conservative: the maximum observed value wins\nand a lower delay applies only after the previous maximum expires from\nthe decay window. The cap and dedupe window move to dedicated keys\n(urlfrontier.robots.delay.max.secs / .decay.secs) instead of reusing the\nback-off ones. A failed RPC no longer self-retries without a higher\npending value: its value is kept as an ambiguous floor for the decay\nwindow - the server may have applied it before the response was lost -\nso the next signal is not deduplicated and reasserts at least that\nvalue instead of lowering a possibly applied delay.\n\nKeyed setDelay/blockQueueUntil are not propagated across URLFrontier\nnodes (crawler-commons/url-frontier#146): multiple configured addresses\nnow log a warning bolt-wide and fail fast only when robots pacing is\nenabled; the single host/port fallback stays valid. Javadoc, README and\nthe asciidoc reference document the dedicated keys, the\nconnected-node-only semantics and the non-atomic getURLs politeness\ngate under concurrent clients. QueueRegulatorBoltTest folds the two\nblind waits into one." }, { "commit": "dc1f70bb88c29050497e9962dc1607816e73683f", "tree": "c430df61e14ba20dd650328661fba1688d42128d", "parents": [ "d193e6cdf0fd9562de304895b0391a5630ce9bb8" ], "author": { "name": "Luca", "email": "15426+lfoppiano@users.noreply.github.com", "time": "Tue Jul 28 08:07:21 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 28 08:07:21 2026 +0200" }, "message": "fix: emit anchored tuples #2021 (#2031)\n\nSigned-off-by: Luca Foppiano \u003cluca@foppiano.org\u003e" }, { "commit": "d193e6cdf0fd9562de304895b0391a5630ce9bb8", "tree": "0aa1d40033b53e88ea91a05b51633e90a33c6610", "parents": [ "725781f12547b23ba5b1f7faf8e30385833a5330" ], "author": { "name": "Richard Zowalla", "email": "13417392+rzo1@users.noreply.github.com", "time": "Mon Jul 27 17:55:18 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 16:55:18 2026 +0100" }, "message": "#2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set (#2027)\n\n* #2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set\n\n* #2026 - Always match http.agent.name against robots.txt, even when http.robots.agents is set" }, { "commit": "725781f12547b23ba5b1f7faf8e30385833a5330", "tree": "3ef25eb44470a1d40a2a6a93e1734511a4ad72f1", "parents": [ "b13ed812cc219d200afcbab44dd87fd298f54b58" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Jul 27 14:14:00 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 14:14:00 2026 +0200" }, "message": "Minor: Regenerated License File for cd5f59c51bdae43202c2c0f408a98fe01b99ae83 (#2030)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "b13ed812cc219d200afcbab44dd87fd298f54b58", "tree": "3ef25eb44470a1d40a2a6a93e1734511a4ad72f1", "parents": [ "cd5f59c51bdae43202c2c0f408a98fe01b99ae83" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Jul 27 14:00:13 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 14:00:13 2026 +0200" }, "message": "Minor: Regenerated License File for f1e35b21261fc43b6990bae7f633c071b7947d57 (#2029)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "cd5f59c51bdae43202c2c0f408a98fe01b99ae83", "tree": "6a2b99d0f1611ead6cd867ddb943f008852c22db", "parents": [ "f1e35b21261fc43b6990bae7f633c071b7947d57" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Mon Jul 27 13:56:51 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 13:56:51 2026 +0200" }, "message": "Minor: Regenerated License File for ee983434cdd00cfbf44d0c6b3d79f124942bf682 (#2028)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" }, { "commit": "f1e35b21261fc43b6990bae7f633c071b7947d57", "tree": "99740735ce542f745e427662e1c706b05d4ba220", "parents": [ "ee983434cdd00cfbf44d0c6b3d79f124942bf682" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Jul 27 13:34:48 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 13:34:48 2026 +0200" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.48.3 to 2.49.3 (#2022)\n\nBumps software.amazon.awssdk:bom from 2.48.3 to 2.49.3.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.49.3\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "ee983434cdd00cfbf44d0c6b3d79f124942bf682", "tree": "04c5d2da7f4baeec9ed7314f7d18f8f5b5d324b2", "parents": [ "8667217f941ae1585c51e4ebc37cb04135697186" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Jul 27 13:32:51 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 13:32:51 2026 +0200" }, "message": "build(deps): bump tika.version from 3.3.1 to 3.3.2 (#2023)\n\nBumps `tika.version` from 3.3.1 to 3.3.2.\n\nUpdates `org.apache.tika:tika-core` from 3.3.1 to 3.3.2\n- [Changelog](https://github.com/apache/tika/blob/3.3.2/CHANGES.txt)\n- [Commits](https://github.com/apache/tika/compare/3.3.1...3.3.2)\n\nUpdates `org.apache.tika:tika-parsers-standard-package` from 3.3.1 to 3.3.2\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.tika:tika-core\n dependency-version: 3.3.2\n dependency-type: direct:production\n update-type: version-update:semver-patch\n- dependency-name: org.apache.tika:tika-parsers-standard-package\n dependency-version: 3.3.2\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "8667217f941ae1585c51e4ebc37cb04135697186", "tree": "53de120cba69146674eec1f16b97ec22a7a7b547", "parents": [ "3c06a12607b1e15a9c2b0bde4ce3b493d3a42b19" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Mon Jul 27 13:31:46 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Mon Jul 27 13:31:46 2026 +0200" }, "message": "build(deps): bump org.apache.maven.plugins:maven-jar-plugin (#2025)\n\nBumps [org.apache.maven.plugins:maven-jar-plugin](https://github.com/apache/maven-jar-plugin) from 3.5.0 to 3.5.1.\n- [Release notes](https://github.com/apache/maven-jar-plugin/releases)\n- [Commits](https://github.com/apache/maven-jar-plugin/compare/maven-jar-plugin-3.5.0...maven-jar-plugin-3.5.1)\n\n---\nupdated-dependencies:\n- dependency-name: org.apache.maven.plugins:maven-jar-plugin\n dependency-version: 3.5.1\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "3c06a12607b1e15a9c2b0bde4ce3b493d3a42b19", "tree": "e9bb4c84b02f740cf4b04d12d8599c05fe851ca3", "parents": [ "54c4a6d3c52b2782428009ebff8ab956322039cc" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Thu Jul 23 18:46:48 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 19:46:48 2026 +0200" }, "message": "Mark HTTPClient as deprecated (#2020)\n\n* Mark HTTPClient as deprecated\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e\n\n* Formatting!\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e\n\n---------\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "54c4a6d3c52b2782428009ebff8ab956322039cc", "tree": "99c0b0f3d2d748bb881d4dd310fa709b36af2bad", "parents": [ "5eb99d0a366473606b319024cb1e0ee0a66ca6db" ], "author": { "name": "Dávid Szigecsán", "email": "sigee@apache.org", "time": "Thu Jul 23 17:46:56 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 16:46:56 2026 +0100" }, "message": "Allow indexer bolts to use a metadata value as the document ID (#2012)\n\n* Allow indexer bolts to use a metadata value as the document ID\n\n* Use the SHA-256 digest of the metadata value\n\n* Fix missing \"SHA-256 digested\" from the description\n\n* Add Note to the documentation\n\n* Extend Javadoc with the caveats\n\n* Add default empty string to the crawler-default.yaml" }, { "commit": "5eb99d0a366473606b319024cb1e0ee0a66ca6db", "tree": "9162fc409e6366fff60f5e5af71c0e3e3e08ddef", "parents": [ "e87846aca880a82320eed31dbb462e84154c2245" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Thu Jul 23 12:50:38 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 12:50:38 2026 +0100" }, "message": "Mark CloudSearch as deprecated. Fixes #2016 (#2018)\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "e87846aca880a82320eed31dbb462e84154c2245", "tree": "3a05255350a66262b79ee4f965690ef138914382", "parents": [ "cc511f9ff73a0799c135cc524fc904e68cfee520" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Thu Jul 23 12:50:15 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 23 12:50:15 2026 +0100" }, "message": "Mark Selenium module as deprecated (#2019)\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "cc511f9ff73a0799c135cc524fc904e68cfee520", "tree": "6da0bd8172ec740408718e70a61cdf19bdb76a9a", "parents": [ "548726f48b63b51156913ef483f5d75e375733a3" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Tue Jul 21 17:51:02 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 17:51:02 2026 +0100" }, "message": "Resolve relative outlinks against \u003cbase href\u003e in JSoupParserBolt (#2011)\n\nRelative links were resolved against the document URL, ignoring any\n\u003cbase href\u003e element. On a page served from a sub-path with a base\npointing at the domain root this doubled the path segment. Resolve\nagainst jsoupDoc.baseUri() instead, which honours \u003cbase\u003e and falls\nback to the document URL when absent." }, { "commit": "548726f48b63b51156913ef483f5d75e375733a3", "tree": "e4ee3753ce27eaddf42a12ef90a3533c1e27b454", "parents": [ "17c14a866b3a10813a0bfc39bee04365a77ea791" ], "author": { "name": "AsifQureshi04", "email": "90105636+AsifQureshi04@users.noreply.github.com", "time": "Tue Jul 21 19:28:49 2026 +0530" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:58:49 2026 +0100" }, "message": "Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest (#2005)" }, { "commit": "17c14a866b3a10813a0bfc39bee04365a77ea791", "tree": "62b30f4ae838eb1815f48fa3ec3a7ac29c176643", "parents": [ "83f90633efc547bf09362d52443e81fb83b52af6" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Tue Jul 21 14:46:10 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:46:10 2026 +0100" }, "message": "build(deps): bump actions/checkout from 7.0.0 to 7.0.1 (#2010)\n\nBumps [actions/checkout](https://github.com/actions/checkout) from 7.0.0 to 7.0.1.\n- [Release notes](https://github.com/actions/checkout/releases)\n- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/actions/checkout/compare/9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0...3d3c42e5aac5ba805825da76410c181273ba90b1)\n\n---\nupdated-dependencies:\n- dependency-name: actions/checkout\n dependency-version: 7.0.1\n dependency-type: direct:production\n update-type: version-update:semver-patch\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "83f90633efc547bf09362d52443e81fb83b52af6", "tree": "28bf6176879123e0e8aaf0d37e4d24e90a17f64f", "parents": [ "04871dab3d97385c13768cc9f40914589fbbb92a" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Jul 21 14:16:21 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:16:21 2026 +0100" }, "message": "Minor: Regenerated License File for 04871dab3d97385c13768cc9f40914589fbbb92a (#2009)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: jnioche \u003cjnioche@users.noreply.github.com\u003e" }, { "commit": "04871dab3d97385c13768cc9f40914589fbbb92a", "tree": "c4f4d7c5f69647960a9d8d79bce4ef0f21a8786b", "parents": [ "18dac8c5f8e17f7a0e8ee89ab588ec2d68953505" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Tue Jul 21 14:06:40 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:06:40 2026 +0100" }, "message": "build(deps): bump langchain4j.version from 1.17.2 to 1.18.0 (#2007)\n\nBumps `langchain4j.version` from 1.17.2 to 1.18.0.\n\nUpdates `dev.langchain4j:langchain4j` from 1.17.2 to 1.18.0\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.17.2...1.18.0)\n\nUpdates `dev.langchain4j:langchain4j-open-ai` from 1.17.2 to 1.18.0\n- [Release notes](https://github.com/langchain4j/langchain4j/releases)\n- [Commits](https://github.com/langchain4j/langchain4j/compare/1.17.2...1.18.0)\n\n---\nupdated-dependencies:\n- dependency-name: dev.langchain4j:langchain4j\n dependency-version: 1.18.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n- dependency-name: dev.langchain4j:langchain4j-open-ai\n dependency-version: 1.18.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "18dac8c5f8e17f7a0e8ee89ab588ec2d68953505", "tree": "84af24397d5464a4dd0e585a6a6334ec57176de7", "parents": [ "0e01b67ed307f4ad8014906f4e1f634bb80035c2" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Tue Jul 21 14:06:11 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:06:11 2026 +0100" }, "message": "build(deps): bump software.amazon.awssdk:bom from 2.47.5 to 2.48.3 (#2008)\n\nBumps software.amazon.awssdk:bom from 2.47.5 to 2.48.3.\n\n---\nupdated-dependencies:\n- dependency-name: software.amazon.awssdk:bom\n dependency-version: 2.48.3\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "0e01b67ed307f4ad8014906f4e1f634bb80035c2", "tree": "1b1e63906612dd9fa562093ae5e6b66459355c1e", "parents": [ "3664644a60fa003917dae1fbdf589fcae02e6f06" ], "author": { "name": "dependabot[bot]", "email": "49699333+dependabot[bot]@users.noreply.github.com", "time": "Tue Jul 21 14:05:34 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 21 14:05:34 2026 +0100" }, "message": "build(deps): bump actions/setup-java from 5.5.0 to 5.6.0 (#2003)\n\nBumps [actions/setup-java](https://github.com/actions/setup-java) from 5.5.0 to 5.6.0.\n- [Release notes](https://github.com/actions/setup-java/releases)\n- [Commits](https://github.com/actions/setup-java/compare/0f481fcb613427c0f801b606911222b5b6f3083a...03ad4de0992f5dab5e18fcb136590ce7c4a0ac95)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-java\n dependency-version: 5.6.0\n dependency-type: direct:production\n update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] \u003csupport@github.com\u003e\nCo-authored-by: dependabot[bot] \u003c49699333+dependabot[bot]@users.noreply.github.com\u003e" }, { "commit": "3664644a60fa003917dae1fbdf589fcae02e6f06", "tree": "33bea974201b80c333fce959a8ab6fe49213aa03", "parents": [ "eaa6869ea60454f09e009c89c9e2d8528594f2f9" ], "author": { "name": "Julien Nioche", "email": "julien@digitalpebble.com", "time": "Sun Jul 19 08:23:56 2026 +0100" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sun Jul 19 08:23:56 2026 +0100" }, "message": "bugfix - max content not applied for robots.txt when redirecting (#2004)\n\nSigned-off-by: Julien Nioche \u003cjulien@digitalpebble.com\u003e" }, { "commit": "eaa6869ea60454f09e009c89c9e2d8528594f2f9", "tree": "0751ebdcef4ee7aedc930db04c48597f3759e8c6", "parents": [ "86647d348cebe6af270a0d514e2b148bab65d98b" ], "author": { "name": "Yurii Bakurov", "email": "45154988+Yurii201811@users.noreply.github.com", "time": "Fri Jul 17 22:34:33 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Fri Jul 17 21:34:33 2026 +0100" }, "message": "#2001 Document adaptive scheduler metadata (#2002)\n\n* Document adaptive scheduler metadata (#2001)\n\n* #2001 Clarify conditional request metadata" }, { "commit": "86647d348cebe6af270a0d514e2b148bab65d98b", "tree": "08d584e02f329e892ada859b3254c15e80e5f7d5", "parents": [ "7b92388e37d9eaccfb511ed62b0d42cf6aeae1c0" ], "author": { "name": "Sebastian Nagel", "email": "snagel@apache.org", "time": "Thu Jul 16 10:30:06 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Thu Jul 16 10:30:06 2026 +0200" }, "message": "[WARC] Format WARC-IP-Address header values following the WARC community recommendations (#1997)\n\n* [WARC] Format WARC-IP-Address header values following the WARC community recommendations\n\n* [WARC] Format WARC-IP-Address header values following the WARC community recommendations\n\nWARC module: add Guava dependency." }, { "commit": "7b92388e37d9eaccfb511ed62b0d42cf6aeae1c0", "tree": "aa6bd8538c61bec16012b4df98bc7cb5649985ec", "parents": [ "d4e7373f2b50d987f5f4ffe806f5d8641ae57db5" ], "author": { "name": "github-actions[bot]", "email": "41898282+github-actions[bot]@users.noreply.github.com", "time": "Tue Jul 14 14:28:34 2026 +0200" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Tue Jul 14 14:28:34 2026 +0200" }, "message": "Minor: Regenerated License File for d4e7373f2b50d987f5f4ffe806f5d8641ae57db5 (#1999)\n\nSigned-off-by: GitHub \u003cnoreply@github.com\u003e\nCo-authored-by: rzo1 \u003crzo1@users.noreply.github.com\u003e" } ], "next": "d4e7373f2b50d987f5f4ffe806f5d8641ae57db5" }