Google GitGoogle Git
Sign in
apache/stormcrawler/HEAD
  1. fd37a2c Minor: Regenerated License File for 82bf0e48b35b54f0181bb3152f1dc6ba68b50829 (#2224) by github-actions[bot] · 8 hours ago main
  2. 82bf0e4 build(deps): bump com.puppycrawl.tools:checkstyle from 14.1.0 to 14.3.0 (#2220) by dependabot[bot] · 8 hours ago
  3. 49eb5b5 build(deps): bump com.google.guava:guava from 33.7.1-jre to 33.7.2-jre (#2219) by dependabot[bot] · 8 hours ago
  4. 2247e86 build(deps): bump org.apache.commons:commons-lang3 from 3.20.0 to 3.21.0 (#2221) by dependabot[bot] · 8 hours ago
  5. d7303a2 build(deps): bump git-code-format-maven-plugin.version from 6.1 to 6.2 (#2218) by dependabot[bot] · 8 hours ago
  6. 2c10fc6 build(deps): bump langchain4j.version from 1.20.0 to 1.20.2 (#2217) by dependabot[bot] · 8 hours ago
  7. 1b4f224 build(deps): bump software.amazon.awssdk:bom from 2.55.5 to 2.55.10 (#2216) by dependabot[bot] · 8 hours ago
  8. 14ec73a Add StatusUpdaterBenchmark to measure the performance of status updater backends (#2213) by Julien Nioche · 3 days ago
  9. 095d609 #2214 Manage the Jackson dataformat/datatype versions in the root pom (#2215) by Tim Allison · 3 days ago
  10. 0ffc02a #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher by Davide Polato · 5 days ago
  11. c39f3d0 #2057 FetcherBolt: extract the response and failure handling by Davide Polato · 7 days ago
  12. bef41bc #2057 FetcherBolt: extract the crawl-delay policy by Davide Polato · 7 days ago
  13. f48c6b4 #2057 FetcherBolt: extract the sitemap autodiscovery by Davide Polato · 7 days ago
  14. df1a57e #2057 FetcherBolt: extract the robots.txt lookup by Davide Polato · 7 days ago
  15. af8cd13 #2057 FetcherBolt: move the fetch queue classes to their own files by Davide Polato · 7 days ago
  16. e57e00d #2057 FetcherBolt: add characterization tests by Davide Polato · 7 days ago
  17. f14e8f7 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs (#2183) by Tim Allison · 4 days ago
  18. fdb16fd Tika 4.1.0 (#2210) by Julien Nioche · 5 days ago
  19. e94fed1 Minor: Regenerated License File for b9f1a44e046a48fb0ea43da4015b09310b016363 (#2195) by github-actions[bot] · 7 days ago
  20. 44b3292 build(deps): bump org.mockito:mockito-core from 5.23.0 to 5.24.0 (#2194) by dependabot[bot] · 7 days ago
  21. b9f1a44 build(deps): bump software.amazon.awssdk:bom from 2.55.0 to 2.55.5 (#2193) by dependabot[bot] · 7 days ago
  22. 681168a build(deps): bump com.github.ben-manes.caffeine:caffeine (#2192) by dependabot[bot] · 7 days ago
  23. d112506 build(deps): bump org.apache:apache from 39 to 40 (#2191) by dependabot[bot] · 7 days ago
  24. 8e5a7bc build(deps): bump git-code-format-maven-plugin.version from 6.0 to 6.1 (#2190) by dependabot[bot] · 7 days ago
  25. d498d9a build(deps-dev): bump de.thetaphi:forbiddenapis from 3.10 to 3.11 (#2189) by dependabot[bot] · 7 days ago
  26. 4b6a813 build(deps): bump org.slf4j:slf4j-simple from 2.0.19 to 2.0.20 (#2188) by dependabot[bot] · 7 days ago
  27. f90c306 #2093 LLM text extractor: keep page markers out of the prompt and clean the reply (#2181) by Richard Zowalla · 9 days ago
  28. 77224e7 fix: count the URLs dropped by fetcher.timeout.queue (#2058) (#2184) by Davide Polato · 10 days ago
  29. 251f27a use the ASF-hosted logo in the README (#2187) by Davide Polato · 10 days ago
  30. 1b32ccc #2098 URLFrontier: optional TLS for the channels to the frontier (#2175) by Richard Zowalla · 12 days ago
  31. 06aa32a #2103 OpenSearchConnection: send the credentials only to the configured addresses (#2173) by Richard Zowalla · 12 days ago
  32. 1a3d88b Minor: Regenerated License File for 94f00e034983d2539f00a886e39f7bf0d3205477 (#2180) by github-actions[bot] · 13 days ago
  33. 94f00e0 Minor: Regenerated License File for 4931f9daaa6072ba59963e518c667eabfabde4ba (#2179) by github-actions[bot] · 14 days ago
  34. c5181aa build(deps): bump com.microsoft.playwright:playwright (#2176) by dependabot[bot] · 2 weeks ago
  35. 4931f9d build(deps): bump software.amazon.awssdk:bom from 2.54.16 to 2.55.0 (#2177) by dependabot[bot] · 2 weeks ago
  36. 220b1b6 #2166 SiteMapParserBolt: cap the number of URLs emitted per sitemap (#2172) by Richard Zowalla · 2 weeks ago
  37. c746b72 #2097 Tika ParserBolt: limit the extracted text with parser.tika.text.maxlength (#2174) by Richard Zowalla · 2 weeks ago
  38. 2b8411f #2100 SiteMapParserBolt: read the content type under the protocol prefix (#2169) by Richard Zowalla · 2 weeks ago
  39. 15007e1 #2091 Protocols: reject metadata keys arriving in a response header (#2171) by Richard Zowalla · 2 weeks ago
  40. d4e8a57 #2099 Fetcher bolts: skip a metrics.* value which is not a number (#2170) by Richard Zowalla · 2 weeks ago
  41. 379d81c #2090 Fetcher bolts: drop a set-header value coming from a response (#2168) by Richard Zowalla · 2 weeks ago
  42. 78ea4a8 #2095 Apply the IP address filter to the requests made by the Playwright browser (#2165) by Richard Zowalla · 2 weeks ago
  43. 0bf87d7 #2101 HostURLFilter: drop the source host memoization shared by all fetcher threads (#2167) by Richard Zowalla · 2 weeks ago
  44. e575776 #2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols (#2135) by Gianluca Graziadei · 2 weeks ago
  45. 6ffb423 #2092 Bound the meta charset look-ahead and catch charset detection failures in JSoupParserBolt (#2157) by Davide Polato · 2 weeks ago
  46. 28119df #2162 Send credential headers from http.custom.headers only to the hosts in http.custom.headers.hosts (#2164) by Richard Zowalla · 2 weeks ago
  47. f859a8f #2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts (#2163) by Richard Zowalla · 2 weeks ago
  48. a011dd0 #2094 Make ignoring HTTPS errors in the Playwright protocol configurable and independent of the proxy (#2158) by Richard Zowalla · 2 weeks ago
  49. 7d744d2 Minor: Regenerated License File for f12f2b589613fce22bf1dd4a094e0d8bb99ffed1 (#2161) by github-actions[bot] · 3 weeks ago
  50. f12f2b5 Minor: Regenerated License File for 515d489e87bb291986dac8309ab313c1919855b1 (#2160) by github-actions[bot] · 3 weeks ago
  51. f164aee Minor: Regenerated License File for 55efcf2f22ec8b482f1d723806e9aec1449eb72a (#2159) by github-actions[bot] · 3 weeks ago
  52. 375800c build(deps): bump langchain4j.version from 1.19.0 to 1.20.0 (#2153) by dependabot[bot] · 3 weeks ago
  53. 515d489 build(deps): bump org.slf4j:slf4j-simple from 2.0.18 to 2.0.19 (#2154) by dependabot[bot] · 3 weeks ago
  54. 863df57 build(deps): bump software.amazon.awssdk:bom from 2.54.12 to 2.54.16 (#2155) by dependabot[bot] · 3 weeks ago
  55. 28b1f0e build(deps): bump actions/setup-java from 6.0.0 to 6.0.1 (#2156) by dependabot[bot] · 3 weeks ago
  56. 55efcf2 #2102 Redact proxy credentials from worker logs (#2151) by Riddhish Bonde · 3 weeks ago
  57. 59350eb Upgrade to Apache Storm 3.1.0 and compile with Java 25 (#2150) by Richard Zowalla · 3 weeks ago
  58. 68b9ecb Run redirect hops through the URL filters (#2128) by Abhinav · 3 weeks ago
  59. 40b59eb Spouts check the scheme of stored URLs before emitting (#2123) by Abhinav · 3 weeks ago
  60. 7ef7d92 SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure (#2125) by Abhinav · 3 weeks ago
  61. 1d455c2 Canonicalise hosts for politeness queues and the robots.txt cache (#2122) by Abhinav · 3 weeks ago
  62. 765cb11 #1998 WARC writer: WARC-Protocol header to follow WARC field proposals (#2034) by Sebastian Nagel · 3 weeks ago
  63. 28a467e URLFrontier StatusUpdaterBolt: make lock objects transient so the bolt can be serialized (#2149) by Richard Zowalla · 3 weeks ago
  64. b4df3e3 Keep the fetcher off private address space by default (#2127) by Abhinav · 3 weeks ago
  65. 151e568 OKHttp: authenticated TLS by default, credentials withheld from unauthenticated servers (#2126) by Abhinav · 4 weeks ago
  66. dcf4e9d #2104 feat: implement MIME type whitelist filtering in ParserBolt for Tika (#2116) by Riddhish Bonde · 4 weeks ago
  67. c6bcf44 Minor: Fix formatting in ColumnNameValidationTest (#2148) by Richard Zowalla · 4 weeks ago
  68. 44f39f8 Minor: Regenerated License File for 1c6f1ae750f812ac085063a6cac5624ef8269f39 (#2147) by github-actions[bot] · 4 weeks ago
  69. 1c6f1ae Use batch endpoint in URLFrontier module for DISCOVERED urls (#2117) by Abhinav · 4 weeks ago
  70. 3c5fcd6 SQL IndexerBolt: only plain identifiers may be used as column names, fixes #2096 (#2146) by Richard Zowalla · 4 weeks ago
  71. ead30fc Upgrade to Tika 4.0.0 (#2118) by Abhinav · 4 weeks ago
  72. 5dd6d99 File scheme off by default; reads confined to file.protocol.root (#2124) by Abhinav · 4 weeks ago
  73. a12bd13 #2129 FetcherBolt: lock-free fetch queues ordered by next fetch time (#2132) by Gianluca Graziadei · 4 weeks ago
  74. e5b05d3 URLFilters: treat an exception from a filter as a rejection (#2120) by Abhinav · 4 weeks ago
  75. c58fd21 Robots.txt fetch no longer replaces the configured content limit (#2121) by Abhinav · 4 weeks ago
  76. c3fa48b IPFilterRules: add anylocal and multicast keywords, fail on unparseable rules (#2119) by Abhinav · 4 weeks ago
  77. 2cb93d1 Minor: Regenerated License File for 096136432d5ce06e94d0b230f402e8db88155e06 (#2144) by github-actions[bot] · 4 weeks ago
  78. 0961364 Minor: Regenerated License File for db885724d5bc2fd730fdd5f0b7ceff3dcbc2f90e (#2143) by github-actions[bot] · 4 weeks ago
  79. 3a49d07 #2130 MetadataTransfer: single pass over the metadata for wildcard keys (#2133) by Gianluca Graziadei · 4 weeks ago
  80. 0661651 build(deps): bump org.netpreserve:jwarc from 0.36.0 to 0.37.0 (#2142) by dependabot[bot] · 4 weeks ago
  81. 8f99237 build(deps): bump org.apache.maven.plugins:maven-surefire-plugin (#2141) by dependabot[bot] · 4 weeks ago
  82. db88572 build(deps): bump org.opensearch.client:opensearch-java (#2140) by dependabot[bot] · 4 weeks ago
  83. 80ae632 build(deps): bump software.amazon.awssdk:bom from 2.54.6 to 2.54.12 (#2139) by dependabot[bot] · 4 weeks ago
  84. f062642 build(deps): bump com.puppycrawl.tools:checkstyle from 14.0.0 to 14.1.0 (#2138) by dependabot[bot] · 4 weeks ago
  85. 770389a build(deps): bump org.yaml:snakeyaml from 2.6 to 2.7 (#2137) by dependabot[bot] · 4 weeks ago
  86. d8d5581 build(deps): bump org.apache.maven.plugins:maven-compiler-plugin (#2136) by dependabot[bot] · 4 weeks ago
  87. fb13910 Make the WARC digest algorithm configurable (#2110) by Abhinav · 4 weeks ago
  88. 3c4f5e7 ParseResult: non-mutating lookup, explicit getOrCreate() and no empty documents emitted (#2112) by Abhinav · 5 weeks ago
  89. 04ee351 CommaSeparatedToMultivaluedMetadata: replace quadratic token append with a single bulk write (#2111) by Abhinav · 5 weeks ago
  90. 0663b8c Escape CR and LF in WARC metadata records and resource Content-Type (#2109) by Abhinav · 5 weeks ago
  91. f7cf183 Regenerated License file after dependency upgrades (#2115) by github-actions[bot] · 5 weeks ago
  92. a915702 build(deps): bump software.amazon.awssdk:bom from 2.54.1 to 2.54.6 (#2113) by dependabot[bot] · 5 weeks ago
  93. 05e1d4a build(deps): bump org.jsoup:jsoup from 1.23.1 to 1.23.2 (#2114) by dependabot[bot] · 5 weeks ago
  94. 93b6e24 Only follow robots.txt redirects within the same origin (#2077) by Richard Zowalla · 5 weeks ago
  95. f520b4e Escape the metadata column of the SQL status store (#2076) by Richard Zowalla · 5 weeks ago
  96. 88fcf3b Scope cookies to the host that set them (#2075) by Richard Zowalla · 5 weeks ago
  97. 2223fe8 Honour ignoreOutsideDomain when set on its own in HostURLFilter (#2074) by Richard Zowalla · 5 weeks ago
  98. e73aaeb build(deps): bump actions/setup-java from 5.7.0 to 6.0.0 (#2078) by dependabot[bot] · 6 weeks ago
  99. 07d4392 Minor: Regenerated License File for 191e5babcaeb9b448607dd10f024cfa7cdae713c (#2072) by github-actions[bot] · 6 weeks ago
  100. 191e5ba build(deps): bump langchain4j.version from 1.18.1 to 1.19.0 (#2064) by dependabot[bot] · 6 weeks ago