Google Git Google Git
Sign in
apache/stormcrawler/HEAD
  1. 2aa44dc Add baseline SUB class for benchmarking (#2279) by Julien Nioche · 17 hours ago main
  2. 4bb9e03 #2266 Remove the dependency on HttpCore 5 (#2270) by Richard Zowalla · 21 hours ago
  3. 94bb6a1 Minor: Regenerated License File for c9ea2a8486a4129194836b9cf47752c960e29134 (#2278) by github-actions[bot] · 25 hours ago
  4. c9ea2a8 build(deps): bump com.microsoft.playwright:playwright (#2277) by dependabot[bot] · 29 hours ago
  5. 9d7ab58 build(deps): bump okhttp.version from 5.5.0 to 5.5.1 (#2275) by dependabot[bot] · 30 hours ago
  6. 9e2377e build(deps): bump com.ibm.icu:icu4j from 78.3 to 79.1 (#2274) by dependabot[bot] · 30 hours ago
  7. 697423f build(deps): bump org.apache.maven.plugins:maven-surefire-plugin (#2273) by dependabot[bot] · 30 hours ago
  8. 4eec171 build(deps): bump software.amazon.awssdk:bom from 2.56.0 to 2.57.3 (#2272) by dependabot[bot] · 30 hours ago
  9. 6392723 build(deps): bump com.mysql:mysql-connector-j from 26.7.0 to 26.10.0 (#2271) by dependabot[bot] · 30 hours ago
  10. a493873 #2198 preserve query strings containing slashes (#2231) by sean0407 · 30 hours ago
  11. a2bd30c #2264 Use entrySet() when iterating maps in WARCSpout and Solr IndexerBolt (#2265) by EL MEHDI EL MOUATE · 2 days ago
  12. 0be39e7 Minor: Regenerated License File for 1c9aeda58de6cb7353879d4cf16664e057aee602 (#2267) by github-actions[bot] · 2 days ago
  13. 36907a2 #158 document the JDK XML limits for sitemaps and feeds (#2263) by Davide Polato · 2 days ago
  14. 1c9aeda #2211 Core now depends only on HttpCore 5 (#2236) by SCORP711 · 2 days ago
  15. db7aff0 #158 remove the DOM from the ParseFilters (#2262) by Julien Nioche · 3 days ago
  16. 466b85c Minor: Regenerated License File for 1ac59f7bf376b7645a764b805a9d8f34b4a8003e (#2259) by github-actions[bot] · 3 days ago
  17. 1ac59f7 Tika ParserBolt: apply the JSoupFilters to the documents parsed by Tika (#2256) by Julien Nioche · 3 days ago
  18. 55b2d83 XPathFilter: skip blank values and try the next expression (#2258) by Julien Nioche · 3 days ago
  19. d6893c3 Use commons-secure-xml for the XPath and DOM factories in core (#2257) by Richard Zowalla · 4 days ago
  20. dbc35c5 #158 remove the DOM-based LDJsonParseFilter and LinkParseFilter (#2253) by Julien Nioche · 4 days ago
  21. 51c23bc #158 jsoup XPathFilter: convert to W3C DOM once per page, not once per expression (#2252) by Julien Nioche · 4 days ago
  22. 9c8b85a #2250 CharsetIdentification: read the content type under the protocol prefix (#2251) by Nico Scandolo · 5 days ago
  23. 0fd8847 Minor: Regenerated License File for 1b56770c5306b0027cc92a763dfc1ea14a52fcdc (#2255) by github-actions[bot] · 5 days ago
  24. 1b56770 #2212 move the opensearch-java implementation into external/opensearch (#2242) by Davide Polato · 5 days ago
  25. 2d6d637 #2212 retry 429 items and flush bulk requests by size (#2241) by Davide Polato · 6 days ago
  26. 56d9ed6 #2145 Punycode the host in politeness, partition and robots.txt keys (#2244) by Omar Bounawara · 7 days ago
  27. d983b71 Minor: Regenerated License File for 37fe1b34f02e3b90bcdc318a034eaa95ef98b1ca (#2247) by github-actions[bot] · 8 days ago
  28. 23f3d1c build(deps): bump actions/setup-java from 6.0.1 to 6.1.0 (#2248) by dependabot[bot] · 8 days ago
  29. 37fe1b3 build(deps): bump com.github.ben-manes.caffeine:caffeine (#2246) by dependabot[bot] · 8 days ago
  30. 208af27 build(deps): bump software.amazon.awssdk:bom from 2.55.13 to 2.56.0 (#2245) by dependabot[bot] · 8 days ago
  31. 3f964e6 MultiProxyManager: thread-safe round-robin rotation (#2228) by Davide Polato · 9 days ago
  32. 0ba9c2e Regenerated License file after dependency upgrades (#2240) by github-actions[bot] · 13 days ago
  33. fca69e9 build(deps): bump software.amazon.awssdk:bom from 2.55.10 to 2.55.13 (#2232) by dependabot[bot] · 13 days ago
  34. 01a0f5a Minor: Regenerated License File for 21f2bf0c214224802139470c0498c02b1db4c26a (#2239) by github-actions[bot] · 13 days ago
  35. 21f2bf0 Minor: Regenerated License File for 58bb8a7dd86dd6fc134bcf432d6447cbc18428ff (#2238) by github-actions[bot] · 13 days ago
  36. e551413 build(deps): bump langchain4j.version from 1.20.2 to 1.22.0 (#2233) by dependabot[bot] · 14 days ago
  37. 58bb8a7 build(deps): bump commons-codec:commons-codec from 1.22.1 to 1.23.0 (#2234) by dependabot[bot] · 14 days ago
  38. 4fdcc40 opensearch-java: fix the search_after values sent by HybridSpout (#2237) by Davide Polato · 14 days ago
  39. e98da1a URLFrontier Spout: report the query time (#2235) by Julien Nioche · 2 weeks ago
  40. 9593874 AbstractStatusUpdaterBolt: report cache hits/misses with a counter, not reset-on-read gauges (#2229) by Julien Nioche · 2 weeks ago
  41. a575f49 #1901 Tika ParserBolt: limit the pages parsed in a PDF with parser.tika.pdf.maxpages (#2227) by Richard Zowalla · 2 weeks ago
  42. d084f82 #2197 BasicURLNormalizer keeps the query when rebuilding a URL (#2207) by Julien Nioche · 3 weeks ago
  43. d8c0d4c Add SpoutBenchmark to measure the read performance of spouts (#2226) by Julien Nioche · 3 weeks ago
  44. 3c8f8c6 URLFrontier StatusUpdaterBolt: cheaper bookkeeping of the acks (#2223) by Julien Nioche · 3 weeks ago
  45. 6b9398b opensearch-java: send bulk requests with the OpenSearchAsyncClient (#2225) by Julien Nioche · 3 weeks ago
  46. 343ae82 StatusUpdaterBenchmark: run several instances of the status updater in parallel (#2222) by Julien Nioche · 3 weeks ago
  47. fd37a2c Minor: Regenerated License File for 82bf0e48b35b54f0181bb3152f1dc6ba68b50829 (#2224) by github-actions[bot] · 3 weeks ago
  48. 82bf0e4 build(deps): bump com.puppycrawl.tools:checkstyle from 14.1.0 to 14.3.0 (#2220) by dependabot[bot] · 3 weeks ago
  49. 49eb5b5 build(deps): bump com.google.guava:guava from 33.7.1-jre to 33.7.2-jre (#2219) by dependabot[bot] · 3 weeks ago
  50. 2247e86 build(deps): bump org.apache.commons:commons-lang3 from 3.20.0 to 3.21.0 (#2221) by dependabot[bot] · 3 weeks ago
  51. d7303a2 build(deps): bump git-code-format-maven-plugin.version from 6.1 to 6.2 (#2218) by dependabot[bot] · 3 weeks ago
  52. 2c10fc6 build(deps): bump langchain4j.version from 1.20.0 to 1.20.2 (#2217) by dependabot[bot] · 3 weeks ago
  53. 1b4f224 build(deps): bump software.amazon.awssdk:bom from 2.55.5 to 2.55.10 (#2216) by dependabot[bot] · 3 weeks ago
  54. 14ec73a Add StatusUpdaterBenchmark to measure the performance of status updater backends (#2213) by Julien Nioche · 3 weeks ago
  55. 095d609 #2214 Manage the Jackson dataformat/datatype versions in the root pom (#2215) by Tim Allison · 4 weeks ago
  56. 0ffc02a #2057 FetcherBolt: move the fetcher internals to o.a.s.fetcher by Davide Polato · 4 weeks ago
  57. c39f3d0 #2057 FetcherBolt: extract the response and failure handling by Davide Polato · 4 weeks ago
  58. bef41bc #2057 FetcherBolt: extract the crawl-delay policy by Davide Polato · 4 weeks ago
  59. f48c6b4 #2057 FetcherBolt: extract the sitemap autodiscovery by Davide Polato · 4 weeks ago
  60. df1a57e #2057 FetcherBolt: extract the robots.txt lookup by Davide Polato · 4 weeks ago
  61. af8cd13 #2057 FetcherBolt: move the fetch queue classes to their own files by Davide Polato · 4 weeks ago
  62. e57e00d #2057 FetcherBolt: add characterization tests by Davide Polato · 4 weeks ago
  63. f14e8f7 2097: Isolate tika parsing via tika-pipes to ensure reliable timeouts and protection from OOMs (#2183) by Tim Allison · 4 weeks ago
  64. fdb16fd Tika 4.1.0 (#2210) by Julien Nioche · 4 weeks ago
  65. e94fed1 Minor: Regenerated License File for b9f1a44e046a48fb0ea43da4015b09310b016363 (#2195) by github-actions[bot] · 4 weeks ago
  66. 44b3292 build(deps): bump org.mockito:mockito-core from 5.23.0 to 5.24.0 (#2194) by dependabot[bot] · 4 weeks ago
  67. b9f1a44 build(deps): bump software.amazon.awssdk:bom from 2.55.0 to 2.55.5 (#2193) by dependabot[bot] · 4 weeks ago
  68. 681168a build(deps): bump com.github.ben-manes.caffeine:caffeine (#2192) by dependabot[bot] · 4 weeks ago
  69. d112506 build(deps): bump org.apache:apache from 39 to 40 (#2191) by dependabot[bot] · 4 weeks ago
  70. 8e5a7bc build(deps): bump git-code-format-maven-plugin.version from 6.0 to 6.1 (#2190) by dependabot[bot] · 4 weeks ago
  71. d498d9a build(deps-dev): bump de.thetaphi:forbiddenapis from 3.10 to 3.11 (#2189) by dependabot[bot] · 4 weeks ago
  72. 4b6a813 build(deps): bump org.slf4j:slf4j-simple from 2.0.19 to 2.0.20 (#2188) by dependabot[bot] · 4 weeks ago
  73. f90c306 #2093 LLM text extractor: keep page markers out of the prompt and clean the reply (#2181) by Richard Zowalla · 4 weeks ago
  74. 77224e7 fix: count the URLs dropped by fetcher.timeout.queue (#2058) (#2184) by Davide Polato · 5 weeks ago
  75. 251f27a use the ASF-hosted logo in the README (#2187) by Davide Polato · 5 weeks ago
  76. 1b32ccc #2098 URLFrontier: optional TLS for the channels to the frontier (#2175) by Richard Zowalla · 5 weeks ago
  77. 06aa32a #2103 OpenSearchConnection: send the credentials only to the configured addresses (#2173) by Richard Zowalla · 5 weeks ago
  78. 1a3d88b Minor: Regenerated License File for 94f00e034983d2539f00a886e39f7bf0d3205477 (#2180) by github-actions[bot] · 5 weeks ago
  79. 94f00e0 Minor: Regenerated License File for 4931f9daaa6072ba59963e518c667eabfabde4ba (#2179) by github-actions[bot] · 5 weeks ago
  80. c5181aa build(deps): bump com.microsoft.playwright:playwright (#2176) by dependabot[bot] · 5 weeks ago
  81. 4931f9d build(deps): bump software.amazon.awssdk:bom from 2.54.16 to 2.55.0 (#2177) by dependabot[bot] · 5 weeks ago
  82. 220b1b6 #2166 SiteMapParserBolt: cap the number of URLs emitted per sitemap (#2172) by Richard Zowalla · 5 weeks ago
  83. c746b72 #2097 Tika ParserBolt: limit the extracted text with parser.tika.text.maxlength (#2174) by Richard Zowalla · 5 weeks ago
  84. 2b8411f #2100 SiteMapParserBolt: read the content type under the protocol prefix (#2169) by Richard Zowalla · 5 weeks ago
  85. 15007e1 #2091 Protocols: reject metadata keys arriving in a response header (#2171) by Richard Zowalla · 5 weeks ago
  86. d4e8a57 #2099 Fetcher bolts: skip a metrics.* value which is not a number (#2170) by Richard Zowalla · 5 weeks ago
  87. 379d81c #2090 Fetcher bolts: drop a set-header value coming from a response (#2168) by Richard Zowalla · 5 weeks ago
  88. 78ea4a8 #2095 Apply the IP address filter to the requests made by the Playwright browser (#2165) by Richard Zowalla · 5 weeks ago
  89. 0bf87d7 #2101 HostURLFilter: drop the source host memoization shared by all fetcher threads (#2167) by Richard Zowalla · 5 weeks ago
  90. e575776 #2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols (#2135) by Gianluca Graziadei · 5 weeks ago
  91. 6ffb423 #2092 Bound the meta charset look-ahead and catch charset detection failures in JSoupParserBolt (#2157) by Davide Polato · 6 weeks ago
  92. 28119df #2162 Send credential headers from http.custom.headers only to the hosts in http.custom.headers.hosts (#2164) by Richard Zowalla · 6 weeks ago
  93. f859a8f #2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts (#2163) by Richard Zowalla · 6 weeks ago
  94. a011dd0 #2094 Make ignoring HTTPS errors in the Playwright protocol configurable and independent of the proxy (#2158) by Richard Zowalla · 6 weeks ago
  95. 7d744d2 Minor: Regenerated License File for f12f2b589613fce22bf1dd4a094e0d8bb99ffed1 (#2161) by github-actions[bot] · 6 weeks ago
  96. f12f2b5 Minor: Regenerated License File for 515d489e87bb291986dac8309ab313c1919855b1 (#2160) by github-actions[bot] · 6 weeks ago
  97. f164aee Minor: Regenerated License File for 55efcf2f22ec8b482f1d723806e9aec1449eb72a (#2159) by github-actions[bot] · 6 weeks ago
  98. 375800c build(deps): bump langchain4j.version from 1.19.0 to 1.20.0 (#2153) by dependabot[bot] · 6 weeks ago
  99. 515d489 build(deps): bump org.slf4j:slf4j-simple from 2.0.18 to 2.0.19 (#2154) by dependabot[bot] · 6 weeks ago
  100. 863df57 build(deps): bump software.amazon.awssdk:bom from 2.54.12 to 2.54.16 (#2155) by dependabot[bot] · 6 weeks ago