Google GitGoogle Git
Sign in
apache/stormcrawler/HEAD
  1. 68b9ecb Run redirect hops through the URL filters (#2128) by Abhinav · 2 days ago main
  2. 40b59eb Spouts check the scheme of stored URLs before emitting (#2123) by Abhinav · 2 days ago
  3. 7ef7d92 SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure (#2125) by Abhinav · 2 days ago
  4. 1d455c2 Canonicalise hosts for politeness queues and the robots.txt cache (#2122) by Abhinav · 2 days ago
  5. 765cb11 #1998 WARC writer: WARC-Protocol header to follow WARC field proposals (#2034) by Sebastian Nagel · 2 days ago
  6. 28a467e URLFrontier StatusUpdaterBolt: make lock objects transient so the bolt can be serialized (#2149) by Richard Zowalla · 2 days ago
  7. b4df3e3 Keep the fetcher off private address space by default (#2127) by Abhinav · 3 days ago
  8. 151e568 OKHttp: authenticated TLS by default, credentials withheld from unauthenticated servers (#2126) by Abhinav · 4 days ago
  9. dcf4e9d #2104 feat: implement MIME type whitelist filtering in ParserBolt for Tika (#2116) by Riddhish Bonde · 4 days ago
  10. c6bcf44 Minor: Fix formatting in ColumnNameValidationTest (#2148) by Richard Zowalla · 4 days ago
  11. 44f39f8 Minor: Regenerated License File for 1c6f1ae750f812ac085063a6cac5624ef8269f39 (#2147) by github-actions[bot] · 4 days ago
  12. 1c6f1ae Use batch endpoint in URLFrontier module for DISCOVERED urls (#2117) by Abhinav · 4 days ago
  13. 3c5fcd6 SQL IndexerBolt: only plain identifiers may be used as column names, fixes #2096 (#2146) by Richard Zowalla · 4 days ago
  14. ead30fc Upgrade to Tika 4.0.0 (#2118) by Abhinav · 5 days ago
  15. 5dd6d99 File scheme off by default; reads confined to file.protocol.root (#2124) by Abhinav · 5 days ago
  16. a12bd13 #2129 FetcherBolt: lock-free fetch queues ordered by next fetch time (#2132) by Gianluca Graziadei · 5 days ago
  17. e5b05d3 URLFilters: treat an exception from a filter as a rejection (#2120) by Abhinav · 5 days ago
  18. c58fd21 Robots.txt fetch no longer replaces the configured content limit (#2121) by Abhinav · 5 days ago
  19. c3fa48b IPFilterRules: add anylocal and multicast keywords, fail on unparseable rules (#2119) by Abhinav · 6 days ago
  20. 2cb93d1 Minor: Regenerated License File for 096136432d5ce06e94d0b230f402e8db88155e06 (#2144) by github-actions[bot] · 7 days ago
  21. 0961364 Minor: Regenerated License File for db885724d5bc2fd730fdd5f0b7ceff3dcbc2f90e (#2143) by github-actions[bot] · 7 days ago
  22. 3a49d07 #2130 MetadataTransfer: single pass over the metadata for wildcard keys (#2133) by Gianluca Graziadei · 7 days ago
  23. 0661651 build(deps): bump org.netpreserve:jwarc from 0.36.0 to 0.37.0 (#2142) by dependabot[bot] · 7 days ago
  24. 8f99237 build(deps): bump org.apache.maven.plugins:maven-surefire-plugin (#2141) by dependabot[bot] · 7 days ago
  25. db88572 build(deps): bump org.opensearch.client:opensearch-java (#2140) by dependabot[bot] · 7 days ago
  26. 80ae632 build(deps): bump software.amazon.awssdk:bom from 2.54.6 to 2.54.12 (#2139) by dependabot[bot] · 7 days ago
  27. f062642 build(deps): bump com.puppycrawl.tools:checkstyle from 14.0.0 to 14.1.0 (#2138) by dependabot[bot] · 7 days ago
  28. 770389a build(deps): bump org.yaml:snakeyaml from 2.6 to 2.7 (#2137) by dependabot[bot] · 7 days ago
  29. d8d5581 build(deps): bump org.apache.maven.plugins:maven-compiler-plugin (#2136) by dependabot[bot] · 7 days ago
  30. fb13910 Make the WARC digest algorithm configurable (#2110) by Abhinav · 8 days ago
  31. 3c4f5e7 ParseResult: non-mutating lookup, explicit getOrCreate() and no empty documents emitted (#2112) by Abhinav · 12 days ago
  32. 04ee351 CommaSeparatedToMultivaluedMetadata: replace quadratic token append with a single bulk write (#2111) by Abhinav · 13 days ago
  33. 0663b8c Escape CR and LF in WARC metadata records and resource Content-Type (#2109) by Abhinav · 13 days ago
  34. f7cf183 Regenerated License file after dependency upgrades (#2115) by github-actions[bot] · 14 days ago
  35. a915702 build(deps): bump software.amazon.awssdk:bom from 2.54.1 to 2.54.6 (#2113) by dependabot[bot] · 2 weeks ago
  36. 05e1d4a build(deps): bump org.jsoup:jsoup from 1.23.1 to 1.23.2 (#2114) by dependabot[bot] · 2 weeks ago
  37. 93b6e24 Only follow robots.txt redirects within the same origin (#2077) by Richard Zowalla · 2 weeks ago
  38. f520b4e Escape the metadata column of the SQL status store (#2076) by Richard Zowalla · 2 weeks ago
  39. 88fcf3b Scope cookies to the host that set them (#2075) by Richard Zowalla · 2 weeks ago
  40. 2223fe8 Honour ignoreOutsideDomain when set on its own in HostURLFilter (#2074) by Richard Zowalla · 2 weeks ago
  41. e73aaeb build(deps): bump actions/setup-java from 5.7.0 to 6.0.0 (#2078) by dependabot[bot] · 2 weeks ago
  42. 07d4392 Minor: Regenerated License File for 191e5babcaeb9b448607dd10f024cfa7cdae713c (#2072) by github-actions[bot] · 3 weeks ago
  43. 191e5ba build(deps): bump langchain4j.version from 1.18.1 to 1.19.0 (#2064) by dependabot[bot] · 3 weeks ago
  44. 4d49466 Removed apache http protocol implementation. Closes #1995 (#2071) by Julien Nioche · 3 weeks ago
  45. 13b1679 Minor: Regenerated License File for e46d752dcdb33ab756ccf678c2aaad9781c4577a (#2070) by github-actions[bot] · 3 weeks ago
  46. e46d752 build(deps): bump software.amazon.awssdk:bom from 2.53.0 to 2.54.1 (#2068) by dependabot[bot] · 3 weeks ago
  47. 6ccccba build(deps): bump com.puppycrawl.tools:checkstyle from 12.3.1 to 14.0.0 (#2069) by dependabot[bot] · 3 weeks ago
  48. e62aac7 build(deps): bump com.google.guava:guava from 33.6.0-jre to 33.7.1-jre (#2067) by dependabot[bot] · 3 weeks ago
  49. 8f7c17d build(deps): bump okhttp.version from 5.4.0 to 5.5.0 (#2065) by dependabot[bot] · 3 weeks ago
  50. 358d761 Minor: Regenerated License File for 00bca0d0a904d3a3bb7a6d84c31fc3ffc9c9e35b (#2060) by github-actions[bot] · 4 weeks ago
  51. b7b0f84 build(deps): bump git-code-format-maven-plugin.version from 5.4 to 6.0 (#1825) by dependabot[bot] · 4 weeks ago
  52. 00bca0d build(deps): bump junit.version from 6.1.2 to 6.1.3 (#2054) by dependabot[bot] · 4 weeks ago
  53. ca71962 build(deps): bump software.amazon.awssdk:bom from 2.51.2 to 2.53.0 (#2055) by dependabot[bot] · 4 weeks ago
  54. 735160e Remove Selenium module, fixes #1994 (#2059) by Julien Nioche · 4 weeks ago
  55. 42b88e0 bugfix set Java 25 for snapshots workflow (#2056) by Julien Nioche · 4 weeks ago
  56. a0838c8 Regenerated License file after dependency upgrades (#2051) by github-actions[bot] · 5 weeks ago
  57. e6c4f7b build(deps): bump software.amazon.awssdk:bom from 2.50.0 to 2.51.2 (#2049) by dependabot[bot] · 5 weeks ago
  58. bbdebac build(deps): bump com.microsoft.playwright:playwright from 1.61.0 to 1.62.0 (#2048) by dependabot[bot] · 5 weeks ago
  59. e1d54a6 refresh NOTICE copyright year to 2026 (#2047) by Davide Polato · 5 weeks ago
  60. e2fcceb Minor: Regenerated License File for 54fcf5b58d6e070b2d0fe89df130b66eafbf0449 (#2046) by github-actions[bot] · 6 weeks ago
  61. 54fcf5b Upgrade to Storm 3 (#2045) by Richard Zowalla · 6 weeks ago
  62. 00b5be0 Minor: Regenerated License File for 738c3024793d609bbb0a5b742c6ec5f50566038b (#2044) by github-actions[bot] · 6 weeks ago
  63. 738c302 Upgrade to Apache Storm 3.0.0 and Java 25 (#2017) by Richard Zowalla · 6 weeks ago
  64. e4d30f7 build(deps): bump org.jsoup:jsoup from 1.22.2 to 1.23.1 by dependabot[bot] · 6 weeks ago
  65. 9b72b5a build(deps): bump com.mysql:mysql-connector-j from 9.7.0 to 26.7.0 by dependabot[bot] · 6 weeks ago
  66. de6977d build(deps): bump langchain4j.version from 1.18.0 to 1.18.1 by dependabot[bot] · 6 weeks ago
  67. caec3bc build(deps): bump commons-codec:commons-codec from 1.22.0 to 1.22.1 by dependabot[bot] · 6 weeks ago
  68. 306474d build(deps): bump actions/setup-java from 5.6.0 to 5.7.0 by dependabot[bot] · 6 weeks ago
  69. 2dddb1d build(deps): bump software.amazon.awssdk:bom from 2.49.3 to 2.50.0 by dependabot[bot] · 6 weeks ago
  70. 8f94a75 Fix broken anchor in Asciidoc by Sebastian Nagel · 6 weeks ago
  71. 747f5d0 Fix broken link to Guava's CacheBuilderSpec by Sebastian Nagel · 10 months ago
  72. 214b23e Fix typo in OpenSearch configuration comment by Sebastian Nagel · 4 months ago
  73. 883b854 [maven-release-plugin] prepare for next development iteration by Davide Polato · 7 weeks ago
  74. 0dc2ab5 [maven-release-plugin] prepare release stormcrawler-3.7.0 by Davide Polato · 7 weeks ago
  75. e3297e4 Regenerated License file after dependency upgrades (#2033) by github-actions[bot] · 7 weeks ago
  76. 6e69083 Upgrade to Storm 2.8.9 (#2032) by Davide Polato · 7 weeks ago
  77. 9304ada Enforce robots.txt Crawl-delay in URLFrontier queues (#2000) by Davide Polato · 7 weeks ago
  78. dc1f70b fix: emit anchored tuples #2021 (#2031) by Luca · 7 weeks ago
  79. d193e6c #2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set (#2027) by Richard Zowalla · 7 weeks ago
  80. 725781f Minor: Regenerated License File for cd5f59c51bdae43202c2c0f408a98fe01b99ae83 (#2030) by github-actions[bot] · 7 weeks ago
  81. b13ed81 Minor: Regenerated License File for f1e35b21261fc43b6990bae7f633c071b7947d57 (#2029) by github-actions[bot] · 7 weeks ago
  82. cd5f59c Minor: Regenerated License File for ee983434cdd00cfbf44d0c6b3d79f124942bf682 (#2028) by github-actions[bot] · 7 weeks ago
  83. f1e35b2 build(deps): bump software.amazon.awssdk:bom from 2.48.3 to 2.49.3 (#2022) by dependabot[bot] · 7 weeks ago
  84. ee98343 build(deps): bump tika.version from 3.3.1 to 3.3.2 (#2023) by dependabot[bot] · 7 weeks ago
  85. 8667217 build(deps): bump org.apache.maven.plugins:maven-jar-plugin (#2025) by dependabot[bot] · 7 weeks ago
  86. 3c06a12 Mark HTTPClient as deprecated (#2020) by Julien Nioche · 8 weeks ago
  87. 54c4a6d Allow indexer bolts to use a metadata value as the document ID (#2012) by Dávid Szigecsán · 8 weeks ago
  88. 5eb99d0 Mark CloudSearch as deprecated. Fixes #2016 (#2018) by Julien Nioche · 8 weeks ago
  89. e87846a Mark Selenium module as deprecated (#2019) by Julien Nioche · 8 weeks ago
  90. cc511f9 Resolve relative outlinks against <base href> in JSoupParserBolt (#2011) by Julien Nioche · 8 weeks ago
  91. 548726f Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest (#2005) by AsifQureshi04 · 8 weeks ago
  92. 17c14a8 build(deps): bump actions/checkout from 7.0.0 to 7.0.1 (#2010) by dependabot[bot] · 8 weeks ago
  93. 83f9063 Minor: Regenerated License File for 04871dab3d97385c13768cc9f40914589fbbb92a (#2009) by github-actions[bot] · 8 weeks ago
  94. 04871da build(deps): bump langchain4j.version from 1.17.2 to 1.18.0 (#2007) by dependabot[bot] · 8 weeks ago
  95. 18dac8c build(deps): bump software.amazon.awssdk:bom from 2.47.5 to 2.48.3 (#2008) by dependabot[bot] · 8 weeks ago
  96. 0e01b67 build(deps): bump actions/setup-java from 5.5.0 to 5.6.0 (#2003) by dependabot[bot] · 8 weeks ago
  97. 3664644 bugfix - max content not applied for robots.txt when redirecting (#2004) by Julien Nioche · 8 weeks ago
  98. eaa6869 #2001 Document adaptive scheduler metadata (#2002) by Yurii Bakurov · 8 weeks ago
  99. 86647d3 [WARC] Format WARC-IP-Address header values following the WARC community recommendations (#1997) by Sebastian Nagel · 9 weeks ago
  100. 7b92388 Minor: Regenerated License File for d4e7373f2b50d987f5f4ffe806f5d8641ae57db5 (#1999) by github-actions[bot] · 9 weeks ago