Escape CR and LF in WARC metadata records and resource Content-Type (#2109)

* Escape CR and LF in WARC metadata records and resource Content-Type

MetadataRecordFormat.format() wrote one line per metadata value into the
application/warc-fields payload without checking for CR or LF. A value
containing CR LF (e.g. feed.description set by FeedParserBolt, or values
of parse.* filters such as the XPath, LDJson and Tika filters) therefore
became additional field lines that look exactly like fields written by
the crawler - for example a fabricated hopsFromSeed or via. Framing
stayed valid because Content-Length is computed from the finished
payload, so WARC readers had no way to detect the injected fields.

- replace CR and LF by spaces in metadata values written into the
  warc-fields payload, and drop metadata keys that are not valid WARC
  field names (printable ASCII without colon, RFC 5322 section 2.2)
- sanitise the server-supplied Content-Type used for resource records in
  WARCRecordFormat.format(), which was appended verbatim into the WARC
  header block
- log MetadataRecordFormat messages under MetadataRecordFormat instead
  of WARCRequestRecordFormat

Fixes #2105

* Validate warc.metadata.keys once on instantiation

The configured metadata keys are fixed topology configuration: check
them for valid WARC field names in the constructor and drop invalid
keys with a single warning, instead of repeating the check and warning
for every record written. Suggested in review.

* Make null check in isValidWarcFieldName explicit

The conjunction already returned false for a null name through
short-circuit evaluation, but the intent was easy to miss. Return false
explicitly and cover the field name and value sanitisation helpers with
unit tests. Suggested in review.
4 files changed
tree: faef42b1a6aa3a33215f746b5075904080156660
  1. .github/
  2. .mvn/
  3. archetype/
  4. core/
  5. docs/
  6. external/
  7. .asf.yaml
  8. .editorconfig
  9. .gitattributes
  10. .gitignore
  11. AGENTS.md
  12. assembly.xml
  13. checkstyle.xml
  14. CONTRIBUTING.md
  15. DISCLAIMER-BINARIES.txt
  16. LICENSE
  17. NOTICE
  18. pom.xml
  19. README.md
  20. RELEASING.md
  21. SECURITY.md
  22. THIRD-PARTY.properties
  23. THIRD-PARTY.txt
README.md

StormCrawler

license Build Status javadoc

Apache StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.

Quickstart

NOTE: These instructions assume that you have Apache Maven installed. You will need to install Apache Storm 3.0.0 to run the crawler.

StormCrawler requires Java 25 or above. To execute tests, it requires you to have a locally installed and working Docker environment.

Once Storm is installed, the easiest way to get started is to generate a new StormCrawler project following the instructions below:

mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=<CURRENT_VERSION>

Be sure to replace <CURRENT_VERSION> with the latest released version, which you can find on search.maven.org.

You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version, a package name and details about the user agent to use.

This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.

Alternatively if you can‘t or don’t want to use the Maven archetype above, you can simply copy the files from archetype-resources.

Have a look at crawler.flux, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.

Getting help

The documentation is a good place to start your investigations but if you are stuck please use the tag stormcrawler on StackOverflow or ask a question in the discussions section.

The project website has a page listing companies providing commercial support for Apache StormCrawler.

Note for developers

Please format your code before submitting a PR with

mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false

You can enable pre-commit format hooks by running:

mvn clean install -Dskip.format.code=false

Building from source

The requirements for building from source are as follows

  • JDK 25+
  • Apache Maven 3
  • Docker (if you want to run tests)

The build itself is straightforward:

mvn clean install

Note: We use some binary files for testing advanced crawler functionality. These files are located exclusively in the src/test directories of the respective modules.

Thanks

alt tag

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.