tree ee2c9c06943181f0faba962de674710377b37f37
parent 5644f9cd20cf436e631a84586af80e9da1935da8
author Yossi Tamari <yossi.tamari@pipl.com> 1557246172 +0300
committer Sebastian Nagel <snagel@apache.org> 1558703578 +0200

NUTCH-2716 Response headers are not stored for a compressed response

Even when store.http.headers=true, the HTTP headers are not saved for a
gzipped or deflated response, because they may contain an incorrect
content-length header.
This causes WARCExporter to generate "resource" (header-less) entries
instead of "response" entries.
The correct behaviour is to store all the headers, and code that uses
them should be aware and careful that they represent the original
headers, not the stored content.

This fixes protocol-http, protocol-selenium, and protocol-htmlunit to
write the raw response headers, and adds logic to WARCExporter and
CommonCrawlDataDumper to fix these headers.

It also fixed NUTCH-2715 (WARCExporter fails on large records), and
upgrades lib-htmlunit to use version 3.141.5 of Selenium, since Eclipse
fails to compile otherwise (conflicts with lib-selenium).