tree ed997318ba59c1901d338885d5bcce7850716036
parent 06b227122fac64533dd7e8912ae24ad2bac41c87
author Sebastian Nagel <snagel@apache.org> 1588244285 +0200
committer Sebastian Nagel <snagel@apache.org> 1597603258 +0200

NUTCH-2002 parse and index checkers to check robots.txt
- applied Julien's patch to recent code base
- also check redirects whether they are allowed
- add command-line parameter `-checkRobotsTxt` enabling this check