tree 6d17459df6baa8314e372acd40a246755d120b49
parent b5541455e0c6e2da4a8539ce61d2f05fc68f472c
author Sebastian Nagel <sebastian@commoncrawl.org> 1557906822 +0200
committer Sebastian Nagel <snagel@apache.org> 1573117474 +0100

NUTCH-2746 Basic URL normalizer to normalize Unicode domain names
- convert domain names IDN to ASCII or ASCII to IDN
  (configured by urlnormalizer.basic.host.idn)
- strip trailing dot in host names
  (if urlnormalizer.basic.host.trim-trailing-dot is true)