tree ed17a02648609c42e0b7413a70aa4f14bf30baf2
parent fa319a60f30dbb0efcd67e306c611d66b7b379f1
author Sebastian Nagel <snagel@apache.org> 1589563020 +0200
committer Sebastian Nagel <snagel@apache.org> 1597603258 +0200

NUTCH-2496 Speed up link inversion step in crawling script

- disable URL filtering and normalizing when calling invertlinks
  in bin/crawl

- add note that the steps invertlinks, dedup, index could also
  be done outside the loop over all segments created in the loop
  iterations

- move webgraph construction (commented out anyway) outside the
  loop because it's done over all available segments