tree 956acdc2823a3b0082c627693d56ac9a3fd9763a
parent 41e88f874132a6bcae3dd034547b735b6a8a4c12
author Michael Bien <mbien42@gmail.com> 1683544649 +0200
committer GitHub <noreply@github.com> 1683544649 +0200
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJkWNpJCRBK7hj4Ov3rIwAA9n4IAJIewEWPDz/k2/w/1QuAETql
 czN/D+Yvsg4Q2AMYU0owz6RWWwszAohGOGZqZgkb62X+VFa+KkZeCCtH+y8sR0v4
 I7aXtbum3yFwokIoruNSXuLS/PHtEPkfkP5vD7IRefVFxomFBjvtWg9Yp4TuzutV
 zlSeT/Hd3qh8d8t0MO4vbYKpfFZyzrzhnCj7z6x/lVgmUmVOUfhpSJsI0N7lDHry
 5PlKz50iEZVQa73e9OQpY+ZgZ4phZW44QYjL8xqhGaQpDdAH91O3v8l+fdzYy0DF
 Q/cdl51iVJHO5wpAAHrVmu/Ty8Cfa12rSNts/lIt8MJBOfJC+ZPiWqIx1n6NvMw=
 =zSK8
 -----END PGP SIGNATURE-----
 

[MINDEXER-185] Filter in index reader and use update request for configuration (#302)

This PR does two things:
 - uses `IndexUpdateRequest` as configuration for `IndexDataReader` (tmp folder, factory etc)
 - moves the filtering from post extraction to the read phase
   - makes filtering really fast (and multi threaded too), it is no longer an extra step
   - has actually an effect on the on-disk index size (since I've learned lucene doesn't really remove things since all files are immutable)
 
I do realize that this is not quite the same behavior as before. To retain the exact same behavior, we could add this as additional filter, one during read (new), one after extraction (old). (edit: done, see second commit)

example filter:
```java
      final Instant cutoff = ZonedDateTime.now().minusYears(2).toInstant();
      iur.setExtractionFilter((doc) -> {
          IndexableField field = doc.getField("m"); // usually never null
          return field != null && Instant.ofEpochMilli(Long.parseLong(field.stringValue())).isAfter(cutoff);
      });
```
results (single threaded, since MT has a index size penalty due to merge overhead):
```
full: 5.6 GB
2y: 2.6 GB
1y: 1.4 GB
```

---

https://issues.apache.org/jira/browse/MINDEXER-185