Google Git
Sign in
apache / lucene / aac856a831af114541346cc53c5cfe3fa1fb7208
commitaac856a831af114541346cc53c5cfe3fa1fb7208[log] [tgz]
authorAdrien Grand <jpountz@gmail.com>Tue May 21 09:12:44 2024 +0200
committerGitHub <noreply@github.com>Tue May 21 09:12:44 2024 +0200
treeadc33fccc89ed62b6dcf40202aae4063384c145f
parent22d50be2eab84c9a75ea65f55fcc4356724faba1 [diff]
Reduce the overhead of `IndexInput#prefetch` when data is cached in RAM. (#13381)

As Robert pointed out and benchmarks confirmed, there is some (small) overhead
to calling `madvise` via the foreign function API, benchmarks suggest it is in
the order of 1-2us. This is not much for a single call, but may become
non-negligible across many calls. Until now, we only looked into using
prefetch() for terms, skip data and postings start pointers which are a single
prefetch() operation per segment per term.

But we may want to start using it in cases that could result into more calls to
`madvise`, e.g. if we start using it for stored fields and a user requests 10k
documents. In #13337, Robert wondered if we could take advantage of `mincore()`
to reduce the overhead of `IndexInput#prefetch()`, which is what this PR is
doing via `MemorySegment#isLoaded()`.

`IndexInput#prefetch` tracks consecutive hits on the page cache and calls
`madvise` less and less frequently under the hood as the number of consecutive
cache hits increases.
3 files changed
tree: adc33fccc89ed62b6dcf40202aae4063384c145f
  1. .github/
  2. buildSrc/
  3. dev-docs/
  4. dev-tools/
  5. gradle/
  6. help/
  7. lucene/
  8. .asf.yaml
  9. .dir-locals.el
  10. .git-blame-ignore-revs
  11. .gitattributes
  12. .gitignore
  13. .hgignore
  14. .lift.toml
  15. build.gradle
  16. CONTRIBUTING.md
  17. gradlew
  18. gradlew.bat
  19. LICENSE.txt
  20. NOTICE.txt
  21. README.md
  22. settings.gradle
  23. versions.lock
  24. versions.props
README.md

Apache Lucene

Lucene Logo

Apache Lucene is a high-performance, full-featured text search engine library written in Java.

Build Status

Online Documentation

This README file only contains basic setup instructions. For more comprehensive documentation, visit:

Building

Basic steps:

  1. Install OpenJDK 21.
  2. Clone Lucene's git repository (or download the source distribution).
  3. Run gradle launcher script (gradlew).

We‘ll assume that you know how to get and set up the JDK - if you don’t, then we suggest starting at https://jdk.java.net/ and learning more about Java, before returning to this README.

See Contributing Guide for details.

Contributing

Bug fixes, improvements and new features are always welcome! Please review the Contributing to Lucene Guide for information on contributing.

Discussion and Support