commit | 6b892a45f02a6d8d0b1d33235d4134d91e3ea111 | [log] [tgz] |
---|---|---|
author | Rajan Dhabalia <rdhabalia@apache.org> | Fri Jul 12 01:03:10 2019 -0700 |
committer | Enrico Olivelli <eolivelli@gmail.com> | Fri Jul 12 10:03:10 2019 +0200 |
tree | 347878cde63f07ebb10d683a83bd4c28db7af8f5 | |
parent | 762a61365408d81a20193ef4ae1d44394746c105 [diff] |
[BK-SERVER] Clean up over-replicated ledgers owned by different bookies ### Motivation As described at: https://github.com/apache/pulsar/issues/4632 - Sometimes due to overreplication, bookie contains ledgers which are not owned by that bookie anymore and that bookie is not part of the ensemble-list of those ledgers. In this case, GC finds out those overreplicated ledgers and - deletes their index from dbStorage (rocksDB) and - tries to delete them from entrylog files. However, bookie doesn't delete them from entry-log files due to change made in [#870](https://github.com/apache/bookkeeper/issues/870) where bookie avoids deleting ledger if znode of that ledger exists. Because of that bookie ends up storing large number entrylog files with ledgers which are owned by different bookies. It also cause OOM when GC tries to deal with large number of entry log files. ### Modification Delete the ledgers if bookie is not part of ensemble list of over-replicated ledgers Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <sijie@apache.org> This closes #2119 from rdhabalia/overRepl
Apache BookKeeper is a scalable, fault tolerant and low latency storage service optimized for append-only workloads.
It is suitable for being used in following scenarios:
You can also read Turning Ledgers into Logs to learn how to turn ledgers into continuous log streams. If you are looking for a high level log stream API, you can checkout DistributedLog.
For filing bugs, suggesting improvements, or requesting new features, help us out by opening a Github issue or opening an Apache jira.
Subscribe or mail the user@bookkeeper.apache.org list - Ask questions, find answers, and also help other users.
Subscribe or mail the dev@bookkeeper.apache.org list - Join development discussions, propose new ideas and connect with contributors.
Join us on Slack - This is the most immediate way to connect with Apache BookKeeper committers and contributors.
We feel that a welcoming open community is important and welcome contributions.
See Developer Setup to get your local environment setup.
Take a look at our open issues: JIRA Issues Github Issues.
Review our coding style and follow our pull requests to learn about our conventions.
Make your changes according to our contribution guide.