apache /
kudu /
58d2338de8d586106ea29cfa281ee653392a9145 KUDU-3371 [fs] Use RocksDB to store LBM metadata
Since the LogBlockContainerNativeMeta stores block records
sequentially in the metadata file, the live blocks maybe
in a very low ratio, so it may cause serious disk space
amplification and long bootstrap times.
This patch introduces a new class LogBlockContainerRdbMeta
which uses RocksDB to store LBM metadata, a new item will
be Put() into RocksDB when a new block is created in LBM,
and the item will be Delete() from RocksDB when the block
is removed from LBM. Data in RocksDB can be maintained by
RocksDB itself, i.e. deleted items will be GCed so it's not
needed to rewrite the metadata as how we do in
LogBlockContainerNativeMeta.
The implementation also reuses most logic of the base class
LogBlockContainer, the main difference with
LogBlockContainerNativeMeta is that LogBlockContainerRdbMeta
stores block records metadata in RocksDB rather than in a
native file. The main implementation of interfaces from
the base class including:
a. Create a container
Data file is created similar to LogBlockContainerNativeMeta,
but the metadata part is stored in RocksDB with keys
constructed as "<container_id>.<block_id>", and values are
the same to the records stored in metadata file of
LogBlockContainerNativeMeta.
b. Open a container
Similar to LogBlockContainerNativeMeta, and it's not needed
to check the metadata part, because it has been checked when
loading containers during the bootstrap phase.
c. Destroy a container
If the container is dead (full and no live blocks), remove
the data file, and clean up metadata part, by deleting all
the keys prefixed by "<container_id>".
d. Load a container (by ProcessRecords())
Iterate the RocksDB in the key range
[<container_id>, <next_container_id>), because dead blocks
have been deleted directly, thus only live block records
will be populated, we can use them as LogBlockContainerNativeMeta.
e. Create blocks in a container
Put() serialized BlockRecordPB records into RocksDB, keys
are constructed the same to the above.
f. Remove blocks from a container
Construct the keys same to the above, Delete() them from RocksDB
in batch.
This patch contains the following changes:
- Adds a new block manager type named 'logr', it uses RocksDB
to store LBM metadata. The new block manager is enabled by setting
--block_manager=logr.
- Related tests add new parameterized value to test the case
of "--block_manager=logr".
It's optional to use RocksDB, we can use the former LBM as
before, we will introduce more tools to convert data between
the two implementations in the future.
The optimization is obvious as shown in JIRA KUDU-3371, it shows that
the time spent to re-open tablet server's metadata when 99.99% of all
the records removed reduced about 9.5 times when using
LogBlockContainerRdbMeta instead of LogBlockContainerNativeMeta.
Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
Reviewed-on: http://gerrit.cloudera.org:8080/18569
Tested-by: Alexey Serbin <alexey@apache.org>
Reviewed-by: Alexey Serbin <alexey@apache.org>
32 files changed
