Decouple out-of-core persistence infrastructure from out-of-core computation
Summary:
This diff proposes the following:
- The persistence layer is decoupled from out-of-core infrastructure. This way one can simply implement different data accessors for various persistence resources. The persistence layer for reading/writing from/to local file system is implemented in this diff.
- Previously, out-of-core data were indexed by string literals. This has changed for more flexibility. Now, data are accessible by a more flexible data indexing mechanism, in which a chain of indices are used to address a particular data.
- With different implementations of data accessor, now there may be more emphasis on having more IO threads. It is important that these IO threads are load-balanced. In this diff, the mechanism to assign partitions to IO threads has changed.
- All the coolness of Kryo's (de)serialization and RandomAccessFile (in D59277) is included in this diff, all at one place.
Test Plan:
mvn clean verify
out-of-core snapshot test passes
Reviewers: dionysis.logothetis, maja.kabiljo, sergey.edunov
Differential Revision: https://reviews.facebook.net/D59691
36 files changed
