This library comes up with a new programming model we call it non-volatile object programming model, this model directly offloads object graphs into a variety of memory-like devices e.g. SSD, NVMe, Off-heap, in this way, it brings some promising features for massive data processing and high performance computing.
Features:
- In-place data storage on local non-volatile memory
- In-place generic Java object persistence
- Object graphs lazy loading & multi-process sharing
- Auto-reclaim memory resources and Mnemonic objects
- Hierarchical cache pool for massive data caching
- Extensible memory services for new device adoption and allocation optimization
- A set of non-volatile data structures (WIP)
- Minimize memory footprint on Java heap
- Reduce GC Overheads as the following chart shown (collected from Apache Spark experiments)
- [Coming major feature]: Distributed Object Graphs (DOG)
- [Coming major feature]: Columnar-aware object graphs & collections (Apache Arrow based optimization)
- [Coming major feature]: Native Massive Object Graph (NMOG) Computing
Mnemonic Way
How to use it ?
Define a Non-Volatile class:
/** * a durable class should be abstract, implement Durable interface and marked with @DurableEntity annotation */ @DurableEntity public abstract class Person<E> implements Durable, Comparable<Person<E>> { E element; // Generic Type /** * callback for this durable object creation */ @Override public void initializeAfterCreate() { System.out.println("Initializing After Created"); } /** * callback for this durable object recovery */ @Override public void initializeAfterRestore() { System.out.println("Initializing After Restored"); } /** * setup generic info manually to avoid performance penalty */ @Override public void setupGenericInfo(EntityFactoryProxy[] efproxies, GenericField.GType[] gftypes) { } @Test public void testOutput() throws RetrieveDurableEntityError { System.out.printf("Person %s, Age: %d ( %s ) \n", getName(), getAge(), null == getMother()? "No Recorded Mother" : "Has Recorded Mother"); } public int compareTo(Person<E> anotherPerson) { int ret = 0; if (0 == ret) ret = getAge().compareTo(anotherPerson.getAge()); if (0 == ret) ret = getName().compareTo(anotherPerson.getName()); return ret; } /** * Getters and Setters for non-volatile fields marked with @DurableGetter and @DurableSetter */ @DurableGetter abstract public Short getAge(); @DurableSetter abstract public void setAge(Short age); @DurableGetter abstract public String getName() throws RetrieveDurableEntityError; @DurableSetter abstract public void setName(String name, boolean destroy) throws OutOfPersistentMemory, RetrieveDurableEntityError; @DurableGetter abstract public Person<E> getMother() throws RetrieveDurableEntityError; @DurableSetter abstract public void setMother(Person<E> mother, boolean destroy) throws RetrieveDurableEntityError; @DurableGetter abstract public Person<E> getFather() throws RetrieveDurableEntityError; @DurableSetter abstract public void setFather(Person<E> mother, boolean destroy) throws RetrieveDurableEntityError; }
Use a non-volatile class:
Setup an allocator for non-volatile object graphs.
// create an allocator instance NonVolatileMemAllocator act = new NonVolatileMemAllocator(1024 * 1024 * 8, "./pobj_person.dat", true); // fetch handler store capacity from this non-volatile storage managed by this allocator KEYCAPACITY = act.handlerCapacity(); .... // close it after use act.close();
Generate structured non-volatile objects.
// create a new non-volatile person object from this specific allocator person = PersonFactory.create(act); // set attributes person.setAge((short)rand.nextInt(50)); person.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // keep this person on non-volatile handler store act.setHandler(keyidx, person.getHandler()); for (int deep = 0; deep < rand.nextInt(100); ++deep) { // create another person as mother mother = PersonFactory.create(act); mother.setAge((short)(50 + rand.nextInt(50))); mother.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // set the person's mother person.setMother(mother, true); person = mother; }
Use the non-volatile objects
for (long i = 0; i < KEYCAPACITY; ++i) { System.out.printf("----------Key %d--------------\n", i); // iterate non-volatile handlers from handler store of this specific allocator val = act.getHandler(i); if (0L == val) { break; } // restore person objects from this specific allocator Person<Integer> person = PersonFactory.restore(act, val, true); while (null != person) { person.testOutput(); // iterate all mother's ancestors person = person.getMother(); } }
How to build it ?
Please see the file LICENSE for information on how this library is licensed.
- mnemonic-core -- the submodule project for core
- mnemonic-collections -- the submodule project for generic collections
- mnemonic-examples -- the submodule project for examples
- mnemonic-memory-services/mnemonic-pmalloc-service -- the submodule project for pmalloc memory service
- mnemonic-memory-services/mnemonic-nvml-vmem-service -- the submodule project for vmem memory service
- mnemonic-memory-services/service-dist -- the location of extensive memory services (auto-generated)
To build this library, you may need to install some required packages on the build system:
- Maven -- the building tool v3.2.1 or above [Required]
- NVML -- the NVM library (Please compile this library that is tagged with 0.1+b16) (http://pmem.io) [Optional if mnemonic-nvml-vmem-service is excluded]
- JDK -- the Java Develop Kit 1.6 or above (please properly configure JAVA_HOME) [Required]
- PMFS -- the PMFS should be properly installed and configured on Linux system if you want to simulate read latency [Optional]
- PMalloc -- a supported durable memory native library at https://github.com/NonVolatileComputing/pmalloc.git [Optional if mnemonic-pmalloc-service is excluded]
Once the build system is setup, this Library is built using this command at the top level:
$ mvn clean package
To exclude a customized memory service for your platform e.g. OSX, note that if you excluded one or both memory services, some or all testcases/examples will fail since their dependent memory services are unavailable.
$ mvn -pl '!mnemonic-memory-services/mnemonic-nvml-vmem-service' clean package
To build and run the unit tests:
$ mvn clean package install -DskipTests=false
To install this package to local repository (required to run examples and testcases):
$ mvn clean install
To run an example:
$ mvn exec:exec -Pexample -pl mnemonic-examples # requires 'vmem' memory service to run, please refer to the code of test cases for more examples.
To run several test cases:
$ mvn -Dtest=DurablePersonNGTest test -pl mnemonic-core -DskipTests=false # a testcase for module "mnemonic-core" that requires 'pmalloc' memory service to pass $ mvn -Dtest=NonVolatileMemAllocatorNGTest test -pl mnemonic-core -DskipTests=false # a testcase for module "mnemonic-core" that requires 'pmalloc' memory service to pass $ mvn -Dtest=VolatileMemAllocatorNGTest test -pl mnemonic-core -DskipTests=false # a testcase for module "mnemonic-core" that requires 'vmem' memory service to pass $ mvn -Dtest=MemClusteringNGTest test -pl mnemonic-core -DskipTests=false # a testcase for module "mnemonic-core" that requires 'vmem memory service to pass $ mvn -Dtest=DurableNodeValueNGTest test -pl mnemonic-collections -DskipTests=false # a testcase for module "mnemonic-collection" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurablePersonNGTest test -pl mnemonic-collections -DskipTests=false # a testcase for module "mnemonic-collection" that requires 'pmalloc' memory service to pass