| commit | 4d1afeed1a4b911f7101a2177330130e4cb013b6 | [log] [tgz] |
|---|---|---|
| author | Wang, Gang(Gary) <gang1.wang@intel.com> | Tue Feb 07 10:08:14 2017 -0800 |
| committer | Wang, Gang(Gary) <gang1.wang@intel.com> | Tue Feb 07 10:14:17 2017 -0800 |
| tree | 118806566e7dbea732b5618d4953a2d2adb2572c | |
| parent | 12fb0f6c73a9969d5357a75908b8c63d54f98fe2 [diff] |
MNEMONIC-188: Verify all partitions that generated by hadoop writer in test case
================================
Apache Mnemonic is an advanced hybrid memory storages oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.
/** * a durable class should be abstract, implement Durable interface and marked with @DurableEntity annotation */ @DurableEntity public abstract class Person<E> implements Durable, Comparable<Person<E>> { E element; // Generic Type /** * callback for this durable object creation */ @Override public void initializeAfterCreate() { System.out.println("Initializing After Created"); } /** * callback for this durable object recovery */ @Override public void initializeAfterRestore() { System.out.println("Initializing After Restored"); } /** * setup generic info manually to avoid performance penalty */ @Override public void setupGenericInfo(EntityFactoryProxy[] efproxies, GenericField.GType[] gftypes) { } @Test public void testOutput() throws RetrieveDurableEntityError { System.out.printf("Person %s, Age: %d ( %s ) \n", getName(), getAge(), null == getMother()? "No Recorded Mother" : "Has Recorded Mother"); } public int compareTo(Person<E> anotherPerson) { int ret = 0; if (0 == ret) ret = getAge().compareTo(anotherPerson.getAge()); if (0 == ret) ret = getName().compareTo(anotherPerson.getName()); return ret; } /** * Getters and Setters for non-volatile fields marked with @DurableGetter and @DurableSetter */ @DurableGetter(Id = 1L) abstract public Short getAge(); @DurableSetter abstract public void setAge(Short age); @DurableGetter(Id = 2L) abstract public String getName() throws RetrieveDurableEntityError; @DurableSetter abstract public void setName(String name, boolean destroy) throws OutOfPersistentMemory, RetrieveDurableEntityError; @DurableGetter(Id = 3L) abstract public Person<E> getMother() throws RetrieveDurableEntityError; @DurableSetter abstract public void setMother(Person<E> mother, boolean destroy) throws RetrieveDurableEntityError; @DurableGetter(Id = 4L) abstract public Person<E> getFather() throws RetrieveDurableEntityError; @DurableSetter abstract public void setFather(Person<E> mother, boolean destroy) throws RetrieveDurableEntityError; }
// create an allocator instance NonVolatileMemAllocator act = new NonVolatileMemAllocator(1024 * 1024 * 8, "./pobj_person.dat", true); // fetch handler store capacity from this non-volatile storage managed by this allocator KEYCAPACITY = act.handlerCapacity(); .... // close it after use act.close();
// create a new non-volatile person object from this specific allocator person = PersonFactory.create(act); // set attributes person.setAge((short)rand.nextInt(50)); person.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // keep this person on non-volatile handler store act.setHandler(keyidx, person.getHandler()); for (int deep = 0; deep < rand.nextInt(100); ++deep) { // create another person as mother mother = PersonFactory.create(act); mother.setAge((short)(50 + rand.nextInt(50))); mother.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // set the person's mother person.setMother(mother, true); person = mother; }
for (long i = 0; i < KEYCAPACITY; ++i) { System.out.printf("----------Key %d--------------\n", i); // iterate non-volatile handlers from handler store of this specific allocator val = act.getHandler(i); if (0L == val) { break; } // restore person objects from this specific allocator Person<Integer> person = PersonFactory.restore(act, val, true); while (null != person) { person.testOutput(); // iterate all mother's ancestors person = person.getMother(); } }
// fetch print service GeneralComputingService gcsvr = Utils.getGeneralComputingService("print"); // instantiate a value info for a value matrix ValueInfo vinfo = new ValueInfo(); // instantiate a object stack List<long[][]> objstack = new ArrayList<long[][]>(); // fill up with all durable object info in order objstack.add(firstnv.getNativeFieldInfo()); objstack.add(person.getNativeFieldInfo()); // configure the Id stack for each level of durable objects long[][] fidinfostack = {{2L, 1L}, {0L, 1L}}; // configure the handler of a value matrix vinfo.handler = handler; // set translate table from handler's allocator vinfo.transtable = m_act.getTranslateTable(); // specify the durable type of value vinfo.dtype = DurableType.SHORT; // generate frames for this value matri from both stacks vinfo.frames = Utils.genNativeParamForm(objstack, fidinfostack); // form an array of value infos ValueInfo[] vinfos = {vinfo}; // perform the print operation gcsvr.perform(vinfos);
Please see the file LICENSE for information on how this library is licensed.
| Features | NVML-VMEM | PMALLOC | NVML-PMEM | SYS-VMEM |
|---|---|---|---|---|
| Fixed Durable K-V Store | NA | O | O | NA |
| Support DOM | O | O | O | O |
| Support DNCM | O | O | O | O |
| Support OS X | NA | O | NA | O |
| Memory Map Sync. | NA | O | O | NA |
| PM Flush | NA | NA | O | NA |
| PM Drain | NA | NA | O | NA |
| PM Persist | NA | NA | O | NA |
| PM Atomic Ops. | NA | NA | O | NA |
| Expected Performance | Average | Average | Slow(on Disk) | Fast |
To build this library, you may need to install some required packages on the build system:
Once the build system is setup, this Library is built using this command at the top level:
$ git clean -xdf # if pull from a git repo. $ mvn clean package install
To exclude a customized memory service for your platform e.g. OSX, note that if you excluded one or both memory services, some or all testcases/examples will fail since their dependent memory services are unavailable.
$ git clean -xdf # if pull from a git repo. $ mvn -pl '!mnemonic-memory-services/mnemonic-nvml-vmem-service' clean package install
To install this package to local repository (required to run examples and testcases):
$ mvn clean install
To run an example:
$ # requires 'vmem' memory service to run, please refer to the code of test cases for more examples. $ mvn exec:exec -Pexample -pl mnemonic-examples
To run several test cases:
$ # a testcase for module "mnemonic-core" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurablePersonNGTest test -pl mnemonic-core -DskipTests=false $ # a testcase for module "mnemonic-core" that requires 'pmalloc' memory service to pass $ mvn -Dtest=NonVolatileMemAllocatorNGTest test -pl mnemonic-core -DskipTests=false $ # a testcase for module "mnemonic-core" that requires 'vmem' memory service to pass $ mvn -Dtest=VolatileMemAllocatorNGTest test -pl mnemonic-core -DskipTests=false $ # a testcase for module "mnemonic-core" that requires 'vmem memory service to pass $ mvn -Dtest=MemClusteringNGTest test -pl mnemonic-core -DskipTests=false $ # a testcase for module "mnemonic-collection" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurableSinglyLinkedListNGTest test -pl mnemonic-collections -DskipTests=false $ # a testcase for module "mnemonic-collection" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurablePersonNGTest test -pl mnemonic-collections -DskipTests=false $ # a testcase for module "mnemonic-computing-services/mnemonic-utilities-service" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurableSinglyLinkedListNGPrintTest test -pl mnemonic-computing-services/mnemonic-utilities-service -DskipTests=false $ # a testcase for module "mnemonic-computing-services/mnemonic-utilities-service" that requires 'pmalloc' memory service to pass $ mvn -Dtest=DurableSinglyLinkedListNGSortTest test -pl mnemonic-computing-services/mnemonic-utilities-service -DskipTests=false
To run sort bench workloads.
$ # generate some input data files with a count parameter that indicates how many random numbers to be generated $ mnemonic-benches/mnemonic-sort-bench/bin/gen_data.py 20000 $ # create a configure file that contains all the absolute paths of generated input data files to work on in a batch $ # call the run.py to run the bench workloads with the configure file $ mnemonic-benches/mnemonic-sort-bench/bin/run.py ./sort-files.conf $ # after finished, you can find the result data from the sort_bench_result.log $ less mnemonic-benches/mnemonic-sort-bench/sort_bench_result.log