| commit | c51c9a21b390792252df191c82309c19d7dc8b8f | [log] [tgz] |
|---|---|---|
| author | Nupur <nupurs@ingeniumsys.com> | Wed Oct 28 15:03:25 2015 +0530 |
| committer | Atri Sharma <atri@apache.org> | Thu Nov 05 17:14:44 2015 +0530 |
| tree | 2926f229863abcd7fe606f0ffbf4fbc2fa241120 | |
| parent | af37c9fe88aa45deadb829b3c4206fbbd02c864f [diff] |
CONCERTED-5:Improve and automated tests
Concerted is a next generation big data engine aimed at supporting massive in memory reads for OLAP support.
The IRC channel is #concerted on Freenode.
The engine is flexible, with no single point of entry and flexibility to use APIs suited for each use case with full scalability.
Concerted is fully ACID compliant.
Concerted allows applications to use the native APIs to have the flexibility to store, access, scale data in memory on demand. No auxillary infrastructure is needed.
Concerted implements its own locking manager and transaction manager utilizing the object oriented model, thus greatly simplifying the management of processes like commit and abort, depending a lot on the OS's native support.
Concerted has to be built with g++, since it uses g++'s built ins.
For running the tests, pthread library is required. It is not required for building or using Concerted itself.
Concerted was built with the principal of “Write Less Read Many”. Although tapes are no longer much in use, the principal of highly scalable reads still applies to big data. In memory scaling of reads can help OLAP engines to have massive read mostly data in memory available on demand and for a high number of concurrent readers.
Imagine a star schema. OLAP engine would like fact table to be present in memory and optimized for high analytical reads. Concerted fills that gap.
Concerted has storage layouts which allow pre aggregation and multi dimensional view of data.
And yes, Concerted's own footprint will not affect your application size!
Concerted works on the principle of Plug and Play. Instead of providing a fixed engine and structure to mould your data and requirements in, Concerted‘s dynamic APIs allow you to use Concerted as and when required in your core application while ensuring massive scalability and all of Concerted’s features whenever you use them!
The way of building Concerted is executing below command:
Cleaning is below:
make all builds and makes libconcerted.a which is present in build folder and should be linked in application using Concerted.
The components of Concerted are:
Locking manager: The locking manager is the component of Concerted which handles all the locking and concurrency in Concerted.
Data store: The data store is the part of Concerted which stores the data and reads and supports queries on the data.
Cache: The cache brings in the data in pages of a fixed size from the disk.
Transaction Manager: The Transaction Manager handles the transactions and the ‘all-or-nothing’ aspect of the transactions and operations.
The locking manager manages the locking in Concerted. It manages the locking with read and write locks and lock upgradation.
The locking manager has locking queues and spinlocks. A lock is granted, and any other request for the lock while the lock is being held by some thread are added to a queue with a flag. The waiting thread checks the value of the flag and once the value of the flag becomes a specific value, the thread can proceed with its operations as it has been granted the lock.
After completing the tasks, the thread which holds the lock currently calls ReleaseLock to release the lock from itself.
The locking manager is designed with NUMA in mind. Hence, the flags for checking whether the lock the thread is waiting for has been granted or not are distributed, so that threads can access the flag in their cache lines, so that the lookup is less expensive.
The data store is the component of Concerted which holds the data present in the database. The data is stored in various data structures and written to disk. The data from a specific data structure is read from disk into main memory when queries based on it are present.
A cache is present which reads the data from disk and keeps it in main memory. The cache has LRU based page replacement algorithm.
The Transaction Manager manages the transactions taking place inside the database and ensures consistency in the data. The transaction manager makes all the changes done to the data in a local copy of the data and writes the changes to the main data store only when commit takes place. Commit takes place when all the operations complete successfully.
For more examples of usage, please see the tests included.
For running any test present in the Tests folder, please use:
g++ [name of code file to compile] -lpthread -lrt