<?xml version="1.0"?> | |
<document> | |
<properties> | |
<title>JCS vs EHCache Performance</title> | |
<author email="asmuts@apache.org">Aaron Smuts</author> | |
</properties> | |
<body> | |
<section name="JCS vs EHCache Memory Performance"> | |
<subsection name="Initial Test Results"> | |
<p> | |
I just built both EHCache (1.2-beta4) and JCS | |
(1.2.7.0) from head, configured both similarly and | |
ran 20 rounds of 50,000 puts and gets, that is | |
1,000,000 puts and gets in total. Using the default | |
LRU Memory Cache, the same algorithm that EHCache | |
uses by default, | |
<b> | |
JCS proved to be nearly twice as fast as EHCache | |
</b> | |
in multiple trials for both puts and gets. I have | |
the log levels for both set at info. I would like to | |
further verify my results, since they completely | |
contradict the information on the EHCache site. | |
</p> | |
<p> | |
From what I can tell so far, JCS is significantly | |
faster than EHCache when you are retrieving items | |
that exist in the cache and when you are putting | |
items into a cache that has not reached its size | |
limit. | |
</p> | |
<p> | |
Additional testing shows that when the size limit it | |
reached, JCS and EHCache perform similarly for puts | |
and gets. Although JCS gets are significantly faster | |
when the items are present, they are almost exactly | |
the same when the items are not in the cache. My | |
initial tests revealed a less than 1% difference, | |
but subsequent runs showed JCS as 20% faster. More | |
tests are needed before the results are conclusive. | |
</p> | |
<p> | |
Since, neither cache will be a relevant bottleneck | |
in any application where a cache would be useful, | |
the differences in performance may be beside the | |
point. Nevertheless, it is important to note that | |
the EHCache web site provides, what appears to be, | |
false test data. | |
</p> | |
<p> | |
The peculiar result is that a few years back EHCache | |
took the JCS source code, removed most of its | |
features, and ended up with something that performs | |
worse. | |
</p> | |
</subsection> | |
<subsection name="Test Data"> | |
<p>Here is the data from the first test:</p> | |
<p> | |
JCS put time for 50000 = 651; millis per = 0.01302 | |
JCS get time for 50000 = 160; millis per = 0.0032 | |
EHCache put time for 50000 = 481; millis per = | |
0.00962 EHCache get time for 50000 = 110; millis per | |
= 0.0022 | |
</p> | |
<p> | |
JCS put time for 50000 = 240; millis per = 0.0048 | |
JCS get time for 50000 = 90; millis per = 0.0018 | |
EHCache put time for 50000 = 491; millis per = | |
0.00982 EHCache get time for 50000 = 120; millis per | |
= 0.0024 | |
</p> | |
<p> | |
JCS put time for 50000 = 241; millis per = 0.00482 | |
JCS get time for 50000 = 80; millis per = 0.0016 | |
EHCache put time for 50000 = 551; millis per = | |
0.01102 EHCache get time for 50000 = 110; millis per | |
= 0.0022 | |
</p> | |
<p> | |
JCS put time for 50000 = 240; millis per = 0.0048 | |
JCS get time for 50000 = 90; millis per = 0.0018 | |
EHCache put time for 50000 = 481; millis per = | |
0.00962 EHCache get time for 50000 = 130; millis per | |
= 0.0026 | |
</p> | |
<p> | |
JCS put time for 50000 = 230; millis per = 0.0046 | |
JCS get time for 50000 = 181; millis per = 0.00362 | |
EHCache put time for 50000 = 520; millis per = | |
0.0104 EHCache get time for 50000 = 101; millis per | |
= 0.00202 | |
</p> | |
<p> | |
JCS put time for 50000 = 220; millis per = 0.0044 | |
JCS get time for 50000 = 90; millis per = 0.0018 | |
EHCache put time for 50000 = 641; millis per = | |
0.01282 EHCache get time for 50000 = 110; millis per | |
= 0.0022 | |
</p> | |
<p> | |
JCS put time for 50000 = 250; millis per = 0.0050 | |
JCS get time for 50000 = 121; millis per = 0.00242 | |
EHCache put time for 50000 = 590; millis per = | |
0.0118 EHCache get time for 50000 = 101; millis per | |
= 0.00202 | |
</p> | |
<p> | |
JCS put time for 50000 = 260; millis per = 0.0052 | |
JCS get time for 50000 = 100; millis per = 0.0020 | |
EHCache put time for 50000 = 581; millis per = | |
0.01162 EHCache get time for 50000 = 100; millis per | |
= 0.0020 | |
</p> | |
<p> | |
JCS put time for 50000 = 290; millis per = 0.0058 | |
JCS get time for 50000 = 121; millis per = 0.00242 | |
EHCache put time for 50000 = 570; millis per = | |
0.0114 EHCache get time for 50000 = 121; millis per | |
= 0.00242 | |
</p> | |
<p> | |
JCS put time for 50000 = 210; millis per = 0.0042 | |
JCS get time for 50000 = 120; millis per = 0.0024 | |
EHCache put time for 50000 = 561; millis per = | |
0.01122 EHCache get time for 50000 = 130; millis per | |
= 0.0026 | |
</p> | |
<p> | |
JCS put time for 50000 = 250; millis per = 0.0050 | |
JCS get time for 50000 = 151; millis per = 0.00302 | |
EHCache put time for 50000 = 560; millis per = | |
0.0112 EHCache get time for 50000 = 111; millis per | |
= 0.00222 | |
</p> | |
<p> | |
JCS put time for 50000 = 250; millis per = 0.0050 | |
JCS get time for 50000 = 100; millis per = 0.0020 | |
EHCache put time for 50000 = 711; millis per = | |
0.01422 EHCache get time for 50000 = 100; millis per | |
= 0.0020 | |
</p> | |
<p> | |
JCS put time for 50000 = 251; millis per = 0.00502 | |
JCS get time for 50000 = 90; millis per = 0.0018 | |
EHCache put time for 50000 = 511; millis per = | |
0.01022 EHCache get time for 50000 = 90; millis per | |
= 0.0018 | |
</p> | |
<p> | |
JCS put time for 50000 = 220; millis per = 0.0044 | |
JCS get time for 50000 = 100; millis per = 0.0020 | |
EHCache put time for 50000 = 491; millis per = | |
0.00982 EHCache get time for 50000 = 90; millis per | |
= 0.0018 | |
</p> | |
<p> | |
JCS put time for 50000 = 230; millis per = 0.0046 | |
JCS get time for 50000 = 80; millis per = 0.0016 | |
EHCache put time for 50000 = 201; millis per = | |
0.00402 EHCache get time for 50000 = 390; millis per | |
= 0.0078 | |
</p> | |
<p> | |
JCS put time for 50000 = 201; millis per = 0.00402 | |
JCS get time for 50000 = 120; millis per = 0.0024 | |
EHCache put time for 50000 = 180; millis per = | |
0.0036 EHCache get time for 50000 = 411; millis per | |
= 0.00822 | |
</p> | |
<p> | |
JCS put time for 50000 = 210; millis per = 0.0042 | |
JCS get time for 50000 = 100; millis per = 0.0020 | |
EHCache put time for 50000 = 210; millis per = | |
0.0042 EHCache get time for 50000 = 381; millis per | |
= 0.00762 | |
</p> | |
<p> | |
JCS put time for 50000 = 240; millis per = 0.0048 | |
JCS get time for 50000 = 90; millis per = 0.0018 | |
EHCache put time for 50000 = 211; millis per = | |
0.00422 EHCache get time for 50000 = 410; millis per | |
= 0.0082 | |
</p> | |
<p> | |
JCS put time for 50000 = 221; millis per = 0.00442 | |
JCS get time for 50000 = 80; millis per = 0.0016 | |
EHCache put time for 50000 = 210; millis per = | |
0.0042 EHCache get time for 50000 = 411; millis per | |
= 0.00822 | |
</p> | |
<p> | |
JCS put time for 50000 = 220; millis per = 0.0044 | |
JCS get time for 50000 = 80; millis per = 0.0016 | |
EHCache put time for 50000 = 190; millis per = | |
0.0038 EHCache get time for 50000 = 411; millis per | |
= 0.00822 | |
</p> | |
<p>Finished 20 loops of 50000 gets and puts</p> | |
<p> | |
Put average for JCS = 256 Put average for EHCache = | |
447 JCS puts took 0.57270694 times the EHCache , the | |
goal is less than 1.0x | |
</p> | |
<p> | |
Get average for JCS = 107 Get average for EHCache = | |
196 JCS gets took 0.54591835 times the EHCache , the | |
goal is less than 1.0x | |
</p> | |
</subsection> | |
<subsection name="A Test Class"> | |
<p>Here is the test class:</p> | |
<source> | |
<![CDATA[ | |
package org.apache.jcs; | |
import junit.framework.TestCase; | |
import net.sf.ehcache.Cache; | |
import net.sf.ehcache.CacheManager; | |
import net.sf.ehcache.Element; | |
import org.apache.commons.logging.Log; | |
import org.apache.commons.logging.LogFactory; | |
import org.apache.jcs.engine.CompositeCacheAttributes; | |
import org.apache.jcs.engine.behavior.ICompositeCacheAttributes; | |
import org.apache.jcs.utils.struct.LRUMap; | |
/** | |
* Compare JCS vs ehcache performance. | |
* | |
* @author Aaron Smuts | |
* | |
*/ | |
public class JCSvsEHCachePerformanceTest | |
extends TestCase | |
{ | |
float ratioPut = 0; | |
float ratioGet = 0; | |
// the jcs to competitor | |
float target = 1.0f; | |
int loops = 20; | |
int tries = 50000; | |
/** | |
* Compare performance between JCS and EHCache. Fail if JCS is not as fast. | |
* Print the ratio. | |
* | |
* @throws Exception | |
* | |
*/ | |
public void testJCSvsEHCache() | |
throws Exception | |
{ | |
Log log = LogFactory.getLog( LRUMap.class ); | |
if ( log.isDebugEnabled() ) | |
{ | |
System.out.println( "The log level must be at info or above for the a performance test." ); | |
return; | |
} | |
doWork(); | |
assertTrue( this.ratioPut < target ); | |
assertTrue( this.ratioGet < target ); | |
} | |
/** | |
* This runs a series of gets and puts for both JCS and EHCache. The test | |
* will fail if JCS is not faster. | |
* | |
* @throws Exception | |
* | |
*/ | |
public void doWork() | |
throws Exception | |
{ | |
int maxSize = 1000000; | |
// create the two caches. | |
CacheManager ehMgr = CacheManager.getInstance(); | |
// Create an ehcache with a max size of maxSize, no swap, with items | |
// that can expire, with maximum idle time to live of 500 seconds, and | |
// maximum idel time of 500 seconds. | |
Cache eh = new Cache( "testJCSvsEHCache", maxSize, false, false, 500, 500 ); | |
ehMgr.addCache( eh ); | |
// Create a similarly configured JCS that uses the LRU memory cache. | |
// maxSize elements that are not eternal. No disk cache is configured. | |
ICompositeCacheAttributes cattr = new CompositeCacheAttributes(); | |
cattr.setMaxObjects( maxSize ); | |
JCS jcs = JCS.getInstance( "testJCSvsEHCache", cattr ); | |
// run settings | |
long start = 0; | |
long end = 0; | |
long time = 0; | |
float tPer = 0; | |
long putTotalJCS = 0; | |
long getTotalJCS = 0; | |
long putTotalEHCache = 0; | |
long getTotalEHCache = 0; | |
String jcsDisplayName = "JCS"; | |
String ehCacheDisplayName = ""; | |
try | |
{ | |
for ( int j = 0; j < loops; j++ ) | |
{ | |
jcsDisplayName = "JCS "; | |
start = System.currentTimeMillis(); | |
for ( int i = 0; i < tries; i++ ) | |
{ | |
jcs.put( "key:" + i, "data" + i ); | |
} | |
end = System.currentTimeMillis(); | |
time = end - start; | |
putTotalJCS += time; | |
tPer = Float.intBitsToFloat( (int) time ) / Float.intBitsToFloat( tries ); | |
System.out | |
.println( jcsDisplayName + " put time for " + tries + " = " + time + "; millis per = " + tPer ); | |
start = System.currentTimeMillis(); | |
for ( int i = 0; i < tries; i++ ) | |
{ | |
jcs.get( "key:" + i ); | |
} | |
end = System.currentTimeMillis(); | |
time = end - start; | |
getTotalJCS += time; | |
tPer = Float.intBitsToFloat( (int) time ) / Float.intBitsToFloat( tries ); | |
System.out | |
.println( jcsDisplayName + " get time for " + tries + " = " + time + "; millis per = " + tPer ); | |
// ///////////////////////////////////////////////////////////// | |
ehCacheDisplayName = "EHCache "; | |
start = System.currentTimeMillis(); | |
for ( int i = 0; i < tries; i++ ) | |
{ | |
Element ehElm = new Element( "key:" + i, "data" + i ); | |
eh.put( ehElm ); | |
} | |
end = System.currentTimeMillis(); | |
time = end - start; | |
putTotalEHCache += time; | |
tPer = Float.intBitsToFloat( (int) time ) / Float.intBitsToFloat( tries ); | |
System.out.println( ehCacheDisplayName + " put time for " + tries + " = " + time + "; millis per = " | |
+ tPer ); | |
start = System.currentTimeMillis(); | |
for ( int i = 0; i < tries; i++ ) | |
{ | |
eh.get( "key:" + i ); | |
} | |
end = System.currentTimeMillis(); | |
time = end - start; | |
getTotalEHCache += time; | |
tPer = Float.intBitsToFloat( (int) time ) / Float.intBitsToFloat( tries ); | |
System.out.println( ehCacheDisplayName + " get time for " + tries + " = " + time + "; millis per = " | |
+ tPer ); | |
System.out.println( "\n" ); | |
} | |
} | |
catch ( Exception e ) | |
{ | |
e.printStackTrace( System.out ); | |
System.out.println( e ); | |
} | |
long putAvJCS = putTotalJCS / loops; | |
long getAvJCS = getTotalJCS / loops; | |
long putAvHashtable = putTotalEHCache / loops; | |
long getAvHashtable = getTotalEHCache / loops; | |
System.out.println( "Finished " + loops + " loops of " + tries + " gets and puts" ); | |
System.out.println( "\n" ); | |
System.out.println( "Put average for " + jcsDisplayName + " = " + putAvJCS ); | |
System.out.println( "Put average for " + ehCacheDisplayName + " = " + putAvHashtable ); | |
ratioPut = Float.intBitsToFloat( (int) putAvJCS ) / Float.intBitsToFloat( (int) putAvHashtable ); | |
System.out.println( jcsDisplayName + " puts took " + ratioPut + " times the " + ehCacheDisplayName | |
+ ", the goal is <" + target + "x" ); | |
System.out.println( "\n" ); | |
System.out.println( "Get average for " + jcsDisplayName + " = " + getAvJCS ); | |
System.out.println( "Get average for " + ehCacheDisplayName + " = " + getAvHashtable ); | |
ratioGet = Float.intBitsToFloat( (int) getAvJCS ) / Float.intBitsToFloat( (int) getAvHashtable ); | |
System.out.println( jcsDisplayName + " gets took " + ratioGet + " times the " + ehCacheDisplayName | |
+ ", the goal is <" + target + "x" ); | |
} | |
} | |
]]> | |
</source> | |
</subsection> | |
</section> | |
<section name="JCS vs EHCache Disk Cache"> | |
<p> | |
It is very difficult to compare the ehcache disk store | |
and the JCS Indexed Disk Cache. | |
</p> | |
<p>The JCS version is much more sophisticated.</p> | |
<p> | |
JCS puts items into a queue called purgatory. While they | |
are in this queue, they are still accessible. This queue | |
gets worked when items are in it. The number of threads | |
used in the system as a whole for disk caches is | |
configurable using the thread pool configuration options | |
in JCS. I could have 1000 regions and only use 3 threads | |
to work the disk queues. From what I can tell EH will | |
use 1 thread per region. This is worse than the JCS | |
default, which uses a queue that kills its threads when | |
they are not used. . . . and much worse than using JCS | |
with a thread pool. | |
</p> | |
<p> | |
The size of JCS purgatory is configurable, so you can | |
avoid catastrophe if something goes wrong with the queue | |
worker. EH doesn't have any such safety. | |
</p> | |
<p> | |
JCS limits the number of keys that can be kept for the | |
disk cache. EH cannot do this. | |
</p> | |
<p> | |
The ehcache disk version is very simple. It puts an | |
unlimited number of items in a temporary store. You can | |
easily fill this up and run out of memory. You can put | |
items into JCS prugatory faster than they can be gc's | |
but it is much more difficult. The EH store is then | |
flushed to disk every 200ms. While EH is flushing the | |
entire disk cache blocks! | |
</p> | |
<p> | |
JCS disk cache is based on a continuous spooling model, | |
not a stop the world model like EH. In most cases the EH | |
model will work out, but not if you put a lot of big | |
items on disk at once. If you want an even distribution | |
of disk cache response times, then you should use JCS. | |
</p> | |
<p> | |
The EH disk store also seems to just keep growing. After | |
several tests, the size of the data file was 10 times | |
that of JCS and EH was taking 10 times as long. | |
</p> | |
<p> | |
You can saturate the EH version much more quickly, since | |
it will hold as many items as you can put in in 200 ms. | |
</p> | |
<p> | |
I tried with 100k and JCS could handle it, but EH died | |
with an out of memory exception. | |
</p> | |
<p> | |
EH cache developed its disk store in response to a bug | |
in the JCS version. This bug was fixed a few years ago . | |
. . The nice thing about JCS is that it is completely | |
pluggable. It would take about 30 minutes to plug a | |
different disk cache implementation into JCS if you so | |
pleased . . . . | |
</p> | |
</section> | |
</body> | |
</document> |