src/site/xdoc/server/feature-performance.xml - james-project - Git at Google

 <?xml version="1.0"?>
 <!--
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.
 -->
 <document>

   <properties>
     <title>Apache James Server 3 - Performance</title>
   </properties>

 <body>

   <section name="Performance">

     <subsection name="Gatling load testing">
       <p>In mail servers, performance do matters. Users expect good, and fast responding commands. Thus performance is a
         priority concern for Apache James developers.</p>

       <p>As part of our work on the project, we need rock-solid tools to benchmark our software. And this across all protocols
         (JMAP, IMAP and SMTP). We used third-party Gatling load testing implementations for :</p>

       <img src="images/performances/gatling_1.png"/>

       <ul>
         <li>Testing main <a href="https://github.com/linagora/james-gatling">JMAP</a> commands</li>
         <li>Testing main <a href="https://github.com/linagora/gatling-imap">IMAP</a> commands</li>
         <li>Testing basic <a href="https://github.com/linagora/james-gatling/tree/master/src/main/scala-2.13/org/apache/james/gatling/smtp">SMTP</a> scenarios</li>
       </ul>

       <img src="images/performances/gatling_2.png"/>

       <p>Load testing was conducted both on top of JPA and Cassandra + OpenSearch implementations. We succeeded to support
         1.000 Thunderbird like users on top of both backends.</p>

       <p>The following results were obtained on a server with an Intel Xeon CPU (E3–1231 3.40GHz 4 cores 8 threads) with 32
         GB of RAM. We run Gatling on the same host, thus it has performance impact on the results. So far Gatling is run
         manually when we take time for it. This might eventually become part of our CI.</p>
     </subsection>

     <subsection name="Metric collection">
       <p> Additionally, we added metrics a bit everywhere in James using the brand new metrics API. We collect and export
         everything in OpenSearch using <a href="http://metrics.dropwizard.io/3.2.1/">Dropwizard metrics</a>. Then we
         graph it all using <a href="http://grafana.com/">Grafana</a>. This allows us to collect all statistics and percentiles.
         Boards can be downloaded <a href="https://github.com/apache/james-project/tree/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting">here</a>.</p>

       <img src="images/performances/metrics.png"/>

       <ul>We track so far:
          <li>Protocols detailed time execution (and counts)</li>
          <li>Percentile of mailet and matcher executions</li>
          <li>Enqueue and Dequeue time</li>
       </ul>

       <ul>All these solutions allow us to identify the components that need improvement. Here are some example of classical
         metrics James can reach with a mono server Guice + Cassandra + OpenSearch installation:
          <li>About 100 users</li>
          <li>So far a number of 1.500.000 e-mails, that we imported using IMAP-sync scripts</li>
          <li>Around 15.000 incoming e-mails a day</li>
          <li>42.500 IMAP commands a day</li>
          <li>20.000 JMAP requests a day</li>
       </ul>

       <p>More details on metrics are available on <a href="metrics.html">the metric page</a>.</p>
     </subsection>

     <subsection name="Measuring dequeueing speed">
       <p>James can spool more than 65 mails per second (configured with Maildir mail storage and the FileMailQueue).</p>

       <p>We ran some SMTP Gatling load testing during 5 minutes (https://github.com/linagora/james-gatling)</p>

       <source>sbt
 gatling:testOnly org.apache.james.gatling.smtp.scenari.NoAuthenticationNoEncryptionScenario</source>

       <p>The result was:</p>
       <ol>
         <li>During these 5 minutes, 29.924 mails where enqueued (~100 email per second)</li>
         <li>The dequeueing of these emails took 7 minutes and 40 seconds (~65 emails per second)</li>
       </ol>

       <p>We thus have a throughput of more than <b>65 mails/second</b>.</p>

       <p>These numbers are obtained with the out-of-the-box configuration (not optimized), on a HDD disk. Better performance is expected with
         better hardware and further JVM tuning (-XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods).</p>
     </subsection>

   </section>

 </body>

 </document>
	<?xml version="1.0"?>
	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->
	<document>

	<properties>
	<title>Apache James Server 3 - Performance</title>
	</properties>

	<body>

	<section name="Performance">

	<subsection name="Gatling load testing">
	<p>In mail servers, performance do matters. Users expect good, and fast responding commands. Thus performance is a
	priority concern for Apache James developers.</p>

	<p>As part of our work on the project, we need rock-solid tools to benchmark our software. And this across all protocols
	(JMAP, IMAP and SMTP). We used third-party Gatling load testing implementations for :</p>

	<img src="images/performances/gatling_1.png"/>

	<ul>
	<li>Testing main <a href="https://github.com/linagora/james-gatling">JMAP</a> commands</li>
	<li>Testing main <a href="https://github.com/linagora/gatling-imap">IMAP</a> commands</li>
	<li>Testing basic <a href="https://github.com/linagora/james-gatling/tree/master/src/main/scala-2.13/org/apache/james/gatling/smtp">SMTP</a> scenarios</li>
	</ul>

	<img src="images/performances/gatling_2.png"/>

	<p>Load testing was conducted both on top of JPA and Cassandra + OpenSearch implementations. We succeeded to support
	1.000 Thunderbird like users on top of both backends.</p>

	<p>The following results were obtained on a server with an Intel Xeon CPU (E3–1231 3.40GHz 4 cores 8 threads) with 32
	GB of RAM. We run Gatling on the same host, thus it has performance impact on the results. So far Gatling is run
	manually when we take time for it. This might eventually become part of our CI.</p>
	</subsection>

	<subsection name="Metric collection">
	<p> Additionally, we added metrics a bit everywhere in James using the brand new metrics API. We collect and export
	everything in OpenSearch using <a href="http://metrics.dropwizard.io/3.2.1/">Dropwizard metrics</a>. Then we
	graph it all using <a href="http://grafana.com/">Grafana</a>. This allows us to collect all statistics and percentiles.
	Boards can be downloaded <a href="https://github.com/apache/james-project/tree/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting">here</a>.</p>

	<img src="images/performances/metrics.png"/>

	<ul>We track so far:
	<li>Protocols detailed time execution (and counts)</li>
	<li>Percentile of mailet and matcher executions</li>
	<li>Enqueue and Dequeue time</li>
	</ul>

	<ul>All these solutions allow us to identify the components that need improvement. Here are some example of classical
	metrics James can reach with a mono server Guice + Cassandra + OpenSearch installation:
	<li>About 100 users</li>
	<li>So far a number of 1.500.000 e-mails, that we imported using IMAP-sync scripts</li>
	<li>Around 15.000 incoming e-mails a day</li>
	<li>42.500 IMAP commands a day</li>
	<li>20.000 JMAP requests a day</li>
	</ul>

	<p>More details on metrics are available on <a href="metrics.html">the metric page</a>.</p>
	</subsection>

	<subsection name="Measuring dequeueing speed">
	<p>James can spool more than 65 mails per second (configured with Maildir mail storage and the FileMailQueue).</p>

	<p>We ran some SMTP Gatling load testing during 5 minutes (https://github.com/linagora/james-gatling)</p>

	<source>sbt
	gatling:testOnly org.apache.james.gatling.smtp.scenari.NoAuthenticationNoEncryptionScenario</source>

	<p>The result was:</p>
	<ol>
	<li>During these 5 minutes, 29.924 mails where enqueued (~100 email per second)</li>
	<li>The dequeueing of these emails took 7 minutes and 40 seconds (~65 emails per second)</li>
	</ol>

	<p>We thus have a throughput of more than <b>65 mails/second</b>.</p>

	<p>These numbers are obtained with the out-of-the-box configuration (not optimized), on a HDD disk. Better performance is expected with
	better hardware and further JVM tuning (-XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods).</p>
	</subsection>

	</section>

	</body>

	</document>