| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <title>Distributed James Server — Architecture :: Apache James</title> |
| <meta name="generator" content="Antora 3.1.2"> |
| <link rel="stylesheet" href="../../../_/css/site.css"> |
| </head> |
| <body class="article"> |
| <header class="header"> |
| <nav class="navbar"> |
| <div class="navbar-brand"> |
| <a class="navbar-item" href="https://james.apache.org"><img src="/_/img/james.svg" alt="james logo"> Apache James</a> |
| <button class="navbar-burger" data-target="topbar-nav"> |
| <span></span> |
| <span></span> |
| <span></span> |
| </button> |
| </div> |
| <div id="topbar-nav" class="navbar-menu"> |
| <div class="navbar-end"> |
| <a class="navbar-item" href="#">Home</a> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a class="navbar-link" href="#">Products</a> |
| <div class="navbar-dropdown"> |
| <div class="navbar-item"><strong>James server</strong></div> |
| <a class="navbar-item" href="https://github.com/apache/james-project">Repository</a> |
| <a class="navbar-item" href="https://issues.apache.org/jira/projects/JAMES/issues">Issue Tracker</a> |
| <hr class="navbar-divider"> |
| <a class="navbar-item" href="https://james.apache.org/mime4j/index.html">Mime4J</a> |
| <a class="navbar-item" href="https://james.apache.org/jsieve/index.html">jSieve</a> |
| <a class="navbar-item" href="https://james.apache.org/jspf/index.html">jSPF</a> |
| <a class="navbar-item" href="https://james.apache.org/jdkim/index.html">jDKIM</a> |
| <a class="navbar-item" href="https://james.apache.org/hupa/index.html">HUPA</a> |
| </div> |
| </div> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a class="navbar-link" href="#">Community</a> |
| <div class="navbar-dropdown"> |
| <!-- Not ideal but dropping the version in the href requires tweaking james-projet docs module first --> |
| <a class="navbar-item" href="/james-project/3.6.0/community/mailing-lists.html">Mailing lists</a> |
| <a class="navbar-item" href="https://gitter.im/apache/james-project"><svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" class="logo-gitter-sign" data-v-44ebcb1a=""><rect x="15" y="5" width="2" height="10"></rect> <rect x="10" y="5" width="2" height="20"></rect> <rect x="5" y="5" width="2" height="20"></rect> <rect width="2" height="15"></rect></svg> Gitter</a> |
| <a class="navbar-item" href="https://twitter.com/ApacheJames"> |
| <span class="icon"> |
| <svg aria-hidden="true" data-icon="twitter" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> |
| <path fill="#57aaee" d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path> |
| </svg> |
| </span> Twitter |
| </a> |
| <a class="navbar-item" href="#"> <svg class="octicon octicon-mark-github v-align-middle" viewBox="0 0 16 16" version="1.1" aria-hidden="true"><path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"></path></svg> Github</a> |
| </div> |
| </div> |
| <!-- <div class="navbar-item"> |
| <span class="control"> |
| <a class="button is-primary" href="#">Download</a> |
| </span> |
| </div> --> |
| </div> |
| </div> |
| </nav> |
| </header> |
| <div class="body"> |
| <div class="nav-container" data-component="james-distributed-app" data-version="3.8.0"> |
| <aside class="nav"> |
| <div class="panels"> |
| <div class="nav-panel-menu is-active" data-panel="menu"> |
| <nav class="nav-menu"> |
| <button class="nav-menu-toggle" aria-label="Toggle expand/collapse all" style="display: none"></button> |
| <h3 class="title"><a href="../index.html">Apache James Distributed Server</a></h3> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="0"> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="1"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../index.html">Distributed James Application</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="2"> |
| <a class="nav-link" href="../objectives.html">Objectives and motivation</a> |
| </li> |
| <li class="nav-item is-current-page" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="index.html">Architecture</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="implemented-standards.html">Implemented standards</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="consistency-model.html">Consistency Model</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="specialized-instances.html">Specialized instances</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../run/index.html">Run</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../run/run-java.html">Run with Java</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../run/run-docker.html">Run with Docker</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../run/run-kubernetes.html">Run with Kubernetes</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-checklist.html">Deployment Checklist</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-logsMetrics.html">Logs & Metrics</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-values.html">values.yaml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-secrets.html">secrets.yaml</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../configure/index.html">Configuration</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Protocols</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/imap.html">imapserver.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/jmap.html">jmap.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/jmx.html">jmx.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/smtp.html">smtpserver.xml & lmtpserver.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/smtp-hooks.html">Packaged SMTP hooks</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/pop3.html">pop3server.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/webadmin.html">webadmin.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/ssl.html">SSL & TLS</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/sieve.html">Sieve & ManageSieve</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Storage dependencies</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/blobstore.html">blobstore.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/cassandra.html">cassandra.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/opensearch.html">opensearch.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/rabbitmq.html">rabbitmq.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/redis.html">redis.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/tika.html">tika.properties</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Core components</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/batchsizes.html">batchsizes.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/dns.html">dnsservice.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/domainlist.html">domainlist.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/healthcheck.html">healthcheck.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailetcontainer.html">mailetcontainer.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailets.html">Packaged Mailets</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/matchers.html">Packaged Matchers</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailrepositorystore.html">mailrepositorystore.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/recipientrewritetable.html">recipientrewritetable.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/search.html">search.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/usersrepository.html">usersrepository.xml</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Extensions</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/vault.html">deletedMessageVault.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/extensions.html">extensions.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/listeners.html">listeners.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/spam.html">Anti-Spam setup</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/remote-delivery-error-handling.html">About RemoteDelivery error handling</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/collecting-contacts.html">Contact collection</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/collecting-events.html">Event collection</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/dsn.html">ESMTP DSN support</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../operate/index.html">Operate</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/guide.html">Operator guide</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/performanceChecklist.html">Performance checklist</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/logging.html">Logging</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/webadmin.html">WebAdmin REST administration API</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/metrics.html">Metrics</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/migrating.html">Migrating existing data</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/cli.html">Command Line Interface</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/cassandra-migration.html">Cassandra migration</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/security.html">Security checklist</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../extending/index.html">Extending server behavior</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/mail-processing.html">Custom mail processing components</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/mailbox-listeners.html">Custom Mailbox Listeners</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/smtp-hooks.html">Custom SMTP hooks</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/webadmin-routes.html">Custom WebAdmin routes</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/imap.html">Custom IMAP processing</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../benchmark/index.html">Performance benchmark</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../benchmark/db-benchmark.html">Database benchmarks</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../benchmark/james-benchmark.html">James benchmarks</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| </div> |
| <div class="nav-panel-explore" data-panel="explore"> |
| <div class="context"> |
| <span class="title">Apache James Distributed Server</span> |
| <span class="version">3.8.0 SNAPSHOT</span> |
| </div> |
| <ul class="components"> |
| <li class="component is-current"> |
| <div class="title"><a href="../index.html">Apache James Distributed Server</a></div> |
| <ul class="versions"> |
| <li class="version is-current is-latest"> |
| <a href="../index.html">3.8.0 SNAPSHOT</a> |
| </li> |
| </ul> |
| </li> |
| <li class="component"> |
| <div class="title"><a href="../../../james-project/3.8.0/index.html">Apache James Server</a></div> |
| <ul class="versions"> |
| <li class="version is-latest"> |
| <a href="../../../james-project/3.8.0/index.html">3.8.0 SNAPSHOT</a> |
| </li> |
| <li class="version"> |
| <a href="../../../james-project/3.6.0/index.html">3.6.0 Snapshot</a> |
| </li> |
| </ul> |
| </li> |
| <li class="component"> |
| <div class="title"><a href="../../../james-site/latest/index.html">Apache James Site</a></div> |
| <ul class="versions"> |
| <li class="version is-latest"> |
| <a href="../../../james-site/latest/index.html">latest</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </aside> |
| </div> |
| <main class="article"> |
| <div class="toolbar" role="navigation"> |
| <button class="nav-toggle"></button> |
| <a href="../../../james-site/latest/homepage.html" class="home-link"></a> |
| <nav class="breadcrumbs" aria-label="breadcrumbs"> |
| <ul> |
| <li><a href="../index.html">Apache James Distributed Server</a></li> |
| <li><a href="../index.html">Distributed James Application</a></li> |
| <li><a href="index.html">Architecture</a></li> |
| </ul> |
| </nav> |
| <div class="edit-this-page"><a href="https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/architecture/index.adoc">Edit this Page</a></div> |
| </div> |
| <div class="content"> |
| <aside class="toc sidebar" data-title="Contents" data-levels="2"> |
| <div class="toc-menu"></div> |
| </aside> |
| <article class="doc"> |
| <h1 class="page">Distributed James Server — Architecture</h1> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>This sections presents the Distributed Server architecture.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_storage"><a class="anchor" href="#_storage"></a>Storage</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>In order to deliver its promises, the Distributed Server leverages the following storage strategies:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="../_images/storage.png" alt="Storage responsibilities for the Distributed Server"> |
| </div> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Cassandra</strong> is used for metadata storage. Cassandra is efficient for a very high workload of small queries following |
| a known pattern.</p> |
| </li> |
| <li> |
| <p>The <strong>blob store</strong> storage interface is responsible of storing potentially large binary data. For instance |
| email bodies, headers or attachments. Different technologies can be used: <strong>Cassandra</strong>, or S3 compatible <strong>Object Storage</strong> |
| (S3 or Swift).</p> |
| </li> |
| <li> |
| <p><strong>OpenSearch</strong> component empowers full text search on emails. It also enables querying data with unplanned access |
| patterns. OpenSearch throughput do not however match the one of Cassandra thus its use is avoided upon regular workloads.</p> |
| </li> |
| <li> |
| <p><strong>RabbitMQ</strong> enables James nodes of a same cluster to collaborate together. It is used to implement connected protocols, |
| notification patterns as well as distributed resilient work queues and mail queue.</p> |
| </li> |
| <li> |
| <p><strong>Tika</strong> (optional) enables text extraction from attachments, thus improving full text search results.</p> |
| </li> |
| <li> |
| <p><strong><a href="https://spamassassin.apache.org/">SpamAssassin</a> or <a href="https://rspamd.com/">Rspamd</a></strong> (optional) can be used for Spam detection and user feedback is supported.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p><a href="consistency-model.html" class="xref page">This page</a> further details Distributed James consistency model.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_protocols"><a class="anchor" href="#_protocols"></a>Protocols</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The following protocols are supported and can be used to interact with the Distributed Server:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>SMTP</strong></p> |
| </li> |
| <li> |
| <p><strong>IMAP</strong></p> |
| </li> |
| <li> |
| <p><a href="../operate/webadmin.html" class="xref page">WebAdmin</a> REST Administration API</p> |
| </li> |
| <li> |
| <p><strong>LMTP</strong></p> |
| </li> |
| <li> |
| <p><strong>POP3</strong></p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>The following protocols should be considered experimental:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>JMAP</strong> (RFC-8620 &RFC-8621 specifications and known limitations of the James implementation are defined <a href="https://github.com/apache/james-project/tree/master/server/protocols/jmap-rfc-8621/doc">here</a>)</p> |
| </li> |
| <li> |
| <p><strong>ManagedSieve</strong></p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>The following protocols should be considered deprecated:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>JMAP-draft</strong> (draft specification as defined <a href="https://github.com/apache/james-project/tree/master/server/protocols/jmap-draft/doc">here</a>)</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Read more on <a href="implemented-standards.html" class="xref page">implemented standards</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_topology"><a class="anchor" href="#_topology"></a>Topology</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>While it is perfectly possible to deploy homogeneous James instances, with the same configuration and thus the same |
| protocols and the same responsibilities one might want to investigate in |
| <a href="specialized-instances.html" class="xref page">'Specialized instances'</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_components"><a class="anchor" href="#_components"></a>Components</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>This section presents the various components of the Distributed server, providing context about |
| their interactions, and about their implementations.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="_high_level_view"><a class="anchor" href="#_high_level_view"></a>High level view</h3> |
| <div class="paragraph"> |
| <p>Here is a high level view of the various server components and their interactions:</p> |
| </div> |
| <div class="imageblock"> |
| <div class="content"> |
| <img src="../_images/server-components.png" alt="Server components mobilized for SMTP & IMAP"> |
| </div> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>The SMTP protocol receives a mail, and enqueue it on the MailQueue</p> |
| </li> |
| <li> |
| <p>The MailetContainer will start processing the mail Asynchronously and will take business decisions like storing the |
| email locally in a user mailbox. The behaviour of the MailetContainer is highly customizable thanks to the Mailets and |
| the Matcher composibility.</p> |
| </li> |
| <li> |
| <p>The Mailbox component is responsible of storing a user’s mails.</p> |
| </li> |
| <li> |
| <p>The user can use the IMAP or the JMAP protocol to retrieve and read his mails.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>These components will be presented more in depth below.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_mail_processing"><a class="anchor" href="#_mail_processing"></a>Mail processing</h3> |
| <div class="paragraph"> |
| <p>Mail processing allows to take asynchronously business decisions on |
| received emails.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Here are its components:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>The <code>spooler</code> takes mail out of the mailQueue and executes mail |
| processing within the <code>mailet container</code>.</p> |
| </li> |
| <li> |
| <p>The <code>mailet container</code> synchronously executes the user defined logic. |
| This <code>logic' is written through the use of `mailet</code>, <code>matcher</code> and |
| <code>processor</code>.</p> |
| </li> |
| <li> |
| <p>A <code>mailet</code> represents an action: mail modification, envelop |
| modification, a side effect, or stop processing.</p> |
| </li> |
| <li> |
| <p>A <code>matcher</code> represents a condition to execute a mailet.</p> |
| </li> |
| <li> |
| <p>A <code>processor</code> is a flow of pair of <code>matcher</code> and <code>mailet</code> executed |
| sequentially. The <code>ToProcessor</code> mailet is a <code>goto</code> instruction to start |
| executing another <code>processor</code></p> |
| </li> |
| <li> |
| <p>A <code>mail repository</code> allows storage of a mail as part of its |
| processing. Standard configuration relies on the following mail |
| repository:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><code>cassandra://var/mail/error/</code> : unexpected errors that occurred |
| during mail processing. Emails impacted by performance related |
| exceptions, or logical bug within James code are typically stored here. |
| These mails could be reprocessed once the cause of the error is fixed. |
| The <code>Mail.error</code> field can help diagnose the issue. Correlation with |
| logs can be achieved via the use of the <code>Mail.name</code> field.</p> |
| </li> |
| <li> |
| <p><code>cassandra://var/mail/address-error/</code> : mail addressed to a |
| non-existing recipient of a handled local domain. These mails could be |
| reprocessed once the user is created, for instance.</p> |
| </li> |
| <li> |
| <p><code>cassandra://var/mail/relay-denied/</code> : mail for whom relay was |
| denied: missing authentication can, for instance, be a cause. In |
| addition to prevent disasters upon miss configuration, an email review |
| of this mail repository can help refine a host spammer blacklist.</p> |
| </li> |
| <li> |
| <p><code>cassandra://var/mail/rrt-error/</code> : runtime error upon Recipient |
| Rewriting occurred. This is typically due to a loop.</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_mail_queue"><a class="anchor" href="#_mail_queue"></a>Mail Queue</h3> |
| <div class="paragraph"> |
| <p>An email queue is a mandatory component of SMTP servers. It is a system |
| that creates a queue of emails that are waiting to be processed for |
| delivery. Email queuing is a form of Message Queuing – an asynchronous |
| service-to-service communication. A message queue is meant to decouple a |
| producing process from a consuming one. An email queue decouples email |
| reception from email processing. It allows them to communicate without |
| being connected. As such, the queued emails wait for processing until |
| the recipient is available to receive them. As James is an Email Server, |
| it also supports mail queue as well.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="_why_mail_queue_is_necessary"><a class="anchor" href="#_why_mail_queue_is_necessary"></a>Why Mail Queue is necessary</h4> |
| <div class="paragraph"> |
| <p>You might often need to check mail queue to make sure all emails are |
| delivered properly. At first, you need to know why email queues get |
| clogged. Here are the two core reasons for that:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Exceeded volume of emails</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Some mailbox providers enforce email rate limits on IP addresses. The |
| limits are based on the sender reputation. If you exceeded this rate and |
| queued too many emails, the delivery speed will decrease.</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Spam-related issues</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Another common reason is that your email has been busted by spam |
| filters. The filters will let the emails gradually pass to analyze how |
| the rest of the recipients react to the message. If there is slow |
| progress, it’s okay. Your email campaign is being observed and assessed. |
| If it’s stuck, there could be different reasons including the blockage |
| of your IP address.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="_why_combining_cassandra_rabbitmq_and_object_storage_for_mailqueue"><a class="anchor" href="#_why_combining_cassandra_rabbitmq_and_object_storage_for_mailqueue"></a>Why combining Cassandra, RabbitMQ and Object storage for MailQueue</h4> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>RabbitMQ ensures the messaging function, and avoids polling.</p> |
| </li> |
| <li> |
| <p>Cassandra enables administrative operations such as browsing, deleting |
| using a time series which might require fine performance tuning (see |
| <a href="http://cassandra.apache.org/doc/latest/operating/index.html">Operating |
| Casandra documentation</a>).</p> |
| </li> |
| <li> |
| <p>Object Storage stores potentially large binary payload.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>However the current design do not implement delays. Delays allow to |
| define the time a mail have to be living in the mailqueue before being |
| dequeued and is used for example for exponential wait delays upon remote |
| delivery retries, or</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_mailbox"><a class="anchor" href="#_mailbox"></a>Mailbox</h3> |
| <div class="paragraph"> |
| <p>Storage for emails belonging for users.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Metadata are stored in Cassandra while headers, bodies and attachments are stored |
| within the <a href="#_blobstore">BlobStore</a>.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="_search_index"><a class="anchor" href="#_search_index"></a>Search index</h4> |
| <div class="paragraph"> |
| <p>Emails are indexed asynchronously in OpenSearch via the <a href="#_event_bus">EventBus</a> |
| in order to empower advanced and fast email full text search.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Text extraction can be set up using <a href="https://tika.apache.org/">Tika</a>, allowing |
| to extract the text from attachment, allowing to search your emails based on the attachment |
| textual content. In such case, the OpenSearch indexer will call a Tika server prior |
| indexing.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="_quotas"><a class="anchor" href="#_quotas"></a>Quotas</h4> |
| <div class="paragraph"> |
| <p>Current Quotas of users are hold in a Cassandra projection. Limitations can be defined via |
| user, domain or globally.</p> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="_event_bus"><a class="anchor" href="#_event_bus"></a>Event Bus</h4> |
| <div class="paragraph"> |
| <p>Distributed James relies on an event bus system to enrich mailbox capabilities. Each |
| operation performed on the mailbox will trigger related events, that can |
| be processed asynchronously by potentially any James node on a |
| distributed system.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Many different kind of events can be triggered during a mailbox |
| operation, such as:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><code>MailboxEvent</code>: event related to an operation regarding a mailbox:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><code>MailboxDeletion</code>: a mailbox has been deleted</p> |
| </li> |
| <li> |
| <p><code>MailboxAdded</code>: a mailbox has been added</p> |
| </li> |
| <li> |
| <p><code>MailboxRenamed</code>: a mailbox has been renamed</p> |
| </li> |
| <li> |
| <p><code>MailboxACLUpdated</code>: a mailbox got its rights and permissions updated</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><code>MessageEvent</code>: event related to an operation regarding a message:</p> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><code>Added</code>: messages have been added to a mailbox</p> |
| </li> |
| <li> |
| <p><code>Expunged</code>: messages have been expunged from a mailbox</p> |
| </li> |
| <li> |
| <p><code>FlagsUpdated</code>: messages had their flags updated</p> |
| </li> |
| <li> |
| <p><code>MessageMoveEvent</code>: messages have been moved from a mailbox to an |
| other</p> |
| </li> |
| </ul> |
| </div> |
| </li> |
| <li> |
| <p><code>QuotaUsageUpdatedEvent</code>: event related to quota update</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Mailbox listeners can register themselves on this event bus system to be |
| called when an event is fired, allowing to do different kind of extra |
| operations on the system, like:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Current quota calculation</p> |
| </li> |
| <li> |
| <p>Message indexation with OpenSearch</p> |
| </li> |
| <li> |
| <p>Mailbox annotations cleanup</p> |
| </li> |
| <li> |
| <p>Ham/spam reporting to Spam filtering system</p> |
| </li> |
| <li> |
| <p>…</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| <div class="sect3"> |
| <h4 id="_deleted_messages_vault"><a class="anchor" href="#_deleted_messages_vault"></a>Deleted Messages Vault</h4> |
| <div class="paragraph"> |
| <p>Deleted Messages Vault is an interesting feature that will help James |
| users have a chance to:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>retain users deleted messages for some time.</p> |
| </li> |
| <li> |
| <p>restore & export deleted messages by various criteria.</p> |
| </li> |
| <li> |
| <p>permanently delete some retained messages.</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>If the Deleted Messages Vault is enabled when users delete their mails, |
| and by that we mean when they try to definitely delete them by emptying |
| the trash, James will retain these mails into the Deleted Messages |
| Vault, before an email or a mailbox is going to be deleted. And only |
| administrators can interact with this component via |
| wref:webadmin.adoc#_deleted-messages-vault[WebAdmin] REST APIs].</p> |
| </div> |
| <div class="paragraph"> |
| <p>However, mails are not retained forever as you have to configure a |
| retention period before using it (with one-year retention by default if |
| not defined). It’s also possible to permanently delete a mail if needed.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_data"><a class="anchor" href="#_data"></a>Data</h3> |
| <div class="paragraph"> |
| <p>Storage for domains and users.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Domains are persisted in Cassandra.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Users can be managed in Cassandra, or via a LDAP (read only).</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_recipient_rewrite_tables"><a class="anchor" href="#_recipient_rewrite_tables"></a>Recipient rewrite tables</h3> |
| <div class="paragraph"> |
| <p>Storage of Recipients Rewriting rules, in Cassandra.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="_mapping_types"><a class="anchor" href="#_mapping_types"></a>Mapping types</h4> |
| <div class="paragraph"> |
| <p>James allows using various mapping types for better expressing the intent of your address rewriting logic:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p><strong>Domain mapping</strong>: Rewrites the domain of mail addresses. Use it for technical purposes, user will not |
| be allowed to use the source in their FROM address headers. Domain mappings can be managed via the CLI and |
| added via <a href="../operate/webadmin.html#_domain_mappings" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Domain aliases</strong>: Rewrites the domain of mail addresses. Express the idea that both domains can be used |
| inter-changeably. User will be allowed to use the source in their FROM address headers. Domain aliases can |
| be managed via <a href="../operate/webadmin.html#_get_the_list_of_aliases_for_a_domain" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Forwards</strong>: Replaces the source address by another one. Vehicles the intent of forwarding incoming mails |
| to other users. Listing the forward source in the forward destinations keeps a local copy. User will not be |
| allowed to use the source in their FROM address headers. Forward can |
| be managed via <a href="../operate/webadmin.html#_address_forwards" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Groups</strong>: Replaces the source address by another one. Vehicles the intent of a group registration: group |
| address will be swapped by group member addresses (Feature poor mailing list). User will not be |
| allowed to use the source in their FROM address headers. Groups can |
| be managed via <a href="../operate/webadmin.html#_address_group" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Aliases</strong>: Replaces the source address by another one. Represents user owned mail address, with which |
| he can interact as if it was his main mail address. User will be allowed to use the source in their FROM |
| address headers. Aliases can be managed via <a href="../operate/webadmin.html#_address_aliases" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Address mappings</strong>: Replaces the source address by another one. Use for technical purposes, this mapping type do |
| not hold specific intent. Prefer using one of the above mapping types…​ User will not be allowed to use the source |
| in their FROM address headers. Address mappings can be managed via the CLI or via |
| <a href="../operate/webadmin.html#_address_mappings" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Regex mappings</strong>: Applies the regex on the supplied address. User will not be allowed to use the source |
| in their FROM address headers. Regex mappings can be managed via the CLI or via |
| <a href="../operate/webadmin.html#_regex_mapping" class="xref page">WebAdmin</a></p> |
| </li> |
| <li> |
| <p><strong>Error</strong>: Throws an error upon processing. User will not be allowed to use the source |
| in their FROM address headers. Errors can be managed via the CLI</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_blobstore"><a class="anchor" href="#_blobstore"></a>BlobStore</h3> |
| <div class="paragraph"> |
| <p>Stores potentially large binary data.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Mailbox component, Mail Queue component, Deleted Message Vault |
| component relies on it.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Supported backends include S3 compatible ObjectStorage (<a href="https://wiki.openstack.org/wiki/Swift">Swift</a>, S3 API).</p> |
| </div> |
| <div class="paragraph"> |
| <p>Encryption can be configured on top of ObjectStorage.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Blobs can currently be deduplicated in order to reduce storage space. This means that two blobs with |
| the same content will be stored one once.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The downside is that deletion is more complicated, and a garbage collection needs to be run. A first implementation |
| based on bloom filters can be used and triggered using the WebAdmin REST API.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_task_manager"><a class="anchor" href="#_task_manager"></a>Task Manager</h3> |
| <div class="paragraph"> |
| <p>Allows to control and schedule long running tasks run by other |
| components. Among other it enables scheduling, progress monitoring, |
| cancellation of long running tasks.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Distributed James leverage a task manager using Event Sourcing and RabbitMQ for messaging.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_event_sourcing"><a class="anchor" href="#_event_sourcing"></a>Event sourcing</h3> |
| <div class="paragraph"> |
| <p><a href="https://martinfowler.com/eaaDev/EventSourcing.html">Event sourcing</a> implementation |
| for the Distributed server stores events in Cassandra. It enables components |
| to rely on event sourcing technics for taking decisions.</p> |
| </div> |
| <div class="paragraph"> |
| <p>A short list of usage are:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Data leak prevention storage</p> |
| </li> |
| <li> |
| <p>JMAP filtering rules storage</p> |
| </li> |
| <li> |
| <p>Validation of the MailQueue configuration</p> |
| </li> |
| <li> |
| <p>Sending email warnings to user close to their quota</p> |
| </li> |
| <li> |
| <p>Implementation of the TaskManager</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </div> |
| </article> |
| </div> |
| </main> |
| </div> |
| <footer class="footer"> |
| <p>This page was built using the Antora default UI.</p> |
| <p>The source code for this UI is licensed under the terms of the MPL-2.0 license.</p> |
| </footer> |
| <script id="site-script" src="../../../_/js/site.js" data-ui-root-path="../../../_"></script> |
| <script async src="../../../_/js/vendor/highlight.js"></script> |
| </body> |
| </html> |