| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <title>Distributed James Server — Consistency Model :: Apache James</title> |
| <meta name="generator" content="Antora 3.1.2"> |
| <link rel="stylesheet" href="../../../_/css/site.css"> |
| </head> |
| <body class="article"> |
| <header class="header"> |
| <nav class="navbar"> |
| <div class="navbar-brand"> |
| <a class="navbar-item" href="https://james.apache.org"><img src="/_/img/james.svg" alt="james logo"> Apache James</a> |
| <button class="navbar-burger" data-target="topbar-nav"> |
| <span></span> |
| <span></span> |
| <span></span> |
| </button> |
| </div> |
| <div id="topbar-nav" class="navbar-menu"> |
| <div class="navbar-end"> |
| <a class="navbar-item" href="#">Home</a> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a class="navbar-link" href="#">Products</a> |
| <div class="navbar-dropdown"> |
| <div class="navbar-item"><strong>James server</strong></div> |
| <a class="navbar-item" href="https://github.com/apache/james-project">Repository</a> |
| <a class="navbar-item" href="https://issues.apache.org/jira/projects/JAMES/issues">Issue Tracker</a> |
| <hr class="navbar-divider"> |
| <a class="navbar-item" href="https://james.apache.org/mime4j/index.html">Mime4J</a> |
| <a class="navbar-item" href="https://james.apache.org/jsieve/index.html">jSieve</a> |
| <a class="navbar-item" href="https://james.apache.org/jspf/index.html">jSPF</a> |
| <a class="navbar-item" href="https://james.apache.org/jdkim/index.html">jDKIM</a> |
| <a class="navbar-item" href="https://james.apache.org/hupa/index.html">HUPA</a> |
| </div> |
| </div> |
| <div class="navbar-item has-dropdown is-hoverable"> |
| <a class="navbar-link" href="#">Community</a> |
| <div class="navbar-dropdown"> |
| <!-- Not ideal but dropping the version in the href requires tweaking james-projet docs module first --> |
| <a class="navbar-item" href="/james-project/3.6.0/community/mailing-lists.html">Mailing lists</a> |
| <a class="navbar-item" href="https://gitter.im/apache/james-project"><svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" class="logo-gitter-sign" data-v-44ebcb1a=""><rect x="15" y="5" width="2" height="10"></rect> <rect x="10" y="5" width="2" height="20"></rect> <rect x="5" y="5" width="2" height="20"></rect> <rect width="2" height="15"></rect></svg> Gitter</a> |
| <a class="navbar-item" href="https://twitter.com/ApacheJames"> |
| <span class="icon"> |
| <svg aria-hidden="true" data-icon="twitter" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> |
| <path fill="#57aaee" d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path> |
| </svg> |
| </span> Twitter |
| </a> |
| <a class="navbar-item" href="#"> <svg class="octicon octicon-mark-github v-align-middle" viewBox="0 0 16 16" version="1.1" aria-hidden="true"><path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"></path></svg> Github</a> |
| </div> |
| </div> |
| <!-- <div class="navbar-item"> |
| <span class="control"> |
| <a class="button is-primary" href="#">Download</a> |
| </span> |
| </div> --> |
| </div> |
| </div> |
| </nav> |
| </header> |
| <div class="body"> |
| <div class="nav-container" data-component="james-distributed-app" data-version="3.8.0"> |
| <aside class="nav"> |
| <div class="panels"> |
| <div class="nav-panel-menu is-active" data-panel="menu"> |
| <nav class="nav-menu"> |
| <button class="nav-menu-toggle" aria-label="Toggle expand/collapse all" style="display: none"></button> |
| <h3 class="title"><a href="../index.html">Apache James Distributed Server</a></h3> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="0"> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="1"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../index.html">Distributed James Application</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="2"> |
| <a class="nav-link" href="../objectives.html">Objectives and motivation</a> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="index.html">Architecture</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="implemented-standards.html">Implemented standards</a> |
| </li> |
| <li class="nav-item is-current-page" data-depth="3"> |
| <a class="nav-link" href="consistency-model.html">Consistency Model</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="specialized-instances.html">Specialized instances</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../run/index.html">Run</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../run/run-java.html">Run with Java</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../run/run-docker.html">Run with Docker</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../run/run-kubernetes.html">Run with Kubernetes</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-checklist.html">Deployment Checklist</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-logsMetrics.html">Logs & Metrics</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-values.html">values.yaml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../run/k8s-secrets.html">secrets.yaml</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../configure/index.html">Configuration</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Protocols</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/imap.html">imapserver.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/jmap.html">jmap.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/jmx.html">jmx.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/smtp.html">smtpserver.xml & lmtpserver.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/smtp-hooks.html">Packaged SMTP hooks</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/pop3.html">pop3server.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/webadmin.html">webadmin.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/ssl.html">SSL & TLS</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/sieve.html">Sieve & ManageSieve</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Storage dependencies</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/blobstore.html">blobstore.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/cassandra.html">cassandra.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/opensearch.html">opensearch.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/rabbitmq.html">rabbitmq.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/redis.html">redis.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/tika.html">tika.properties</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Core components</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/batchsizes.html">batchsizes.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/dns.html">dnsservice.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/domainlist.html">domainlist.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/healthcheck.html">healthcheck.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailetcontainer.html">mailetcontainer.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailets.html">Packaged Mailets</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/matchers.html">Packaged Matchers</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/mailrepositorystore.html">mailrepositorystore.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/recipientrewritetable.html">recipientrewritetable.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/search.html">search.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/usersrepository.html">usersrepository.xml</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <button class="nav-item-toggle"></button> |
| <span class="nav-text">Extensions</span> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/vault.html">deletedMessageVault.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/extensions.html">extensions.properties</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/listeners.html">listeners.xml</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/spam.html">Anti-Spam setup</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/remote-delivery-error-handling.html">About RemoteDelivery error handling</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/collecting-contacts.html">Contact collection</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/collecting-events.html">Event collection</a> |
| </li> |
| <li class="nav-item" data-depth="4"> |
| <a class="nav-link" href="../configure/dsn.html">ESMTP DSN support</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../operate/index.html">Operate</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/guide.html">Operator guide</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/performanceChecklist.html">Performance checklist</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/logging.html">Logging</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/webadmin.html">WebAdmin REST administration API</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/metrics.html">Metrics</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/migrating.html">Migrating existing data</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/cli.html">Command Line Interface</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/cassandra-migration.html">Cassandra migration</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../operate/security.html">Security checklist</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../extending/index.html">Extending server behavior</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/mail-processing.html">Custom mail processing components</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/mailbox-listeners.html">Custom Mailbox Listeners</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/smtp-hooks.html">Custom SMTP hooks</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/webadmin-routes.html">Custom WebAdmin routes</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../extending/imap.html">Custom IMAP processing</a> |
| </li> |
| </ul> |
| </li> |
| <li class="nav-item" data-depth="2"> |
| <button class="nav-item-toggle"></button> |
| <a class="nav-link" href="../benchmark/index.html">Performance benchmark</a> |
| <ul class="nav-list"> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../benchmark/db-benchmark.html">Database benchmarks</a> |
| </li> |
| <li class="nav-item" data-depth="3"> |
| <a class="nav-link" href="../benchmark/james-benchmark.html">James benchmarks</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </nav> |
| </div> |
| <div class="nav-panel-explore" data-panel="explore"> |
| <div class="context"> |
| <span class="title">Apache James Distributed Server</span> |
| <span class="version">3.8.0 SNAPSHOT</span> |
| </div> |
| <ul class="components"> |
| <li class="component is-current"> |
| <div class="title"><a href="../index.html">Apache James Distributed Server</a></div> |
| <ul class="versions"> |
| <li class="version is-current is-latest"> |
| <a href="../index.html">3.8.0 SNAPSHOT</a> |
| </li> |
| </ul> |
| </li> |
| <li class="component"> |
| <div class="title"><a href="../../../james-project/3.8.0/index.html">Apache James Server</a></div> |
| <ul class="versions"> |
| <li class="version is-latest"> |
| <a href="../../../james-project/3.8.0/index.html">3.8.0 SNAPSHOT</a> |
| </li> |
| <li class="version"> |
| <a href="../../../james-project/3.6.0/index.html">3.6.0 Snapshot</a> |
| </li> |
| </ul> |
| </li> |
| <li class="component"> |
| <div class="title"><a href="../../../james-site/latest/index.html">Apache James Site</a></div> |
| <ul class="versions"> |
| <li class="version is-latest"> |
| <a href="../../../james-site/latest/index.html">latest</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </aside> |
| </div> |
| <main class="article"> |
| <div class="toolbar" role="navigation"> |
| <button class="nav-toggle"></button> |
| <a href="../../../james-site/latest/homepage.html" class="home-link"></a> |
| <nav class="breadcrumbs" aria-label="breadcrumbs"> |
| <ul> |
| <li><a href="../index.html">Apache James Distributed Server</a></li> |
| <li><a href="../index.html">Distributed James Application</a></li> |
| <li><a href="index.html">Architecture</a></li> |
| <li><a href="consistency-model.html">Consistency Model</a></li> |
| </ul> |
| </nav> |
| <div class="edit-this-page"><a href="https://github.com/apache/james-project/blob/master/server/apps/distributed-app/docs/modules/ROOT/pages/architecture/consistency-model.adoc">Edit this Page</a></div> |
| </div> |
| <div class="content"> |
| <aside class="toc sidebar" data-title="Contents" data-levels="2"> |
| <div class="toc-menu"></div> |
| </aside> |
| <article class="doc"> |
| <h1 class="page">Distributed James Server — Consistency Model</h1> |
| <div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>This page presents the consistency model used by the Distributed Server and |
| points to the tools built around it.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_data_replication"><a class="anchor" href="#_data_replication"></a>Data Replication</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The Distributed Server relies on different storage technologies, all having their own |
| consistency models.</p> |
| </div> |
| <div class="paragraph"> |
| <p>These data stores replicate data in order to enforce some level of availability. We call |
| this process replication. By consistency, we mean the ability for all replica to hold the |
| same data. By availability, we mean the ability for a replica to answer a request.</p> |
| </div> |
| <div class="paragraph"> |
| <p>In distributed systems, <a href="https://en.wikipedia.org/wiki/CAP_theorem">according to the CAP theorem</a>, |
| as we will necessarily encounter network partitions, then trade-offs need to be made between |
| consistency and availability.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This section details this trade-off for data stores used by the Distributed Server.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="_cassandra_consistency_model"><a class="anchor" href="#_cassandra_consistency_model"></a>Cassandra consistency model</h3> |
| <div class="paragraph"> |
| <p><a href="https://cassandra.apache.org/">Cassandra</a> is an |
| <a href="https://en.wikipedia.org/wiki/Eventual_consistency">eventually consistent</a> data store. |
| This means that replica can hold diverging data, but are guaranteed to converge over time.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Several mechanisms are built in Cassandra to enforce this convergence, and need to be |
| leveraged by <strong>Distributed Server Administrator</strong>. Namely |
| <a href="https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html">nodetool repair</a>, |
| <a href="https://cassandra.apache.org/doc/latest/operating/hints.html">Hinted hand-off</a> and |
| <a href="https://cassandra.apache.org/doc/latest/operating/read_repair.html">Read repair</a>.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Distributed Server tries to mitigate inconsistencies by relying on |
| <a href="https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/dml/dmlConfigConsistency.html">QUORUM</a> read and write levels. |
| This means that a majority of replica are needed for read and write operations to be performed.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Critical business operations, like UID allocation, rely on strong consistency mechanisms brought by |
| <a href="https://www.datastax.com/blog/2013/07/lightweight-transactions-cassandra-20">lightweight transaction</a>.</p> |
| </div> |
| <div class="sect3"> |
| <h4 id="_about_multi_data_center_setups"><a class="anchor" href="#_about_multi_data_center_setups"></a>About multi data-center setups</h4> |
| <div class="paragraph"> |
| <p>As strong consistency is required for some operation, especially regarding IMAP monotic UID and MODSEQ generation, |
| and as lightweight transactions are slow across data centers, running James with a |
| <a href="https://docs.datastax.com/en/ddac/doc/datastax_enterprise/production/DDACmultiDCperWorkloadType.html">multi data-center</a> |
| Cassandra setup is discouraged.</p> |
| </div> |
| <div class="paragraph"> |
| <p>However, <a href="../configure/cassandra.html" class="xref page">this page</a> enables setting alternative read level, |
| which could be acceptable regarding limited requirements. <code>LOCAL_QUORUM</code> coupled with <code>LOCAL_SERIAL</code> |
| is likely the only scalable setup. Some options were added to turn off SERIAL consistency usage for message |
| and mailbox management. However, the use of Lightweight Transaction cannot be disabled for UIDs and ModSeqs.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Running the Distributed Server IMAP server in a multi datacenter setup will likely result either in data loss, |
| or very slow operations - as we rely on monotic UID generation, without strong consistency, UIDs could be allocated |
| several times.</p> |
| </div> |
| <div class="paragraph"> |
| <p>We did wire a multi-DC friendly distributed, POP3 only server that leverages acceptable performance while staying |
| consistent. This is achieved by having a reduced feature set - supporting only the POP3 server and using messageIds as |
| identifiers (generated without synchronisation using TimeUUIDs). You can find this application |
| <a href="https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app">on GitHub</a>. In the future, |
| JMAP support could be added, but requires followup developments as some components critically depends on UIDs |
| (for instance the search).</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_opensearch_consistency_model"><a class="anchor" href="#_opensearch_consistency_model"></a>OpenSearch consistency model</h3> |
| <div class="paragraph"> |
| <p>OpenSearch relies on <a href="https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch">strong consistency</a> |
| with home grown algorithm.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The 6.x release line, that the distributed server is using is known to be slow to recover from failures.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Be aware that data is asynchronously indexed in OpenSearch, changes will be eventually visible.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_rabbitmq_consistency_model"><a class="anchor" href="#_rabbitmq_consistency_model"></a>RabbitMQ consistency model</h3> |
| <div class="paragraph"> |
| <p>The Distributed Server relies out of the box on a single RabbitMQ server, thus consistency concerns |
| are not (yet) applicable. Availability concerns are applicable.</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_denormalization"><a class="anchor" href="#_denormalization"></a>Denormalization</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>In Cassandra, data needs to be structured to match the read patterns. To support several conflicting |
| read patterns, the data needs to be duplicated into different structures. This process is called |
| denormalization.</p> |
| </div> |
| <div class="paragraph"> |
| <p>While data can be consistent at the table level, some inconsistencies can sneak in at the applicative |
| level across denormalization tables.</p> |
| </div> |
| <div class="paragraph"> |
| <p>We write to a "table of truth" first, then duplicate the data to denormalization tables.</p> |
| </div> |
| <div class="paragraph"> |
| <p>The Distributed server offers several mechanisms to mitigate these inconsistencies:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Writes to denormalization tables are retried.</p> |
| </li> |
| <li> |
| <p>Some <a href="../operate/guide.html#_solving_cassandra_inconsistencies" class="xref page">SolveInconsistencies tasks</a> are exposed and are able to heal a given denormalization table. |
| They reset the "deduplication tables" content to the "table of truth" content.</p> |
| </li> |
| <li> |
| <p><a href="https://github.com/apache/james-project/blob/master/src/adr/0042-applicative-read-repairs.md">Read repairs</a>, |
| when implemented for a given denormalization, enables auto-healing. When an inconsistency is detected, They reset the |
| "deduplication tables" entry to the "table of truth" entry.</p> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_consistency_across_data_stores"><a class="anchor" href="#_consistency_across_data_stores"></a>Consistency across data stores</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The Distributed Server leverages several data stores:</p> |
| </div> |
| <div class="ulist"> |
| <ul> |
| <li> |
| <p>Cassandra is used for metadata storage</p> |
| </li> |
| <li> |
| <p>OpenSearch for search</p> |
| </li> |
| <li> |
| <p>Object Storage for large object storage</p> |
| </li> |
| </ul> |
| </div> |
| <div class="paragraph"> |
| <p>Thus the Distributed Server also offers mechanisms to enforce consistency across data stores.</p> |
| </div> |
| <div class="sect2"> |
| <h3 id="_write_path_organisation"><a class="anchor" href="#_write_path_organisation"></a>Write path organisation</h3> |
| <div class="paragraph"> |
| <p>The primary data stores are composed of Cassandra for metadata and Object storage for binary data.</p> |
| </div> |
| <div class="paragraph"> |
| <p>To ensure the data referenced in Cassandra is pointing to a valid object in the object store, we write |
| the object store payload first, then write the corresponding metadata in Cassandra.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Such a procedure avoids metadata pointing to unexisting blobs, however might lead to some unreferenced |
| blobs.</p> |
| </div> |
| </div> |
| <div class="sect2"> |
| <h3 id="_cassandra_opensearch"><a class="anchor" href="#_cassandra_opensearch"></a>Cassandra <⇒ OpenSearch</h3> |
| <div class="paragraph"> |
| <p>After being written to the primary stores (namely Cassandra & Object Storage), email content is |
| asynchronously indexed into OpenSearch.</p> |
| </div> |
| <div class="paragraph"> |
| <p>This process, called the EventBus, which retries temporary errors, and stores transient errors for |
| later admin-triggered retries is described further <a href="../operate/guide.html#_mailbox_event_bus" class="xref page">here</a>. |
| His role is to spread load and limit inconsistencies.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Furthermore, some <a href="../operate/guide.html#_usual_troubleshooting_procedures" class="xref page">re-indexing tasks</a> |
| enables to re-synchronise OpenSearch content with the primary data stores</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </article> |
| </div> |
| </main> |
| </div> |
| <footer class="footer"> |
| <p>This page was built using the Antora default UI.</p> |
| <p>The source code for this UI is licensed under the terms of the MPL-2.0 license.</p> |
| </footer> |
| <script id="site-script" src="../../../_/js/site.js" data-ui-root-path="../../../_"></script> |
| <script async src="../../../_/js/vendor/highlight.js"></script> |
| </body> |
| </html> |
