Date: 2022-01-07
Accepted (lazy consensus).
Implemented.
Provides an alternative to ADR-31 Distributed MailQueue (RabbitMQ + Cassandra).
MailQueue is a central component of SMTP infrastructure allowing asynchronous mail processing. This enables a short SMTP reply time despite a potentially longer mail processing time. It also works as a buffer during SMTP peak workload to not overload a server.
Furthermore, when used as a Mail Exchange server (MX), the ability to add delays being observed before dequeing elements allows, among others:
A mailqueue also enables advanced administration operations like traffic review, discarding emails, resetting wait delays, purging the queue, etc.
Distributed James currently ships a distributed MailQueue composing the following software with the following responsibilities:
This implementation suffers from the following pitfall:
Apache Pulsar is a cloud-native, distributed messaging and streaming platform. It is horizontally scalable, low latency with durability, persistent, multi-tenant, geo replicated. The count of topics can reach several millions, making it suitable for all queuing usages existing in James, including the one of the Event Bus (cf ADR 37 and ADR 38).
Pulsar supports advanced features like delayed messages, priorities, for instance, making it suitable to a MailQueue implementation.
Helm charts to ease deployments are available.
Pulsar is however complex to deploy and relies on the following components:
This would make it suitable for large to very-large deployments or PaaS.
The Pulsar SDK is handy and handles natively reactive calls, retries, dead lettering, making implementation less boiler plate.
Provide a distributed mail queue implemented on top of Pulsar for email metadata, using the blobStore to store email content.
Package this mail queue in a simple artifact dedicated to distributed mail processing.
We expect an easier way to operate a cheaper and more reliable MailQueue.
We expect delays being supported as well.
Pulsar technology would benefit from a broader adoption in James, eventually becoming the de-facto standard solution backing Apache James messaging capabilities.
To reach this status the following work needs to be under-taken:
We could then create a new artifact relying solely on Pulsar, and deprecate the RabbitMQ based artifact.
A broader adoption of Pulsar would benefit from performance insights.
This work could be continued, for instance under the form of a Google Summer of Code for 2022.
The support of deduplicated blobs in a queue, that is short lived have less benefits in terms of storage space. Yet it enables to do a single blob creation accross the full message lifecycle for message bodies.
Priorities are not yet supported by the current implementation. See JIRA-XXXX.
[[This section requires a deep review]]
Akka actor system is used in single node mode as a processing framework.
The MailQueue relies on the following topology:
The consumers on out topic and scheduled topic use the same subscription name and shared consumers. On filter topic, each consumer uses a unique subscription name and will therefore receive a copy of every messages in the topic. this ensures a full distribution of the filter state to all nodes in the cluster.
Upon enqueue, the blobs are first saved, then the Pulsar message payload is generated and published to the relevant topic (out or scheduled).
Scheduled messages have their deliveredAt
property set to the desired value. When the delay is expired, the message will be consumed and thus moved to the out topic. Flushes simply copy content of the scheduled topic to the out topic then reset the offset of the scheduled queue, atomically. Expired filters are removed.
Note that in current versions of pulsar there is a scheduled job that handles scheduled messages, the accuracy of scheduling is limited by the frequency at which this job runs.
The size of the mail queue can be simply computed from the out and scheduled topics.
Upon deletes, the condition of this deletion, as well as the sequence before which it applies is synchronized across nodes an in-memory datastructures wrapped in an actor. Each instance uses a unique subscription and thus will maintain a set of all deletions ever performed. This mechanism is repeated for both the out and the scheduled topic, using the respective sequence values for each set of filters.
Upon dequeues, messages of the out and scheduled topics are filtered using that in-memory data structure, then exposed as a reactive publisher.
Upon browsing, both the out and scheduled topics are read from the consumption offset and filtering is applied.
Upon clear, the out topic is deleted.
Miscellaneous remarks:
Materials regarding this ADR: