Date: 2021-07-21
Accepted (lazy consensus).
Implemented.
JMAP allows users to upload binary content called blobs to be later referenced via method calls. This includes but is not limited to Email/set
for specifying the blobId of attachments and Email/import
.
The specification strongly encourages enforcing the cleanup of these uploads:
A blob that is not referenced by a JMAP object (e.g., as a message attachment) MAY be deleted by the server to free up resources. Uploads (see below) are initially unreferenced blobs. [...] An unreferenced blob MUST NOT be deleted for at least 1 hour from the time of upload; if reuploaded, the same blobId MAY be returned, but this SHOULD reset the expiry time.
Deleting such uploads in a timely manner is important as:
Today, uploads are stored along side email attachments. This means:
We need to create a separate interface UploadRepository
in data-jmap
to store uploads for each user. We would provide a memory implementation as well as a distributed implementation of it.
The distributed implementation would host metadata of the upload in Cassandra, and the content using the BlobStore API, so object storage.
This UploadRepository
would be used by JMAP RFC-8620 to back uploads (instead of the attachment manager), we will provide a BlobResolver
to enable interactions with the uploaded blob. Similarly, we will use the UploadRepository
to back uploads of JMAP draft.
We will implement cleanup of the distributed UploadRepository
. This will be done via:
Upon migrating to the UploadRepository
, previous uploads will not be carried over. No migration plan is provided as the impact is minimal. Upload prior this change will never be cleaned up. This is acceptable as JMAP implementations are marked as experimental.
We can clean up attachment storage within the mailbox-api
and its implementation:
attachmentOwners
cassandra tablegetOwners
storeAttachmentForOwner
methods in the Attachment mapperstoreAttachmentsForMessage*
-> storeAttachments*
in attachment mapperStoreAttachmentManager
(looking message ownership is then enough)attachmentMessageId
and attachmentV2
table, attachmentMessageId
to be dropped in next release, attachmentV2
can be altered to add the referencing messageId
, and a migration task will be provided to populate it. In the meantime a fallback strategy can be supplied: If the messageId cell is null we should default to reading the (old) attachmentMessageId
table.JMAP blob draft had been proposed to have the clients explicitly delete its uploads once the blob had been used to create other entities, as this extension introduce a mean to delete blobs.
However, relying on clients to enforce effective deletion seems brittle as: