Date: 2019-10-17
Accepted (lazy consensus) & implemented
Additional performance testing is required for adoption.
Our queries are mostly bounded to a mailbox or a user. We can easily limit the number of ElasticSearch nodes involved in a given query by grouping the underlying documents on the same node using a routing key.
Without a routing key, each shard needs to execute the query. The coordinator needs also to be waiting for the slowest shard.
Using the routing key unlocks significant throughput enhancement (proportional to the number of shards) and also a possible high percentile latency enhancement.
As most requests are restricted to a single coordination, most search requests will hit a single shard, as opposed to non routed searches which would have hit each shards (each shard would return the number of searched documents, to be ordered and limited again in the coordination node). This allows to be more linearly scalable.
Enable ElasticSearch routing.
Messages should be indexed by mailbox.
Quota Ratio should be indexed by user.
A data reindex is needed.
On a single ElasticSearch node with 5 shards, we noticed latency reduction for mailbox search (2x mean time and 3x 99 percentile reduction)