ninja-fix: date amd image links on src/_posts/2020-09-03-improving-resiliency.markdown
diff --git a/src/_posts/2020-09-03-improving-resiliency.markdown b/src/_posts/2020-09-03-improving-resiliency.markdown
index 1acea24..254fb22 100644
--- a/src/_posts/2020-09-03-improving-resiliency.markdown
+++ b/src/_posts/2020-09-03-improving-resiliency.markdown
@@ -1,7 +1,7 @@
---
layout: post
title: "Improving Apache Cassandra’s Front Door and Backpressure"
-date: 2020-09-03 09:00:00 -0700
+date: 2020-09-03 09:00:00 0200
author: the Apache Cassandra Community
categories: blog
---
@@ -12,43 +12,43 @@
Let us take the scenario of a client application sending requests to C* cluster. For the purpose of this blog, let us focus on one of the C* coordinator nodes.
-
+
Below is the microscopic view of client-server interaction at the C* coordinator node. Each client connection to Cassandra node happens over a netty channel, and for efficiency purposes, each Netty eventloop thread is responsible for more than one netty channel.
-
+
The eventloop threads read requests coming off of netty channels and enqueue them into a bounded inbound queue in the Cassandra node.
-
+
A thread pool dequeues requests from the inbound queue, processes them asynchronously and enqueues the response into an outbound queue. There exist multiple outbound queues, one for each eventloop thread to avoid races.
-
+
-
+
-
+
The same eventloop threads that are responsible for enqueuing incoming requests into the inbound queue, are also responsible for dequeuing responses off from the outbound queue and shipping responses back to the client.
-
+
-
+
#### Issue with this workflow
Let us take a scenario where there is a spike in operations from the client. The eventloop threads are now enqueuing requests at a much higher rate than the rate at which the requests are being processed by the native transport thread pool. Eventually, the inbound queue reaches its limit and says it cannot store any more requests in the queue.
-
+
Consequently, the eventloop threads get into a blocked state as they try to enqueue more requests into an already full inbound queue. They wait until they can successfully enqueue the request in hand, into the queue.
-
+
As noted earlier, these blocked eventloop threads are also supposed to dequeue responses from the outbound queue. Given they are in blocked state, the outbound queue (which is unbounded) grows endlessly, with all the responses, eventually resulting in C* going out of memory. This is a vicious cycle because, since the eventloop threads are blocked, there is no one to ship responses back to the client; eventually client side timeout triggers, and clients may send more requests due to retries. This is an unfortunate situation to be in, since Cassandra is doing all the work of processing these requests as fast as it can, but there is no one to ship the produced responses back to the client.
-
+
So far, we have built a fair understanding of how the front door of C* works with regard to handling client requests, and how blocked eventloop threads can affect Cassandra.
@@ -82,24 +82,24 @@
If the inbound queue is full (i.e. the thresholds are met).
-
+
C* sets autoread to false on the netty channel, which means it will stop reading bytes off of the netty channel.
-
+
Consequently, the kernel socket inbound buffer becomes full since no bytes are being read off of it by netty eventloop.
-
+
Once the Kernel Socket Inbound Buffer is full on the server side, things start getting piled up in the Kernel Socket Outbound Buffer on the client side, and once this buffer gets full, client will start experiencing backpressure.
-
+
#### **THROW_ON_OVERLOAD = true**
If the inbound queue is full (i.e. the thresholds are met), eventloop threads do not enqueue the request into the Inbound Queue. Instead, the eventloop thread creates an OverloadedException response message and enqueues it into the flusher queue, which will then be shipped back to the client.
-
+
This way, Cassandra is able to serve very large throughput, while protecting itself from getting into memory starvation issues. This patch has been vetted through thorough performance benchmarking. Detailed performance analysis can be found [here](https://issues.apache.org/jira/browse/CASSANDRA-15013?focusedCommentId=16881762&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16881762).
