What to do when a request failed on a node: retry (same or other node), rethrow, or ignore.
advanced.retry-policy in the configuration. Default policy retries at most once, in cases that have a high chance of success; you can also write your own.When a query fails, it sometimes makes sense to retry it: the error might be temporary, or the query might work on a different node. The driver uses a retry policy to determine when and how to retry. It is defined in the configuration:
datastax-java-driver.advanced.retry-policy {
class = DefaultRetryPolicy
}
The behavior of the default policy will be detailed in the sections below. You can also use your own policy by specifying the fully-qualified name of a class that implements RetryPolicy.
The policy has several methods that cover different error cases. Each method returns a decision to indicate what to do next:
session.execute call, or as a failed future if using the asynchronous API);A request reached the coordinator, but there weren't enough live replicas to achieve the requested consistency level. The coordinator replied with an UNAVAILABLE error.
If the policy rethrows the error, the user code will get an UnavailableException. You can inspect the exception's fields to get the amount of replicas that were known to be alive when the error was triggered, as well as the amount of replicas that where required by the requested consistency level.
The default policy triggers a maximum of one retry, to the next node in the query plan. The rationale is that the first coordinator might have been network-isolated from all other nodes (thinking they're down), but still able to communicate with the client; in that case, retrying on the same node has almost no chance of success, but moving to the next node might solve the issue.
A read request reached the coordinator, which initially believed that there were enough live replicas to process it. But one or several replicas were too slow to answer within the predefined timeout (read_request_timeout_in_ms in cassandra.yaml); therefore the coordinator replied to the client with a READ_TIMEOUT error.
This could be due to a temporary overloading of these replicas, or even that they just failed or were turned off. During reads, Cassandra doesn‘t request data from every replica to minimize internal network traffic; instead, some replicas are only asked for a checksum of the data. A read timeout may occur even if enough replicas responded to fulfill the consistency level, but only checksum responses were received (the method’s dataPresent parameter allow you to check if you're in that situation).
If the policy rethrows the error, the user code will get a ReadTimeoutException (do not confuse this error with DriverTimeoutException, which happens when the coordinator didn't reply at all to the client).
The default policy triggers a maximum of one retry (to the same node), and only if enough replicas had responded to the read request but data was not retrieved amongst those. That usually means that enough replicas are alive to satisfy the consistency, but the coordinator picked a dead one for data retrieval, not having detected that replica as dead yet. The reasoning is that by the time we get the timeout, the dead replica will likely have been detected as dead and the retry has a high chance of success.
This is similar to onReadTimeout, but for write operations. The reason reads and writes are handled separately is because a read is obviously a non mutating operation, whereas a write is likely to be. If a write times out at the coordinator level, there is no way to know whether the mutation was applied or not on the non-answering replica.
If the policy rethrows the error, the user code will get a WriteTimeoutException.
This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.
The default policy triggers a maximum of one retry (to the same node), and only for a BATCH_LOG write. The reasoning is that the coordinator tries to write the distributed batch log against a small subset of nodes in the local datacenter; a timeout usually means that none of these nodes were alive but the coordinator hadn't detected them as dead yet. By the time we get the timeout, the dead nodes will likely have been detected as dead, and the retry has a high chance of success.
The request was aborted before we could get a response from the coordinator. This can happen in two cases:
This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.
The default policy retries on the next node if the connection was closed, and rethrows (assuming a driver bug) in all other cases.
The coordinator replied with an error other than READ_TIMEOUT, WRITE_TIMEOUT or UNAVAILABLE. Namely, this covers OverloadedException, ServerError, TruncateException, ReadFailureException and WriteFailureException.
This method is only invoked for idempotent statements. Otherwise, the driver bypasses the retry policy and always rethrows the error.
The default policy rethrows read and write failures, and retries other errors on the next node.
There are a few cases where retrying is always the right thing to do. These are not covered by RetryPolicy, but instead hard-coded in the driver:
Similarly, some errors have no chance of being solved by a retry. They will always be rethrown directly to the user. These include:
The retry policy can be overridden in execution profiles:
datastax-java-driver {
advanced.retry-policy {
class = DefaultRetryPolicy
}
profiles {
custom-retries {
advanced.retry-policy {
class = CustomRetryPolicy
}
}
slow {
request.timeout = 30 seconds
}
}
}
The custom-retries profile uses a dedicated policy. The slow profile inherits the default profile's. Note that this goes beyond configuration inheritance: the driver only creates a single DefaultRetryPolicy instance and reuses it (this also occurs if two sibling profiles have the same configuration).
Each request uses its declared profile‘s policy. If it doesn’t declare any profile, or if the profile doesn‘t have a dedicated policy, then the default profile’s policy is used.