Improve dist diagnostics
* Monitor nodes membership in mem3_distribution and log unexpected `nodedown`
reasons.
* Enhance mem3_distribution gen_sever to keep track of the last few events for
each node. This can help detect nodes flapping or disconnecting often.
* Expose the last few events from mem3_distribution in the .../_system
endpoint, besides the already existing dist packet stats metrics.
* Add `ping_nodes(...)` debug function. Use it to ping all connected nodes.
This can help detect a still connected but slow network connection.
* Add `dead_nodes(...)` debug function. Use it to detect a partitioned
cluster, where some nodes may not be connected to each other.
5 files changed