How to get notified when a node is down

[SteveLauC]

Hi, thanks for the project!

I would like to know how I can get notified when a node is down, searching the issues shows me the server_metrics() API, but since an unavailable node triggers backoff retry, i.e., infinite number of retries, that node will remain in the cluster(which makes sense), there won't be any server state changes, so this API won't work IMHO.

Is there any way I can implement this? Thanks!

[drmingdrmer]

When assessing the status of a follower node in a Raft cluster, it's important to consider the difference between the leader's last_applied index and the follower's replication state. If this difference becomes significantly large, it may indicate that the follower is struggling to keep up with the leader or potentially offline.

To determine this:

1. Get the leader's last_applied index:

   let leader_last_applied = raft_metrics.last_applied.index;

   (Reference: raft_metrics.rs#L36)

2. Get the follower's replication state:

   let follower_replication = raft_metrics.replication.get(follower_node_id);

   (Reference: raft_metrics.rs#L79)

3. Calculate the replication lag:

   let replication_lag = leader_last_applied - follower_replication.index;

4. Define a threshold for considering a follower as potentially offline:

   const OFFLINE_THRESHOLD: u64 = /* define your threshold */;
   
   if replication_lag > OFFLINE_THRESHOLD {
       // Consider the follower as potentially offline
   }

It's crucial to note that in a distributed system, you can't definitively determine if a node is offline. However, a large replication lag suggests a higher probability that the follower node is experiencing issues or may be offline.

By monitoring this metric, you can make informed decisions about the health of your Raft cluster and take appropriate actions when followers fall too far behind.

[SteveLauC]

Hi, thanks for the reply!

If this difference becomes significantly large, it may indicate that the follower is struggling to keep up with the leader or potentially offline.

I guess this won't work in the case where no changes will be made to the state machine after a follower node becomes offline as the applied log indexes will remain the same, thus the difference will not change.

It's crucial to note that in a distributed system, you can't definitively determine if a node is offline.

Yeah, I think we have to guess through all kinds of metrics.

Thanks again for the reply!

[drmingdrmer]

I guess this won't work in the case where no changes will be made to the state machine after a follower node becomes offline as the applied log indexes will remain the same, thus the difference will not change.

You're correct. To better assess the connectivity status of a follower node, it would be beneficial to add a metric that tracks the timestamp of the last acknowledged heartbeat from each follower. This additional information would provide a more accurate and timely indication of the follower's connectivity status. Here's how we could refine this idea:

1. Add a new field to the RaftMetrics struct specifically for follower heartbeat information:

pub struct RaftMetrics<C: RaftTypeConfig> {
    // ... existing fields ...
    pub follower_heartbeats: HashMap<C::NodeId, Instant>,
}

2. Update this field whenever a follower acknowledges a heartbeat or successfully replicates an entry:

impl<C: RaftTypeConfig> RaftMetrics<C> {
    pub fn update_follower_heartbeat(&mut self, follower_id: C::NodeId) {
        self.follower_heartbeats.insert(follower_id, Instant::now());
    }
}

3. In the leader's routine that checks follower health:

const HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(/* define your timeout */);

for (follower_id, last_heartbeat) in &raft_metrics.follower_heartbeats {
    if last_heartbeat.elapsed() > HEARTBEAT_TIMEOUT {
        // Consider this follower as potentially disconnected
    }
}

[SteveLauC]

pub struct RaftMetrics<C: RaftTypeConfig> {
   // ... existing fields ...
   pub follower_heartbeats: HashMap<C::NodeId, Instant>,
}

To better assess the connectivity status of a follower node, it would be beneficial to add a metric that tracks the timestamp of the last acknowledged heartbeat from each follower.

So we are going to add a new metric?

[drmingdrmer]

Yes

[SteveLauC]

Thanks! :)

[SteveLauC]

Close as completed in #1177.