Optimize the write performance when host is down

[luyade]

What type of PR is this?

• enhancement

What problem(s) does this PR solve?

Issue(s) number: #5570

Description:

Optimize the write performance after some host is down.

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

• Unit test(positive and negative cases)
• Function test
• Performance test
• N/A

Affects:

• Documentation affected (Please add the label if documentation needs to be modified.)
• Incompatibility (If it breaks the compatibility, please describe it and add the label.）
• If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
• Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

[codecov-commenter]

Codecov Report

Patch coverage: 46.42% and project coverage change: +1.26 🎉

Comparison is base (e469b32) 76.96% compared to head (ccaa956) 78.22%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5571      +/-   ##
==========================================
+ Coverage   76.96%   78.22%   +1.26%     
==========================================
  Files        1130     1130              
  Lines       85345    85368      +23     
==========================================
+ Hits        65686    66783    +1097     
+ Misses      19659    18585    -1074     

Impacted Files	Coverage Δ
src/kvstore/raftex/Host.h	87.80% <ø> (+4.87%)	⬆️
src/kvstore/raftex/RaftPart.cpp	70.67% <ø> (+0.47%)	⬆️
src/kvstore/raftex/Host.cpp	70.23% <46.42%> (-2.11%)	⬇️

... and 82 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[luyade]

@critical27 any additional thoughts?

[critical27]

If a host is down, the leader will still continue to send appendLog requests to the bad follower. As time goes on, the log gap will become large which will make the performance of write request bad.

The key points that make performance regression is that, when log gap become large, the building of the WAL iterator from specific logId is too slow. (In the same wal file, the target log is seek in sequential)

I'm ok with this PR for now. If you are interested in the problem I said, could try to fix it. The problem is the biggest one in current implementation. It will make raft much more robust and faster in many scenarios.

[critical27]

One point you need to check is that, when a node is elected as leader, should we reset all paused_ of its host to false. WDYT?

[luyade]

If a host is down, the leader will still continue to send appendLog requests to the bad follower. As time goes on, the log gap will become large which will make the performance of write request bad.

The key points that make performance regression is that, when log gap become large, the building of the WAL iterator from specific logId is too slow. (In the same wal file, the target log is seek in sequential)

I'm ok with this PR for now. If you are interested in the problem I said, could try to fix it. The problem is the biggest one in current implementation. It will make raft much more robust and faster in many scenarios.

Actually I fixed the issue you mentioned (at least from my view) in our implementation and it almost had no peformance regression even if the log gap is very large. ^_^

[luyade]

One point you need to check is that, when a node is elected as leader, should we reset all paused_ of its host to false. WDYT?

Yes, the current implementation will call host->resume to reset paused_ as false. I think it should be fine. At the beginning, the new leader doesn't know its peer is down. It can eventually set the right status of the peer by sending HB request.