Storaged go insane after feeded with a ChainAddEdge request of 409600 batch size of edges #3465
Comments
kikimo
changed the title
|
I guess this is because this log is too big that raft can't return in 1 min(default raft rpc timeout). In this case, leader will think its replication to follower failed, then goes into infinite loop. we add some log to check this later. |
|
tested with raft kvput，and batch size = 4096， we reproduce this problem in nearly 20mins. |
We perform a double check for this problem and found that this only happends when we are inserting edges. |
|
in this case, TOSS will make a batch, consist of the original request and 409600 locks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Please check the FAQ documentation before raising an issue
Storage go insane after feeded with a ChainAddEdge request of 409600 batch size of edges. storaged respond every subsequent add edge request with
Code:E_WRITE_WRITE_CONFLICT, this situation can be reproduced steadily:root@graph1:~/nebula-chaos-cluster/scripts# ./tossEdge I1214 16:27:13.847096 1546 raft.go:156] found leader of term: 370, leader: store2 putting kvs... insert resp: ExecResponse({Result_:ResponseCommon({FailedParts:[PartitionResult_({Code:E_WRITE_WRITE_CONFLICT PartID:1 Leader:<nil>})] LatencyInUs:172 LatencyDetailUs:map[]})}), err: <nil> done putting kvs...from the storage log:
We still cannot insert edge after we restart the whole cluster:
root@graph1:~/nebula-chaos-cluster/scripts# ./tossEdge I1214 16:55:22.207335 383 raft.go:156] found leader of term: 371, leader: store1 putting kvs... insert resp: ExecResponse({Result_:ResponseCommon({FailedParts:[PartitionResult_({Code:E_WRITE_WRITE_CONFLICT PartID:1 Leader:<nil>})] LatencyInUs:115 LatencyDetailUs:map[]})}), err: <nil> done putting kvs... root@graph1:~/nebula-chaos-cluster/scripts# ./tossEdge I1214 16:55:22.822024 393 raft.go:156] found leader of term: 371, leader: store1 putting kvs... insert resp: ExecResponse({Result_:ResponseCommon({FailedParts:[PartitionResult_({Code:E_WRITE_WRITE_CONFLICT PartID:1 Leader:<nil>})] LatencyInUs:97 LatencyDetailUs:map[]})}), err: <nil> done putting kvs... root@graph1:~/nebula-chaos-cluster/scripts# ./tossEdge I1214 16:55:23.505508 402 raft.go:156] found leader of term: 371, leader: store1 putting kvs... insert resp: ExecResponse({Result_:ResponseCommon({FailedParts:[PartitionResult_({Code:E_WRITE_WRITE_CONFLICT PartID:1 Leader:<nil>})] LatencyInUs:98 LatencyDetailUs:map[]})}), err: <nil> done putting kvs...store1 is the new leader, and from its log we can see that:
Your Environments (required)
uname -ag++ --versionorclang++ --versionlscpua3ffc7d8)How To Reproduce(required)
Steps to reproduce the behavior:
Expected behavior
Additional context
The text was updated successfully, but these errors were encountered: