Changes to heartbeat broke test TestLimitNumberOfConcurrentRemoteBootstraps #4328
Assignees
Labels
Comments
hectorgcr
changed the title
hectorgcr
added a commit
that referenced
this issue
May 6, 2020
Summary: Before the changes from #2236, this test was succeeding even though the master had not marked the paused tserver as failed. For some unknown but correct reason, after the changes from #2236, the test started failing. This diff addresses this issue by modifying the flag tserver_unresponsive_timeout_ms to be less than follower_unavailable_considered_failed_sec. It needs to be less, otherwise if the peer gets removed from the raft config before the master considers this peer unresponsive, the balancer will start assigning tablets to the peer even though it's considered failed by its peers, this will cause several RBS to accumulate if the peer were to come online soon after that, Test Plan: ybd release --cxx-test integration-tests_remote_bootstrap-itest --gtest_filter "*TestLimitNumberOfConcurrentRemoteBootstraps*" -n 100 Reviewers: nicolas, bogdan Reviewed By: bogdan Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D8402
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Changes from #2236 caused the test TestLimitNumberOfConcurrentRemoteBootstraps to start failing because the number of concurrent remote bootstrap sessions was going above the number specified by the flag
load_balancer_max_concurrent_tablet_remote_bootstraps.After further investigation, we determined that the load balancer kept issuing AddServer requests even though one of the tservers was down. This was caused by a misconfigured flag (
tserver_unresponsive_timeout_ms). It's unclear why it worked before the changes from #4328, but we know that the behavior after the changes is correct. So only the test needs to be fixed by setting the flagtserver_unresponsive_timeout_msto a value lower thanfollower_unavailable_considered_failed_secThe text was updated successfully, but these errors were encountered: