[DocDB] Flaky test XClusterTabletSplitITest.SplittingOnProducerAndConsumer

[arybochkin]

Jira Link: DB-8923

Description

mac12-clang-release failure:

I1115 11:34:04.604979 309831168 tablet-split-itest-base.cc:511] Tablet splitting did not complete. Crashing test with core dump. Received error: Timed out (yb/util/backoff_waiter.cc:78): Operation 'Wait for tablet split to be completed' didn't complete within 40145ms
*** Aborted at 1700048044 (unix time) try "date -d @1700048044" if you are using GNU date ***
PC: @                0x0 yb::XClusterAutomaticTabletSplitITest_AutomaticTabletSplitting_Test::TestBody()::$_4::operator()()::LOG_THROTTLER_1079
*** SIGSEGV (@0x1107c9060) received by PID 63181 (TID 0x11277a600) stack trace: ***
    @     0x7ff806e60dfd _sigtramp
    @     0x7ff806d1ebeb notify_check_tz
    @     0x7ff806d592d8 raise
    @        0x10a4aca5d yb::TabletSplitITest::WaitForTabletSplitCompletion()
    @        0x109ff735c yb::XClusterTabletSplitITest::SplitAllTablets()
    @        0x109ff8868 yb::XClusterTabletSplitITest_SplittingOnProducerAndConsumer_Test::TestBody()
    @        0x10b65c948 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @        0x10b65c89f testing::Test::Run()
    @        0x10b65dea0 testing::TestInfo::Run()
    @        0x10b65eb80 testing::TestSuite::Run()
    @        0x10b66e999 testing::internal::UnitTestImpl::RunAllTests()
    @        0x10b66dfe8 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @        0x10b66df6c testing::UnitTest::Run()
    @        0x10a63812f main
    @        0x1126ff52e (unknown)

centos-clang17-tsan failure:

I1115 12:22:47.330473  3494 tablet_split_manager.cc:870] Processing split tablet result for table 69c43ff05af845909a6df8a81cfef110, split tablet ids: { source: 7de1416c39724038803a2cd0cd1288c6 children: {14a6829614e740238b5bad3fb043a77d, 3f10dbe3988b4913a1600977fb731177} }
F1115 12:22:47.332132  3493 xcluster_consumer_registry_service.cc:146] Illegal state (yb/master/xcluster_consumer_registry_service.cc:145): Could not find any consumer tablets with overlapping key range for producer tablet d2474a9c175f4e5897605c79c3f154b8, partition_key_start: C9D5 and partition_key_end: 
Fatal failure details written to ${BUILD_ROOT}/yb-test-logs/tests-integration-tests__xcluster-tablet-split-itest/XClusterTabletSplitITest_SplittingOnProducerAndConsumer.fatal_failure_details.2023-11-15T12_22_47.pid2354.txt
F20231115 12:22:47 ../../src/yb/master/xcluster_consumer_registry_service.cc:146] Illegal state (yb/master/xcluster_consumer_registry_service.cc:145): Could not find any consumer tablets with overlapping key range for producer tablet d2474a9c175f4e5897605c79c3f154b8, partition_key_start: C9D5 and partition_key_end: 

alma8-clang16-asan failure:

[C-m-1] I1115 12:42:33.465422  5752 tablet_split_manager.cc:828] vlog2: MaybeDoSplitting: Skipping splitting run because enable_automatic_tablet_splitting is not set
I1115 12:42:33.467732  5404 tablet-split-itest-base.cc:484] vlog0: num_peers_running: 6
I1115 12:42:33.467804  5404 tablet-split-itest-base.cc:485] vlog0: num_peers_split: 0
I1115 12:42:33.467830  5404 tablet-split-itest-base.cc:486] vlog0: num_peers_leader_ready: 2
F1115 12:42:33.468516  6395 xcluster_consumer_registry_service.cc:146] Illegal state (yb/master/xcluster_consumer_registry_service.cc:145): Could not find any consumer tablets with overlapping key range for producer tablet db5d0f940be247d585e14f8235069ffe, partition_key_start: CA8C and partition_key_end: 
Fatal failure details written to ${BUILD_ROOT}/yb-test-logs/tests-integration-tests__xcluster-tablet-split-itest/XClusterTabletSplitITest_SplittingOnProducerAndConsumer.fatal_failure_details.2023-11-15T12_42_33.pid5404.txt
F20231115 12:42:33 ../../src/yb/master/xcluster_consumer_registry_service.cc:146] Illegal state (yb/master/xcluster_consumer_registry_service.cc:145): Could not find any consumer tablets with overlapping key range for producer tablet db5d0f940be247d585e14f8235069ffe, partition_key_start: CA8C and partition_key_end: 

Stability report: https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&class=XClusterTabletSplitITest&name=SplittingOnProducerAndConsumer

Issue Type

kind/failing-test

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.