TSAN Failure in QLDmlTest.ReadFollower test

[rao-vasireddy]

Jira Link: DB-2572

WARNING: ThreadSanitizer: data race (pid=12708)
1827 | Atomic read of size 8 at 0x7b0400078b90 by thread T92:
1828 | #0 __tsan_atomic64_load $YB_THIRDPARTY_DIR/build/common/llvm-7.0.1.src/../../../src/llvm-7.0.1.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:540 (ql-dml-test+0x4b0751)
1829 | #1 std::__1::__atomic_base<cds::intrusive::basket_queue::node<cds::gc::DHP, cds::opt::none>*, false>::load(std::__1::memory_order) const $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/atomic:929:17 (libyrpc.so+0x1edded)
1830 |   | I | 10.901675 | 12892 | consensus_meta.cc:248 | T TABLET4 P PEER_C: Updating active role from FOLLOWER to FOLLOWER. Consensus state: current_term: 3 leader_uuid: "" config { opid_index: -1 peers { permanent_uuid: "PEER_C" member_type: VOTER last_known_private_addr { host: "127.0.0.4" port: 16226 } last_known_broadcast_addr { host: "127.0.0.5.ip.yugabyte" port: 16226 } cloud_info { placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" } } peers { permanent_uuid: "PEER_B" member_type: VOTER last_known_private_addr { host: "127.0.0.2" port: 15915 } last_known_broadcast_addr { host: "127.0.0.3.ip.yugabyte" port: 15915 } cloud_info { placement_cloud: "cloud1" placement_region: "rack1" placement_zone: "zone" } } peers { permanent_uuid: "PEER_D" member_type: VOTER last_known_private_addr { host: "127.0.0.6" port: 17875 } last_known_broadcast_addr { host: "127.0.0.7.ip.yugabyte" port: 17875 } cloud_info { placement_cloud: "cloud2" placement_region: "rack3" placement_zone: "zone" } } }, has_pending_config = 0
1831 |   | I | 10.902498 | 12892 | raft_consensus.cc:2462 | T TABLET4 P PEER_C [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate PEER_B for term 3 because replica has last-logged OpId of term: 1 index: 63, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 56.
1832 | #2 std::__1::atomic<cds::details::marked_ptr<cds::intrusive::basket_queue::node<cds::gc::DHP, cds::opt::none>, 1> >::load(std::__1::memory_order) const $YB_THIRDPARTY_DIR/installed/tsan/include/cds/details/marked_ptr.h:284 (libyrpc.so+0x1edded)
1833 |   | I | 10.965516 | 12930 | leader_election.cc:379 | T TABLET4 P PEER_B [CANDIDATE]: Term 3 election: Vote denied by peer PEER_C. Message: Invalid argument (yb/consensus/raft_consensus.cc:2463): T TABLET4 P PEER_C [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate PEER_B for term 3 because replica has last-logged OpId of term: 1 index: 63, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 56.
1834 | #3 cds::intrusive::BasketQueue<cds::gc::DHP, cds::container::details::make_basket_queue<cds::gc::DHP, yb::rpc::ThreadPoolTask*, cds::container::basket_queue::traits>::node_type, cds::container::details::make_basket_queue<cds::gc::DHP, yb::rpc::ThreadPoolTask*, cds::container::basket_queue::traits>::intrusive_traits>::enqueue(cds::container::details::make_basket_queue<cds::gc::DHP, yb::rpc::ThreadPoolTask*, cds::container::basket_queue::traits>::node_type&) $YB_THIRDPARTY_DIR/installed/tsan/include/cds/intrusive/basket_queue.h:642:47 (libyrpc.so+0x1f112b)
1835 | #4 cds::container::BasketQueue<cds::gc::DHP, yb::rpc::ThreadPoolTask*, cds::container::basket_queue::traits>::enqueue(yb::rpc::ThreadPoolTask* const&) $YB_THIRDPARTY_DIR/installed/tsan/include/cds/container/basket_queue.h:296:30 (libyrpc.so+0x1f0f90)
1836 | #5 cds::container::BasketQueue<cds::gc::DHP, yb::rpc::ThreadPoolTask*, cds::container::basket_queue::traits>::push(yb::rpc::ThreadPoolTask* const&) $YB_THIRDPARTY_DIR/installed/tsan/include/cds/container/basket_queue.h:339:20 (libyrpc.so+0x1f0f40)
1837 | #6 yb::rpc::ThreadPool::Impl::Enqueue(yb::rpc::ThreadPoolTask*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/thread_pool.cc:183:36 (libyrpc.so+0x1ed771)
1838 | #7 yb::rpc::ThreadPool::Enqueue(yb::rpc::ThreadPoolTask*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/thread_pool.cc:278:17 (libyrpc.so+0x1e9c58)
1839 | #8 yb::rpc::OutboundCall::InvokeCallback() /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/outbound_call.cc:341:28 (libyrpc.so+0x176d44)
1840 | #9 yb::rpc::OutboundCall::SetResponse(yb::rpc::CallResponse&&) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/outbound_call.cc:393:5 (libyrpc.so+0x176fa5)
1841 | #10 yb::rpc::Connection::HandleCallResponse(yb::rpc::CallData*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/connection.cc:316:9 (libyrpc.so+0x1476c0)
1842 | #11 yb::rpc::YBOutboundConnectionContext::HandleCall(std::__1::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/yb_rpc.cc:388:22 (libyrpc.so+0x1f4551)
1843 | #12 non-virtual thunk to yb::rpc::YBOutboundConnectionContext::HandleCall(std::__1::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/yb_rpc.cc (libyrpc.so+0x1f45a9)
1844 | #13 yb::rpc::BinaryCallParser::Parse(std::__1::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, boost::container::new_allocator<iovec> > const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/binary_call_parser.cc:79:5 (libyrpc.so+0x13fbeb)
1845 | #14 yb::rpc::YBOutboundConnectionContext::ProcessCalls(std::__1::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, boost::container::new_allocator<iovec> > const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/yb_rpc.cc:401:19 (libyrpc.so+0x1f48f1)
1846 | #15 yb::rpc::Connection::ProcessReceived(boost::container::small_vector<iovec, 4ul, boost::container::new_allocator<iovec> > const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/connection.cc:279:27 (libyrpc.so+0x147353)
1847 | #16 yb::rpc::TcpStream::TryProcessReceived() /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/tcp_stream.cc:341:17 (libyrpc.so+0x1e5528)
1848 | #17 yb::rpc::TcpStream::ReadHandler() /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/tcp_stream.cc:290:31 (libyrpc.so+0x1e4e91)
1849 | #18 yb::rpc::TcpStream::Handler(ev::io&, int) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/tcp_stream.cc:238:14 (libyrpc.so+0x1e4cc7)
1850 | #19 void ev::base<ev_io, ev::io>::method_thunk<yb::rpc::TcpStream, &(yb::rpc::TcpStream::Handler(ev::io&, int))>(ev_loop*, ev_io*, int) $YB_THIRDPARTY_DIR/installed/common/include/ev++.h:479:7 (libyrpc.so+0x1e841a)
1851 | #20 ev_invoke_pending <null> (libev.so.4+0x49aa)
1852 | #21 yb::rpc::Reactor::RunThread() /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/rpc/reactor.cc:479:9 (libyrpc.so+0x199c63)
1853 | #22 decltype(*(std::__1::forward<yb::rpc::Reactor*&>(fp0)).*fp()) std::__1::__invoke<void (yb::rpc::Reactor::*&)(), yb::rpc::Reactor*&, void>(void (yb::rpc::Reactor::*&)(), yb::rpc::Reactor*&) $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/type_traits:4286:1 (libyrpc.so+0x1a5983)
1854 | #23 std::__1::__bind_return<void (yb::rpc::Reactor::*)(), std::__1::tuple<yb::rpc::Reactor*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::*)(), std::__1::tuple<yb::rpc::Reactor*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::rpc::Reactor::*)(), std::__1::tuple<yb::rpc::Reactor*>, 0ul, std::__1::tuple<> >(void (yb::rpc::Reactor::*&)(), std::__1::tuple<yb::rpc::Reactor*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/functional:2216 (libyrpc.so+0x1a5983)
1855 | #24 std::__1::__bind_return<void (yb::rpc::Reactor::*)(), std::__1::tuple<yb::rpc::Reactor*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::*)(), std::__1::tuple<yb::rpc::Reactor*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>::operator()<>() $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/functional:2249 (libyrpc.so+0x1a5983)
1856 | #25 decltype(std::__1::forward<std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/type_traits:4345 (libyrpc.so+0x1a5983)
1857 | #26 void std::__1::__invoke_void_return_wrapper<void>::__call<std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/__functional_base:349 (libyrpc.so+0x1a5983)
1858 | #27 std::__1::__function::__func<std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>, std::__1::allocator<std::__1::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&> >, void ()>::operator()() $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/functional:1562:12 (libyrpc.so+0x1a588d)
1859 | #28 std::__1::function<void ()>::operator()() const $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/functional:1913:12 (libtserver.so+0x174094)
1860 | #29 yb::Thread::SuperviseThread(void*) /net/centos-gcp-cloud-jenkins-slave-8a5/share/jenkins/workspace/yugabyte-centos-master-clang-tsan/build/tsan-clang-dynamic-enterprise-ninja/../../src/yb/util/thread.cc:669:3 (libyb_util.so+0x31b3d9)
1861

[rthallamko3]

This test is no longer failing
https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=QLDmlTest&fail_tag=all&name=ReadFollower&platform=linux