fix: deadlock in balance shutdown

[bukata-sa]

What

Calling bm.sleeperTask.IfStarted -> bm.sleeperTask.WakeUp uses two read locks, while bm.sleeperTask.WakeUpIfStarted uses only one read lock

Why

From time to time balancer monitor can't shutdown. Because there can be exclusive lock (taken on service.Stop) between these two read locks.
Imagine: (g1 and g2 - goroutines)
g1:OnNewLongestChain -> g1:bm.sleeperTask.IfStarted (RLock1 taken) -> g2:bm.sleeperTask.Stop (WLock tried to take, but blocked, waiting for RLock1) -> g1:bm.sleeperTask.WakeUp (RLock2 tried to be taken, but blocked because have pending WLock) => deadlock (RLock2 -(waits)> WLock -(waits)> RLock1)

Extra

I had no idea before, but if you have RLock, then pending WLock, then you can't have new RLocks

This test fails

func TestStrangeRWMutex(t *testing.T) {
	mtx := sync.RWMutex{}
	mtx.RLock()
	go func() {
		fmt.Println("before write lock")
		mtx.Lock()
		assert.Fail(t, "should not have been called")
	}()
	time.Sleep(time.Second)
	fmt.Println("before second read lock")
	// mtx.Lock() - uncomment this and test will stuck with deadlock like in balanceMonitor
	assert.True(t, mtx.TryRLock())
}

Requires Dependencies

smartcontractkit/chainlink-common#706

Resolves Dependencies

#13647

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube