Cannot remove taskId from the queue shard conductor.test.QUEUE.HTTP.a

[manan164]

Hi @apanicker-nflx @kishorebanala ,
We are facing a lot of these warnings in conductor.

cannot add 2b5a87f2-c97f-4294-b883-57869f5fd65d from the queue shard conductor.test.QUEUE.HTTP.a
cannot remove b3ce517b-eeef-474f-94e9-3468e6b0927b from the queue shard conductor.test.QUEUE._deciderQueue.a
cannot add b3ce517b-eeef-474f-94e9-3468e6b0927b from the queue shard conductor.test.QUEUE._deciderQueue.a

We are running Redis cluster setup. Also, enabled workflow execution locking a few days back but that did not help much.
We also see the deciderQueue getting piled up and we run Sweeper. But running Sweeper also does not help much because sweeper also does pop from redis and I see that in pop itself these exception comes.
Can you please help us with this?
Does it cause any harm in running workflows?
We also see PENDING_WORKFLOWS set also gets piled up and we terminate all old workflows based on the workflow creation time.

[apanicker-nflx]

@smukil Can you comment on these errors? These errors seem to be exceptions thrown by dyno queues.

[manan164]

This is also the reason for Sweeper not working correctly.

116101-2020-03-31 23:54:06 DEBUG WorkflowSweeper:82 - Sweeper's current deciderqueue size: 2806.
116102:2020-03-31 23:54:06 DEBUG WorkflowSweeper:84 - Sweeper retrieved 0 workflows from the decider queue.```
It never receive the workflow from dynoQueue because it is not able to pop.

[saurabh-narain]

@apanicker-nflx Any updates on this issue?

[Mohit177]

@apanicker-nflx @smukil any update on this issue?