Calls stuck in StackExchange.Redis calls during Azure Redis Cache maintenace

[sudheergn]

Hi,
I am using 2.1.28 StackExchange.Redis package in a service with Redis hosted in Azure Redis Cache. (Cluster with 2 shards)
We are seeing that during Azure maintenance (Failover), the calls are stuck inside StackExchange.Redis calls for up to 8 mins. A similar behavior is seen when I reboot both nodes on the Azure Redis cluster, though the calls are stuck only for about 2-3 minutes in this case.
I have seen both StringGetSync and HashGetAll getting stuck.

SyncTimeout is set to 8 secs.
ConnectRetryPolicy is set to be Exponential with with deltaBackoff as 1 sec and maxDeltaBackoff as 3 secs.

When the master is down, calls are fine. The calls seem to have switched to slave and continue to work.
Later master comes back and slave goes down, but now calls get stuck and do not switch to the previous master that got restored. Once the slave comes back, then the calls return (some fail with RedisTimeoutException and some may have succeeded)

Are the calls expected to be stuck when the SyncTimeout is 8 secs?
What is the reason that Redis connections are not being switched to the new master.

Any help will be appreciated.

[NickCraver]

We've done some improvements here in partnership with the Azure Redis Cache team to make the client aware of failovers and proactively reconfigure based on messages the server publishes. In Linux especially some socket configurations can lead to bad timeouts because they'll wait up to 15 minutes to fail. With the changes in #1876, and grabbing the latest SE.Redis client, these situations should be solved. Sorry it took a while to get this in, but it wasn't a quick deployment effort getting to both sides. I hope this helps!