Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RedisCommandException during Azure Cluster failover #2181

Open
MarcinFrankowski opened this issue Jul 7, 2022 · 0 comments
Open

RedisCommandException during Azure Cluster failover #2181

MarcinFrankowski opened this issue Jul 7, 2022 · 0 comments

Comments

@MarcinFrankowski
Copy link

Configuration:

  • StackExchange.Redis 2.5.61
  • Azure Redis with clustering enabled, 2 shards (Redis version 6.0.14)

I've noticed RedisCommandException being thrown in WriteMessageToServerInsideWriteLock during the scheduled maintenance window.

"id": "3923059",
"outerId": "17459436",
"type": "StackExchange.Redis.RedisCommandException",
"message": "Command cannot be issued to a replica: HDEL <KEY>",
"parsedStack": [{
  "level": 0,
  "method": "StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock",
  "assembly": "StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46",
  "fileName": "/_/src/StackExchange.Redis/PhysicalBridge.cs",
  "line": 1411
}]

This issue seems to be related to #1504 which should be resolved by #1876.
It looks like it's caused by the planned failover. The topology change was not recognized by the library at the time the command was sent. We didn't have any retry policy set up for that particular exception so it caused a brief downtime of a part of the system (less than 30s).

The question here is: should we still expect this exception when rebooting one of the shards in a cluster? Maybe it's worth adding a guideline on how to handle that to the docs.

I've already updated SE.Redis to the latest version and put a retry policy for that in place but wanted to report anyway.
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant