Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is using ForceReconnect still necessary for using Azure Redis? #1996

Closed
navidmatin opened this issue Feb 15, 2022 · 11 comments
Closed

Is using ForceReconnect still necessary for using Azure Redis? #1996

navidmatin opened this issue Feb 15, 2022 · 11 comments

Comments

@navidmatin
Copy link

navidmatin commented Feb 15, 2022

While reading Microsoft Best Practices around Azure Redis I came upon this https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-connection#using-forcereconnect-with-stackexchangeredis which talks about using "ForceReconnect" approach which appears to be a very specific implementation of using StackExchange.Redis. However, I can't find anything on StackExchange.Redis that talks about this, and Microsoft explanation is very vague so I have the following questions:

  1. Is it necessary to use Microsoft's ForceReconnect approach in order to avoid issues when Microsoft patches Azure Redis instances or is this something that StackExchange.Redis has fixed?
  2. If it is still necessary to use ForceReconnect, Can someone use ServerMaintenanceEvents to handle Microsoft Azure patching more gracefully instead of implementing the ForceReconnect.

I started looking into this while trying to troubleshoot an issue with our servers unsubscribing from Redis Pub/Sub channel without any exception after Microsoft patching our Azure Redis instances. I'm trying to find the best way to prevent this from happening in the future.

@mgravell
Copy link
Collaborator

The idea is that when everything is in place, the library should deal with those server maintenance events for you - we added them in discussion with the Azure redis team. As for ForceReconnect: honestly I don't know, but I can try to find out; maybe @philon-msft knows something?

@philon-msft
Copy link
Collaborator

Hi @navidmatin we developed ForceReconnect as a backstop for rare cases where a StackExchange.Redis ConnectionMultiplexer instance got stuck in a state where its internal connection restoration was unable to reconnect. Those cases are becoming even more rare as we continue to track down issues that can interfere with reconnection inside SE.Redis.

The Azure Cache for Redis team still recommends ForceReconnect for Redis client apps that are having problems with connection stability, or need an additional layer of protection to reinitialize stalled connections. But with the latest versions of SE.Redis, it's becoming less important. We recommend the same pattern for other client libraries too, but we don't currently have sample projects demonstrating it in other languages.

The specific issue you mention about pub/sub connections not being restored should be fixed by #1947. In the upcoming release with that change, you should see all connections restored reliably without needing ForceReconnect or ServerMaintenanceEvents.

@navidmatin
Copy link
Author

Thank you both this information has been informative.
@philon-msft it sounds like even if we implement Force-Reconnect but without a version of the library that contains #1947 , we might still face issues when we have a long-running subscription to Redis Pub/Sub and there is a Redis patch getting applied?

@NickCraver
Copy link
Collaborator

@navidmatin FWIW a prelease is up on NuGet with the pub/sub fixes - if you're willing to give it a go we'd love any reports back!

@philon-msft
Copy link
Collaborator

+1 to Nick's suggestion for trying the new 2.5.27-prerelease version to eliminate the issues you're seeing with pub/sub connections.
If that's not an option, you could add some code to periodically check/restore the subscription connection. ForceReconnect shouldn't be necessary unless you're seeing cases where the ConnectionMultiplexer is failing all commands for an extended duration (not just losing the pub/sub connection).

@navidmatin
Copy link
Author

@NickCraver I added the pre-release version to the app, the only comment I have so far is that I've noticed when the Redis server is unavailable and the flag AbortOnConnectFail is set to false, subscribing using subscriber.SubscribeAsync(MESSAGING_CHANNEL, HandleResponse) throws a RedistTimeoutException. However, it successfully subscribes to the channel, so when the Redis server becomes available again the pub/sub works without the need to subscribe to it again. I found this behavior to be a bit unexpected due to the exception that gets thrown from that method.

StackExchange.Redis.RedisTimeoutException: 'The timeout was reached before the message could be written to the output buffer, and it was not sent

Other than that everything appears to be working as expected.

@NickCraver
Copy link
Collaborator

@navidmatin ah great note - is this a case where you can't connect for a while on startup that succeeds later?

@navidmatin
Copy link
Author

@NickCraver yes that is correct. An easy way to reproduce this is to just use the local Redis instance and start the application and attempt to subscribe to a channel before starting the Redis service.

@NickCraver
Copy link
Collaborator

@navidmatin Thanks for the info btw - I'm not sure how to best handle this. I consider the current to be a net win in that it will work and subscriptions are much improved on reliability vs. before, but indeed if it doesn't succeed right then, the command will fail. I feel this is correct but happy to be shown otherwise here.

I'm thinking maybe some beefy documentation clarification on the method pointing this out with example scenarios? Thoughts?

@navidmatin
Copy link
Author

I think this sounds good. I'm going to close this.

@czb182
Copy link

czb182 commented May 23, 2023

I had basically this same question, so asking here instead of another ticket. Is ForceReconnectAsync still recommended for Azure Cache for Redis using latest version of this client? It seems enhancements are being made on client and server, so wondering. My use case is MSAL distributed token cache using Redis on Azure.

@philon-msft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants