What is the root cause of this RedisTimeoutException?

[MarcDrexler]

Hi,

We use a single Redis Server instance (v 4.0.9) as a cache layer for our services.
There are 5 service instances.
Each of these clients is regularly logging timeout exceptions:

Timeout performing GET (1000ms), next: GET solution:2097, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 102209, out-pipe: 0, serverEndpoint: 172.22.0.246:6379, mgr: 10 of 10 available, clientName: XXXXXXXXX, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=0,Free=32767,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

I read the article at https://stackexchange.github.io/StackExchange.Redis/Timeouts, but I can not determine the cause of this error.
Could someone give me any advice?

[tnachshon]

Facing the same issue since we've upgraded to version 2.0.601 from 1.2.6.
We've tried to downgrade the version back to 2.0.593 and now we face this issue every single day in production.
The last truly stable release, in our case, was 1.2.6
Please help us to solve this problem.

[damtur]

We have upgraded from version 1.2.1 to 2.0.601. Timeouts increased from ~100 an hour to ~1500 an hour.

I've investigated them and they seem to happen after ~15 simultaneous calls of GetMultipleKeys when asking for 200 different keys each. All calls happen at the same time but they are being returned with increased latency.

E.g.
First call - 5ms, second 8ms, then 14,23,26,30 ... eventually 65ms (which is over our timeout of 50ms)

The average response for calls is still ~ 1ms-2ms so 50 ms seems excessive.

[humayun-ahmed]

have you found any solution?

[tnachshon]

In our case,
We had a lot of gets and sets for keys with a large value.
Apparently this behavior caused a lot of timeouts.
We split those "problematic" keys to a few keys with a smaller amount of data.
So far it looks like the problem has been fixed.
In any case, we've started looking for a new Redis client, maybe ServiceStack.

[jacobrastad]

@tnachshon how large is "a large value" in your case? We have the same issue, currently using 2.0.519 (downgraded from 2.0.601). We use Redis primarly as a cache store for output cache and those objects can be around 200kb-300kb.

[zhangxin511]

I don't think it is key/command related. I have a sample console application example on
#1327

It fails for just simply SetWithExpiry for a int string. One thing I do notice that it seems the 16 is the magic number for parrallel to see this error, on my local

[zhangxin511]

Updates, for my case and demo code in #1327

The reason was the issue discussed on https://stackexchange.github.io/StackExchange.Redis/Timeouts.html

That the busy thread count is more than min count. Adjusted the min thread account helps my case.

[Sandeep321]

We are performing multiple operations(GET/SET) in parallel on a Redis server and we are hitting timeout issues.
I read the article at https://stackexchange.github.io/StackExchange.Redis/Timeouts, but I can not determine the cause of this error. Could someone give me any advice?

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET KEYXXXXXX, inst: 0, qu: 0, qs: 10, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: endpoint.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: SYSTEMNAME, IOCP: (Busy=0,Free=1000,Min=12,Max=1000), WORKER: (Busy=10,Free=32757,Min=12,Max=32767), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2826 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 2374

[pawelklimczyk]

@Sandeep321 was you able to figure out the root cause?

[NickCraver]

There are many timeout threads (there are many environmental and in-app causes for this), so going to close this out to tidy up/consolidate. For anyone finding this: please use the latest version of the library. Adding information to the timeout message to help advise on what's happening is something we've done a lot and current timeout messages will provide additional info like thread pool stats (e.g. how many items in queue) on .NET 5.0+ builds. If you're still having a timeout problem please feel free to open an issue here, but many of the existing causes (that aren't environmental - which most are) all have better logging so we have help advise in these cases. Those newer error messages would help greatly in our ability to provide any help.