fix: Use Redis ConnectionManager to reconnect on failures

[sbernauer]

Previously trino-lb would not recover from a closed connection

2024-06-21T08:00:17.924954Z  INFO Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Updated query counters from 1 remote clusters
2024-06-21T08:00:22.720512Z ERROR Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Failed to get last QueryCountFetcher update timestamp err=RedisError { source: GetLastQueryCountFetcherUpdate { source: broken pipe } }
2024-06-21T08:00:27.719737Z ERROR Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Failed to get last QueryCountFetcher update timestamp err=RedisError { source: GetLastQueryCountFetcherUpdate { source: broken pipe } }
2024-06-21T08:00:32.720312Z ERROR Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Failed to get last QueryCountFetcher update timestamp err=RedisError { source: GetLastQueryCountFetcherUpdate { source: broken pipe } }

Now the first call to redis will fail, but the following calls will work again

2024-06-21T07:58:57.264989Z  INFO Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Updated query counters from 1 remote clusters
2024-06-21T07:59:02.085295Z ERROR Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Failed to get last QueryCountFetcher update timestamp err=RedisError { source: GetLastQueryCountFetcherUpdate { source: broken pipe } }
2024-06-21T07:59:07.086128Z  INFO Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Updated query counters from 0 remote clusters
2024-06-21T07:59:12.085915Z  INFO Fetching current query counters: trino_lb::maintenance::query_count_fetcher: QueryCountFetcher: Updated query counters from 0 remote clusters

[Techassi]

How many times does the connection manager try to reconnect? Is there an exponential timeout? Are these parameters configurable?

Because these reconnect attempts can cause performance issues on the Redis server side which could lead to DoS.

[sbernauer]

Seems like they are using https://docs.rs/tokio-retry/latest/tokio_retry/ for that. They have some default values in https://docs.rs/redis/latest/src/redis/aio/connection_manager.rs.html#101-109, which I guess should be a good starting point

[maltesander]

LGTM if sascha is happy :)

[Techassi]

Seems like they are using https://docs.rs/tokio-retry/latest/tokio_retry/ for that. They have some default values in https://docs.rs/redis/latest/src/redis/aio/connection_manager.rs.html#101-109, which I guess should be a good starting point

Thanks! That looks reasonable.

[maltesander]

LGTM!

[Techassi]

LGTM, just one question (non-blocking).