Rolling Update Protection

[rhefner1]

Currently when you update the RedisFailover settings (either sentinels or the main Redis nodes), the operator performs a rolling update. Each pod is updated and then Kubernetes waits for the readiness probe to pass before moving on.

The problem with this is that the readiness probe doesn't indicate that the node has joined the cluster and is actually an active node.

I propose the following changes in the statefulset or deployment YAML unless you think there's a better way to solve this:

spec:
  minReadySeconds: 120
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
[...]

This makes sure that only one node is down during a rolling update and that it waits 120 seconds after the readiness probe passes before moving onto the next pod. There's no guarantee that the node will actually be ready after that time but it's better than not having it.

What do you think?

cc: @jchanam

[jchanam]

Hi!

I think that could be an option, but it would cause the startup of a redis-failover to be much slower.

Another option could be to set a better configuration on the readiness/liveness probes to only set as running a Sentinel that it's not checking itself, but again, would cause to have a slower startup (but safer).

What do you think it's better?

I'll try to do some tests with this. If you want to do it too and send a PR, it will be highly appreciated.