Redis Cluster Persistence

[rhefner1]

So I have a cluster set up running on the latest 0.5.0 version (thanks for that release!). I have persistence enabled in the config and I can confirm that the persistent volume claims are created as expected. However:

Actual behaviour

When all three Redis nodes (data nodes, not sentinel) disappear, the cluster loses all state. Here's what happens:

1. I have a simple script that increments a counter every 1 second and retrieves it.
2. The counter is at a certain number - 50, for example.
3. I delete all of the Redis pods at once (simulating an outage of three separate Kubernetes nodes - unlikely, but still possible).
4. The pods restart but sit idle for about 3 minutes. The redis operator does a heal operation and a new master is elected.
5. The counter starts incrementing again but it is back at 0.

Expected behaviour

When the cluster recovers, the counter resumes from 50.

---

So I have a few questions:

1. Is this a bug or is the failure of all of the nodes a destructive action?
2. What do I gain by enabling persistence? If one Redis node dies and respawns, does it recover the data from disk first and then catch up with the current master? Or does a restarted node always start from scratch?

Thanks in advance! Great project! cc: @jchanam

[jchanam]

Hi @rhefner1,

One quick question, do you have keepAfterDeletion set to true under the storage section of the Redis spec?

redis-operator/example/redisfailover/persistant-storage-no-pvc-deletion.yaml

Lines 10 to 11 in 7bba835

	storage:
	keepAfterDeletion: true

[rhefner1]

@jchanam Yes, I do. Here's the file I'm using (templated for use with Helm): https://gist.github.com/rhefner1/5ea5e547743dff47f200494d17961698#file-redis-yaml-L49

And .Values.redis.persistence.enabled is set to true.

[jchanam]

Hi!

I've found the issue. It's needed to say to Redis that it has to save the data on disk. More info: https://redis.io/topics/persistence

I'll set a default SAVE configuration values, so it's done periodically. Also, the pre-stop script would need to have a BGSAVE command so the data is saved.

It's very important to have into account that some data loss could happen.

[jchanam]

Hi @rhefner1,

Have you tested the new version proposed? It's available here: quay.io/spotahome/redis-operator:devops-819-add-configuration-to-persist-data

Thanks!

[rhefner1]

@jchanam just tried it out and it looks good to me! Thanks!