Utilize the 'Enabled' parameter to toggle the bootstrap mode on or off for Redis nodes. #32

rurkss · 2023-12-22T17:57:42Z

This Pull Request introduces a parameter to the bootstrap configuration, designed to simplify the process of enabling or disabling the bootstrap mode for Redis nodes.

indiebrain

Can we add some examples on how to configure the CR: https://github.com/powerhome/redis-operator/tree/master/example/redisfailover

...and update the README:

redis-operator/docs/README.md

Lines 312 to 340 in 2233e9c

    
           ### Bootstrapping from pre-existing Redis Instance(s) 
        
           If you are wanting to migrate off of a pre-existing Redis instance, you can provide a `bootstrapNode` to your `RedisFailover` resource spec. 
        
           This `bootstrapNode` can be configured as follows: 
        
           |       Key      | Type         | Description                                                                                                                                                                               | Example File                                                                                 | 
        
           |:--------------:|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------| 
        
           | host           | **required** | The IP of the target Redis address or the ClusterIP of a pre-existing Kubernetes Service targeting Redis pods                                                                             | [bootstrapping.yaml](example/redisfailover/bootstrapping.yaml)                               | 
        
           | port           | _optional_   | The Port that the target Redis address is listening to. Defaults to `6379`.                                                                                                               | [bootstrapping-with-port.yaml](example/redisfailover/bootstrapping-with-port.yaml)           | 
        
           | allowSentinels | _optional_   | Allow the Operator to also create the specified Sentinel resources and point them to the target Node/Port. By default, the Sentinel resources will **not** be created when bootstrapping. | [bootstrapping-with-sentinels.yaml](example/redisfailover/bootstrapping-with-sentinels.yaml) | 
        
           #### What is Bootstrapping? 
        
           When a `bootstrapNode` is provided, the Operator will always set all of the defined Redis instances to replicate from the provided `bootstrapNode` host value. 
        
           This allows for defining a `RedisFailover` that replicates from an existing Redis instance to ease cutover from one instance to another. 
        
           **Note: Redis instance will always be configured with `replica-priority 0`. This means that these Redis instances can _never_ be promoted to a `master`.** 
        
           Depending on the configuration provided, the Operator will launch the `RedisFailover` in two bootstrapping states: without sentinels and with sentinels. 
        
           #### Default Bootstrapping Mode (Without Sentinels) 
        
           By default, if the `RedisFailover` resource defines a valid `bootstrapNode`, **only the redis instances will be created**. 
        
           This allows for ease of bootstrapping from an existing `RedisFailover` instance without the Sentinels intermingling with each other. 
        
           #### Bootstrapping With Sentinels 
        
           When `allowSentinels` is provided, the Operator will also create the defined Sentinel resources. These sentinels will be configured to point to the provided 
        
           `bootstrapNode` as their monitored master. 
        
           ### Default versions 
        
           The image versions deployed by the operator can be found on the [defaults file](api/redisfailover/v1/defaults.go).

api/redisfailover/v1/types.go

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

powerhome-portal · 2023-12-22T20:06:24Z

A change to documentation files was detected in your PR. Please visit this link to preview changes: https://portal-staging.powerapp.cloud/docs?filters[kind]=all&filters[user]=all&filters[namespaceFilter]=enablebootstrapparameter

api/redisfailover/v1/types.go

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

CHANGELOG.md

docs/README.md

operator/redisfailover/ensurer_test.go

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

This PR resolves the state of the Redis cluster after it transitions to replica mode ### Problem: While initializing a new Redis cluster, it might operate as a replica, pulling data from a source. The proper configuration for a replica Redis cluster involves excluding all sentinels and their associated resources. However, when converting a source cluster into replica mode, the redis-operator fails to remove sentinels and their related resources, as it typically does during the initial bootstrap mode. This oversight leads to issues with the sentinel monitoring process. ### Why should we care about converting source into replica and vise versa? The primary reason for the dynamic conversion of a source Redis cluster into a replica, and vice versa, is centered around disaster recovery procedures. If datacenter 'A', which serves as the source, experiences downtime, it becomes essential to transform the Redis cluster in the standby datacenter 'B' into the new source. Once datacenter 'A' is back online, the initial step to revert control to it involves synchronizing data across all storage systems, such as MySQL and Redis. This synchronization is facilitated by initially setting datacenter 'A' as a replica, allowing data alignment, and then reinstating it as the source ### Testing 1. Start redis cluster as replica with following configuration: ```yaml apiVersion: databases.spotahome.com/v1 kind: RedisFailover metadata: name: nitro-id-redis spec: bootstrapNode: host: redis-nitro-id-redis.op-1.svc.cluster.local enabled: true redis: replicas: 2 sentinel: replicas: 3 ``` 2. Check state, no sentinels and its dependend resources should be created, such as configmap, pdb and services ```yaml artur.zheludkov@artur ~ % k --context minikube -n op-2 get po NAME READY STATUS RESTARTS AGE redis-operator-f8d7f564b-npzfw 1/1 Running 0 51m rfr-nitro-id-redis-0 1/1 Running 0 73s rfr-nitro-id-redis-1 1/1 Running 0 73s artur.zheludkov@artur ~ % k --context minikube -n op-2 get pdb NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE rfr-nitro-id-redis 1 N/A 1 111s ... ``` 3. Check connection on source datacenter, make sure that replica nodes connected to the slave node of the source datacenter: ``` 172.17.0.10:6379> info replication # Replication role:slave master_host:172.17.0.9 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 slave_read_repl_offset:101138330 slave_repl_offset:101138330 slave_priority:100 slave_read_only:1 replica_announced:1 connected_slaves:2 slave0:ip=172.17.0.12,port=6379,state=online,offset=101138330,lag=1 slave1:ip=172.17.0.11,port=6379,state=online,offset=101138330,lag=0 master_failover_state:no-failover ``` 4. Convert replica datacenter into source ``` artur.zheludkov@artur ~ % k --context minikube -n op-2 patch RedisFailover nitro-id-redis -p '{"spec":{"bootstrapNode":{"enabled":false}}}' --type merge redisfailover.databases.spotahome.com/nitro-id-redis patched ``` 5. Check state of redis cluster, sentinels and its related resources (pdb, configmap, service) should be created by redis-operator: ```yaml artur.zheludkov@artur ~ % k --context minikube -n op-2 get po NAME READY STATUS RESTARTS AGE redis-operator-f8d7f564b-npzfw 1/1 Running 0 69m rfr-nitro-id-redis-0 1/1 Running 0 19m rfr-nitro-id-redis-1 1/1 Running 0 19m rfs-nitro-id-redis-77688cc867-bhh5b 1/1 Running 0 58s rfs-nitro-id-redis-77688cc867-cm754 1/1 Running 0 58s rfs-nitro-id-redis-77688cc867-lf9tw 1/1 Running 0 58s ... ``` 6. Convert redis cluster into replica mode again: ``` artur.zheludkov@artur ~ % k --context minikube -n op-2 patch RedisFailover nitro-id-redis -p '{"spec":{"bootstrapNode":{"enabled":true}}}' --type merge redisfailover.databases.spotahome.com/nitro-id-redis patched ``` 7. Check state of redis cluster, redis-operator should remove sentinels and its related resources ```yaml artur.zheludkov@artur ~ % k --context minikube -n op-2 get po NAME READY STATUS RESTARTS AGE redis-operator-6576967bfc-2bqb4 1/1 Running 0 10m rfr-nitro-id-redis-0 1/1 Running 0 8m38s rfr-nitro-id-redis-1 1/1 Running 0 8m38s artur.zheludkov@artur ~ % k --context minikube -n op-2 get pdb NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE rfr-nitro-id-redis 1 N/A 1 9m3s artur.zheludkov@artur ~ % k --context minikube -n op-2 get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rfrm-nitro-id-redis ClusterIP 10.107.179.26 <none> 6379/TCP 9m17s rfrs-nitro-id-redis ClusterIP 10.105.242.204 <none> 6379/TCP 9m17s artur.zheludkov@artur ~ % k --context minikube -n op-2 get configmap NAME DATA AGE kube-root-ca.crt 1 19d redis-failover-lease 0 19d rfr-nitro-id-redis 1 9m33s rfr-readiness-nitro-id-redis 1 9m33s rfr-s-nitro-id-redis 1 9m33s ``` --------- Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

enable parameter

06ae090

rurkss self-assigned this Dec 22, 2023

rurkss requested a review from a team as a code owner December 22, 2023 17:57

Update CHANGELOG.md

5655457

indiebrain reviewed Dec 22, 2023

View reviewed changes

api/redisfailover/v1/types.go Show resolved Hide resolved

rurkss and others added 4 commits December 22, 2023 13:30

Update api/redisfailover/v1/types.go

305ce56

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

fix test

a37475d

add example

d139eed

Update README.md

32328b6

rurkss requested a review from indiebrain December 22, 2023 20:20

indiebrain reviewed Jan 4, 2024

View reviewed changes

api/redisfailover/v1/types.go Outdated Show resolved Hide resolved

rurkss and others added 2 commits January 5, 2024 08:21

Update api/redisfailover/v1/types.go

59983cb

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

update crd

faf0a93

indiebrain reviewed Jan 9, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

docs/README.md Outdated Show resolved Hide resolved

operator/redisfailover/ensurer_test.go Show resolved Hide resolved

rurkss and others added 3 commits January 9, 2024 12:33

Update docs/README.md

3c6bb79

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

Update CHANGELOG.md

5faafb0

Co-authored-by: Aaron Kuehler <aaron.kuehler@powerhrg.com>

indiebrain approved these changes Jan 9, 2024

View reviewed changes

rurkss merged commit bd12c9a into master Jan 9, 2024
11 checks passed

rurkss deleted the enable_bootstrap_parameter branch January 9, 2024 22:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Utilize the 'Enabled' parameter to toggle the bootstrap mode on or off for Redis nodes. #32

Utilize the 'Enabled' parameter to toggle the bootstrap mode on or off for Redis nodes. #32

rurkss commented Dec 22, 2023

indiebrain left a comment

powerhome-portal bot commented Dec 22, 2023

	### Bootstrapping from pre-existing Redis Instance(s)
	If you are wanting to migrate off of a pre-existing Redis instance, you can provide a `bootstrapNode` to your `RedisFailover` resource spec.

	This `bootstrapNode` can be configured as follows:
	\| Key \| Type \| Description \| Example File \|
	\|:--------------:\|--------------\|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|----------------------------------------------------------------------------------------------\|
	\| host \| required \| The IP of the target Redis address or the ClusterIP of a pre-existing Kubernetes Service targeting Redis pods \| [bootstrapping.yaml](example/redisfailover/bootstrapping.yaml) \|
	\| port \| _optional_ \| The Port that the target Redis address is listening to. Defaults to `6379`. \| [bootstrapping-with-port.yaml](example/redisfailover/bootstrapping-with-port.yaml) \|
	\| allowSentinels \| _optional_ \| Allow the Operator to also create the specified Sentinel resources and point them to the target Node/Port. By default, the Sentinel resources will not be created when bootstrapping. \| [bootstrapping-with-sentinels.yaml](example/redisfailover/bootstrapping-with-sentinels.yaml) \|

	#### What is Bootstrapping?
	When a `bootstrapNode` is provided, the Operator will always set all of the defined Redis instances to replicate from the provided `bootstrapNode` host value.
	This allows for defining a `RedisFailover` that replicates from an existing Redis instance to ease cutover from one instance to another.

	Note: Redis instance will always be configured with `replica-priority 0`. This means that these Redis instances can _never_ be promoted to a `master`.

	Depending on the configuration provided, the Operator will launch the `RedisFailover` in two bootstrapping states: without sentinels and with sentinels.

	#### Default Bootstrapping Mode (Without Sentinels)
	By default, if the `RedisFailover` resource defines a valid `bootstrapNode`, only the redis instances will be created.
	This allows for ease of bootstrapping from an existing `RedisFailover` instance without the Sentinels intermingling with each other.

	#### Bootstrapping With Sentinels
	When `allowSentinels` is provided, the Operator will also create the defined Sentinel resources. These sentinels will be configured to point to the provided
	`bootstrapNode` as their monitored master.

	### Default versions

	The image versions deployed by the operator can be found on the [defaults file](api/redisfailover/v1/defaults.go).

Utilize the 'Enabled' parameter to toggle the bootstrap mode on or off for Redis nodes. #32

Utilize the 'Enabled' parameter to toggle the bootstrap mode on or off for Redis nodes. #32

Conversation

rurkss commented Dec 22, 2023

indiebrain left a comment

Choose a reason for hiding this comment

powerhome-portal bot commented Dec 22, 2023