How to set client-output-buffer-limit?

[rhefner1]

Hi @jchanam, thanks again for the great project. We've been running in production for a few months now and things seem to be pretty good.

However, we recently had some trouble with our redis cluster due to the slaves exceeding their output buffer limits, so I need to set this in the operator config. Here is the relevant part of my config file:

    redis:
      replicas: 3
      image: "redis"
      version: "3.2-alpine"
  
      customConfig:
        - "tcp-keepalive 60"
        - client-output-buffer-limit \"normal 0 0 0 slave 1000000000 1000000000 0 pubsub 33554432 8388608 60\"

Note that I've tried a number of combinations (without \", with it, etc.) and each time I get the error message below in the operator logs:

time="2019-02-21T16:41:25Z" level=error msg="Error processing staging/staging-sandbox-redis-ha: ERR Wrong number of arguments for CONFIG set" controller=redisfailover operator=redisfailover src="generic.go:223"

Running config set client-output-buffer-limit "normal 0 0 0 slave 1000000000 1000000000 0 pubsub 33554432 8388608 60" directly in redis-cli works just fine. How can I express this in the operator?

[jchanam]

Hi @rhefner1,

I'll try to reproduce and update you if I find a fix. Have you tried with the following?

    redis:
      replicas: 3
      image: "redis"
      version: "3.2-alpine"
  
      customConfig:
        - "tcp-keepalive 60"
        - "client-output-buffer-limit 'normal 0 0 0 slave 1000000000 1000000000 0 pubsub 33554432 8388608 60'"

[rhefner1]

@jchanam No but that's a really great idea. One sec, I'll try it now.

[rhefner1]

@jchanam nope, same error "ERR Wrong number of arguments for CONFIG set"

[jchanam]

Ok, maybe a change in the code has to be done. I'll try to check it soon ok?

[benjaminjb]

Not sure this would work, @rhefner1 -- and I don't have a setup I can easily test right now -- but have you tried something like:

      customConfig:
        - "tcp-keepalive 60"
        - "client-output-buffer-limit normal 0 0 0"
        - "client-output-buffer-limit slave 1000000000 1000000000 0"
        - "client-output-buffer-limit pubsub 33554432 8388608 60"

[rhefner1]

@benjaminjb that doesn't work either 😢

[rhefner1]

I'm suspecting that the error is here [1] (but I'm not sure). The command is split by a space and then sent to NewBoolCmd, which will work for key value, but maybe not for key value value. Of note, I think that the quotes are really important here. In redis-cli, running config set client-output-buffer-limit normal 0 0 0 produces the "Wrong number of arguments" error. But running config set "client-output-buffer-limit normal 0 0 0" returns OK.

I don't really want to try this in my cluster due to #89 and #78 (could cause downtime in production), but I'm tempted to change the redisSetCommand to "CONFIG~set~%s" and the string split to strings.split(command, "~") and see if that would fix it. Obviously, that's a pretty hacky fix.

[1] https://github.com/spotahome/redis-operator/blob/master/service/redis/client.go#L273

[jchanam]

Hi @rhefner1,

I'm working on a fix, on the branch https://github.com/spotahome/redis-operator/tree/fix-multiple-values-custom-config

If you want to test it, you can do it using this image: quay.io/spotahome/redis-operator:fix-multiple-values-custom-config

[rhefner1]

@jchanam Great! I will check it out tomorrow. Thanks!

[rhefner1]

@jchanam Just tried the new 0.5.6 version and it looks good. Thanks!

Edit: by "this looks good" I mean that the settings are propagated correctly to the redis instances without any restarts, etc.