Set a rate limit on syslog messages from all Docker containers

[jleveque]

Enable rate limiting on rsyslog messages originating from each Docker container. Each container is limited to 10,000 messages per 5-minute interval.

I averaged the rates of messages of a few switches configured with 120+ ports. During normal operation, the highest rate of messages occurs at boot. I found with these devices, the average maximum rate of total messages (base image + all containers) is ~12,000 per 5 minute interval. The average maximum rate per container at boot is as follows:

Container	Max Messages Per 5-Minute Interval
syncd	3871
swss	2761
teamd	2097
lldp	415
snmp	325
pmon	110
dhcp_relay	67
bgp	31
radv	13
database	9

Thus, setting the per-container rate limit to 10,000 messages per 5-minute interval should not interfere with boot messages. If we are concerned at all, I think we could increase the limit to 20,000 or even a bit higher. The idea here is simply to prevent a rogue process from spamming a remote syslog server, so there is a bit of flexibility.

Note: 3/1/2019 -- Increased threshold to 20,000 messages per 5-minute interval.

[lguohan]

can you describe the test you have done to enable this limit?

[jleveque]

@lguohan: Updated PR description above.

[nikos-github]

I hadn't noticed you had a PR already with some numbers. Continuing my commenting here. Given the current number, debugging something that sends the messages to syslog, won't work if the voulme exceeds the threshold, correct?

[jleveque]

@nikos-github: If you are debugging something in a container which is sending > 10,000 (or whatever number we settle upon) messages within a 5-minute span, that container will stop sending messages to the syslog for the remainder of the interval (in this case, whatever remains of the 5-minute period). Once that period expires, the container will begin logging messages again, until it once again exceeds the threshold.

[jleveque]

Retest this please

[nikos-github]

@jleveque I recommend considering a slightly higher limit to catch any bootup issues which may produce errors and as such fall outside the limit and get dropped.

[jleveque]

@nikos-github: Do you think 20,000 messages in 5 minutes is good, or more?

[nikos-github]

@jleveque It seems like a better starting point. What do you think?

[jleveque]

@nikos-github: Increased threshold to 20,000 mesages per 5-minute interval in 3a0958b

[nikos-github]

Thank you for the change.

[yxieca]

Made to 201811 branch on 3/6/2019