-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automatically detect number of cpus from cgroup in Scylla Operator binaries #1612
Conversation
…naries Golang runtime doesn't understand cgroups hence it assumes all cpus are available for running goroutines. Since applications on Kubernetes are running under restricted resources we should detect cgroups settings and align Go runtime to discovered settings. This should lower resource consumption, and allow us to run on even more limited resources.
Did we actually hit any throttling-related issues in the past? |
I don't recall hitting latency issues with the operator itself. As this is well know and we haven't seen an impact, so far we have been waiting for a golang to fix golang/go#33803. I suppose we could as well use the uber lib in the meantime although the impact may be blurry. |
No, we haven't, as we don't have any observability, nor any container running on low resources having a liveness/readiness probe so any slowness is not visible. As part of other PR I will add such low resource container with probes. I hit multiple times issue with such container being started but not even printing starting log message. After I configured Go runtime properly I haven't noticed it again. Can't tell whether I was lucky or it indeed fixed what I saw. This is a known issue, fixing it wouldn't hurt. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if it helps you I am fine going ahead with it and reverting this when golang/go#33803 gets fixed
/approve
/lgtm
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tnozicka, zimnx The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Of course, I just wanted to know why do it now, since I remember we discussed in the past and never followed through. Thanks. |
Flake: #1028 (comment) |
Description of your changes:
Golang runtime doesn't understand cgroups hence it assumes all cpus are
available for running goroutines.
Since applications on Kubernetes are running under restricted resources
we should detect cgroups settings and align Go runtime to discovered settings.
This should lower resource consumption, and allow us to run on even more limited resources.