MCO-371: metrics: aggregate os_image_url_override metric to avoid unbounded cardinality #3363
Conversation
openshift-ci
bot
added
the
approved
…rdinality For context, see openshift/cluster-monitoring-operator#1784
|
From a local cluster, aggregated data looks fine when both master and worker pool has overridedn osImageURL. See screenshot @cgwalters PTAL, does this look good to go? |
|
I'll be honest, I don't know the Prometheus stuff very well. Something that wasn't clear to me was why we don't have labels/annotations/warning stuff, but I guess that's because it's not an alert? One question here - I imagine we'll want to display something about this in the GUI console - do we expect them to trigger off this metric for example? |
|
Yeah, the other stuff that was in there was for alerting. There are different "types" of rules, it just so happens that the other rules that are there for the MCC are for the paused pool alert and not recording rules. I personally like that we're doing the aggregation of the pool metrics at the "rules level" rather than emitting a separate metric from the MCC, but what do I know 😄 |
| @@ -33,6 +33,10 @@ spec: | |||
| summary: "Paused machine configuration pool '{{$labels.pool}}' is blocking a necessary certificate rotation and must be unpaused before the current kube-apiserver-to-kubelet-signer certificate expires in {{ $value | humanizeDuration }}." | |||
| description: "Machine config pools have a 'pause' feature, which allows config to be rendered, but prevents it from being rolled out to the nodes. This alert indicates that a certificate rotation has taken place, and the new kubelet-ca certificate bundle has been rendered into a machine config, but because the pool '{{$labels.pool}}' is paused, the config cannot be rolled out to the nodes in that pool. You will notice almost immediately that for nodes in pool '{{$labels.pool}}', pod logs will not be visible in the console and interactive commands (oc log, oc exec, oc debug, oc attach) will not work. You must unpause machine config pool '{{$labels.pool}}' to let the certificates through before the kube-apiserver-to-kubelet-signer certificate expires. You have approximately {{ $value | humanizeDuration }} remaining before this happens and nodes in '{{$labels.pool}}' cease to function properly." | |||
| runbook_url: https://github.com/openshift/blob/master/alerts/machine-config-operator/MachineConfigControllerPausedPoolKubeletCA.md | |||
| - name: os-image-override.rules | |||
There was a problem hiding this comment.
Our "standards" for prometheus metrics/rules have changed over time.
- is this
name.rulesnaming for the group a convention we should be following for future rules (or just recording rules?) - do we need to worry about retrofitting?
There was a problem hiding this comment.
Our "standards" for prometheus metrics/rules have changed over time.
The rules for the rules
There was a problem hiding this comment.
I am not aware of if this is a rule or not. I picked this convention from existing rules that I came across in CMO e.g. https://github.com/openshift/cluster-monitoring-operator/blob/master/assets/cluster-monitoring-operator/prometheus-rule.yaml#L14 . I know CMO team is working on a doc, so we may get better idea of best practices once available.
+1 on what John said. Also, have found https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules helpful to understand difference between recording and alerting rules.
For console based on what we would want to show, we can use alerting with |
|
@simonpasquier @jan--f can I get a review and lgtm from one of you :) This is follow-up from openshift/cluster-monitoring-operator#1784 |
|
👍 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jan--f, sinnykumari The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test e2e-aws |
|
@sinnykumari: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
For context, see openshift/cluster-monitoring-operator#1784