[Stack Monitoring] [Test Scenario] Out of the box alerting

[simianhacker]

Summary

Stack Monitoring provides a set of out-of-the-box alerts, created by simply loading the Stack Monitoring UI within Kibana. The default action for each alert is a server log and the action messaging is controlled by the Stack Monitoring UI code directly.

PRs

Original, and CPU alert: #68805
Disk usage alert: #75419
JVM memory usage alert: #79039
Missing monitoring data alert: #78208
Threadpool rejections alert: #79433

Testing

Creation

• Ensure alerts are created once visiting the Stack Monitoring UI
• Ensure a user with the minimum set of monitoring permissions is able to create and manage alerts
  • ⚠️ this requires extra permissions, that should be documented (and reflected in the test scenario)

Management

• Ensure that you can view and manage these alerts when Setup Mode is active
  • 🪲 Alert editing fly-out shows outdated alert parameters after editing ([Stack Monitoring] Alert editing fly-out shows outdated alert parameters after editing #94440)
• Ensure you can properly add an additional action (to any alert) and it works as expected

UX

• Ensure you can see all created alerts while in Setup Mode
• Ensure you can properly navigate between display modes of alerts [Monitoring] Some progress on making alerts better in the UI #81569
  • 🪲 Alert grouping toggle never renders as "enabled" ([Stack Monitoring] Alert grouping toggle never renders as "enabled" #94556)
• Ensure a toast appears in the UI when you first visit SM telling you about new alerts [Monitoring] Add toast for newly created alerts #89202

Specific alerts

• Ensure you can properly trigger and see the server log for the CPU usage alert
• Ensure you can properly trigger and see the server log for the disk usage alert
• Ensure you can properly trigger and see the server log for the jvm memory alert
• Ensure you can properly trigger and see the server log for the missing monitoring data alert (Note: This alert is only concerned with Elasticsearch now and no longer looks at other stack products [Monitoring] Missing monitoring data alert firing for version upgrade and configuration changes for Kibana in Cloud #83309)
• Ensure you can properly trigger and see the server log for both threadpool rejection alerts
  • 🤔 I was unable to create the conditions for this.

Information in reproducting legacy alerts -> #87377

• Ensure you can properly trigger and see the server log for the legacy cluster health alert
• Ensure you can properly trigger and see the server log for the legacy nodes change alert
• Ensure you can properly trigger and see the server log for the legacy Elasticsearch version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy Kibana version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy Logstash version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy license expiration alert (This cannot be tested on cloud)
  • 🤔 I was unable to create the conditions for this.

Edge cases

• Ensure a CCS environment works as expected
• Ensure Setup Mode works as expected on cloud ([Monitoring] Ensure setup mode works on cloud but only for alerts #73127)
• Ensure a dedicated monitoring cluster environment works as expected
• Ensure a quality UX with multiple Kibanas (we recommend a dedicated Kibana for a dedicated monitoring cluster) ("Duplicate" alerts can potentially exist based on what the user is doing in each Kibana but they should be able to easily disable the ones they don't want)
  • 🤔 Not sure what that means. "Quality UX" is quite subjective, after all.
• Ensure Stack Monitoring UI works properly when alerts are not available (both because ssl is disabled and an encryption key is not set: [Monitoring] Fix UI error when alerting is not available #77179)
• Ensure Stack Monitoring alerts are not editable or createable in the Alerts & Management UI ([Monitoring] Prevent edit/create for Stack Monitoring alerts in Alerts Management #77097)
  • ⚠️ this doesn't apply anymore after [Monitoring] Reverting alert management #94167

Previous issue: #85841

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[neptunian]

@igoristic

Ensure alerts are created once visiting the Stack Monitoring UI

Can you advise on the best way to ensure the alerts are created? After visiting Stack Monitoring i went to Stack Management -> Rules and Connectors and saw the alerts in the summary. Though there were man others as well. For "Threadpool rejections" there were "Thread pool search rejections" and "Thread pool write rejections". What is meant by the "Original" alert as described in the summary?

Ensure a user with the minimum set of monitoring permissions is able to create and manage alerts

Is creating a user and assigning it to the monitoring role fulfill this requirement? I'm not sure what qualifies as the "minimum set of monitoring permissions".

[neptunian]

I'm not sure this is related, but after the default alerts were created I started getting this in the logs:

which appears to be coming from:

I'm using the release cluster

[neptunian]

Ensure you can properly trigger and see the server log for both threadpool rejection alerts

Is there any guidance for how to create the conditions for this?

[neptunian]

I was not able to get any legacy alerts to fire using the gists. I'm using the release cluster (enterprise license) with the elastic user.

[neptunian]

Ensure Stack Monitoring alerts are not editable or createable in the Alerts & Management UI

Can this be removed? There is a comment below it that says it no longer applies. Can the other comments that were copied over also be addressed?

[simianhacker]

Already shipped