[Stack Monitoring] create alert per node, index, or cluster instead of always per cluster

[neptunian]

Resolves #100136. This is part of Stack Monitoring alerting alignment where we want to create an alert on where the alert occured and not always on the cluster. This means we also change the alert ids to reflect that.

• Creates an alert per node or per index for some alerts instead of all per cluster.
• Action Messaging Template Changes for each Rule type
• Alert messaging changes to reflect the new context (per node, per index)
• Bug fixes with alerts

Changes

Bug fix for Threadpool alerts

I don't think Threadpool search and write rejection alerts are working in production. Testing on master will give you errors due to the wrong parameters being passed into executeActions. This was probably not caught because it's never been checked off in any test scenarios due to testers not having an easy way to trigger the alert. This will be resolved in this PR.

Bug fix for Threadpool alerts UI message

Threadpool rejection counts were not showing up in the UI message Before/after:

CPU Usage Alert Messaging

internalShortMessage, before and after. The first alert is the old message, where with new alerting changes there would only ever be 1 node in the list. The last couple lines are the new message.

internalFullMessage, before and after. The top messages are the old messages, where with new alerting changes there would only ever be 1 node in the list. The last messages are the new messages. Note that the link was changed to a global state link, like other alerts, and instead of taking the user to the node view

Remove template variables nodes and count, though they will still continue to be sent "behind the scenes" for users already using them, where count always equals 1 and nodes is always one node.

Add template variable node, which functions the same as nodes but will only have one node in the value, eg: mynodeId:cpuUsage

Disk Usage Alert Messaging

All the same changes as CPU Usage rule above, but replace "CPU usage" with "Disk usage"

Eg of internalShortMessage, before and after

Eg of internalFullMessage, before and after

Memory Usage (JVM) Alert Messaging

All the same changes as CPU Usage rule above, but replace "CPU usage" with "Memory usage"

Eg of internalShortMessage, before and after

Eg of internalFullMessage, before and after

Missing Monitoring Data Alert Messaging

All the same changes as CPU Usage rule above, but replace "CPU usage" with "Missing monitoring data"

Eg of internalShortMessage, before and after

Eg of internalFullMessage, before and after

Thread pool Search Rejections Alert Messaging

All the same changes as CPU Usage rule above, but replace "CPU usage" with "threadpool search rejections"

Eg of internalShortMessage, before and after

Eg of internalFullMessage, before and after

Thread pool Write Rejections Alert Messaging

All the same changes as Thread pool Write Rejections rule above

internalShortMessage

internalFullMessage

Shard Size Alert ID and Messaging change

This alert becomes a per index alert:

Before

Now the alert id is the instance_id returned from elasticsearch: clusterId:shardIndex.

Messaging changes:
internalShortMessage
Before:
Large shard size alert is firing for the following indices: apm-8.0.0-error-000001, apm-8.0.0-metric-000001, apm-8.0.0-onboarding-2021.06.22, apm-8.0.0-profile-000001, apm-8.0.0-span-000001, apm-8.0.0-transaction-000001, metricbeat-8.0.0-2021.06.24-000001, twitter. Investigate indices with large shard sizes.

After:
Large shard size alert is firing for the following index: apm-8.0.0-onboarding-2021.06.22. Investigate indices with large shard sizes.
Large shard size alert is firing for the following index: apm-8.0.0-metric-000001. Investigate indices with large shard sizes.
Large shard size alert is firing for the following index: apm-8.0.0-error-000001. Investigate indices with large shard sizes.
Large shard size alert is firing for the following index: apm-8.0.0-span-000001. Investigate indices with large shard sizes.
Large shard size alert is firing for the following index: twitter. Investigate indices with large shard sizes.

internalFullMessage
Before:
Large shard size alert is firing for the following indices: apm-8.0.0-error-000001, apm-8.0.0-metric-000001, apm-8.0.0-onboarding-2021.06.22, apm-8.0.0-profile-000001, apm-8.0.0-span-000001, apm-8.0.0-transaction-000001, metricbeat-8.0.0-2021.06.24-000001, twitter. [View index shard size stats](http://localhost:5603/ssg/app/monitoring#/elasticsearch/indices?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ))

After:
Large shard size alert is firing for the following index: apm-8.0.0-error-000001. [View index shard size stats](http://localhost:5603/ssg/app/monitoring#/elasticsearch/indices/apm-8.0.0-error-000001?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ))

Large shard size alert is firing for the following index: apm-8.0.0-transaction-000001. [View index shard size stats](http://localhost:5603/ssg/app/monitoring#/elasticsearch/indices/apm-8.0.0-transaction-000001?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ))

Large shard size alert is firing for the following index: metricbeat-8.0.0-2021.06.24-000001. [View index shard size stats](http://localhost:5603/ssg/app/monitoring#/elasticsearch/indices/metricbeat-8.0.0-2021.06.24-000001?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ))

Large shard size alert is firing for the following index: apm-8.0.0-metric-000001. [View index shard size stats](http://localhost:5603/ssg/app/monitoring#/elasticsearch/indices/apm-8.0.0-metric-000001?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ))

CCR Read Rejections alert ID and Messaging change

This alert becomes a per index alert.

Before/after (the bottom ID is the old id, the new id is the top two). Since an index could have the same name in different remote clusters we use the remote cluster in the ID as well. We do not use the "leader" cluster as discussed with @jasonrhodes the alerting framework is not across clusters.

Messaging changes:
internalShortMessage

Before:
CCR read exceptions alert is firing for the following remote clusters: remoteCluster1, remoteCluster2. Verify follower and leader index relationships across the affected remote clusters.
After:
CCR read exceptions alert is firing for the following remote cluster: remoteCluster2. Verify follower and leader index relationships on the affected remote cluster.

CCR read exceptions alert is firing for the following remote cluster: remoteCluster1. Verify follower and leader index relationships on the affected remote cluster.

internalFullMessage

Before:
CCR read exceptions alert is firing for the following remote clusters: remoteCluster1, remoteCluster2. Current 'follower_index' indices are affected: followerIndex, followerIndex2. [View CCR stats](http://localhost:5601/ssg/app/monitoring#/elasticsearch/ccr?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ,ccs:ccs))
After:
CCR read exceptions alert is firing for the following remote cluster: remoteCluster1. Current 'follower_index' index affected: followerIndex. [View CCR stats](http://localhost:5601/ssg/app/monitoring#/elasticsearch/ccr?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ,ccs:ccs))
CCR read exceptions alert is firing for the following remote cluster: remoteCluster2. Current 'follower_index' index affected: followerIndex2. [View CCR stats](http://localhost:5601/ssg/app/monitoring#/elasticsearch/ccr?_g=(cluster_uuid:22FsxZ7mRs6tINBAPq-9lQ,ccs:ccs))

Elasticsearch cluster health, Elasticsearch version mismatch, Kibana version mismatch, Logstash version mismatch, Elasticsearch nodes changed, Elasticsearch license expiration

These alerts remain cluster level and the alert id is the cluster id, as they were previously, minus a colon that was at the end of id. Everything should function and look the same.

Test

• The UI should look as it did previously. Though we only created one alert we still displayed it to the user as multiple alerts. For eg one alert for a cluster with 4 nodes firing is still displayed as 4 alerts in the UI listing each node the alert takes place on. Now we actually do have 4 alerts, one per node, and each node should be listed as an alert. Whether looking at an old alert, or a new "per node" alert, the API fetch handles them the same, so no changes for "backward compatibility" are necessary.

• When viewing the alerts in Stack Management, you should see alerts per node (with id being the node id) and not per cluster. In this example the the active alerts are the new alerts and the old alert with the old ID is no longer active

Helpful for generating the alerts:
#87377 . Threadpool and CCR alerts, I still need to find an easier way to reproduce but here are some notes: #93072 (comment), #85908

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[neptunian]

@elasticmachine merge upstream

[neptunian]

Here is where I've moved the triggering of the new alert into the inner for loop that loops through each node, instead of in the cluster for loop.

[neptunian]

Some alerts are cluster level and do not have a nodeId or instanceId property, so use the clusterUuid. This should be typed and improved at some point or have a different function for cluster vs node level processing.

[neptunian]

This was getting the wrong data was not able to destructure the values further down resulting in a bug for Threadpool alerts.

[estermv]

LGTM 🚀

I couldn't fire all the alarms but the code looks good.

[ravikesarwani]

@neptunian If I understand this correctly we have changed the alert ID to be just the node ID.

When viewing the alerts in Stack Management, you should see alerts per node (with id being the node id) and not per cluster. In this example the the active alerts are the new alerts and the old alert with the old ID is no longer active

In most customer use cases monitoring cluster is used to consolidate many production clusters monitoring data. Each production cluster can have an instance say "instance-0000000001". If the alert id is only nodeID then it may not be unique. Do we need the "clusterid:nodeID" to make the alert ID unique?

[neptunian]

@ravikesarwani: "instance-0000000001" is the node name but the node id should be unique. You can see here in the url, the ID used bMihNvXbR7iUs4A2U-s-UQ:

@jasonrhodes Do you see any issue with just using the node ID?

[jasonrhodes]

I actually don't know what generates the node IDs but I imagine it's a UUID as well? If so, it should be safe/unique.

[simianhacker]

@jasonrhodes @ravikesarwani I believe the Alerts team said that those ids only need to be unique per alert.

[jasonrhodes]

@simianhacker yes, that's correct. I think Ravi is wondering whether, if you have multiple nodes whose CPU is spiking at the same time, across different clusters, could those nodes each have the same Node ID? It would all be within the one rule, but if those Node IDs aren't unique, there could theoretically be a collision. I assume they're UUIDs but I'm not sure, it'd be nice to confirm that.

[neptunian]

We are using node_stats.node_id which matches the source_node.uuid. poIczYSkR-aSlh3jXwK6QQ in this example. So I think it's safe to assume it's a uuid.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 0a0b773
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #134908 succeeded 0a0b773
• 💚 Build #134257 succeeded f3034ca
• 💔 Build #134239 failed 5c9f377
• 💔 Build #133755 failed 4957b68
• 💔 Build #133181 failed e08158e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream