Alerting rules can end up in a state where they stop running indefinitely until a user intervenes to fix the problem

[mikecote]

Alerting rules should operate continually. When a user enables the alerting rule, it shouldn't stop running or fail indefinitely until the rule is disabled.

We should audit our codebase and identify scenarios where rules stop running indefinitely. Then, based on the findings, we should propose fixes or mitigations for the scenarios that can be.

As a starting point, I am aware of the following scenarios where rules stop running indefinitely. Since I only did a brief research, we should still analyze our code in-depth and prioritize after the fact so we can start discussing solutions and priorities for each.

• An alerting rule's task tasks get deleted
• An alerting rule's task gets marked as failed
• An alerting rule's task cannot find its associated rule
• An alerting rule's API key is deleted
• An alerting rule is unable to decrypt the API key due to the Kibana encrypted saved objects encryption key changed
• The alerting rule's type no longer exists
• An alerting rule was created without an API key (security features disabled) and later on the security features got enabled making the rule miss an API key (relevant issue)
• An alerting rule's task becomes unrecognizable

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

There is still an open question with regards to rule actions

Should we consider rule actions (ex: Email, PagerDuty, etc.) part of these requirements, or would the rule actions portion be its dedicated non-functional requirement? (ex: "Users need to know when rule actions stop executing indefinitely")

@arisonl and I are trying to get an answer to this question but should be part of this research issue once we have an answer.

[arisonl]

I believe that in practice, if an action is set up, receiving the actions in the integrated system is an integral part of the rule from a use case perspective. We should assume that if it goes out, it is very important and there is a workflow associated with it, and hence having the rule running but the action failing should be anticipated to be just as bad as the rule not running at all, at least for a number of use cases.

[gmmorris]

Added a link to a related issue (it was already described in the issue, but wasn't linked to the source issue).

#118520

[ymao1]

After some research, we've concluded that there are two types of problems that alerting rules can encounter:

• Task manager related problems - These are problems with the task that backs an alerting rule. If these tasks get deleted, duplicated, or into a state where they are no longer picked up by task manager, alerting rule execution will be impacted. This is especially egregious because we can only identify these issues by inspecting the task manager index. The rule management UI doesn't provide any indication of issues occurring at the task level
• Alerting task runner related problems - These are alerting rules that execute at the requested interval but throw an error on every execution. Often these execution errors cannot be automatically fixed and require manual intervention. These errors already do show up on the rule management UI and error status is stored in the rule SO.

As part of the research, we've identified the scenarios that can lead to the specified problems. More details are available in the research document. As a result of this research, the following issues have been created:

• [Alerting] Smarter retry interval for ES Connectivity errors #122390
• Detect and prevent the use of mismatched encryption keys #92654 - With this we envision working with Kibana security to implement their vision as proposed via an RFC
• [Task Manager] Investigate better handling of unregistered task types  #122389
• [Alerting] Add telemetry to count number of failed or unrecognized rule tasks #122985
• [Alerting] Investigate recurring task to detect broken rules #122984
• [Alerting] Gracefully restore failed rules from pre-7.11 #117593 - Existing bug issue
• [alerts] provide an "explain" capability to show elasticsearch queries that alerts will run #84417 - This is only tangentially related but we briefly discussed providing an API for ad-hoc rule execution, which we could then use as part of the recurring task in 122984 to try to detect errors that otherwise only show up during rule type execution.

Closing this research issue in favor of the linked issues.