[ResponseOps] Update flapping logic order to determine whether an alert is flapping after it's returned for notification

[doakalexi]

Resolves #151135

Summary

Moved the logic around to check flapping after an alert is already returned for notification. I also updated the event log test cases to match examples in the rfc.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

To Verify

The new test cases capture this example but if you want to test it locally:

• Create a connector and make sure to save the connector id

curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'

• Create the rule and let it run

curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

• Verify that you see the following, we want to make sure that aren't missing any notifications
  

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 8b84af2

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	428	430	+2

Total ESLint disabled count

id	before	after	diff
securitySolution	506	508	+2

History

• 💔 Build #110822 failed de0b2d6
• 💚 Build #110730 succeeded f80335b
• 💚 Build #110351 succeeded 271b602
• 💛 Build #110141 was flaky 3a125b5
• 💔 Build #110114 failed ab85a88
• 💔 Build #110080 failed 0980e2d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[maryam-saeidi]

Moved the logic around to check flapping after an alert is already returned for notification.

I am trying to understand this description. How was the flapping status generation different in the previous implementation?

[doakalexi]

Moved the logic around to check flapping after an alert is already returned for notification.

I am trying to understand this description. How was the flapping status generation different in the previous implementation?

The flapping status generation didn't change. I just moved things around in this pr to match what the RFC is expecting. Before we would check to see whether an alert was flapping and then determine if that alert is going to be reported as a notification to users. The flapping status affects whether an alert is reported to the user, and so by checking flapping first we were missing notifications in examples from the RFC that were expecting them. In this pr, I moved the flapping check to be after we determine which alert is going to be reported to the user which aligns more with the RFC

[maryam-saeidi]

AO changes LGTM

[pmuellr]

LGTM, testing it locally seems to work as expected

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.7

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation