Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ResponseOps] Update flapping logic order to determine whether an alert is flapping after it's returned for notification #151135

Closed
doakalexi opened this issue Feb 14, 2023 · 1 comment · Fixed by #151148
Assignees
Labels
Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@doakalexi
Copy link
Contributor

doakalexi commented Feb 14, 2023

To align with the RFC, we just need to change the order of setting flapping on an alert and determining which alerts to return for notification.

@doakalexi doakalexi added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Feb 14, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@doakalexi doakalexi self-assigned this Feb 14, 2023
doakalexi added a commit that referenced this issue Mar 1, 2023
…rt is flapping after it's returned for notification (#151148)

Resolves #151135

## Summary

Moved the logic around to check flapping after an alert is already
returned for notification. I also updated the event log test cases to
match examples in the rfc.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### To Verify
The new test cases capture this example but if you want to test it
locally:
- Create a connector and make sure to save the connector id
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'
```
- Create the rule and let it run
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

```
- Verify that you see the following, we want to make sure that aren't
missing any notifications

![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Mar 1, 2023
…rt is flapping after it's returned for notification (elastic#151148)

Resolves elastic#151135

## Summary

Moved the logic around to check flapping after an alert is already
returned for notification. I also updated the event log test cases to
match examples in the rfc.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### To Verify
The new test cases capture this example but if you want to test it
locally:
- Create a connector and make sure to save the connector id
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'
```
- Create the rule and let it run
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

```
- Verify that you see the following, we want to make sure that aren't
missing any notifications

![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)

(cherry picked from commit 5187a6f)
kibanamachine referenced this issue Mar 1, 2023
…an alert is flapping after it's returned for notification (#151148) (#152507)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[ResponseOps] Update flapping logic order to determine whether an
alert is flapping after it's returned for notification
(#151148)](#151148)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Alexi
Doak","email":"109488926+doakalexi@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-03-01T20:19:38Z","message":"[ResponseOps]
Update flapping logic order to determine whether an alert is flapping
after it's returned for notification (#151148)\n\nResolves
https://github.com/elastic/kibana/issues/151135\r\n\r\n##
Summary\r\n\r\nMoved the logic around to check flapping after an alert
is already\r\nreturned for notification. I also updated the event log
test cases to\r\nmatch examples in the rfc.\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### To
Verify\r\nThe new test cases capture this example but if you want to
test it\r\nlocally:\r\n- Create a connector and make sure to save the
connector id\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/actions/connector\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"connector_type_id\":
\".server-log\",\r\n \"name\": \"server log\",\r\n \"config\": {},\r\n
\"secrets\": {}\r\n}'\r\n```\r\n- Create the rule and let it
run\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/alerting/rule/\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"rule_type_id\":
\"example.pattern\",\r\n \"name\": \"pattern\",\r\n \"schedule\": {\r\n
\"interval\": \"5s\"\r\n },\r\n \"actions\": [\r\n { \"group\":
\"default\", \"id\": {connector id}, \"params\": { \"message\":
\"{{alert.id}} active on run {{context.runs}} step
{{context.patternIndex}} flapping {{alert.flapping}}\"}},\r\n {
\"group\": \"recovered\", \"id\": {connector id}, \"params\": {
\"message\": \"{{alert.id}} recovered on run flapping
{{alert.flapping}}\"}}\r\n ],\r\n \"consumer\": \"alerts\",\r\n
\"tags\": [],\r\n \"notify_when\": \"onActionGroupChange\",\r\n
\"params\": {\r\n \"patterns\": {\r\n \"instA\": \" a - - a - a - a - a
- - - - - - - - \"\r\n }\r\n }\r\n}'\r\n\r\n```\r\n- Verify that you see
the following, we want to make sure that aren't\r\nmissing any
notifications\r\n\r\n![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)","sha":"5187a6f9aa706a94dbf759b12cc11940698a81b5","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","v8.7.0","v8.8.0"],"number":151148,"url":"https://github.com/elastic/kibana/pull/151148","mergeCommit":{"message":"[ResponseOps]
Update flapping logic order to determine whether an alert is flapping
after it's returned for notification (#151148)\n\nResolves
https://github.com/elastic/kibana/issues/151135\r\n\r\n##
Summary\r\n\r\nMoved the logic around to check flapping after an alert
is already\r\nreturned for notification. I also updated the event log
test cases to\r\nmatch examples in the rfc.\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### To
Verify\r\nThe new test cases capture this example but if you want to
test it\r\nlocally:\r\n- Create a connector and make sure to save the
connector id\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/actions/connector\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"connector_type_id\":
\".server-log\",\r\n \"name\": \"server log\",\r\n \"config\": {},\r\n
\"secrets\": {}\r\n}'\r\n```\r\n- Create the rule and let it
run\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/alerting/rule/\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"rule_type_id\":
\"example.pattern\",\r\n \"name\": \"pattern\",\r\n \"schedule\": {\r\n
\"interval\": \"5s\"\r\n },\r\n \"actions\": [\r\n { \"group\":
\"default\", \"id\": {connector id}, \"params\": { \"message\":
\"{{alert.id}} active on run {{context.runs}} step
{{context.patternIndex}} flapping {{alert.flapping}}\"}},\r\n {
\"group\": \"recovered\", \"id\": {connector id}, \"params\": {
\"message\": \"{{alert.id}} recovered on run flapping
{{alert.flapping}}\"}}\r\n ],\r\n \"consumer\": \"alerts\",\r\n
\"tags\": [],\r\n \"notify_when\": \"onActionGroupChange\",\r\n
\"params\": {\r\n \"patterns\": {\r\n \"instA\": \" a - - a - a - a - a
- - - - - - - - \"\r\n }\r\n }\r\n}'\r\n\r\n```\r\n- Verify that you see
the following, we want to make sure that aren't\r\nmissing any
notifications\r\n\r\n![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)","sha":"5187a6f9aa706a94dbf759b12cc11940698a81b5"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/151148","number":151148,"mergeCommit":{"message":"[ResponseOps]
Update flapping logic order to determine whether an alert is flapping
after it's returned for notification (#151148)\n\nResolves
https://github.com/elastic/kibana/issues/151135\r\n\r\n##
Summary\r\n\r\nMoved the logic around to check flapping after an alert
is already\r\nreturned for notification. I also updated the event log
test cases to\r\nmatch examples in the rfc.\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### To
Verify\r\nThe new test cases capture this example but if you want to
test it\r\nlocally:\r\n- Create a connector and make sure to save the
connector id\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/actions/connector\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"connector_type_id\":
\".server-log\",\r\n \"name\": \"server log\",\r\n \"config\": {},\r\n
\"secrets\": {}\r\n}'\r\n```\r\n- Create the rule and let it
run\r\n```\r\ncurl -X POST -u {username}:{password}
\"http://localhost:5601/api/alerting/rule/\" -H 'kbn-xsrf: true' -H
'Content-Type: application/json' -d'\r\n{\r\n \"rule_type_id\":
\"example.pattern\",\r\n \"name\": \"pattern\",\r\n \"schedule\": {\r\n
\"interval\": \"5s\"\r\n },\r\n \"actions\": [\r\n { \"group\":
\"default\", \"id\": {connector id}, \"params\": { \"message\":
\"{{alert.id}} active on run {{context.runs}} step
{{context.patternIndex}} flapping {{alert.flapping}}\"}},\r\n {
\"group\": \"recovered\", \"id\": {connector id}, \"params\": {
\"message\": \"{{alert.id}} recovered on run flapping
{{alert.flapping}}\"}}\r\n ],\r\n \"consumer\": \"alerts\",\r\n
\"tags\": [],\r\n \"notify_when\": \"onActionGroupChange\",\r\n
\"params\": {\r\n \"patterns\": {\r\n \"instA\": \" a - - a - a - a - a
- - - - - - - - \"\r\n }\r\n }\r\n}'\r\n\r\n```\r\n- Verify that you see
the following, we want to make sure that aren't\r\nmissing any
notifications\r\n\r\n![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)","sha":"5187a6f9aa706a94dbf759b12cc11940698a81b5"}}]}]
BACKPORT-->

Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com>
sloanelybutsurely pushed a commit to sloanelybutsurely/kibana that referenced this issue Mar 8, 2023
…rt is flapping after it's returned for notification (elastic#151148)

Resolves elastic#151135

## Summary

Moved the logic around to check flapping after an alert is already
returned for notification. I also updated the event log test cases to
match examples in the rfc.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### To Verify
The new test cases capture this example but if you want to test it
locally:
- Create a connector and make sure to save the connector id
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'
```
- Create the rule and let it run
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

```
- Verify that you see the following, we want to make sure that aren't
missing any notifications

![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
…rt is flapping after it's returned for notification (elastic#151148)

Resolves elastic#151135

## Summary

Moved the logic around to check flapping after an alert is already
returned for notification. I also updated the event log test cases to
match examples in the rfc.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### To Verify
The new test cases capture this example but if you want to test it
locally:
- Create a connector and make sure to save the connector id
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'
```
- Create the rule and let it run
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

```
- Verify that you see the following, we want to make sure that aren't
missing any notifications

![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)
nkhristinin pushed a commit that referenced this issue Mar 22, 2023
…rt is flapping after it's returned for notification (#151148)

Resolves #151135

## Summary

Moved the logic around to check flapping after an alert is already
returned for notification. I also updated the event log test cases to
match examples in the rfc.


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### To Verify
The new test cases capture this example but if you want to test it
locally:
- Create a connector and make sure to save the connector id
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/actions/connector" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "connector_type_id": ".server-log",
  "name": "server log",
  "config": {},
  "secrets": {}
}'
```
- Create the rule and let it run
```
curl -X POST -u {username}:{password} "http://localhost:5601/api/alerting/rule/" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "rule_type_id": "example.pattern",
  "name": "pattern",
  "schedule": {
    "interval": "5s"
  },
  "actions": [
    { "group": "default", "id": {connector id}, "params": { "message": "{{alert.id}} active on run {{context.runs}} step {{context.patternIndex}} flapping {{alert.flapping}}"}},
    { "group": "recovered", "id": {connector id}, "params": { "message": "{{alert.id}} recovered on run flapping {{alert.flapping}}"}}
  ],
  "consumer": "alerts",
  "tags": [],
  "notify_when": "onActionGroupChange",
  "params": {
    "patterns": {
      "instA": " a - - a - a - a - a - - - - - - - - "
    }
  }
}'

```
- Verify that you see the following, we want to make sure that aren't
missing any notifications

![image](https://user-images.githubusercontent.com/109488926/221615405-48061d10-4b80-4b98-812a-6951da4178da.png)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
2 participants