[Security Solution]Incorrect Total Alert count under Group alert by filter

[ghost]

Describe the bug
Incorrect Total Alert count under Group alert by filter

Build Details:

Version: 8.7.0-BC1
Commit: d3b239d76aa04f073836f6100782134ac86887e2
Build: 60614

Preconditions

• Elastic Agent with Elastic Defend integrations should be installed
• Generate some Alert data on the build

Steps

• Go to Security App
• Navigate to Alert Page
• Apply the Group Alert by Filter let say choose Group Alert by: Username
• Grouped Alert Will show up
• Observer the Total Alert count on alert table middle left position ( let say this as A )
• Observe the count of each group and sum it ( let say this as B )
• Observed that Count of A and B is not matching

Additionally in one of the group internal table alert count and group alert count is also not matching

Screen-Cast

- Group Alert by : None : 12241 Alerts
- Group Alert by : Source IP: 12341 Alerts ( increase of 100 in count ) ❌
- 5 Group Alerts Total : ( 927 + 372 +7 +7 + 2 : 1315) ❌

if below video not works try this link

Alerts.-.Kibana.Mozilla.Firefox.2023-02-10.15-19-44.mp4

- Group outer count : 927
- Expanded Group alert table count: 964

Inspect Information

Request

  "size": 0,
  "aggs": {
    "stackByMultipleFields0": {
      "terms": {
        "field": "source.ip",
        "size": 10000
      },
      "aggs": {
        "bucket_truncate": {
          "bucket_sort": {
            "from": 0,
            "size": 25
          }
        },
        "alertsCount": {
          "cardinality": {
            "field": "kibana.alert.uuid"
          }
        },
        "rulesCountAggregation": {
          "cardinality": {
            "field": "kibana.alert.rule.rule_id"
          }
        },
        "countSeveritySubAggregation": {
          "cardinality": {
            "field": "kibana.alert.severity"
          }
        },
        "severitiesSubAggregation": {
          "terms": {
            "field": "kibana.alert.severity"
          }
        },
        "usersCountAggregation": {
          "cardinality": {
            "field": "host.name"
          }
        }
      }
    },
    "alertsCount": {
      "terms": {
        "field": "kibana.alert.rule.producer",
        "exclude": [
          "alerts"
        ]
      }
    },
    "groupsNumber": {
      "cardinality": {
        "field": "source.ip"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "term": {
                  "kibana.alert.workflow_status": "open"
                }
              }
            ],
            "should": [],
            "must_not": [
              {
                "exists": {
                  "field": "kibana.alert.building_block_type"
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2023-02-09T18:30:00.000Z",
              "lte": "2023-02-10T18:29:59.999Z"
            }
          }
        }
      ]
    }
  },
  "runtime_mappings": {
    "Enhanced Time": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['@timestamp'].value.getDayOfWeekEnum().toString())"
      }
    }
  },
  "_source": false
}

Response

  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "groupsNumber": {
      "value": 5
    },
    "alertsCount": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "siem",
          "doc_count": 12341
        }
      ]
    },
    "stackByMultipleFields0": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "172.31.9.210",
          "doc_count": 927,
          "alertsCount": {
            "value": 927
          },
          "rulesCountAggregation": {
            "value": 1
          },
          "severitiesSubAggregation": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "medium",
                "doc_count": 927
              }
            ]
          },
          "countSeveritySubAggregation": {
            "value": 1
          },
          "usersCountAggregation": {
            "value": 1
          }
        },
        {
          "key": "127.0.0.1",
          "doc_count": 372,
          "alertsCount": {
            "value": 372
          },
          "rulesCountAggregation": {
            "value": 1
          },
          "severitiesSubAggregation": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "medium",
                "doc_count": 372
              }
            ]
          },
          "countSeveritySubAggregation": {
            "value": 1
          },
          "usersCountAggregation": {
            "value": 1
          }
        },
        {
          "key": "59.89.33.107",
          "doc_count": 7,
          "alertsCount": {
            "value": 7
          },
          "rulesCountAggregation": {
            "value": 1
          },
          "severitiesSubAggregation": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "medium",
                "doc_count": 7
              }
            ]
          },
          "countSeveritySubAggregation": {
            "value": 1
          },
          "usersCountAggregation": {
            "value": 1
          }
        },
        {
          "key": "223.178.210.153",
          "doc_count": 7,
          "alertsCount": {
            "value": 7
          },
          "rulesCountAggregation": {
            "value": 1
          },
          "severitiesSubAggregation": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "medium",
                "doc_count": 7
              }
            ]
          },
          "countSeveritySubAggregation": {
            "value": 1
          },
          "usersCountAggregation": {
            "value": 1
          }
        },
        {
          "key": "185.167.96.150",
          "doc_count": 2,
          "alertsCount": {
            "value": 2
          },
          "rulesCountAggregation": {
            "value": 1
          },
          "severitiesSubAggregation": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "medium",
                "doc_count": 2
              }
            ]
          },
          "countSeveritySubAggregation": {
            "value": 1
          },
          "usersCountAggregation": {
            "value": 1
          }
        }
      ]
    }
  }
}

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[MadameSheema]

@karanbirsingh-qasource may you please confirm if you have any other filter activated? i.e. building block alerts? Per the attached screencast seems that there is also a mismatch between the Severity Levels card and Alerts by Type. Can you please take a look at that as well?? Thanks! :)

[ghost]

Hi @MadameSheema

No additional filter is applied please find the new screen-cast and for severity there is an another issue here #150278

group by : 3048
Alert Table count: 22604

Alerts.-.Kibana.Mozilla.Firefox.2023-02-10.18-15-09.mp4

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[stephmilovic]

@karanbirsingh-qasource the issue was fixed. Can you please confirm the fix in BC4? Thanks!

[ghost]

Hi @stephmilovic

We have validated this issue on 8.7 BC4 and found the issue to be fixed ✔️ . Correct count is showing on complete alert page including the charts , groups .

Build Details:

Version: 8.7.0 BC4
Commit: de22cd9361a0dbf429f9648d3c7b7c45aa862e90
Build: 60949

screen-cast

Alerts.-.Kibana.Mozilla.Firefox.2023-03-06.11-29-12.mp4

Hence we are closing this issue and adding "QA:Validated"