Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution][Detections][Threshold Rules] Filtering by Cardinality may miss alerts when bucket count is high #95258

Closed
madirey opened this issue Mar 24, 2021 · 4 comments
Closed

[Security Solution][Detections][Threshold Rules] Filtering by Cardinality may miss alerts when bucket count is high #95258

madirey opened this issue Mar 24, 2021 · 4 comments
Assignees
@madirey
Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Alerts Security Solution Detection Alerts Feature impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0

Comments

@madirey
Copy link
Contributor

madirey commented Mar 24, 2021

When filtering by cardinality, Threshold Rules use a cardinality leaf aggregation with a bucket_selector aggregation to limit the result set. Since the terms aggregation may return at most 10,000 buckets (the default ES limit), some buckets may be excluded in the outer aggregation. If a cardinality is provided, the user might expect that higher cardinality buckets would be returned as signals first, but some of those may have been excluded in the original terms agg results.

To prevent this, we should sort the terms agg buckets by cardinality, using something like the following:
image (8)
@madirey madirey added bug Fixes for quality problems that affect the customer experience Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Detection Alerts Security Solution Detection Alerts Feature v7.12.1 labels Mar 24, 2021
@madirey madirey self-assigned this Mar 24, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@MadameSheema
Copy link
Member

@deepikakeshav-qasource can you please validate this ticket on 7.12.1BC? Thanks :)

@MadameSheema MadameSheema added the Team:Detections and Resp Security Detection Response Team label Apr 20, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@ghost
Copy link

ghost commented Apr 20, 2021

Hi @MadameSheema,

We have validated this ticket on the 7.12.1 BC3 build. Please find our below observations:

Build Details:

Version:7.12.1 BC3
Commit:d03a18355d794e508183a34734141626ce74b2c4
Build:39452

  1. We have run the below post command:
    POST /.siem-signals-default-000001/_search

  2. Then we run the below GET hit with field: "event.category":
    GET /_search
    {
    "aggs": {
    "genres": {
    "terms" : {
    "field": "event.category",
    "order": {"max_play_count": "desc"}
    },
    "aggs": {
    "max_play_count": { "max": {"field": "play_count"}}
    }
    }
    }
    }

Response: 

{
  "took" : 1025,
  "timed_out" : false,
  "_shards" : {
    "total" : 51,
    "successful" : 51,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "x9NA7ngB_S0EmzxWPGmH",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:06.583Z",
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "log.origin" : {
            "file.line" : 18,
            "file.name" : "warn/warn.go"
          },
          "message" : "The Elastic Agent is currently in BETA and should not be used in production",
          "agent" : {
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b"
          },
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            }
          },
          "log.level" : "info",
          "ecs" : {
            "version" : "1.8.0"
          },
          "host" : {
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "name" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705"
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "4NNA7ngB_S0EmzxWPGmH",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.710Z",
          "log.level" : "error",
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-watcher-json.log"""
            }
          },
          "log.origin" : {
            "file.name" : "cmd/watch.go",
            "file.line" : 61
          },
          "message" : """failed to load markeropen C:\Program Files\Elastic\Agent\data\.update-marker: The system cannot find the file specified.""",
          "host" : {
            "os" : {
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "name" : "qasource-deepika",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "agent" : {
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b"
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "5NNA7ngB_S0EmzxWPGmH",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.709Z",
          "agent" : {
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e"
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "message" : "Detecting execution mode",
          "log.level" : "info",
          "log.origin" : {
            "file.name" : "application/application.go",
            "file.line" : 65
          },
          "input" : {
            "type" : "log"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "snapshot" : false,
            "version" : "7.12.1",
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e"
          },
          "host" : {
            "architecture" : "x86_64",
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "name" : "qasource-deepika"
          },
          "log" : {
            "offset" : 232,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            }
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "79NA7ngB_S0EmzxWPGmH",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.711Z",
          "log.level" : "info",
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "version" : "7.12.1",
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false
          },
          "message" : "Agent is managed locally",
          "log.origin" : {
            "file.name" : "application/application.go",
            "file.line" : 74
          },
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "agent" : {
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "host" : {
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "name" : "qasource-deepika",
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows"
            }
          },
          "log" : {
            "offset" : 427,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            }
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "9NNA7ngB_S0EmzxWPGmH",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.711Z",
          "log.level" : "info",
          "log" : {
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            },
            "offset" : 622
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "host" : {
            "name" : "qasource-deepika",
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ]
          },
          "agent" : {
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika"
          },
          "log.origin" : {
            "file.name" : "capabilities/capabilities.go",
            "file.line" : 59
          },
          "message" : """capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml""",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "version" : "7.12.1",
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "YNNA7ngB_S0EmzxWP2rk",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.711Z",
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-watcher-json.log.1"""
            }
          },
          "log.level" : "error",
          "log.origin" : {
            "file.name" : "cmd/watch.go",
            "file.line" : 61
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "host" : {
            "name" : "qasource-deepika",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705"
          },
          "agent" : {
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika"
          },
          "message" : """failed to load markeropen C:\Program Files\Elastic\Agent\data\.update-marker: The system cannot find the file specified.""",
          "ecs" : {
            "version" : "1.8.0"
          },
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "dataset" : "elastic_agent",
            "namespace" : "default",
            "type" : "logs"
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "Y9NA7ngB_S0EmzxWP2rk",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.711Z",
          "ecs" : {
            "version" : "1.8.0"
          },
          "log.level" : "info",
          "log.logger" : "composable",
          "message" : "EXPERIMENTAL - Inputs with variables are currently experimental and should not be used in production",
          "agent" : {
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "log" : {
            "offset" : 877,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            }
          },
          "log.origin" : {
            "file.name" : "composable/controller.go",
            "file.line" : 44
          },
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "host" : {
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500"
            },
            "name" : "qasource-deepika",
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "ip" : [
              "10.0.6.178"
            ]
          }
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "bdNA7ngB_S0EmzxWP2rk",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.712Z",
          "log" : {
            "offset" : 1172,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log.1"""
            }
          },
          "log.level" : "info",
          "data_stream" : {
            "type" : "logs",
            "dataset" : "elastic_agent",
            "namespace" : "default"
          },
          "host" : {
            "name" : "qasource-deepika",
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64"
          },
          "log.origin" : {
            "file.name" : "docker/docker.go",
            "file.line" : 43
          },
          "message" : "Docker provider skipped, unable to connect: protocol not available",
          "input" : {
            "type" : "log"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "agent" : {
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "log.logger" : "composable.providers.docker"
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "btNA7ngB_S0EmzxWP2rk",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.712Z",
          "log.origin" : {
            "file.name" : "warn/warn.go",
            "file.line" : 18
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "host" : {
            "mac" : [
              "00:50:56:b1:6d:a6"
            ],
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)",
              "build" : "18363.1500",
              "type" : "windows"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "name" : "qasource-deepika",
            "ip" : [
              "10.0.6.178"
            ]
          },
          "agent" : {
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b",
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log"""
            }
          },
          "log.level" : "info",
          "input" : {
            "type" : "log"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "elastic_agent"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "message" : "The Elastic Agent is currently in BETA and should not be used in production"
        }
      },
      {
        "_index" : ".ds-logs-elastic_agent-default-2021.04.20-000001",
        "_type" : "_doc",
        "_id" : "cdNA7ngB_S0EmzxWP2rk",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-04-20T07:48:08.712Z",
          "data_stream" : {
            "dataset" : "elastic_agent",
            "namespace" : "default",
            "type" : "logs"
          },
          "event" : {
            "dataset" : "elastic_agent"
          },
          "elastic_agent" : {
            "id" : "b0ead940-a1ac-11eb-91e8-5f2a1c592c4e",
            "snapshot" : false,
            "version" : "7.12.1"
          },
          "ecs" : {
            "version" : "1.8.0"
          },
          "host" : {
            "hostname" : "qasource-deepika",
            "architecture" : "x86_64",
            "os" : {
              "build" : "18363.1500",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.1500 (WinBuild.160101.0800)"
            },
            "id" : "4143c277-074e-47a9-b37d-37f94b508705",
            "name" : "qasource-deepika",
            "ip" : [
              "10.0.6.178"
            ],
            "mac" : [
              "00:50:56:b1:6d:a6"
            ]
          },
          "log" : {
            "offset" : 232,
            "file" : {
              "path" : """C:\Program Files\Elastic\Agent\data\elastic-agent-b5ea30\logs\elastic-agent-json.log"""
            }
          },
          "log.level" : "info",
          "log.origin" : {
            "file.name" : "application/application.go",
            "file.line" : 65
          },
          "message" : "Detecting execution mode",
          "agent" : {
            "id" : "77f4ae4e-1975-4199-adb4-8b0b43ec7c7e",
            "name" : "qasource-deepika",
            "type" : "filebeat",
            "version" : "7.12.1",
            "hostname" : "qasource-deepika",
            "ephemeral_id" : "de33a496-e1d2-422b-b0c5-197e80f05b3b"
          },
          "input" : {
            "type" : "log"
          }
        }
      }
    ]
  },
  "aggregations" : {
    "genres" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 42,
      "buckets" : [
        {
          "key" : "authentication",
          "doc_count" : 4432,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "file",
          "doc_count" : 483,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "host",
          "doc_count" : 27,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "iam",
          "doc_count" : 5909,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "intrusion_detection",
          "doc_count" : 9,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "library",
          "doc_count" : 155,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "malware",
          "doc_count" : 9,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "network",
          "doc_count" : 537,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "process",
          "doc_count" : 507,
          "max_play_count" : {
            "value" : null
          }
        },
        {
          "key" : "registry",
          "doc_count" : 2266,
          "max_play_count" : {
            "value" : null
          }
        }
      ]
    }
  }
}

Screenshot:
Get_hit

Please let us know if we are missing something. It would be really helpful if you can provide more information to validate this ticket.

Thanks!!

@peluja1012 peluja1012 added v7.14.0 impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. and removed v7.12.1 labels May 12, 2021
@peluja1012 peluja1012 added v7.15.0 and removed v7.14.0 labels Jul 7, 2021
@madirey madirey mentioned this issue Jul 7, 2021
Closed
13 tasks
madirey added a commit to madirey/kibana that referenced this issue Sep 8, 2021
@madirey
Fix elastic#95258 - cardinality sort bug
ee7ce2a
@madirey madirey mentioned this issue Sep 9, 2021
Merged
9 tasks
madirey added a commit to madirey/kibana that referenced this issue Sep 14, 2021
@madirey
[RAC][Security Solution] Adds Threshold rule type and removes relianc…
5a555d1 
…e on outputIndex (elastic#111437)

* Initial commit

* Properly handle signal history

* Fix elastic#95258 - cardinality sort bug

* Init threshold rule

* Create working threshold rule

* Fix threshold signal generation

* Fix tests

* Update mappings

* ALERT_TYPE_ID => RULE_TYPE_ID

* Add tests

* Fix types
madirey added a commit that referenced this issue Sep 23, 2021
@madirey
[RAC][Security Solution] Adds Threshold rule type and removes relianc…
82dc3ec 
…e on outputIndex (#111437) (#112161)

* Initial commit

* Properly handle signal history

* Fix #95258 - cardinality sort bug

* Init threshold rule

* Create working threshold rule

* Fix threshold signal generation

* Fix tests

* Update mappings

* ALERT_TYPE_ID => RULE_TYPE_ID

* Add tests

* Fix types
madirey added a commit that referenced this issue Oct 26, 2021
@madirey @marshallmain
[RAC][Security Solution] Adds migration to new SecuritySolution rule …
117efdf 
…types (#112113)

* Initial commit

* Properly handle signal history

* Fix #95258 - cardinality sort bug

* Init threshold rule

* Create working threshold rule

* Fix threshold signal generation

* Fix tests

* Update mappings

* ALERT_TYPE_ID => RULE_TYPE_ID

* Add tests

* Fix types

* Adds RAC rule type migration

* Fix threshold tests (remove outputIndex)

* Add threshold rule type to ruleTypeMappings

* Add kbn-securitysolution-rules package for sharing with alerting framework

* Fix type errors

* Fix find_rules tests

* First round of test fixes

* Fix issues from merge conflicts

* Use ruleDataClient getReader() for reading

* Fixes to 'generating_signals' tests

* Remove more refs to legacy schema

* Linting

* Quick type fix

* Bug fixes

* Add saved query rule type

* Linting

* Fix types

* Signal generation tests

* Test updates

* Update some more refs

* build_alert tests

* Cleanup

* Ref updates

* Revert "Ref updates"

This reverts commit 4d1473d.

* Update status field

* Test fixes

* Another test

* Got a little too aggressive with search/replace

* let's see where we're at

* Fix

* Test fixes

* cleanup

* Fix cases API integration test config, flaky DE tests

* Move flattenWithPrefix to package / skip signal migration tests

* Fix unit tests

* Use new schema for bulk rule creation

* event: { kind } => event.kind

* Fix signal migration API tests

* Fix ml integration test

* Fix threat match integration tests

* Fix ML rule type tests and add correct producer to all rule types

* Update threat match API integration test

* Remove dupe properties

* Type fix

* Fix ML producer in functional test

* Fix generating_signals tests

* Remove usage of RuleDataClient-based execution log client

* Don't check output index version if rule registry enabled

* Fix bulk duplicate rule

* Fix duplicate rule test

* Fix readPrivileges and timestamp check logic

* Fixes for eql and exceptions tests... disable open_close_signals

* Type fixes / keyword test fixes

* Additional test fixes

* Unit test fixes + signal -> kibana.alert

* Test fixes for exceptions

* Fix read_resolve_rules test

* Various test fixes with marshallmain

* Sort search results

* Fix create_rules tests

* Disable writer cache for integration tests

* Disable writer cache for cases integration tests

* Fix types in rule_data_plugin_service

* Fix ordering in exceptions tests

* Remove rule_registry.enabled flag

* Fix signals migration tests

* Don't check signals index before creation

* Fix cypress config

* Fix type error

* create_migrations tests

* Skip flaky test

* Helpful comment

* Fixes from merge conflicts

* Pretend that signals index exists

* Fix type errors

* Skip flaky tests

* Fix threat matching test

* Clean up

* Reverting default ruleRegistry experimental flag (breaks unit tests)

* Reenable rule registry experimental feature by default

* Execute DE rule migration in 8.0

Co-authored-by: Marshall Main <marshall.main@elastic.co>
@madirey madirey mentioned this issue May 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@madirey madirey

Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Alerts Security Solution Detection Alerts Feature impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@madirey @peluja1012 @elasticmachine @MadameSheema