[Security Solution][Detections] Support querying with multiple timestamps in Detection Rules

[spong]

With the introduction of the Timestamp Override feature to Detection Rules in 7.9, we added the ability for users to select another timestamp field (other than @timestamp) to be used when querying for data during Rule execution. This functionality is key to minimizing gaps when data is delayed (endpoint offline for multiple days) as we can now ensure Detection Rules will use the timestamp closest to ingest time like event.ingested.

While this helps minimize gaps, support for event.ingested is still sparse, and so in effort to support the transition to using this new timestamp field, the Detection Engine will need the ability to search using multiple timestamp fields in case some data is missing the specified timestamp override field.

For example, a certain rule may query against the winlogbeat-* index, but only a subset of winlogbeat modules may support event.ingested. So if the timestamp override is configured to use event.ingested, the rule will now only fire against the data from these updated modules, which could very well result in missed alerts.

One potential solution as @tsg mentioned:

It might be possible to do the logical equivalent of WHERE event.ingested in range OR @timestamp in range. That way we error on the side of including more data and duplicates are ok because we have de-duplication.

Following this direction, we'll want ensure that the query also sorts the timestamps accordingly such that it conforms with our gap detection mitigation efforts, and also that our de-duplication logic is not affected by these changes (should be good here as we use the index+id to create a unique composite key, and not the timestamp).

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[peluja1012]

Currently, one of our prebuilt rules fails to execute if one of the default indices is missing event.ingested. See error message here:

Running the following query will show more details about the failures:

GET "apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,winlogbeat-*"/_search
{
  "sort" : ["event.ingested"]
}

    "failures" : [
      {
        "shard" : 0,
        "index" : ".ds-logs-elastic.agent.filebeat-default-000001",
        "node" : "m88rV4HIS7uEbH2ZTe4G1g",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "No mapping found for [event.ingested] in order to sort on",
          "index_uuid" : "BQ6n1mF1QCKzcb0mgVV-dw",
          "index" : ".ds-logs-elastic.agent.filebeat-default-000001"
        }
      },
      {
        "shard" : 0,
        "index" : ".ds-logs-elastic.agent.metricbeat-default-000001",
        "node" : "m88rV4HIS7uEbH2ZTe4G1g",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "No mapping found for [event.ingested] in order to sort on",
          "index_uuid" : "1WbUSz4CT-2vkaGIL--jlA",
          "index" : ".ds-logs-elastic.agent.metricbeat-default-000001"
        }
      }
  ]

[dhurley14]

@spong @peluja1012

I have a working solution for this here. However, I could use some input / ideas on how to communicate this logic to the end user. Specifically regarding the following situation:

If I as a customer have a rule defined to query two indices named index_with_timestamp_override_field and index_with_missing_timestamp_override_field where the override field is event.ingested, I will find signals in the index with the override timestamp field defined, and will not find any signals in the index with the timestamp override field missing. When this happens should I just write the signals I do find from the index_with_timestamp_override_field and an error to the logs saying "couldn't query the following index pattern index_with_missing_timestamp_override_field because it was missing the timestamp override field: event.ingested and default field of @timestamp"? Should I write the signals but set the current status of the rule to "failed"? I think we need some way to tell the user "hey we found some signals in some of the indices you gave us but we couldn't query the other indices because they are missing the timestamp override field (and obviously don't have @timestamp defined)" Maybe we need a "warning" status? What are your thoughts?