[Security Solution][Detections] Rule fails when timestamp override field is not mapped

[spong]

In 7.11 support was added (#86368) for querying multiple timestamps and falling back to @timestamp if the timestampOverride field did not exist in the source index. If this fallback occurred, the rule would write a partial failure informing the user of the missing field, but still write any alerts from the fallback query using @timestamp. In testing #91553, a permutation was found that resulted in a full rule failure instead of a partial, and would prevent alerts from being written for the fallback query using @timestamp.

This can be recreated using any of the Elastic Endgame detection rules as the endgame-* index does not include a mapping for event.ingested, and the resulting ES error from this request is in a different format than what is currently expected.

1. Modify an Elastic Endgame rule to use event.ingested as the timestamp override field, and enable the rule. Note: since this field doesn't exist and there is client side validation, you must duplicate, then export the rule, modify the ndjson, and re-import. Or just use this rule here... 🙂

event_ingested_failure_rule.ndjson.zip

2. Once enabled, you should see the rule fail with the below error:

{
  "error": {
    "root_cause": [
      {
        "type": "query_shard_exception",
        "reason": "No mapping found for [event.ingested] in order to sort on",
        "index_uuid": "vU38GWfHQZGG3nbn1xQk3w",
        "index": "endgame-00001"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "endgame-00001",
        "node": "UmFIfD_iQfepBl6sUyEk2g",
        "reason": {
          "type": "query_shard_exception",
          "reason": "No mapping found for [event.ingested] in order to sort on",
          "index_uuid": "vU38GWfHQZGG3nbn1xQk3w",
          "index": "endgame-00001"
        }
      }
    ]
  },
  "status": 400
}

This is the result of a different ES error response format, and the following line:

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts

Line 139 in 4584a8b

: timestampFieldCapsResponse.body.fields[timestampField].unmapped.indices

Kibana version: 7.11.x

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)