[SIEM] @timestamp values without timezone designator are instantiated in browser local time

[spong]

As highlighted in this discuss topic, when @timestamp values are in a format that do not contain timezone designators, they will be instantiated in browser local time before being formatted as per the users dateFormat:tz Kibana Advanced Setting, which can result in incorrect datetimes throughout the SIEM App.

This can be resolved similarly to what Discover does by specifying

  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],

when querying ES, that way we ensure a consistent date format.

Note: This should be the case for all datetime fields not explicitly mapped to ISO 8601 with timezone designators.

Steps to reproduce

Run these commands in Kibana Dev Tools

• Create new index with @timestamp mapping that doesn't contain timezone indicators

PUT timestamp-without-tz-designator
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      }
    }
  }
}

• Add a document without tz designator

PUT timestamp-without-tz-designator/_doc/1
{ "@timestamp": "2020-02-11 23:59:55" }

• Add timestamp-without-tz-designator index to siem:defaultIndex
• Ensure dateformat:tz is not set to Browser (use UTC for full effect :)
• Go to Hosts -> Events Table and verify time difference between Table & Inspect/Network Panel

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[wesleyraptor]

Thanks for filing this ticket Garrett! Looking forward to this fix.

[FrankHassanabad]

"Reviewed by Frank Hassanabad on 7/29/2020", This looks complete and will be shippable at this point. Going to close this and associated tickets. Any new bugs/features should be new tickets.

Other notes is that new features, new developers, existing developers with endpoints and data might from time to time forget to add the prescribed fixes as it is counter-intuitive to the natural defaults of date time stamps of features of Elastic Search. If you encounter a new issue, please open that new issue against the new feature/API, or existing one as a regression rather than re-opening this one and we will fix it in those areas and keep a watchful 👁️ to hopefully not introduce this again.