Unable to open index in Discover and Visualize when date format is epoch_millis

[pheyos]

Kibana version: 6.x (currently 6.4) and master (currently 7.0)

Elasticsearch version: 6.x (currently 6.4) and master (currently 7.0)

Server OS version: Ubuntu 18.04

Browser version: Firefox latest, Chromium latest

Browser OS version: Ubuntu 18.04

Original install method (e.g. download page, yum, from source, etc.): Download archives from staging

Describe the bug:
When selecting an index pattern for an index with a date field of format epoch_millis, an error message is shown and no data is loaded (see screenshot of error message below).

Steps to reproduce:

1. Create two indices as follows (only difference: first one with time_format explicitely set to epoch_millis and second one with automatic time_format):

PUT test_epochmilli_1
{
  "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0
  },
  "mappings": {
    "doc": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "epoch_millis"
        },
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT test_epochmilli_2
{
  "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0
  },
  "mappings": {
    "doc": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}

2. Create the same document with epoch millisecond date in both indices:

PUT test_epochmilli_1/doc/1
{
  "@timestamp": 1477954800000,
  "name": "some_name"
}

PUT test_epochmilli_2/doc/1
{
  "@timestamp": 1477954800000,
  "name": "some_name"
}

3. Create two corresponding index patterns: test_epochmilli_1 and test_epochmilli_2 (with time field @timestamp)
4. Open test_epochmilli_2 in Discover (or render a data histogram in Visualize)

• => This works

5. Open test_epochmilli_1 in Discover (or render a data histogram in Visualize)

• => An error message is shown

Expected behavior:

• test_epochmilli_1 should work like test_epochmilli_2

Screenshots (if relevant):

Any additional context:

• In version 6.3.0-bc11 this error didn't occur
• Searching the index works fine (via GET test_epochmilli_1/_search)

[timroes]

Looking at elastic/elasticsearch#22621 it seems like ES should have denied a non UTC time zone with format epoch_millis already since a long time. I don't understand how this would work in 6.3.

@elastic/kibana-discovery @elastic/kibana-management @elastic/kibana-visualizations can anyone see a reason, why this recently started failing? Did we earlier respect the epoch_millis setting in the index management, and somehow now accidentally don't anymore?

I will already mark that as a blocker for 6.4, until we have figured out more, how, why and since when this is failing. If we find out that it's not a new issue, we might lift that again, but I want to make sure this issue gets enough attention.

[timroes]

I checked your example above and we haven't changed anything (important) in the request we send.

Request in 6.3

{
	"index": ["test_epochmilli_1"],
	"ignore_unavailable": true,
	"preference": 1529311514977
} {
	"version": true,
	"size": 500,
	"sort": [{
		"@timestamp": {
			"order": "desc",
			"unmapped_type": "boolean"
		}
	}],
	"_source": {
		"excludes": []
	},
	"aggs": {
		"2": {
			"date_histogram": {
				"field": "@timestamp",
				"interval": "1M",
				"time_zone": "Europe/Berlin",
				"min_doc_count": 1
			}
		}
	},
	"stored_fields": ["*"],
	"script_fields": {},
	"docvalue_fields": ["@timestamp"],
	"query": {
		"bool": {
			"must": [{
				"match_all": {}
			}, {
				"range": {
					"@timestamp": {
						"gte": 1371545482540,
						"lte": 1529311882540,
						"format": "epoch_millis"
					}
				}
			}],
			"filter": [],
			"should": [],
			"must_not": []
		}
	},
	"highlight": {
		"pre_tags": ["@kibana-highlighted-field@"],
		"post_tags": ["@/kibana-highlighted-field@"],
		"fields": {
			"*": {}
		},
		"fragment_size": 2147483647
	}
}

Request in master

{
	"index": "test_epochmilli_1",
	"ignore_unavailable": true,
	"timeout": 30000,
	"preference": 1529311749746
} {
	"version": true,
	"size": 500,
	"sort": [{
		"@timestamp": {
			"order": "desc",
			"unmapped_type": "boolean"
		}
	}],
	"_source": {
		"excludes": []
	},
	"aggs": {
		"2": {
			"date_histogram": {
				"field": "@timestamp",
				"interval": "1M",
				"time_zone": "Europe/Berlin",
				"min_doc_count": 1
			}
		}
	},
	"stored_fields": ["*"],
	"script_fields": {},
	"docvalue_fields": ["@timestamp"],
	"query": {
		"bool": {
			"must": [{
				"match_all": {}
			}, {
				"range": {
					"@timestamp": {
						"gte": 1371545422761,
						"lte": 1529311822761,
						"format": "epoch_millis"
					}
				}
			}],
			"filter": [],
			"should": [],
			"must_not": []
		}
	},
	"highlight": {
		"pre_tags": ["@kibana-highlighted-field@"],
		"post_tags": ["@/kibana-highlighted-field@"],
		"fields": {
			"*": {}
		},
		"fragment_size": 2147483647
	}
}

So the only thing that changes, is that we now have added a timeout parameter and write the index name directly instead of encapsulating it in an array.

For whatever reasons that seems to be failing against a master snapshot of ES. So I assume something changed there, but haven't found any related issue.

@elastic/es-search-aggs Do you have any idea what changed here?

[cbuescher]

Maybe related: elastic/elasticsearch#29310

[cbuescher]

I can confirm this happens for me in 6.x but not in 6.3. I suspect its an unwanted side effect of the bugfix mentioned above, but I will try to confirm that by reverting it on 6.x

[cbuescher]

Hmm, I still get the error on 6.x after reverting the commit from elastic/elasticsearch#29310, so maybe its unrelated after all. Not sure though. One thing to note: the only non-test change in #29310 was in RangeFieldMapper, which I'm not sure atm how it can affect the date queries or aggregations, so maybe its something else after all. Keep digging.

[cbuescher]

Unfortunately #29310 seems unrelated, at least from what I can tell after doing some debugging.

[cbuescher]

Tracking this further down I found elastic/elasticsearch#30534 which. When reverting this commit, I don't see the reported issue any more. Will need to check if the problem is a side effect of something else or an actual bug and if we can fix it without reverting.

[cbuescher]

I opened elastic/elasticsearch#31392 to see how to fix this. Maybe this will require specifying the "format" option explicitely on the kibana side for the aggregation. Would this work @timroes ?

[cbuescher]

I opened elastic/elasticsearch#31407 with which I hope is a fix.
@pheyos @timroes great catch, glad we noticed this on an unreleased version.

[cbuescher]

I commited what I think is a fix in elastic/elasticsearch#31392 to master and the 6.4 branch.
@pheyos @timroes can you verify this also fixes the problem you are seeing in Kibana? Then we can close this.

[pheyos]

Awesome @cbuescher ! 🎉
I tested the simplified example from above (test_epochmilli_1) again on master and 6.x: the error didn't occur any more.
I also tested the index which caused my investigation and this issue in the first place (ML data snapshot it_ops) with the same result on master and 6.x: the error didn't occur any more.

Not sure if more checks for Kibana are needed @timroes ?
From my point of view this can be closed.

[cbuescher]

@pheyos Great. Again thanks for catching this.

[timroes]

Awesome. No, I don't have any different test set, so if you consider that issue resolved, I will close it. Thanks @cbuescher for taking care of this so quickly.