Sorting on timestamp inconsistent in 3.1

[matthughes]

Here is my mapping for:

"@timestamp" : {
"format" : "dateOptionalTime",
"type" : "date"
},

If I instruct Kibana to sort newest to oldest, the results seem to come in completely random order. If I use the curl query Kibana gives and look at the JSON results, they are sorted in the correct order, newest to oldest.

[rashidkpc]

I've not been able to duplicate this in Kibana 3.1, however all development has moved to Kibana 4. Are you seeing this issue in master?

[matthughes]

I have not tried v4 yet. Looking at the code though, I have a question:

from table/module.js

          // Sort the data
          $scope.data = _.sortBy($scope.data, function(v){
            if(!_.isUndefined(v.sort)) {
              return v.sort[0];
            } else {
              return v._score;
            }
          });

          // Reverse if needed
          if($scope.panel.sort[1] === 'desc') {
            $scope.data.reverse();
          }

Why does Kibana sort this data again? The JSON is already sorted and in the correct order AFAICT. Besides being an expensive operation, this -- in my case and others judging by the the number of timestamp issues reported -- is causing incorrect sorting.

[rashidkpc]

The resorting is necessary for the sequential querying we do. We query one index at a time, and need to resort the results once they hit Kibana. Its possible if you have any indexes in which @timestamp is not a date that the sort order could appear incorrect.

[matthughes]

Interesting. I can only see one search query getting posted to /_search/_all.

I did a bit more digging and was hoping to find this return surprising results:

_.sortBy([1,2,3], function(v) { return null });

As that is what is happening in my case, but it does not appear to disturb the original order when returning null.

[rashidkpc]

If you point kibana at a specific instead of _all does the problem continue to occur?

[matthughes]

I'm back to blaming elasticsearch. I have a daily index with an alias for those indicies. Mapping for the timestamp field is the same on all. Looking at the results again, Kibana isn't the problem. The sort value is wrong:

0: "2014-10-24T16:27:25.180Z 1414168045180"
1: "2014-10-24T16:27:25.180Z 1414168045180"
2: "2014-10-24T16:27:29.450Z 1414168049450"
3: "2014-10-24T16:27:28.372Z 1414168048372"
4: "2014-10-24T16:27:28.372Z 1414168048372"

For each hit, i printed out the timestamp and the sort array. I didn't think there should even be a sort value but apparently there is. As you can see the sort order doesn't make sense. If I just target an individual index instead of the alias, it works fine.

Seems to be related to elastic/elasticsearch#7343 but I haven't confirmed that yet.

[rashidkpc]

gotcha, I'll go ahead and close this.

[matthughes]

FYI: I wrote up the real issue here: elastic/elasticsearch#8226

It appears that if you have an empty index, it affects sort order.