Empty index corrupts sort order

[matthughes]

I've been trying to track down some strange sorting results and have narrowed it down to having an empty index. I have multiple indexes all aliased to the same value. Sort order works fine, but once you add an empty index, the order is corrupted. Reproduction below:

#!/bin/bash

# Create index with no docs with alias test
curl -s -XPOST "http://localhost:9200/test-1"     -d '{ "aliases" : { "test" : {} }}' > /dev/null

# Then create a bunch of indices with alias test
curl -s XPOST "http://localhost:9200/test-2"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-2/doc" -d "{ \"entry\": 1 }"

curl -s -XPOST "http://localhost:9200/test-3"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-3/doc" -d "{ \"entry\": 2 }"

curl -s -XPOST "http://localhost:9200/test-4"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-4/doc" -d "{ \"entry\": 3 }"

curl -s -XPOST "http://localhost:9200/test-5"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-5/doc" -d "{ \"entry\": 4 }"

curl -s -XPOST "http://localhost:9200/test-6"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-6/doc" -d "{ \"entry\": 5 }"

curl -s -XPOST "http://localhost:9200/test-7"     -d '{ "aliases" : { "test" : {} }}' > /dev/null
curl -s -XPOST "http://localhost:9200/test-7/doc" -d "{ \"entry\": 6 }"

sleep 2

# Perform a sorted query, descending on field 'entry'
curl -XPOST 'http://localhost:9200/test/_search?pretty' -d '{
  "query": {
    "filtered": {
      "query": {
        "bool": {
          "should": [
            {
              "query_string": {
                "query": "*"
              }
            }
          ]
        }
      },
      "filter": {
        "bool": {
          "must": [
            {
              "match_all": {}
            }
          ]
        }
      }
    }
  },
  "highlight": {
    "fields": {},
    "fragment_size": 2147483647,
    "pre_tags": [
      "@start-highlight@"
    ],
    "post_tags": [
      "@end-highlight@"
    ]
  },
  "size": 500,
  "sort": [
    {
      "entry": {
        "order": "desc",
        "ignore_unmapped": true
      }
    }
  ]
}' | jq ".hits.hits[] | ._source.entry"

Results:

5
1
3
2
6
4

I can fix sort order in two ways: 1) remove the empty index or 2) change "ignore_unmapped" to false.

But IMO an empty index should no affect sort results. I'm also confused on why "ignore_unmapped": false, fixes things. I would think you would want the reverse, but it fails when true.

[s1monw]

I can reproduce this on 1.3 branch with:

public void testIssue8226() {
        int numIndices = between(5, 10);
        for (int i = 0; i < numIndices; i++) {
            assertAcked(prepareCreate("test_" + i).addAlias(new Alias("test")));
            if (i > 0) {
                client().prepareIndex("test_" + i, "foo", "" + i).setSource("{\"entry\": " + i + "}").get();
            }
        }
        ensureYellow();
        refresh();
        SearchResponse searchResponse = client().prepareSearch()
                .addSort(new FieldSortBuilder("entry").order(SortOrder.DESC).ignoreUnmapped(true))
                .setSize(10).get();
        assertSearchResponse(searchResponse);

        for (int j = 1; j < searchResponse.getHits().hits().length; j++) {
            assertThat(searchResponse.toString(), searchResponse.getHits().hits()[j].getId(), lessThan(searchResponse.getHits().hits()[j-1].getId()));
        }
    }

[s1monw]

It seems this doesn't happen in master, 1.x and neither on 1.4 but the test fails on 1.3 on every run so either we forgot to backport a fix or it's been fixed due to the cut-over to lucene comparators or so?

[s1monw]

alright I found the issue - nasty... will open a PR soon

[jpountz]

It seems this doesn't happen in master, 1.x and neither on 1.4 but the test fails on 1.3 on every run so either we forgot to backport a fix or it's been fixed due to the cut-over to lucene comparators or so?

I believe it is because this change #7039 is only on 1.4+

[s1monw]

fixed in 1.3.5