INDEX DELETE with wildcard doesn't delete all matching indexes #7295

andrassy · 2014-08-15T14:52:20Z

When trying to delete multiple indexes, Issuing a DELETE with a wildcard seems to delete 20 indexes at a time and then returns with an {"acknowledge":"true"}. If this is correct behavior then this should be added to the documentation index\delete documentation, otherwise it needs to be fixed.

For example...

DELETE http://192.168.10.73:55590/*event.2013-* HTTP/1.1

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=UTF-8
Content-Length: 21

{"acknowledged":true}

clintongormley · 2014-08-18T09:24:50Z

@andrassy Could you try this instead:

DELETE /*event.2013*

I suspect that the - is being interpreted as a negative

clintongormley · 2014-08-18T09:26:47Z

Related to #6736

OlegYch · 2014-09-17T15:12:57Z

i'm facing the same issue using 1.3.2
here is my request:
DELETE logstash_,.marvel_,-.marvel-kibana,-logstash-2014.09.17,-logstash-2014.09.16,-logstash-2014.09.15,-logstash-2014.09.14,-logstash-2014.09.13,-logstash-2014.09.12,-logstash-2014.09.11,-logstash-2014.09.10,-logstash-2014.09.09,-logstash-2014.09.08,-.marvel-2014.09.17,-.marvel-2014.09.16,-.marvel-2014.09.15,-.marvel-2014.09.14,-.marvel-2014.09.13,-.marvel-2014.09.12,-.marvel-2014.09.11,-.marvel-2014.09.10,-.marvel-2014.09.09,-.marvel-2014.09.08

andrassy · 2014-09-25T09:20:34Z

We still face the same issue with DELETE/event.2013 so it's not explicitly the wildcarding.

Sometimes I get an acknowledged:true and other times I get "ProcessClusterEventTimeoutException[failed to process cluster event (delete-index [.......................]) within 30s]".

Might be related to #7799 where executions are rejected as the master is stressed (which is why I'm trying to delete lots of historic indexes).

clintongormley · 2014-10-31T11:28:09Z

@andrassy When 1.4 is out, please could you check whether the delete index command works as expected (ie has been fixed y #7799) and let us know

thanks

bobrik · 2014-12-02T17:10:03Z

@clintongormley index delete even without wildcard is failing to remove all indices.

Creating 25 indices hey-{1..25}, removing all them in one request:

# for i in {1..25}; do curl -s -X PUT http://web245:9200/hey-$i?pretty; done && sleep 10 && curl -s 'http://web245:9200/_cat/indices/hey-*?h=index' | sort && curl -s -X DELETE http://web245:9200/hey-1,hey-2,hey-3,hey-4,hey-5,hey-6,hey-7,hey-8,hey-9,hey-10,hey-11,hey-12,hey-13,hey-14,hey-15,hey-16,hey-17,hey-18,hey-19,hey-20,hey-21,hey-22,hey-23,hey-24,hey-25?pretty && curl -s 'http://web245:9200/_cat/indices/hey-*?v=index' | sort
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
hey-1
hey-10
hey-11
hey-12
hey-13
hey-14
hey-15
hey-16
hey-17
hey-18
hey-19
hey-2
hey-20
hey-21
hey-22
hey-23
hey-24
hey-25
hey-3
hey-4
hey-5
hey-6
hey-7
hey-8
hey-9
{
  "acknowledged" : true
}
health status index  pri rep docs.count docs.deleted store.size pri.store.size
yellow open   hey-16   5   1          0            0       812b           575b
yellow open   hey-17   5   1          0            0       575b           575b
yellow open   hey-18   5   1          0            0       575b           575b
yellow open   hey-19   5   1          0            0       575b           575b
yellow open   hey-20   5   1          0            0       575b           575b
yellow open   hey-21   5   1          0            0       575b           575b
yellow open   hey-22   5   1          0            0       575b           575b
yellow open   hey-23   5   1          0            0       575b           575b
yellow open   hey-24   5   1          0            0       575b           575b
yellow open   hey-25   5   1          0            0       575b           575b

Same, but removing with wildcard:

# for i in {1..25}; do curl -s -X PUT http://web245:9200/hey-$i?pretty; done && sleep 10 && curl -s 'http://web245:9200/_cat/indices/hey-*?h=index' | sort && curl -s -X DELETE http://web245:9200/hey-*?pretty && curl -s 'http://web245:9200/_cat/indices/hey-*?v=index' | sort
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
hey-1
hey-10
hey-11
hey-12
hey-13
hey-14
hey-15
hey-16
hey-17
hey-18
hey-19
hey-2
hey-20
hey-21
hey-22
hey-23
hey-24
hey-25
hey-3
hey-4
hey-5
hey-6
hey-7
hey-8
hey-9
{
  "acknowledged" : true
}
green  open   hey-15   5   1          0            0       970b           575b
green  open   hey-16   5   1          0            0       970b           575b
green  open   hey-17   5   1          0            0       970b           575b
health status index  pri rep docs.count docs.deleted store.size pri.store.size
yellow open   hey-10   5   1          0            0       891b           575b
yellow open   hey-11   5   1          0            0       891b           575b
yellow open   hey-12   5   1          0            0       891b           575b
yellow open   hey-13   5   1          0            0       891b           575b
yellow open   hey-14   5   1          0            0       891b           575b

I'm on 1.4.1

bobrik · 2014-12-02T17:47:57Z

Logs are here:

[2014-12-02 21:08:52,701][DEBUG][action.admin.indices.delete] [statistics04] [hey-15] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [hey-15]) within 30s
    at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:263)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-12-02 21:08:52,701][DEBUG][action.admin.indices.delete] [statistics04] [hey-14] failed to delete index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [hey-14]) within 30s
    at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:263)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

This is probably not the best idea to tell user that index is deleted when it is not.

clintongormley · 2014-12-02T18:15:00Z

I could replicate this in master by starting two nodes, then running this shell script:

for i in {1..50}; do curl -s -X PUT http://localhost:9200/hey-$i?pretty; done && sleep 3 && curl -s 'http://localhost:9200/_cat/indices/hey-*?h=index' | sort && curl -s -X DELETE 'http://localhost:9200/hey-*?master_timeout=1s&pretty' && echo "DONE"  && curl -s 'http://localhost:9200/_cat/indices/hey-*?v=index' | sort

In the logs were lines like:

[2014-12-02 19:07:22,183][DEBUG][action.admin.indices.delete] [Hephaestus] [hey-34] failed to delete index org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (delete-index [hey-34]) within 1s
at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

The delete request returns acknowledged: true even though not all matched indices have been deleted. Not sure what the right thing to do here is.

bobrik · 2014-12-02T18:20:01Z

Will it return an error if only one index was requested and removal is timed out? What if I pass two indices and both timed out?

I think it's better to return an error so clients can implement logic that is sane for them.

javanna · 2015-03-22T17:34:27Z

I had a look at this, I see the problem and I will work on fixing it. It all boils down to how delete index works internally. One cluster state update per index gets processed by TransportDeleteIndexAction, which tends to "forget" about previous indices failures. If the last delete index task succeeds, a positive response is returned, whereas if the last delete index task fails (or times out), then a failure is returned. This is bad :( I am on it

yodog · 2015-03-31T15:38:59Z

also happens with curl -XDELETE 'http://localhost:9200/_all'

javanna · 2015-09-03T18:22:00Z

This issue will be solved when we get #11189 in, which is a good fix but needs a proper test written for it. With that fix we will return a reliable last failure at least. That said #11258 (or something along those lines) would improve things even better, as it would treat the request as a single cluster state update task. The annoying part in fact is that we have one single failure in the response at the moment, but that's only the last failure, while we could have more potentially.

javanna · 2015-10-27T15:14:29Z

Fixed via #11258 . Closing

clintongormley added the discuss label Aug 18, 2014

clintongormley added feedback_needed and removed discuss labels Oct 31, 2014

clintongormley added discuss and removed feedback_needed labels Dec 2, 2014

javanna added >bug and removed discuss labels Mar 22, 2015

javanna self-assigned this Mar 22, 2015

javanna mentioned this issue May 15, 2015

Deleting multiple indices can fail to report all failures #11189

Closed

memcmp mentioned this issue May 20, 2015

Bulk cluster state updates on index deletion #11258

Merged

javanna removed their assignment Sep 3, 2015

javanna closed this as completed Oct 27, 2015

javanna added v2.2.0 v5.0.0-alpha1 labels Oct 27, 2015

clintongormley removed v2.2.0 v5.0.0-alpha1 labels Oct 29, 2015

javanna mentioned this issue Dec 16, 2015

-XDELETE wouldn't delete all the indexes specified #15487

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INDEX DELETE with wildcard doesn't delete all matching indexes #7295

INDEX DELETE with wildcard doesn't delete all matching indexes #7295

andrassy commented Aug 15, 2014

clintongormley commented Aug 18, 2014

clintongormley commented Aug 18, 2014

OlegYch commented Sep 17, 2014

andrassy commented Sep 25, 2014

clintongormley commented Oct 31, 2014

bobrik commented Dec 2, 2014

bobrik commented Dec 2, 2014

clintongormley commented Dec 2, 2014

bobrik commented Dec 2, 2014

javanna commented Mar 22, 2015

yodog commented Mar 31, 2015

javanna commented Sep 3, 2015

javanna commented Oct 27, 2015

INDEX DELETE with wildcard doesn't delete all matching indexes #7295

INDEX DELETE with wildcard doesn't delete all matching indexes #7295

Comments

andrassy commented Aug 15, 2014

clintongormley commented Aug 18, 2014

clintongormley commented Aug 18, 2014

OlegYch commented Sep 17, 2014

andrassy commented Sep 25, 2014

clintongormley commented Oct 31, 2014

bobrik commented Dec 2, 2014

bobrik commented Dec 2, 2014

clintongormley commented Dec 2, 2014

bobrik commented Dec 2, 2014

javanna commented Mar 22, 2015

yodog commented Mar 31, 2015

javanna commented Sep 3, 2015

javanna commented Oct 27, 2015