Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unicode index names causing DELETE index to hang #8254

Closed
clintongormley opened this issue Oct 28, 2014 · 6 comments
Closed

Unicode index names causing DELETE index to hang #8254

clintongormley opened this issue Oct 28, 2014 · 6 comments
Assignees
@s1monw
Labels
>bug

Comments

@clintongormley
Copy link

The test in https://github.com/elasticsearch/elasticsearch/blob/master/rest-api-spec/test/index/10_with_id.yaml uses Unicode characters in the index name: test-weird-index-中文. After running the test, the test runner tries to clear up the created indices with a DELETE /_all.

At least on OSX, the name of the index is being changed to test-weird-index-??.

In 1.4, the test succeeds but Elasticsearch logs errors about not being able to delete the index, but DELETE all at least returns. In 1.x, the DELETE all just hangs.

Both versions leave the test-weird-index-?? in place.

Relates to #6736

Logs from 1.x:

[2014-10-28 14:07:54,695][INFO ][node                     ] [Turbo] version[1.5.0-SNAPSHOT], pid[60278], build[${build/NA]
[2014-10-28 14:07:54,696][INFO ][node                     ] [Turbo] initializing ...
[2014-10-28 14:07:54,701][INFO ][plugins                  ] [Turbo] loaded [], sites []
[2014-10-28 14:07:57,345][INFO ][node                     ] [Turbo] initialized
[2014-10-28 14:07:57,345][INFO ][node                     ] [Turbo] starting ...
[2014-10-28 14:07:57,457][INFO ][transport                ] [Turbo] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.5.103:9300]}
[2014-10-28 14:07:57,479][INFO ][discovery                ] [Turbo] elasticsearch/aprLOBxNRXKldBazmm8VTg
[2014-10-28 14:08:01,253][INFO ][cluster.service          ] [Turbo] new_master [Turbo][aprLOBxNRXKldBazmm8VTg][Slim.local][inet[/192.168.5.103:9300]], reason: zen-disco-join (elected_as_master)
[2014-10-28 14:08:01,270][INFO ][http                     ] [Turbo] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.5.103:9200]}
[2014-10-28 14:08:01,270][INFO ][node                     ] [Turbo] started
[2014-10-28 14:08:01,283][INFO ][gateway                  ] [Turbo] recovered [0] indices into cluster_state
[2014-10-28 14:10:37,337][INFO ][cluster.metadata         ] [Turbo] [test-weird-index-中文] creating index, cause [auto(index api)], shards [5]/[1], mappings []
[2014-10-28 14:10:37,586][WARN ][gateway.local.state.meta ] [Turbo] [test-weird-index-中文]: failed to write index state
java.nio.file.NoSuchFileException: /Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-中文/_state/state-1.st.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:176)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:334)
    at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:313)
    at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.write(MetaDataStateFormat.java:117)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.writeIndex(LocalGatewayMetaState.java:371)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:218)
    at org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:207)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:437)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:10:37,624][WARN ][gateway.local.state.meta ] [Turbo] [test-weird-index-中文]: failed to write index state
java.nio.file.NoSuchFileException: /Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-中文/_state/state-1.st.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:176)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:334)
    at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:313)
    at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.write(MetaDataStateFormat.java:117)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.writeIndex(LocalGatewayMetaState.java:371)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:218)
    at org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:207)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:437)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:10:37,633][WARN ][gateway.local.state.meta ] [Turbo] [test-weird-index-中文]: failed to write index state
java.nio.file.NoSuchFileException: /Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-中文/_state/state-1.st.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:176)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:334)
    at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:313)
    at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.write(MetaDataStateFormat.java:117)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.writeIndex(LocalGatewayMetaState.java:371)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:218)
    at org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:207)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:437)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:10:37,702][WARN ][index.mapper             ] [Turbo] [test-weird-index-中文] Type [weird.type] contains a '.', it is recommended not to include it within a type name
[2014-10-28 14:10:37,725][WARN ][index.mapper             ] [Turbo] [test-weird-index-中文] Type [weird.type] contains a '.', it is recommended not to include it within a type name
[2014-10-28 14:10:37,727][INFO ][cluster.metadata         ] [Turbo] [test-weird-index-中文] update_mapping [weird.type] (dynamic)
[2014-10-28 14:10:37,732][WARN ][gateway.local.state.meta ] [Turbo] [test-weird-index-中文]: failed to write index state
java.nio.file.NoSuchFileException: /Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-中文/_state/state-2.st.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:176)
    at java.nio.channels.FileChannel.open(FileChannel.java:287)
    at java.nio.channels.FileChannel.open(FileChannel.java:334)
    at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:313)
    at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.write(MetaDataStateFormat.java:117)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.writeIndex(LocalGatewayMetaState.java:371)
    at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:218)
    at org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:207)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:437)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:10:37,791][INFO ][cluster.metadata         ] [Turbo] [test-weird-index-中文] deleting index

Logs from 1.4:

[2014-10-28 14:12:34,362][INFO ][node                     ] [Burstarr] version[1.4.0-SNAPSHOT], pid[60351], build[${build/NA]
[2014-10-28 14:12:34,362][INFO ][node                     ] [Burstarr] initializing ...
[2014-10-28 14:12:34,367][INFO ][plugins                  ] [Burstarr] loaded [], sites []
[2014-10-28 14:12:36,930][INFO ][node                     ] [Burstarr] initialized
[2014-10-28 14:12:36,930][INFO ][node                     ] [Burstarr] starting ...
[2014-10-28 14:12:37,065][INFO ][transport                ] [Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.5.103:9300]}
[2014-10-28 14:12:37,080][INFO ][discovery                ] [Burstarr] elasticsearch/M_KcoO3qQCOBp9VuHZ31AA
[2014-10-28 14:12:40,854][INFO ][cluster.service          ] [Burstarr] new_master [Burstarr][M_KcoO3qQCOBp9VuHZ31AA][Slim.local][inet[/192.168.5.103:9300]], reason: zen-disco-join (elected_as_master)
[2014-10-28 14:12:40,869][INFO ][http                     ] [Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.5.103:9200]}
[2014-10-28 14:12:40,869][INFO ][node                     ] [Burstarr] started
[2014-10-28 14:12:40,883][INFO ][gateway                  ] [Burstarr] recovered [0] indices into cluster_state
[2014-10-28 14:12:50,301][INFO ][cluster.metadata         ] [Burstarr] [test-weird-index-中文] creating index, cause [auto(index api)], shards [5]/[1], mappings []
[2014-10-28 14:12:50,580][INFO ][gateway.local.state.meta ] [Burstarr] [test-weird-index-??] dangling index, exists on local file system, but not in cluster metadata, scheduling to delete in [2h], auto import to cluster state [YES]
[2014-10-28 14:12:50,581][INFO ][gateway.local.state.meta ] [Burstarr] dangled index directory name is [test-weird-index-??], state name is [test-weird-index-中文], renaming to directory name
[2014-10-28 14:12:50,611][INFO ][gateway.local.state.meta ] [Burstarr] dangled index directory name is [test-weird-index-??], state name is [test-weird-index-中文], renaming to directory name
[2014-10-28 14:12:50,617][INFO ][gateway.local.state.meta ] [Burstarr] dangled index directory name is [test-weird-index-??], state name is [test-weird-index-中文], renaming to directory name
[2014-10-28 14:12:50,624][INFO ][gateway.local.state.meta ] [Burstarr] auto importing dangled indices [test-weird-index-??/OPEN] from [[Burstarr][M_KcoO3qQCOBp9VuHZ31AA][Slim.local][inet[/192.168.5.103:9300]]]
[2014-10-28 14:12:50,682][WARN ][index.mapper             ] [Burstarr] [test-weird-index-中文] Type [weird.type] contains a '.', it is recommended not to include it within a type name
[2014-10-28 14:12:50,709][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][2] shard is locked, releasing lock
[2014-10-28 14:12:50,733][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][3] shard is locked, releasing lock
[2014-10-28 14:12:50,760][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][1] shard is locked, releasing lock
[2014-10-28 14:12:50,782][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][4] shard is locked, releasing lock
[2014-10-28 14:12:50,828][WARN ][index.mapper             ] [Burstarr] [test-weird-index-中文] Type [weird.type] contains a '.', it is recommended not to include it within a type name
[2014-10-28 14:12:50,831][INFO ][cluster.metadata         ] [Burstarr] [test-weird-index-中文] update_mapping [weird.type] (dynamic)
[2014-10-28 14:12:50,837][INFO ][cluster.metadata         ] [Burstarr] [test-weird-index-??] deleting index
[2014-10-28 14:12:55,730][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][2] Could not lock IndexWriter isLocked [true]
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,732][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][2] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][2] failed recovery
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [test-weird-index-??][2] failed to create engine
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:288)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    ... 3 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    ... 6 more
[2014-10-28 14:12:55,749][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][3] Could not lock IndexWriter isLocked [true]
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,750][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][3] failed recovery
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [test-weird-index-??][3] failed to create engine
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:288)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    ... 3 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    ... 6 more
[2014-10-28 14:12:55,778][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][1] Could not lock IndexWriter isLocked [true]
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,795][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-??][4] Could not lock IndexWriter isLocked [true]
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/4/index/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1448)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,800][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][3] sending failed shard for [test-weird-index-??][3], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][3] failed recovery]; nested: EngineCreationFailureException[[test-weird-index-??][3] failed to create engine]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index/write.lock]; ]]
[2014-10-28 14:12:55,800][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][3] received shard failed for [test-weird-index-??][3], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][3] failed recovery]; nested: EngineCreationFailureException[[test-weird-index-??][3] failed to create engine]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index/write.lock]; ]]
[2014-10-28 14:12:55,800][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][2] sending failed shard for [test-weird-index-??][2], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][2] failed recovery]; nested: EngineCreationFailureException[[test-weird-index-??][2] failed to create engine]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index/write.lock]; ]]
[2014-10-28 14:12:55,801][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][2] received shard failed for [test-weird-index-??][2], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][2] failed recovery]; nested: EngineCreationFailureException[[test-weird-index-??][2] failed to create engine]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index/write.lock]; ]]
[2014-10-28 14:12:55,806][INFO ][cluster.metadata         ] [Burstarr] [test-weird-index-中文] deleting index
[2014-10-28 14:12:55,808][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-中文][0] failed to rollback writer on close
org.apache.lucene.store.NoSuchDirectoryException: directory '/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index' does not exist
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:424)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:460)
    at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2134)
    at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2074)
    at org.elasticsearch.index.engine.internal.InternalEngine.close(InternalEngine.java:1295)
    at org.elasticsearch.index.service.InternalIndexService.removeShard(InternalIndexService.java:419)
    at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,813][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-中文][2] failed to rollback writer on close
org.apache.lucene.store.NoSuchDirectoryException: directory '/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index' does not exist
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:424)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:460)
    at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2134)
    at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2074)
    at org.elasticsearch.index.engine.internal.InternalEngine.close(InternalEngine.java:1295)
    at org.elasticsearch.index.service.InternalIndexService.removeShard(InternalIndexService.java:419)
    at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,815][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-中文][1] failed to rollback writer on close
org.apache.lucene.store.NoSuchDirectoryException: directory '/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index' does not exist
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:424)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:460)
    at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2134)
    at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2074)
    at org.elasticsearch.index.engine.internal.InternalEngine.close(InternalEngine.java:1295)
    at org.elasticsearch.index.service.InternalIndexService.removeShard(InternalIndexService.java:419)
    at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,817][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-中文][4] failed to rollback writer on close
org.apache.lucene.store.NoSuchDirectoryException: directory '/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/4/index' does not exist
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:424)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:460)
    at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2134)
    at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2074)
    at org.elasticsearch.index.engine.internal.InternalEngine.close(InternalEngine.java:1295)
    at org.elasticsearch.index.service.InternalIndexService.removeShard(InternalIndexService.java:419)
    at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,819][WARN ][index.engine.internal    ] [Burstarr] [test-weird-index-中文][3] failed to rollback writer on close
org.apache.lucene.store.NoSuchDirectoryException: directory '/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index' does not exist
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
    at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:424)
    at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:460)
    at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2134)
    at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2074)
    at org.elasticsearch.index.engine.internal.InternalEngine.close(InternalEngine.java:1295)
    at org.elasticsearch.index.service.InternalIndexService.removeShard(InternalIndexService.java:419)
    at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
[2014-10-28 14:12:55,822][INFO ][gateway.local.state.meta ] [Burstarr] auto importing dangled indices [test-weird-index-??/OPEN] from [[Burstarr][M_KcoO3qQCOBp9VuHZ31AA][Slim.local][inet[/192.168.5.103:9300]]]
[2014-10-28 14:12:55,881][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][2] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][2] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][2] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index), type=MERGE, rate=20.0)]): files: []
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
[2014-10-28 14:12:55,900][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][0] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index), type=MERGE, rate=20.0)]): files: []
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
[2014-10-28 14:12:55,913][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][3] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][3] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index), type=MERGE, rate=20.0)]): files: []
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
[2014-10-28 14:12:55,929][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][4] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][4] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][4] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/4/index), type=MERGE, rate=20.0)]): files: []
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
[2014-10-28 14:12:55,929][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][3] sending failed shard for [test-weird-index-??][3], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][3] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][3] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,930][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][3] received shard failed for [test-weird-index-??][3], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][3] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][3] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/3/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,930][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][0] sending failed shard for [test-weird-index-??][0], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,931][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][0] received shard failed for [test-weird-index-??][0], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,931][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][2] sending failed shard for [test-weird-index-??][2], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][2] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][2] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,931][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][2] received shard failed for [test-weird-index-??][2], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][2] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][2] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/2/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,932][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][4] sending failed shard for [test-weird-index-??][4], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][4] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][4] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/4/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,932][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][4] received shard failed for [test-weird-index-??][4], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][4] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][4] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/4/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,934][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][4] received shard failed for [test-weird-index-??][4], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [master [Burstarr][M_KcoO3qQCOBp9VuHZ31AA][Slim.local][inet[/192.168.5.103:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2014-10-28 14:12:55,951][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][2] received shard failed for [test-weird-index-??][2], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [master [Burstarr][M_KcoO3qQCOBp9VuHZ31AA][Slim.local][inet[/192.168.5.103:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2014-10-28 14:12:55,953][WARN ][indices.cluster          ] [Burstarr] [test-weird-index-??][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][1] failed to fetch index version after copying it over
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:158)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][1] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:132)
    ... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index), type=MERGE, rate=20.0)]): files: []
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
    at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:85)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:123)
    ... 4 more
[2014-10-28 14:12:55,958][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][1] sending failed shard for [test-weird-index-??][1], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][1] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][1] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-10-28 14:12:55,958][WARN ][cluster.action.shard     ] [Burstarr] [test-weird-index-??][1] received shard failed for [test-weird-index-??][1], node[M_KcoO3qQCOBp9VuHZ31AA], [P], s[INITIALIZING], indexUUID [FsbgmgXZTY6ksoOUAIBgOQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][1] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[test-weird-index-??][1] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(niofs(/Users/clinton/workspace/elasticsearch/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index), type=MERGE, rate=20.0)]): files: []]; ]]
@clintongormley
Copy link
Author

This appears to be an artifact of running Elasticsearch in Eclipse. And the DELETE _all doesn't hang - it times out after 30 sec and returns {acknowledged: false}.

Closing

@uschindler
Copy link
Contributor

If there would have been an error like this, forbidden-apis should have catched it :-)

@clintongormley
Copy link
Author

@s1monw has a fix for this in the works

@s1monw
Copy link
Contributor

s1monw commented Nov 12, 2014

@clintongormley is this still happening?

@clintongormley
Copy link
Author

@s1monw This still happens in 1.3 and 1.4, in master it is worse.

This request no longer logs an error:

curl -XPOST 'http://localhost:9200/test-weird-index-%E4%B8%AD%E6%96%87/weird.type/1?pretty=1' -d '
{
   "foo" : "bar"
}
'

However it creates two separate directories, one for the segments and one for the translogs:

data/elasticsearch/nodes/0/indices/test-weird-index-??
├── 0
│   ├── index
│   └── translog
│       └── translog-1415797929651
├── 1
│   ├── index
│   └── translog
│       └── translog-1415797929667
├── 2
│   ├── index
│   └── translog
│       └── translog-1415797929681
├── 3
│   ├── index
│   └── translog
│       └── translog-1415797929655
└── 4
    ├── index
    └── translog
        └── translog-1415797929711
data/elasticsearch/nodes/0/indices/test-weird-index-中�\226\207
├── 0
│   ├── _state
│   │   └── state-2.st
│   └── index
│       ├── segments_1
│       └── write.lock
├── 1
│   ├── _state
│   │   └── state-2.st
│   └── index
│       ├── segments_1
│       └── write.lock
├── 2
│   ├── _state
│   │   └── state-2.st
│   └── index
│       ├── segments_1
│       └── write.lock
├── 3
│   ├── _state
│   │   └── state-2.st
│   └── index
│       ├── _0.cfe
│       ├── _0.cfs
│       ├── _0.si
│       ├── segments_1
│       └── write.lock
├── 4
│   ├── _state
│   │   └── state-2.st
│   └── index
│       ├── segments_1
│       └── write.lock
└── _state
    ├── state-1.st
    └── state-2.st

@s1monw s1monw closed this as completed in a6510f9 Dec 2, 2014
@javanna
Copy link
Member

javanna commented Dec 3, 2014

Hye @s1monw this was closed by a commit that went only to master if I'm not mistaken, but it looks like an actual problem in 1.4 and 1.x too (also 1.3). Is there a way to fix this there as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s1monw s1monw

Labels
>bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@clintongormley @javanna @s1monw @uschindler