AssertionError "map must be empty" when call _flush/synced after deleting document

[floragunn]

Elasticsearch version (bin/elasticsearch --version):
5.6.5 (5.6.4 works fine, so maybe related to #27534 and #27516 and #20498 )

Plugins installed: []

JVM version (java -version):
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version (uname -a if on a Unix-like system):
Mac and Ubuntu

Description of the problem including expected versus actual behavior:
"java.lang.AssertionError: map must be empty" when it not must be thrown

Steps to reproduce:

• Make sure assertions are enabled (export ES_JAVA_OPTS=-ea)
• Delete a non existent document in an non existent index (curl -Ss -XDELETE "http://localhost:9200/x/y/1")
• Make a synced flush (curl -Ss -XPOST "http://localhost:9200/_flush/synced")

Provide logs (if relevant):

[2017-12-17T15:24:39,706][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[Cj7lPWN][flush][T#3]], exiting
java.lang.AssertionError: map must be empty
	at org.elasticsearch.index.engine.LiveVersionMap.adjustMapSizeUnderLock(LiveVersionMap.java:43) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.index.engine.InternalEngine.syncFlush(InternalEngine.java:997) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.index.shard.IndexShard.syncFlush(IndexShard.java:757) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.indices.flush.SyncedFlushService.performSyncedFlush(SyncedFlushService.java:423) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.indices.flush.SyncedFlushService.access$1100(SyncedFlushService.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.indices.flush.SyncedFlushService$SyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:704) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.indices.flush.SyncedFlushService$SyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:700) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.5.jar:5.6.5]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

[colings86]

@boaz are you able to take a look at this?

[bleskes]

I already did ;) The problem is that Lucene doesn't consider a delete of a non existent doc a "pending uncommitted change", so we skip executing a commit on the pre-synced flush commit and we also don't fail the synced flush due to pending op. We do however insert the delete into the version map even if the doc is not found due to external versioning semantics. I need to discuss it with @s1monw, I don't' have a solution I'm happy with at the moment