Cancel replica recovery on another sync option copy found

[kimchy]

When a replica is initializing from the primary, and we find a better node that has full sync id match, it is better to cancel the existing replica allocation and allocate it to the new node with sync id match (eventually)

[bleskes]

we miss an L here REAL_L_OCATED

[kimchy]

:)

[bleskes]

This is awesome. LGTM. Left some small suggestions...

[bleskes]

OH. We probably need to document this. I'm also wondering if a release highlight is in order...

[kimchy]

@bleskes pushed changes, @clintongormley I am not sure where to document this?

[bleskes]

still awesome

[pickypg]

This is awesome and it's definitely a worthwhile release highlight!

[clintongormley]

@kimchy should be documented here i think: https://www.elastic.co/guide/en/elasticsearch/reference/master/delayed-allocation.html

I can add a note if you like. What do you mean by (eventually) in your PR description?

[kimchy]

thanks @clintongormley!, I just pushed it. What I meant by eventually is that we cancel the replica allocation to the node, and then we will try and allocate it to the node with the sync id match, but on that node, we might be throttling allocation, so it will end up getting assigned to it, but not right away, in order to respect the allocation deciders.

[clintongormley]

Docs added in c22e179