[BUG] New shards allocation failure for shards present in timed out batches in batch allocator

[imRishN]

Describe the bug

With #14848, we introduced capability to make reroute iteration time-bound. Currently OpenSearch uses two primary allocators -ExistingShardsAllocator which is responsible for allocating existing shards, and BalancedShardsAllocator which handles the allocation of all unassigned shards.

When a new index is created, the responsibility for allocating its shards falls to BalancedShardsAllocator. However, if the ExistingShardsAllocator times out, the remaining shards are currently marked as ignored, causing them to be skipped in the current round of allocation. Ideally, before marking these shards as ignored, the system should verify whether the ExistingShardsAllocator is actually responsible for their allocation. If it's not those shards should not be added to the ignore list.

As a result, when ExistingShardsAllocator times out, shards from newly created indices included in the timed out batches fails to get assigned.

Related component

Other

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Newly created shards should be allocated by BalancedShardsAllocator.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.