[BUG] Gradle check on 2.x are failing due to RemoteStoreRefreshListenerTests tests

[dreamer-89]

Gradle check on 2.x are failing due to RemoteStoreRefreshListenerTests consistently and blocking the backport PRs. Opening up this issue to first mute these tests and secondly to root cause to prevent it in future. This is preventing the backport PRs needed for 2.9.0 release (code freeze today i.e. July 11).
CC @sachinpkale @ashking94 @gbbafna

Few recend gradle check failures

1. https://build.ci.opensearch.org/job/gradle-check/19907/testReport/
2. https://build.ci.opensearch.org/job/gradle-check/19926/testReport/
3. https://build.ci.opensearch.org/job/gradle-check/19925/testReport/
4. https://build.ci.opensearch.org/job/gradle-check/19928/testReport/

Unit tests causing failures

1. org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit
2. https://build.ci.opensearch.org/job/gradle-check/19928/testReport/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testAfterRefresh/
3. https://build.ci.opensearch.org/job/gradle-check/19928/testReport/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testAfterMultipleCommits/

[ashking94]

@dreamer-89 - @Rishikesh1159 has already made changes on main. Please backport the same. PR - #8626

Also, this was done on account of constant failures introduced as part of #7119. I have raised the fix in #8632 and have run the class tests for 100+ iterations without any failure. You can go ahead with backporting @Rishikesh1159's PR on muting and then if possible, review my PR, merge & backport accordingly.

[dreamer-89]

@dreamer-89 - @Rishikesh1159 has already made changes on main. Please backport the same. PR - #8626

Also, this was done on account of constant failures introduced as part of #7119. I have raised the fix in #8632 and have run the class tests for 100+ iterations without any failure. You can go ahead with backporting @Rishikesh1159's PR on muting and then if possible, review my PR, merge & backport accordingly.

Hey @ashking94,
The backport to 2.x PR #8634 for #7119 is still open. So, I don't think the failures were introduces as part of #7119. Am I missing something ?

[ashking94]

@dreamer-89 This PR had the fix for all flaky tests in RemoteStoreRefreshListenerTests. Anyways, @sachinpkale has raised a separate PR for the same which is backporting just the fix to 2.9 branch.

[dreamer-89]

Closing this issue as #8656 is merged & backported.

[dblock]

Saw this again in #11940

[sachinpkale]

2.x build passes now. Closing.