[BUG] org.opensearch.search.aggregations.bucket.DiversifiedSamplerIT.testNestedSamples {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky

[reta]

Describe the bug
The test case org.opensearch.search.aggregations.bucket.DiversifiedSamplerIT.testNestedSamples {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky:

org.opensearch.search.aggregations.bucket.DiversifiedSamplerIT.testNestedSamples {p0={"search.concurrent_segment_search.enabled":"true"}}

java.lang.AssertionError: 
Expected: a value less than or equal to <4L>
     but: <5L> was greater than <4L>
	at __randomizedtesting.SeedInfo.seed([D30D5DEAC663DE87:9ECE7E7BADC6DF04]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.search.aggregations.bucket.DiversifiedSamplerIT.testNestedSamples(DiversifiedSamplerIT.java:248)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)

To Reproduce
Steps to reproduce the behavior:

• https://build.ci.opensearch.org/job/gradle-check/25586/
• https://build.ci.opensearch.org/job/gradle-check/25543/

Expected behavior
Test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
Add any other context about the problem here.

[reta]

@sohami @neetikasinghal fyi :(

[dblock]

#10256 (comment)

[andrross]

#10388 (comment)

[ashking94]

#10492 (comment) - testSimpleTerminateAfterTrackTotalHitsUpTo is flaky as well

[shwetathareja]

#10711 (comment) - org.opensearch.search.aggregations.bucket.DiversifiedSamplerIT.testNestedSamples {p0={"search.concurrent_segment_search.enabled":"true"}}

[amkhar]

#10670 (comment)

[jed326]

DiversifiedSamplerIT.testNestedSamples is the test specified in this issue and was muted as a part of #10437 for the concurrent search case.

Looks like the mute backport is still open actually -- #10472

Based on the other reported occurrences, it seems like the following tests are also flaky:

• DiversifiedSamplerIT.testNestedDiversity
• DiversifiedSamplerIT.testNestedSamples

Ref: https://build.ci.opensearch.org/job/gradle-check/28555/testReport/junit/

[jed326]

Looks like the problem here is the maxDocsPerValue is being evaluated at the slice level for concurrent segment search so at the shard level we are getting more than expected docs. Added a 100% reproduction POC here: jed326@09e9d49.

Will work on publishing a fix for this.