[Segment Replication][BUG] Failing test: SegmentReplicationIndexShardTests.testSegmentReplication_Index_Update_Delete

[nknize]

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.shard.SegmentReplicationIndexShardTests.testSegmentReplication_Index_Update_Delete" -Dtests.seed=98A07D624AEFB534 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nl-NL -Dtests.timezone=Etc/GMT+4 -Druntime.java=17

Stacktrace
java.lang.AssertionError: expected:<[doc{id='3 seqNo=3 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='5 seqNo=5 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='6 seqNo=6 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='7 seqNo=7 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='8 seqNo=8 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='9 seqNo=9 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='11 seqNo=11 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='12 seqNo=12 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='13 seqNo=13 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='18 seqNo=18 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='19 seqNo=19 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='20 seqNo=20 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='24 seqNo=24 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='26 seqNo=26 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='28 seqNo=28 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='29 seqNo=29 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='30 seqNo=30 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='31 seqNo=31 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='34 seqNo=34 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='37 seqNo=37 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='42 seqNo=42 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='47 seqNo=47 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='48 seqNo=48 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='49 seqNo=49 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='52 seqNo=52 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='54 seqNo=54 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='55 seqNo=55 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='56 seqNo=56 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='60 seqNo=60 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='65 seqNo=65 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='67 seqNo=67 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='68 seqNo=68 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='70 seqNo=70 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='72 seqNo=72 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='73 seqNo=73 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='76 seqNo=76 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='77 seqNo=77 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='79 seqNo=79 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='80 seqNo=80 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='81 seqNo=81 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='82 seqNo=82 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='83 seqNo=83 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='85 seqNo=85 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='87 seqNo=87 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='88 seqNo=88 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='90 seqNo=90 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='92 seqNo=92 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='93 seqNo=93 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='94 seqNo=94 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='96 seqNo=96 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='97 seqNo=97 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='98 seqNo=98 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='99 seqNo=99 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='100 seqNo=100 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='101 seqNo=101 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='102 seqNo=102 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='104 seqNo=104 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='107 seqNo=107 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='108 seqNo=108 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='110 seqNo=110 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='111 seqNo=111 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='112 seqNo=112 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='113 seqNo=113 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='114 seqNo=114 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='115 seqNo=115 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='116 seqNo=116 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='117 seqNo=117 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='118 seqNo=118 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='119 seqNo=119 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='121 seqNo=121 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='122 seqNo=122 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='123 seqNo=123 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='125 seqNo=125 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='126 seqNo=126 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='127 seqNo=127 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='131 seqNo=131 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='0 seqNo=132 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='1 seqNo=133 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='2 seqNo=134 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='4 seqNo=135 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='10 seqNo=136 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='14 seqNo=137 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='15 seqNo=138 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='16 seqNo=139 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='17 seqNo=140 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='21 seqNo=141 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='22 seqNo=142 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='23 seqNo=143 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='25 seqNo=144 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='27 seqNo=145 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='32 seqNo=146 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='33 seqNo=147 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='35 seqNo=148 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='36 seqNo=149 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='38 seqNo=150 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='39 seqNo=151 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='40 seqNo=152 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='41 seqNo=153 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='43 seqNo=154 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='44 seqNo=155 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='45 seqNo=156 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='46 seqNo=157 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='50 seqNo=158 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='51 seqNo=159 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='53 seqNo=160 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='57 seqNo=161 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='58 seqNo=162 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='59 seqNo=163 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='61 seqNo=164 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='62 seqNo=165 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='63 seqNo=166 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='64 seqNo=167 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='66 seqNo=168 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='69 seqNo=169 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='71 seqNo=170 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='74 seqNo=171 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='75 seqNo=172 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='78 seqNo=173 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='84 seqNo=174 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='86 seqNo=175 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='89 seqNo=176 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='91 seqNo=177 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='95 seqNo=178 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='103 seqNo=179 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='105 seqNo=180 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='106 seqNo=181 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='109 seqNo=182 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='120 seqNo=183 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='124 seqNo=184 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='128 seqNo=185 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='129 seqNo=186 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='130 seqNo=187 primaryTerm=35 version=2 source= { "foo" : "baz" }}]> but was:<[doc{id='3 seqNo=3 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='6 seqNo=6 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='7 seqNo=7 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='8 seqNo=8 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='9 seqNo=9 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='11 seqNo=11 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='18 seqNo=18 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='19 seqNo=19 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='26 seqNo=26 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='28 seqNo=28 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='30 seqNo=30 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='37 seqNo=37 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='42 seqNo=42 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='47 seqNo=47 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='49 seqNo=49 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='60 seqNo=60 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='65 seqNo=65 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='67 seqNo=67 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='68 seqNo=68 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='72 seqNo=72 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='77 seqNo=77 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='79 seqNo=79 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='82 seqNo=82 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='85 seqNo=85 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='87 seqNo=87 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='88 seqNo=88 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='94 seqNo=94 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='112 seqNo=112 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='114 seqNo=114 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='115 seqNo=115 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='116 seqNo=116 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='117 seqNo=117 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='118 seqNo=118 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='119 seqNo=119 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='121 seqNo=121 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='122 seqNo=122 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='125 seqNo=125 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='131 seqNo=131 primaryTerm=35 version=1 source= {"foo": "bar"}}, doc{id='0 seqNo=132 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='2 seqNo=134 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='10 seqNo=136 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='14 seqNo=137 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='15 seqNo=138 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='25 seqNo=144 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='27 seqNo=145 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='32 seqNo=146 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='36 seqNo=149 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='39 seqNo=151 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='43 seqNo=154 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='44 seqNo=155 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='50 seqNo=158 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='53 seqNo=160 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='57 seqNo=161 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='58 seqNo=162 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='59 seqNo=163 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='61 seqNo=164 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='62 seqNo=165 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='63 seqNo=166 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='66 seqNo=168 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='71 seqNo=170 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='74 seqNo=171 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='75 seqNo=172 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='78 seqNo=173 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='84 seqNo=174 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='89 seqNo=176 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='103 seqNo=179 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='105 seqNo=180 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='109 seqNo=182 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='124 seqNo=184 primaryTerm=35 version=2 source= { "foo" : "baz" }}, doc{id='129 seqNo=186 primaryTerm=35 version=2 source= { "foo" : "baz" }}]>
	at __randomizedtesting.SeedInfo.seed([98A07D624AEFB534:4ECD147A083F9EF]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.opensearch.index.shard.SegmentReplicationIndexShardTests.testSegmentReplication_Index_Update_Delete(SegmentReplicationIndexShardTests.java:136)

[tlfeng]

Tried reproducing the test failure, along with another failure of #4976, but I couldn't.

I used the script mentioned in #1715 (comment) to run the tests in a loop.

I ran the following commands, but no test failure was found during the loops.
for i in $(seq 0 1000) ; do echo "Iteration: $i" && ./gradlew ':server:test' --tests "org.opensearch.index.shard.SegmentReplicationIndexShardTests.testSegmentReplication_Index_Update_Delete" -Dtests.seed=98A07D624AEFB534 >> test-output.txt 2>&1 ; done

for i in $(seq 0 1000) ; do echo "Iteration: $i" && ./gradlew ':server:test' --tests "org.opensearch.index.shard.SegmentReplicationIndexShardTests.testReplicaReceivesLowerGeneration" -Dtests.seed=25A1BC6ABB78D54B >> test-output.txt 2>&1 ; done

for i in $(seq 0 1000) ; do echo "Iteration: $i" && ./gradlew ':server:test' --tests "org.opensearch.index.shard.SegmentReplicationIndexSh ardTests" >> test-output.txt 2>&1 ; done

[Rishikesh1159]

Tried looking through code to reproduce this failure. But couldn't reproduce this anytime. Closing this issue for now. If anyone find this test failing again feel free to reopen the issue.