Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] ReplicaShardAllocatorIT testPreferCopyWithHighestMatchingOperations failing #74081

Closed
benwtrent opened this issue Jun 14, 2021 · 1 comment · Fixed by #75170
Closed

[CI] ReplicaShardAllocatorIT testPreferCopyWithHighestMatchingOperations failing #74081

benwtrent opened this issue Jun 14, 2021 · 1 comment · Fixed by #75170
Assignees
Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@benwtrent
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/5x3nuaw4ehzzm/tests/:server:internalClusterTest/org.elasticsearch.gateway.ReplicaShardAllocatorIT/testPreferCopyWithHighestMatchingOperations

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.gateway.ReplicaShardAllocatorIT.testPreferCopyWithHighestMatchingOperations" -Dtests.seed=E84FA673AF3C8429 -Dtests.locale=sk -Dtests.timezone=Brazil/East -Druntime.java=8

Applicable branches:
7.13

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.gateway.ReplicaShardAllocatorIT&tests.test=testPreferCopyWithHighestMatchingOperations

Failure excerpt:

java.lang.AssertionError: Unexpected ShardFailures: [[test][0] failed, reason [BroadcastShardOperationFailedException[]; nested: NoNodeAvailableException[unknown node [dgqOGFY8TDWR0VEbZ_OOeA]]; ]]
Expected: <0>
     but: was <1>

  at __randomizedtesting.SeedInfo.seed([E84FA673AF3C8429:258ABCBDFE6E1B4A]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures(ElasticsearchAssertions.java:333)
  at org.elasticsearch.test.ESIntegTestCase.indexRandom(ESIntegTestCase.java:1592)
  at org.elasticsearch.gateway.ReplicaShardAllocatorIT.testPreferCopyWithHighestMatchingOperations(ReplicaShardAllocatorIT.java:266)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)

@benwtrent benwtrent added :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >test-failure Triaged test failures from CI labels Jun 14, 2021
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Jun 14, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner self-assigned this Jun 16, 2021
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jul 9, 2021
In elastic#74081 this test failed with a `NoNodeAvailableException` within the
`indexRandom()` call immediately after stopping a node. This could
happen if the `node-left` event wasn't fully applied before calling
`indexRandom()` with an empty list of docs but with `forceRefresh` set
to true: since there's no docs, the replica wouldn't be marked as stale,
so the final refresh would detect the missing node, failing its
`assertNoFailures` wrapper.

This commit avoids calling `indexRandom()` with no docs in this
location. It also enhances `assertNoFailures` to report the details of
each failure, rather than just the summary.

Closes elastic#74081
DaveCTurner added a commit that referenced this issue Jul 13, 2021
In #74081 this test failed with a `NoNodeAvailableException` within the
`indexRandom()` call immediately after stopping a node. This could
happen if the `node-left` event wasn't fully applied before calling
`indexRandom()` with an empty list of docs but with `forceRefresh` set
to true: since there's no docs, the replica wouldn't be marked as stale,
so the final refresh would detect the missing node, failing its
`assertNoFailures` wrapper.

This commit avoids calling `indexRandom()` with no docs in this
location. It also enhances `assertNoFailures` to report the details of
each failure, rather than just the summary.

Closes #74081
DaveCTurner added a commit that referenced this issue Jul 13, 2021
In #74081 this test failed with a `NoNodeAvailableException` within the
`indexRandom()` call immediately after stopping a node. This could
happen if the `node-left` event wasn't fully applied before calling
`indexRandom()` with an empty list of docs but with `forceRefresh` set
to true: since there's no docs, the replica wouldn't be marked as stale,
so the final refresh would detect the missing node, failing its
`assertNoFailures` wrapper.

This commit avoids calling `indexRandom()` with no docs in this
location. It also enhances `assertNoFailures` to report the details of
each failure, rather than just the summary.

Closes #74081
DaveCTurner added a commit that referenced this issue Jul 13, 2021
In #74081 this test failed with a `NoNodeAvailableException` within the
`indexRandom()` call immediately after stopping a node. This could
happen if the `node-left` event wasn't fully applied before calling
`indexRandom()` with an empty list of docs but with `forceRefresh` set
to true: since there's no docs, the replica wouldn't be marked as stale,
so the final refresh would detect the missing node, failing its
`assertNoFailures` wrapper.

This commit avoids calling `indexRandom()` with no docs in this
location. It also enhances `assertNoFailures` to report the details of
each failure, rather than just the summary.

Closes #74081
DaveCTurner added a commit that referenced this issue Jul 13, 2021
In #74081 this test failed with a `NoNodeAvailableException` within the
`indexRandom()` call immediately after stopping a node. This could
happen if the `node-left` event wasn't fully applied before calling
`indexRandom()` with an empty list of docs but with `forceRefresh` set
to true: since there's no docs, the replica wouldn't be marked as stale,
so the final refresh would detect the missing node, failing its
`assertNoFailures` wrapper.

This commit avoids calling `indexRandom()` with no docs in this
location. It also enhances `assertNoFailures` to report the details of
each failure, rather than just the summary.

Closes #74081
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants