Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: SearchableSnapshotsBlobStoreCacheIntegTests testBlobStoreCache failure #61801

Closed
nik9000 opened this issue Sep 1, 2020 · 3 comments
Closed
Assignees
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@nik9000
Copy link
Member

nik9000 commented Sep 1, 2020

Build scan:
https://gradle-enterprise.elastic.co/s/x2jjcc4jufimu

Repro line:

./gradlew ':x-pack:plugin:searchable-snapshots:test' \
  --tests "org.elasticsearch.blobstore.cache.SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache" \
  -Dtests.seed=CC89BE29420C144B -Dtests.security.manager=true -Dtests.locale=fr-CH \
  -Dtests.timezone=Pacific/Apia -Druntime.java=14 -Dtests.fips.enabled=true

Reproduces locally?:
Nope.

Applicable branches:
7.x

Failure history:
https://build-stats.elastic.co/goto/4b69af8574c4548e11c08a824c38f3e7

Failed five times in the past week and never before.

Failure excerpt:


org.elasticsearch.blobstore.cache.SearchableSnapshotsBlobStoreCacheIntegTests > testBlobStoreCache FAILED
    java.lang.AssertionError: timed out waiting for green state
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:952)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:905)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:894)
        at org.elasticsearch.blobstore.cache.SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache(SearchableSnapshotsBlobStoreCacheIntegTests.java:146)

    java.lang.AssertionError: ClusterHealthResponse has timed out - returned: [{"cluster_name":"SUITE-TEST_WORKER_VM=[843]-CLUSTER_SEED=[-2553043917039051597]-HASH=[24B456B2CB3]-cluster","status":"green","timed_out":true,"number_of_nodes":5,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}]
    Expected: is <false>
         but: was <true>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoTimeout(ElasticsearchAssertions.java:111)
        at org.elasticsearch.test.ESIntegTestCase.ensureClusterSizeConsistency(ESIntegTestCase.java:1050)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:575)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2233)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
@nik9000 nik9000 added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Sep 1, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Sep 1, 2020
@tlrx tlrx self-assigned this Sep 2, 2020
@tlrx tlrx added the :Delivery/Build Build or test infrastructure label Sep 3, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Build)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Sep 3, 2020
@tlrx tlrx removed the :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs label Sep 3, 2020
@elasticmachine elasticmachine removed the Team:Distributed Meta label for distributed team label Sep 3, 2020
@tlrx tlrx added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Delivery/Build Build or test infrastructure labels Sep 3, 2020
@elasticmachine elasticmachine added Team:Distributed Meta label for distributed team and removed Team:Core/Infra Meta label for core/infra team labels Sep 3, 2020
@tlrx
Copy link
Member

tlrx commented Sep 3, 2020

This failure is due to a bug in system indices for which I opened #61785. Build stats also show more failures that are likely caused by a change in build plugins, for which I opened #61888.

@tlrx tlrx closed this as completed Sep 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

3 participants