Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RunDataFrameAnalyticsIT failures #61852

Closed
jkakavas opened this issue Sep 2, 2020 · 2 comments · Fixed by #61868
Closed

RunDataFrameAnalyticsIT failures #61852

jkakavas opened this issue Sep 2, 2020 · 2 comments · Fixed by #61868
Assignees
@dimitris-athanasiou
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@jkakavas
Copy link
Member

jkakavas commented Sep 2, 2020 •
edited
Loading

There are a few tests that fail:

org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT testOutlierDetectionWithEnoughDocumentsToScroll
org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT testOutlierDetectionWithFewDocuments
org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT testOutlierDetectionWithMultipleSourceIndices
org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT testOutlierDetectionWithPreExistingDestIndex

with the same error.

Build scan:
[7.10.0] https://gradle-enterprise.elastic.co/s/dp2u6q52kyxus
[7.9.1] https://gradle-enterprise.elastic.co/s/dlgajwjbzcmli
[7.10.0] https://gradle-enterprise.elastic.co/s/w57smywps3wlq
[7.9.1] https://gradle-enterprise.elastic.co/s/at3whtu5np6r6
https://gradle-enterprise.elastic.co/s/c5egkz647juis

and

[7.9.1] https://gradle-enterprise.elastic.co/s/jbaq7v3utks6q
[7.10.0] https://gradle-enterprise.elastic.co/s/bnbd27r4httos
[7.10.0] https://gradle-enterprise.elastic.co/s/si6k6omt5gjfe
[7.9.1] https://gradle-enterprise.elastic.co/s/blgn5kzzmp7pg

[7.10.0] https://gradle-enterprise.elastic.co/s/d5w6ctq3li7va
https://gradle-enterprise.elastic.co/s/3o4d53arbbfia

Repro line:

Example:

REPRODUCE WITH: gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTest' --tests "org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testOutlierDetectionWithFewDocuments" \
  -Dtests.seed=E6859593F0B2A900 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-EC \
  -Dtests.timezone=Canada/Newfoundland \
  -Druntime.java=11

Reproduces locally?:
Didn't try ( windows)

Applicable branches:
master

Failure history:

FAiled twice today but hasnt been failing recently with this assertion error.

Failure excerpt:


org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT > testOutlierDetectionWithFewDocuments FAILED
    java.lang.AssertionError: progress is complete: [{"phase":"reindexing","progress_percent":1}, {"phase":"loading_data","progress_percent":100}, {"phase":"computing_outliers","progress_percent":100}, {"phase":"writing_results","progress_percent":100}]
    Expected: is <true>
         but: was <false>
        at __randomizedtesting.SeedInfo.seed([E6859593F0B2A900:58D5761152F12CEA]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.assertProgressComplete(MlNativeDataFrameAnalyticsIntegTestCase.java:227)
        at org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testOutlierDetectionWithFewDocuments(RunDataFrameAnalyticsIT.java:143)
@jkakavas jkakavas added >test-failure Triaged test failures from CI :ml Machine learning labels Sep 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@dimitris-athanasiou dimitris-athanasiou self-assigned this Sep 2, 2020
@jkakavas jkakavas changed the title RunDataFrameAnalyticsIT.testOutlierDetectionWithFewDocuments failure in windows RunDataFrameAnalyticsIT failures Sep 2, 2020
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 2, 2020
Closed
@dimitris-athanasiou
Copy link
Contributor

I'm working on a fix for this. I should have a PR shortly.

dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[ML] Update reindexing task progress before persisting job progress
32d3427 
This fixes a bug introduced by elastic#61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes elastic#61852
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 2, 2020
Merged
dimitris-athanasiou added a commit that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[ML] Update reindexing task progress before persisting job progress (#…
76fa500 
…61868)

This fixes a bug introduced by #61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes #61852
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[7.x][ML] Update reindexing task progress before persisting job progr…
44bf6c3 
…ess (elastic#61868)

This fixes a bug introduced by elastic#61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes elastic#61852

Backport of elastic#61868
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 2, 2020
Merged
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[7.9][ML] Update reindexing task progress before persisting job progr…
9ba16de 
…ess (elastic#61868)

This fixes a bug introduced by elastic#61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes elastic#61852

Backport of elastic#61868
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Sep 2, 2020
Merged
dimitris-athanasiou added a commit that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[7.x][ML] Update reindexing task progress before persisting job progr…
ec40597 
…ess (#61868) (#61875)

This fixes a bug introduced by #61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes #61852

Backport of #61868
dimitris-athanasiou added a commit that referenced this issue Sep 2, 2020
@dimitris-athanasiou
[7.9][ML] Update reindexing task progress before persisting job progr…
83a0fbd 
…ess (#61868) (#61876)

This fixes a bug introduced by #61782. In that PR I thought I could
simplify the persistence of progress by using the progress straight
from the stats holder in the task instead of calling the get
stats action. However, I overlooked that it is then possible to
have stale progress for the reindexing task as that is only updated
when the get stats API is called.

In this commit this is fixed by updating reindexing task progress
before persisting the job progress. This seems to be much more
lightweight than calling the get stats request.

Closes #61852

Backport of #61868
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimitris-athanasiou dimitris-athanasiou

Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
3 participants
@dimitris-athanasiou @jkakavas @elasticmachine