Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LEAK: resource was not cleaned up before it was garbage-collected #104522

Closed
albertzaharovits opened this issue Jan 18, 2024 · 6 comments · Fixed by #104524 or #104546
Closed

[LEAK: resource was not cleaned up before it was garbage-collected #104522

albertzaharovits opened this issue Jan 18, 2024 · 6 comments · Fixed by #104524 or #104546
Assignees
@original-brownbear
Labels
blocker :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team >test-failure Triaged test failures from CI

Comments

@albertzaharovits
Copy link
Contributor

albertzaharovits commented Jan 18, 2024 •
edited
Loading

This only fails for FIPS JVM.

CI Link

https://gradle-enterprise.elastic.co/s/i3kbauaznos3u

Repro line

./gradlew ':x-pack:plugin:async-search:test' --tests "org.elasticsearch.xpack.search.AsyncSearchTaskTests.testWithFailureAndGetResponseFailureDuringReduction" -Dtests.seed=AC371F6873FDAB2 -Dtests.locale=ar-QA -Dtests.timezone=Pacific/Funafuti -Druntime.java=21 -Dtests.fips.enabled=true

Does it reproduce?

No

Applicable branches

main

Failure history

No response

Failure excerpt

Might be related to #104000

https://buildkite.com/elastic/elasticsearch-pull-request/builds/7908#018d1c62-8d1b-42cd-bab0-448a6e5ce3f9

org.elasticsearch.xpack.search.AsyncSearchTaskTests > testWithFailureAndGetResponseFailureDuringReduction FAILED
    java.lang.AssertionError:
    Expected: an empty collection
         but: <[LEAK: resource was not cleaned up before it was garbage-collected.
    Recent access records:
    #1:
        org.elasticsearch.xpack.core.search.action.AsyncSearchResponse$1.closeInternal(AsyncSearchResponse.java:53)
        org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:70)
        org.elasticsearch.transport.LeakTracker$2.decRef(LeakTracker.java:126)
        org.elasticsearch.xpack.core.search.action.AsyncSearchResponse.decRef(AsyncSearchResponse.java:135)
        org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289)
        org.elasticsearch.xpack.search.AsyncSearchTask.addCompletionListener(AsyncSearchTask.java:206)
        org.elasticsearch.xpack.search.AsyncSearchTaskTests.lambda$assertCompletionListeners$13(AsyncSearchTaskTests.java:437)
        java.base/java.lang.Thread.run(Thread.java:1583)
    #2:
        org.elasticsearch.xpack.search.MutableSearchResponse.toAsyncSearchResponse(MutableSearchResponse.java:291)
        org.elasticsearch.xpack.search.AsyncSearchTask.getResponse(AsyncSearchTask.java:348)
        org.elasticsearch.xpack.search.AsyncSearchTask.getResponseWithHeaders(AsyncSearchTask.java:339)
        org.elasticsearch.xpack.search.AsyncSearchTask.addCompletionListener(AsyncSearchTask.java:206)
        org.elasticsearch.xpack.search.AsyncSearchTaskTests.lambda$assertCompletionListeners$13(AsyncSearchTaskTests.java:437)
        java.base/java.lang.Thread.run(Thread.java:1583)

https://buildkite.com/elastic/elasticsearch-pull-request/builds/7909#018d1c6c-5fce-4be3-ab74-e2acb8271de0

java.lang.AssertionError:
    Expected: an empty collection
         but: <[LEAK: resource was not cleaned up before it was garbage-collected.
    Recent access records:
    Created at:
        org.elasticsearch.search.SearchHits.<init>(SearchHits.java:79)
        org.elasticsearch.search.SearchHits.<init>(SearchHits.java:61)
        org.elasticsearch.xpack.inference.registry.ModelRegistryTests.mockSearchResponse(ModelRegistryTests.java:317)
        org.elasticsearch.xpack.inference.registry.ModelRegistryTests.testGetUnparsedModelMap_ThrowsIllegalStateException_WhenUnableToFindInferenceEntry(ModelRegistryTests.java:97)
        java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        java.base/java.lang.reflect.Method.invoke(Method.java:580)
        com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        java.base/java.lang.Thread.run(Thread.java:1583), LEAK: resource was not cleaned up before it was garbage-collected.
    Recent access records:
    Created at:
        org.elasticsearch.search.SearchHits.<init>(SearchHits.java:79)
        org.elasticsearch.search.SearchHits.<init>(SearchHits.java:61)
        org.elasticsearch.xpack.inference.registry.ModelRegistryTests.mockSearchResponse(ModelRegistryTests.java:317)
        org.elasticsearch.xpack.inference.registry.ModelRegistryTests.testGetUnparsedModelMap_ThrowsIllegalStateException_WhenUnableToFindInferenceSecretsEntry(ModelRegistryTests.java:114)
        java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        java.base/java.lang.reflect.Method.invoke(Method.java:580)
        com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        org.junit.rules.RunRules.evaluate(RunRules.java:20)
        com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        java.base/java.lang.Thread.run(Thread.java:1583), LEAK: resource was not cleaned up before it was garbage-collected.
    Recent access records:
@albertzaharovits albertzaharovits added :Core/Infra/Core Core issues without another label >test-failure Triaged test failures from CI labels Jan 18, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:Core/Infra Meta label for core/infra team labels Jan 18, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@albertzaharovits
Copy link
Contributor Author

@DaveCTurner Can you take a look, please? Is this somehow related to: #104000 ?

@DaveCTurner DaveCTurner added :Search/Search Search-related issues that do not fall into other categories and removed :Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team labels Jan 18, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Search Meta label for search team label Jan 18, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Jan 18, 2024 •
edited
Loading

Is this somehow related to: #104000 ?

I don't think so, I suspect it might be a consequence of #103763. Redirecting to the search team.

#104000 relates to the lifecycle of requests, but the leak here relates to responses.

@albertzaharovits
Copy link
Contributor Author

Good point! Thanks!

@albertzaharovits albertzaharovits mentioned this issue Jan 18, 2024
Closed
@original-brownbear original-brownbear self-assigned this Jan 18, 2024
@original-brownbear original-brownbear mentioned this issue Jan 18, 2024
Merged
@albertzaharovits
Copy link
Contributor Author

There's one 🐛 test that was not squashed:

https://gradle-enterprise.elastic.co/s/liwmdrom3rl6o/tests/:x-pack:plugin:inference:test/org.elasticsearch.xpack.inference.services.huggingface.embeddings.HuggingFaceEmbeddingsModelTests/testThrowsURISyntaxException_ForInvalidUrl

https://gradle-enterprise.elastic.co/s/qthosjmkgv4kg/tests/:x-pack:plugin:inference:test/org.elasticsearch.xpack.inference.services.huggingface.embeddings.HuggingFaceEmbeddingsModelTests/testThrowsURISyntaxException_ForInvalidUrl

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 18, 2024
@original-brownbear
Fix SearchResponse leaks in ModelRegistryTests
79ec29e 
It's in the title, closes elastic#104522
@original-brownbear original-brownbear mentioned this issue Jan 18, 2024
Merged
albertzaharovits pushed a commit to albertzaharovits/elasticsearch that referenced this issue Jan 18, 2024
@original-brownbear @albertzaharovits
Fix SearchResponse leaks in ModelRegistryTests
585a64c 
It's in the title, closes elastic#104522
original-brownbear added a commit that referenced this issue Jan 18, 2024
@original-brownbear
Fix SearchResponse leaks in ModelRegistryTests (#104546)
75169ce 
It's in the title, closes #104522
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
blocker :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@albertzaharovits @DaveCTurner @original-brownbear @elasticsearchmachine