[Flaky Test]: TestLongRunningAgentForLeaks/TestHandleLeak – Condition never satisfied

[rdner]

Failing test case

TestLongRunningAgentForLeaks/TestHandleLeak

Error message

Condition never satisfied

Build

https://buildkite.com/elastic/elastic-agent/builds/7876#018e5c3a-00ae-4972-b890-37cad38f2a96

OS

Windows

Stacktrace and notes

agent_long_running_leak_test.go:232: component state: Healthy: communicating with pid '3216'
    agent_long_running_leak_test.go:223: unit ID: winlog-fleet-default-output
    agent_long_running_leak_test.go:223: unit ID: winlog-fleet-default-output-winlog-system-e3f438c4-571c-44b0-a8d5-25e4ea556eef
    agent_long_running_leak_test.go:232: component state: Healthy: communicating with pid '2344'
    agent_long_running_leak_test.go:210: 
        	Error Trace:	C:/Users/windows/agent/testing/integration/agent_long_running_leak_test.go:210
        	            				C:/Users/windows/agent/testing/integration/agent_long_running_leak_test.go:142
        	Error:      	Condition never satisfied
        	Test:       	TestLongRunningAgentForLeaks/TestHandleLeak
        	Messages:   	install never became healthy
--- FAIL: TestLongRunningAgentForLeaks/TestHandleLeak (180.00s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

@fearful-symmetry the log message for this needs to be more specific, identifying what is missing or what specifically is unhealthy.

[rdner]

It failed in the daily build too, perhaps it's a real leak then https://buildkite.com/elastic/elastic-agent/builds/7884#018e5e50-9390-41d6-855d-2979484718c7

[rdner]

More failures in PR builds here:

• https://buildkite.com/elastic/elastic-agent/builds/7887#018e6028-a8c6-4a57-b671-3d66843e15a8
• https://buildkite.com/elastic/elastic-agent/builds/7888#018e6039-9e5f-4d9f-97eb-685da5920054

[rdner]

I skipped the tests for now to avoid more failures, needs to be investigated ASAP
#4452

[cmacknz]

Are we sure this is a bug in this test, and it is not that this test is finding a real bug in the underlying inputs it is testing?

[rdner]

@cmacknz it can be a real failure, we need to properly investigate of course. The most concerning fact is that it started failing without any connection to a new change in the code.

It first started failing after #4447 was merged. Initially, I thought this change introduced something that fails the test, despite the PR being green.

However, then I opened a new PR to revert this change #4451 and in that PR the leak tests also failed which proves that the failure is not related to the change but just came out of nowhere.

[cmacknz]

It is making sure the underlying Beat inputs are healthy, it is entirely possibly an integration or a change in Beats triggered this failure.