Elastic-agent should not swallows stderr and panics

[mtojek]

Hi Team,

this issue proved that we need a more user-friendly method to collect all panics. Otherwise we won't be able to grab anything from customers.

In the mentioned case I used strace to listen what filebeat is sending to /dev/null, but we need a more trival method.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mtojek]

@jlind23 I suggest giving it a higher priority as it may result in a pile of mysterious SDH issues: the process doesn't run, don't see any panics, no error, no logs collected, etc.

[jlind23]

@ruflin @ph From your opinion would it be something easy to put in place?

[ph]

I don't think it would be much effort and this would align with v2 changes for logging. IE: We should be able to stream stderr from a subprocess into the Elastic Agent Appender.

[ruflin]

I like the idea of starting to feed this into the elastic-agent logs. This will help us to already gain some experience around how it will work in v2 as @ph pointed out.

[AndersonQ]

jotting down ideas:

• do we have panics (stderr) logged to systemd/jornald?
• could we collect systemd/jornald using the agent's diagnostics collect command?
• if the logger is set up when the agent panics, we could recover and properly log the panic
• is there any trade-off for always collecting systemd/jornald logs for the agent on the diagnostics collect?

[ph]

Funny thing is, if Agent recover and we are running journald input I presume we would get that information in Elasticsearch.

[mtojek]

Awesome, thanks @AndersonQ!