Add Kubernetes autodiscover test cases for standalone Agent

[jsoriano]

Continue with #1064 to cover also autodiscover test cases with standalone Agent.

Current tests use the file output to write the events, if it is not possible to use this output with Agent we will need to refactor this part to write the events in Elasticsearch or other supported output.

// cc: @mdelapenya @ChrsMark

[ChrsMark]

I had a look into this and tried to figure out what is needed here. I see some possible blockers here:

0. I'm assuming that we are gonna use standalone Agent for simplicity since we only want to test the internal functionality of the k8s provider.
1. The tests for beats' autodiscovery follow a simplistic (but really handy) approach of writing the output to a file. Unfortunately this is not possible with elastic-agent since it does not support this output yet. So the only alternative is to write the output to ES.
2. This brings us to the problem of where this ES could run. The best option would be to be deployed as a Pod so as to be in the same network with Agent's pod and be directly accessible. Is this doable within this framework though?
3. Last but not least we would need to verify the events, most probably using the ES API?

@jsoriano @mdelapenya let me know what you think about the above and if this scenario would be reasonable and doable. Also any pointers on top of it are more than welcome.

[jsoriano]

Hey @ChrsMark, thanks for looking into this. Yes, I agree with your assumptions, we should simplify the deployment using standalone Agent, and if Agent is only able to output to ES, we are going to need a ES deployed.
Also for simplicity, this ES could be deployed as a single node in a pod in the same K8s cluster where the test is run, then the output of the Agent could be configured to use this node, and events could be collected from there with HTTP requests.
Maybe this ES pod can be even part of the Agent manifest, or even a container in the same pod. Then current framework could be enough, though it will take a bit more of time to start 🙂 With this we would only need to add something to retrieve and check the collected events.

[ChrsMark]

Thanks for your feedback @jsoriano. I tried to add a simple case at #1645 and it seems to work. Only tricky part is that I use a background script to fetch events from ES api and write them to the expected file. Other option would be to tune the suite so as to be able to fetch/check events directly from ES but I'm not sure what effort it would require. Let me know what you think.

[jsoriano]

@ChrsMark if it works, I think we can start with this. If we see this trick is a burden to maintain we can reconsider in the future.