Support more resources in Kubernetes composable provider

[exekias]

Kubernetes composable provider supports discovering Pods, we should extend this to other resources, starting with Nodes and Services, to match what Beats can do today. This issue is to discuss the possible approach for this.

I don't see a big technical challenge (perhaps fixing #21842) as this is a problem we have solved before, this is mostly about the user experience we want to provide. This is my initial proposal on how things would look for the kubernetes composable provider, I focused on the common use cases:

Note: this is also tied to the "Running Agent on Kubernetes discussion"

providers.kubernetes:
  unique_election: true  # Enables leader election between all Agent pods (we could decide hiding this and making it always true)
  ...
  resources:   # List of resources to watch
   # logs & metrics common use case, scoped to the current node, watch all Pods (and their containers)
   - resource: pod
     scope: node
     namespaces: []

   # Single team without cluster wide permissions, they normally want to have a deployment that watches a whole namespace (for metrics only)
   - resource: pod
     scope: cluster
     namespaces: ["myapp"]

   # hearbeat monitoring the whole cluster
   - resource: service
     scope: cluster
     unique: true   # Flag to emit events only from the leader

# kube-state-metrics
input:
 condition:
   - {{kubernetes.unique}} == true  # Launch this input only in the leader, different from the Heartbeat case 

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

cc @blakerouse ?

[blakerouse]

I think with the solving of #21842 we should just add them as part of the kubernetes dynamic provider.

To match beats today we need to add Services. I think that kubernetes.service.* will work correctly with #21842 solved.

@exekias is their other types to add? maybe kubernetes.ingress.*?

[exekias]

The other one needed to match beats would be kubernetes.node.

We didn't discuss ingress as of today, these are partially covered by service, but happy to discuss more.

I'm personally ok to keeping the scope in node, pod & service for now.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!