Integrate Packetbeat with Agent

[andrewstucki]

Describe the enhancement:

Integrate packetbeat with agent.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

Relates: #22227

[andrewstucki]

See discussion in #22227 for details about the spec file format.

Packages

We probably want to have multiple packages -- a single package for something like Network Traffic that we can use to enable flow data as well as potentially our protocol dissectors. Additionally, we should be able to have packet-type inputs in other packages that would allow another entrypoint for integration to specific products.

Indexing strategy

Really, the indexing strategy is an implementation detail of the packages, but it will require updating the default index patterns that get created when you use agent. I'm leaning towards having things coming from Packetbeat wind up in their own network-* indices, but we should continue that conversation.

[ruflin]

Related to indexing strategy, the part I'm most interested about is how you think of the dataset part?

[andrewstucki]

@ruflin dataset generally just defaults to {package}.{stream} unless its explicitly overridden (which I don't believe a single package is currently doing) -- so it doesn't seem to me like Packetbeat indexing strategy should diverge from what other packages are doing.

Was there an attempt to move away from this in other packages?

The main reason I brought up network-* as potentially a new indexing pattern is that network traffic events don't quite make sense to try and jam into a logs-* or metrics-* index, and adopting network-* (I believe) would require changes to add support for creation of those index patterns on the Kibana side.

Any thoughts @andrewkroh ?

[ruflin]

@andrewstucki I think there are parts in endpoint package which do not fully follow this logic but probably not relevant in this context here. We need to make a difference here between inputs and the specific config for this input. For example the logfile input is used in many packages and has stream configs for it. So this ends up being nginx.access, nginx.error etc. Even though both use the logfile input, the resulting dataset structure is different because of the ingest pipeline processing.

How does fit packetbeat into this? Is http just the same as logfile input or is it the dataset itself? In the case of nginx and we fetch http traffic on the machine for nginx, what should the dataset name be in this case?