Add event.dataset to events from cloudfoundry filebeat input

[blakerouse]

What does this PR do?

Adds event.dataset to events generated by the cloudfoundry filebeat input. This is so the events match the same format that metricbeat automatically sets for events.

Why is it important?

Helps with filtering and building dashboards for cloudfoundry.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes Add event.dataset to filebeat for cloud foundry fields #17000

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

I think that Filebeat inputs don't set event.dataset, this is set in modules (e.g. the s3 input doesn't add event.dataset, but aws filesets add it). input.type is set in any case.

If we inclue this change we also have to think:

• What event.dataset would be set on modules that use the cloudfoundry input?
• Should we set event.dataset in all inputs?

Maybe as an alternative, we can set for this case input.type to the values proposed, and leave event.dataset for filesets as in other cases. Would it also solve the problem?

[blakerouse]

@jsoriano The field is trying to get the same field that metricbeat uses, instead of 2 different fields event.dataset for metricbeat and input.type in the other.

If event.dataset is only set by the module we could add a cloudfoundry module to filebeat that just sets up the input and adds the event.dataset. That seems like a little bit over engineering to just set one field, but maybe that would be a good base to maybe expand on the module later? But then again I think having a module might be less confusing to the user because at the moment its an input in filebeat and a module in metricbeat. So having 1 module in each, might provide a better user experience. Thoughts?

If the module is a no and don't want to set the event.dataset field in the input it, I could just go back to the idea of cloudfoundry.type on the event, that way it's the same from metricbeat and filebeat. But thats not really using a ECS field for that.

[jsoriano]

@jsoriano The field is trying to get the same field that metricbeat uses, instead of 2 different fields event.dataset for metricbeat and input.type in the other.

Yes, the problem I see is that filebeat uses event.dataset to indicate the module and fileset, and input.type to indicate the input, we should also be consistent with this. I am not against setting event.dataset by default to some value decided by the input, but then we should consider doing it for other inputs too.

If event.dataset is only set by the module we could add a cloudfoundry module to filebeat that just sets up the input and adds the event.dataset. That seems like a little bit over engineering to just set one field, but maybe that would be a good base to maybe expand on the module later?

In the case of azure or AWS there are filesets for specific services or kinds of logs, could something like this make sense on cloudfoundry? In that case I think we could have a cloudfoundry module to collect data about cloudfoundry itself.

Also take into account that filebeat modules allow to override their inputs, so you could use for example the redis module with the S3 input (for logs stored in S3), in this case event.dataset would be redis.log and the input.type would be s3.

If the cloudfoundry input can collect plain logs from services deployed there we could also use the redis module with this input, in this case event.dataset would also be redis.log, and input.type would be cloudfoundry.

But then again I think having a module might be less confusing to the user because at the moment its an input in filebeat and a module in metricbeat. So having 1 module in each, might provide a better user experience. Thoughts?

Yeah, I think that having a module for the use case of collecting data about cloudfoundry itself can make sense.

If the module is a no and don't want to set the event.dataset field in the input it, I could just go back to the idea of cloudfoundry.type on the event, that way it's the same from metricbeat and filebeat. But thats not really using a ECS field for that.

If there are different kinds of events collected by the same input it can make sense to have a field specific for this input, like cloudfoundry.type in this case. (There is also an event.kind field in ECS, but it has a limited number of accepted values, and not sure if this is thought for this).

[blakerouse]

@jsoriano Now that I understand better that the event.dataset would be more of a module that does parsing on top of an input I have change to using cloudfoundry.type as the type information is coming directly from Cloud Foundry.

I think this actually makes the most sense from the input and will be shared with the module in metricbeat as well.

It's ready for another look.

[blakerouse]

@jsoriano Fixed, ready for another review.