[Filebeat] Add cloudwatch fileset in aws module

[kaiyan-sheng]

What does this PR do?

This PR is to add cloudwatch fileset into aws module to parse logs from CloudWatch AWS. There are different kinds of logs can be sent to CloudWatch from different services. This PR is only focusing on EC2 logs.

Why is it important?

Users can use Amazon CloudWatch Logs to monitor, store, and access log files from Amazon EC2 instances, AWS CloudTrail, Route 53, and other sources. This fileset enables users to export logs into s3 bucket and parse logs using this fileset.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes [Filebeat] Add support for AWS CloudWatch logs #13716

[exekias]

I understand cloudwatch is more like an input, right @kaiyan-sheng? Where the actual logs (coming from log groups) can be of many kinds.

Would it make sense to create specific filesets for these different cloudwatch logs sources? For instance, the examples you are putting here probably map to the system module, that can be reused if configured with the right input.

Then we have a plethora of other services that are AWS specific and we don't yet support as filesets.

[kaiyan-sheng]

I understand cloudwatch is more like an input, right @kaiyan-sheng? Where the actual logs (coming from log groups) can be of many kinds.

Yes, cloudwatch is more like an input. Different services can send logs into CW.

Would it make sense to create specific filesets for these different cloudwatch logs sources? For instance, the examples you are putting here probably map to the system module, that can be reused if configured with the right input.

That would be better for sure! The examples are logs sent to CloudWatch from EC2 instance. Should we rename this fileset something like cloudwatch_ec2?

Then we have a plethora of other services that are AWS specific and we don't yet support as filesets.

Great! That's why I added %{TIMESTAMP_ISO8601:_tmp.timestamp} %{GREEDYDATA:message} in grok pattern.

[exekias]

That would be better for sure! The examples are logs sent to CloudWatch from EC2 instance. Should we rename this fileset something like cloudwatch_ec2?

That sounds great! I wonder if we should keep the “cloudwatch” prefix or do just ec2. At the end, I foresee these filesets being shared between Filebeat & Functionbeat at some point (through the integrations project). If you call it ec2 now you can just make it clear in the docs & configs that this one comes from cloudwatch

[kaiyan-sheng]

That sounds great! I wonder if we should keep the “cloudwatch” prefix or do just ec2. At the end, I foresee these filesets being shared between Filebeat & Functionbeat at some point (through the integrations project). If you call it ec2 now you can just make it clear in the docs & configs that this one comes from cloudwatch

Yeah if we ever decides to add cloudwatch as a separate Filebeat input, these filesets can be shared there too. I will change the fileset name to ec2 in next commit. Thanks! Will you be ok with cloudwatch as the fileset name for the default %{TIMESTAMP_ISO8601:_tmp.timestamp} %{GREEDYDATA:message} format logs?

[exekias]

Yeah if we ever decides to add cloudwatch as a separate Filebeat input, these filesets can be shared there too. I will change the fileset name to ec2 in next commit. Thanks! Will you be ok with cloudwatch as the fileset name for the default %{TIMESTAMP_ISO8601:_tmp.timestamp} %{GREEDYDATA:message} format logs?

I think this is a good idea!

[kaiyan-sheng]

@exekias @mtojek This PR is ready for another review (when you get a chance). Thanks!

[exekias]

btw this needs a make update