Skip to content

Commit

Permalink
[Filebeat] Add dashboard for vpc flow log (#16007)
Browse files Browse the repository at this point in the history
* Add dashboard for vpc flow log
* update doc for aws filebeat module
  • Loading branch information
kaiyan-sheng committed Feb 3, 2020
1 parent 583a89c commit 249929e
Show file tree
Hide file tree
Showing 5 changed files with 685 additions and 25 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
*Filebeat*
- Set event.outcome field based on googlecloud audit log output. {pull}15731[15731]
- Add dashboard for AWS ELB fileset. {pull}15804[15804]

- Add dashboard for AWS vpcflow fileset. {pull}16007[16007]

*Heartbeat*

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 52 additions & 12 deletions filebeat/docs/modules/aws.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -15,22 +15,14 @@ beta[]
This is a module for aws logs. It uses filebeat s3 input to get log files from
AWS S3 buckets with SQS notification. This module supports reading s3 server
access logs with `s3access` fileset, ELB access logs with `elb` fileset, VPC
flow logs with `vpc` fileset, and CloudTrail logs with `cloudtrail` fileset.
flow logs with `vpcflow` fileset, and CloudTrail logs with `cloudtrail` fileset.

Access logs contain detailed information about the requests made to these
services. VPC flow logs captures information about the IP traffic going to and
from network interfaces in AWS VPC. CloudTrail logs contain events
from network interfaces in AWS VPC. ELB access logs captures detailed information
about requests sent to the load balancer. CloudTrail logs contain events
that represent actions taken by a user, role or AWS service.

[float]
=== Example dashboard

This module comes with several predefined dashboards. For example, here is the
dashboard for `s3access` fileset:

[role="screenshot"]
image::./images/filebeat-aws-s3access-overview.png[]

[float]
=== Module configuration

Expand Down Expand Up @@ -112,12 +104,60 @@ Filename of AWS credential file.

AWS credential profile name.

=== CloudTrail fileset
=== cloudtrail fileset

CloudTrail monitors events for the account. If user creates a trail, it
delivers those events as log files to a specific Amazon S3 bucket.
The `cloudtrail` fileset does not read the CloudTrail Digest files
that are delivered to the S3 bucket when Log File Integrity is turned
on, it only reads the CloudTrail logs.

=== elb fileset

Elastic Load Balancing provides access logs that capture detailed information
about requests sent to the load balancer. Each log contains information such
as the time the request was received, the client's IP address, latencies,
request paths, and server responses. Users can use these access logs to analyze
traffic patterns and to troubleshoot issues.

Please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html[enable access logs for classic load balancer]
for sending Classic ELB access logs to S3 bucket.
For application load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging[enable access log for application load balancer].
For network load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html[enable access log for network load balancer].

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-elb-overview.png[]

=== s3access fileset

Server access logging provides detailed records for the requests that are made
to a bucket. Server access logs are useful for many applications. For example,
access log information can be useful in security and access audits. It can also
help you learn about customer base and understand Amazon S3 bill.

Please follow https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html#server-access-logging-overview[how to enable server access logging]
for sending server access logs to S3 bucket.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-s3access-overview.png[]

=== vpcflow fileset

VPC Flow Logs is a feature in AWS that enables users to capture information
about the IP traffic going to and from network interfaces in VPC. Flow log data
needs to be published to Amazon S3 in order for `vpcflow` fileset to retrieve.
Flow logs can help users to monitor traffic that is reaching each instance and
determine the direction of the traffic to and from the network interfaces.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-vpcflow-overview.png[]


[float]
=== Fields
Expand Down
64 changes: 52 additions & 12 deletions x-pack/filebeat/module/aws/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,14 @@ beta[]
This is a module for aws logs. It uses filebeat s3 input to get log files from
AWS S3 buckets with SQS notification. This module supports reading s3 server
access logs with `s3access` fileset, ELB access logs with `elb` fileset, VPC
flow logs with `vpc` fileset, and CloudTrail logs with `cloudtrail` fileset.
flow logs with `vpcflow` fileset, and CloudTrail logs with `cloudtrail` fileset.

Access logs contain detailed information about the requests made to these
services. VPC flow logs captures information about the IP traffic going to and
from network interfaces in AWS VPC. CloudTrail logs contain events
from network interfaces in AWS VPC. ELB access logs captures detailed information
about requests sent to the load balancer. CloudTrail logs contain events
that represent actions taken by a user, role or AWS service.

[float]
=== Example dashboard

This module comes with several predefined dashboards. For example, here is the
dashboard for `s3access` fileset:

[role="screenshot"]
image::./images/filebeat-aws-s3access-overview.png[]

[float]
=== Module configuration

Expand Down Expand Up @@ -107,8 +99,56 @@ Filename of AWS credential file.

AWS credential profile name.

=== CloudTrail fileset
=== cloudtrail fileset

CloudTrail monitors events for the account. If user creates a trail, it
delivers those events as log files to a specific Amazon S3 bucket.
The `cloudtrail` fileset does not read the CloudTrail Digest files
that are delivered to the S3 bucket when Log File Integrity is turned
on, it only reads the CloudTrail logs.

=== elb fileset

Elastic Load Balancing provides access logs that capture detailed information
about requests sent to the load balancer. Each log contains information such
as the time the request was received, the client's IP address, latencies,
request paths, and server responses. Users can use these access logs to analyze
traffic patterns and to troubleshoot issues.

Please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html[enable access logs for classic load balancer]
for sending Classic ELB access logs to S3 bucket.
For application load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#enable-access-logging[enable access log for application load balancer].
For network load balancer, please follow https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html[enable access log for network load balancer].

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-elb-overview.png[]

=== s3access fileset

Server access logging provides detailed records for the requests that are made
to a bucket. Server access logs are useful for many applications. For example,
access log information can be useful in security and access audits. It can also
help you learn about customer base and understand Amazon S3 bill.

Please follow https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html#server-access-logging-overview[how to enable server access logging]
for sending server access logs to S3 bucket.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-s3access-overview.png[]

=== vpcflow fileset

VPC Flow Logs is a feature in AWS that enables users to capture information
about the IP traffic going to and from network interfaces in VPC. Flow log data
needs to be published to Amazon S3 in order for `vpcflow` fileset to retrieve.
Flow logs can help users to monitor traffic that is reaching each instance and
determine the direction of the traffic to and from the network interfaces.

This fileset comes with a predefined dashboard:

[role="screenshot"]
image::./images/filebeat-aws-vpcflow-overview.png[]
Loading

0 comments on commit 249929e

Please sign in to comment.