[Filebeat][New Fileset] Cisco Umbrella support (elastic#21504)

* initial MVP push for cisco umbrella

* Initial commit for Cisco Umbrella ready for review

* updating some wording in the cisco docs for umbrella

(cherry picked from commit 641d02f)

CHANGELOG.next.asciidoc

@@ -689,6 +689,7 @@ field. You can revert this change by configuring tags for the module and omittin
 - Add related.hosts ecs field to all modules {pull}21160[21160]
 - Keep cursor state between httpjson input restarts {pull}20751[20751]
 - Convert aws s3 to v2 input {pull}20005[20005]
+- New Cisco Umbrella dataset {pull}21504[21504]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -26579,6 +26579,153 @@ type: keyword
 --
 This key captures values or decorators used within a registry entry
 
+type: keyword
+
+--
+
+[float]
+=== cisco.umbrella
+
+Fields for Cisco Umbrella.
+
+
+
+*`cisco.umbrella.identities`*::
++
+--
+An array of the different identities related to the event.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.categories`*::
++
+--
+The security or content categories that the destination matches.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.policy_identity_type`*::
++
+--
+The first identity type matched with this request. Available in version 3 and above.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.identity_types`*::
++
+--
+The type of identity that made the request. For example, Roaming Computer or Network.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.blocked_categories`*::
++
+--
+The categories that resulted in the destination being blocked. Available in version 4 and above.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.content_type`*::
++
+--
+The type of web content, typically text/html.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.sha_sha256`*::
++
+--
+Hex digest of the response content.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.av_detections`*::
++
+--
+The detection name according to the antivirus engine used in file inspection.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.puas`*::
++
+--
+A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_disposition`*::
++
+--
+The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_malware_name`*::
++
+--
+If Malicious, the name of the malware according to AMP.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_score`*::
++
+--
+The score of the malware from AMP. This field is not currently used and will be blank.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.datacenter`*::
++
+--
+The name of the Umbrella Data Center that processed the user-generated traffic.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.origin_id`*::
++
+--
+The unique identity of the network tunnel.
+
+
 type: keyword
 
 --

filebeat/docs/modules/cisco.asciidoc

@@ -10,15 +10,15 @@ This file is generated! See scripts/docs_collector.py
 
 == Cisco module
 
-beta[]
-
-This is a module for Cisco network device's logs. It includes the following
+This is a module for Cisco network device's logs and Cisco Umbrella. It includes the following
 filesets for receiving logs over syslog or read from a file:
 
 - `asa` fileset: supports Cisco ASA firewall logs.
 - `ftd` fileset: supports Cisco Firepower Threat Defense logs.
 - `ios` fileset: supports Cisco IOS router and switch logs.
 - `nexus` fileset: supports Cisco Nexus switch logs.
+- `meraki` fileset: supports Cisco Meraki logs.
+- `umbrella` fileset: supports Cisco Umbrella logs.
 
 Cisco ASA devices also support exporting flow records using NetFlow, which is
 supported by the {filebeat-ref}/filebeat-module-netflow.html[netflow module] in
@@ -34,6 +34,8 @@ The module is by default configured to run via syslog on port 9001 for ASA and
 port 9002 for IOS. However it can also be configured to read from a file path.
 See the following example.
 
+Cisco Umbrella publishes its logs in a compressed CSV format to a S3 bucket.
+
 ["source","yaml",subs="attributes"]
 -----
 - module: cisco
@@ -381,6 +383,59 @@ will be found under `rsa.raw`. The default is false.
 
 :fileset_ex!:
 
+[float]
+==== `umbrella` fileset settings
+
+The Cisco Umbrella fileset primarily focuses on reading CSV files from an S3 bucket using the filebeat S3 input.
+
+To configure Cisco Umbrella to log to either your own S3 bucket or one that is managed by Cisco please follow the https://docs.umbrella.com/deployment-umbrella/docs/log-management[Cisco Umbrella User Guide.]
+
+This fileset supports all 4 log types:
+- Proxy
+- Cloud Firewall
+- IP Logs
+- DNS logs
+
+The Cisco Umbrella fileset depends on the original file path structure being followed. This structure is documented https://docs.umbrella.com/deployment-umbrella/docs/log-formats-and-versioning[Umbrella Log Formats and Versioning]:
+
+<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz
+dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz
+
+When configuring the fileset, please ensure that the Queue URL is set to the root folder that includes each of these subfolders above.
+
+Example config:
+
+[source,yaml]
+----
+- module: cisco
+  umbrella:
+    enabled: true
+    var.input: s3
+    var.queue_url: https://sqs.us-east-1.amazonaws.com/ID/CiscoQueue
+    var.access_key_id: 123456
+    var.secret_access_key: PASSWORD
+----
+
+*`var.input`*::
+
+The input from which messages are read. Can be S3 or file.
+
+*`var.queue_url`*::
+
+The URL to the SQS queue if the input type is S3.
+
+*`var.access_key_id`*::
+
+The ID for the access key used to read from the SQS queue.
+
+*`var.secret_access_key`*::
+
+The secret token used for authenticating to the SQS queue.
+
+:has-dashboards!:
+
+:fileset_ex!:
+
 [float]
 === Example dashboard
 

x-pack/filebeat/module/cisco/_meta/docs.asciidoc

@@ -5,15 +5,15 @@
 
 == Cisco module
 
-beta[]
-
-This is a module for Cisco network device's logs. It includes the following
+This is a module for Cisco network device's logs and Cisco Umbrella. It includes the following
 filesets for receiving logs over syslog or read from a file:
 
 - `asa` fileset: supports Cisco ASA firewall logs.
 - `ftd` fileset: supports Cisco Firepower Threat Defense logs.
 - `ios` fileset: supports Cisco IOS router and switch logs.
 - `nexus` fileset: supports Cisco Nexus switch logs.
+- `meraki` fileset: supports Cisco Meraki logs.
+- `umbrella` fileset: supports Cisco Umbrella logs.
 
 Cisco ASA devices also support exporting flow records using NetFlow, which is
 supported by the {filebeat-ref}/filebeat-module-netflow.html[netflow module] in
@@ -29,6 +29,8 @@ The module is by default configured to run via syslog on port 9001 for ASA and
 port 9002 for IOS. However it can also be configured to read from a file path.
 See the following example.
 
+Cisco Umbrella publishes its logs in a compressed CSV format to a S3 bucket.
+
 ["source","yaml",subs="attributes"]
 -----
 - module: cisco
@@ -376,6 +378,59 @@ will be found under `rsa.raw`. The default is false.
 
 :fileset_ex!:
 
+[float]
+==== `umbrella` fileset settings
+
+The Cisco Umbrella fileset primarily focuses on reading CSV files from an S3 bucket using the filebeat S3 input.
+
+To configure Cisco Umbrella to log to either your own S3 bucket or one that is managed by Cisco please follow the https://docs.umbrella.com/deployment-umbrella/docs/log-management[Cisco Umbrella User Guide.]
+
+This fileset supports all 4 log types:
+- Proxy
+- Cloud Firewall
+- IP Logs
+- DNS logs
+
+The Cisco Umbrella fileset depends on the original file path structure being followed. This structure is documented https://docs.umbrella.com/deployment-umbrella/docs/log-formats-and-versioning[Umbrella Log Formats and Versioning]:
+
+<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz
+dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz
+
+When configuring the fileset, please ensure that the Queue URL is set to the root folder that includes each of these subfolders above.
+
+Example config:
+
+[source,yaml]
+----
+- module: cisco
+  umbrella:
+    enabled: true
+    var.input: s3
+    var.queue_url: https://sqs.us-east-1.amazonaws.com/ID/CiscoQueue
+    var.access_key_id: 123456
+    var.secret_access_key: PASSWORD
+----
+
+*`var.input`*::
+
+The input from which messages are read. Can be S3 or file.
+
+*`var.queue_url`*::
+
+The URL to the SQS queue if the input type is S3.
+
+*`var.access_key_id`*::
+
+The ID for the access key used to read from the SQS queue.
+
+*`var.secret_access_key`*::
+
+The secret token used for authenticating to the SQS queue.
+
+:has-dashboards!:
+
+:fileset_ex!:
+
 [float]
 === Example dashboard
 

x-pack/filebeat/module/cisco/umbrella/_meta/fields.yml

@@ -0,0 +1,61 @@
+- name: cisco.umbrella
+  type: group
+  description: >
+    Fields for Cisco Umbrella.
+  fields:
+  - name: identities
+    type: keyword
+    description: >
+      An array of the different identities related to the event.
+  - name: categories
+    type: keyword
+    description: >
+      The security or content categories that the destination matches.
+  - name: policy_identity_type
+    type: keyword
+    description: >
+      The first identity type matched with this request. Available in version 3 and above.
+  - name: identity_types
+    type: keyword
+    description: >
+      The type of identity that made the request. For example, Roaming Computer or Network.
+  - name: blocked_categories
+    type: keyword
+    description: >
+      The categories that resulted in the destination being blocked. Available in version 4 and above.
+  - name: content_type
+    type: keyword
+    description: >
+      The type of web content, typically text/html.
+  - name: sha_sha256
+    type: keyword
+    description: >
+      Hex digest of the response content.
+  - name: av_detections
+    type: keyword
+    description: >
+      The detection name according to the antivirus engine used in file inspection.
+  - name: puas
+    type: keyword
+    description: >
+      A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
+  - name: amp_disposition
+    type: keyword
+    description: >
+      The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
+  - name: amp_malware_name
+    type: keyword
+    description: >
+      If Malicious, the name of the malware according to AMP.
+  - name: amp_score
+    type: keyword
+    description: >
+      The score of the malware from AMP. This field is not currently used and will be blank.
+  - name: datacenter
+    type: keyword
+    description: >
+      The name of the Umbrella Data Center that processed the user-generated traffic.
+  - name: origin_id
+    type: keyword
+    description: >
+      The unique identity of the network tunnel.