[Rule Tuning] Roshal Archive (RAR) or PowerShell File Downloaded from the Internet

[willemdh]

https://www.elastic.co/guide/en/security/current/roshal-archive-rar-or-powershell-file-downloaded-from-the-internet.html

Description

The way this rule currently works is highly ineffective and only searches in packetbeat data and also uses regex with front wildcard. it would be way more efficient if it would (also) query NGFW data, for example from panw.panos. This dataset contains all download url's and also has a field url.extension.

Instead of having to do a front wildcard Lucene regex query, like:

event.category:(network OR network_traffic) AND network.protocol:http AND url.path:/.*(rar|ps1)/ AND source.ip:(10.0.0.0\/8 OR 172.16.0.0\/12 OR 192.168.0.0\/16)

This could be changed to a more efficient KQL query like:

event.category:(network OR network_traffic) and url.extension:(ps1 or rar) and not source.ip:(10.0.0.0/8 or 127.0.0.0/8 or 169.254.0.0/16 or 172.16.0.0/12 or 192.168.0.0/16 or 224.0.0.0/4 or "::1" or "FE80::/10" or "FF00::/8") and destination.ip:(10.0.0.0/8 or 172.16.0.0/12 or 192.168.0.0/16)

Example Data

{
    "destination": {
      "nat": {},
      "geo": {
        "continent_name": "North America",
        "country_iso_code": "US",
        "country_name": "United States",
        "name": "Belgium",
        "location": {
          "lon": -97.822,
          "lat": 37.751
        }
      },
      "as": {
        "number": 35994,
        "organization": {
          "name": "Akamai Technologies, Inc."
        }
      },
      "address": "14.117.77.16",
      "port": 80,
      "ip": "14.117.77.16"
    },
    "rule": {
      "name": "DMZ-Public-to-Internet"
    },
    "source": {
      "nat": {},
      "geo": {
        "continent_name": "Europe",
        "country_iso_code": "BE",
        "country_name": "Belgium",
        "name": "Belgium",
        "location": {
          "lon": 24.35,
          "lat": 20.85
        }
      },
      "as": {
        "number": 688,
        "organization": {
          "name": "Tlent BVBA"
        }
      },
      "address": "8.143.72.17",
      "port": 36795,
      "ip": "8.143.72.17"
    },
    "syslog": {
      "priority": 14,
      "facility": 1,
      "severity_label": "Informational",
      "facility_label": "user-level"
    },
    "type": "mon-elastic-beats-filebeat",
    "panw": {
      "panos": {
        "sub_type": "url",
        "flow_id": "35116964",
        "destination": {
          "nat": {
            "port": 0,
            "ip": "0.0.0.0"
          },
          "zone": "Internet",
          "interface": "ae2.498"
        },
        "ruleset": "DMZ-Public-to-Internet",
        "action": "alert",
        "source": {
          "nat": {
            "port": 0,
            "ip": "0.0.0.0"
          },
          "zone": "Internet",
          "interface": "ethernet1/15.451"
        },
        "threat": {
          "resource": "corporate.updates.pandasecurity.com/aether/nano/pavsignano/nano_1/av_1/firmdig_0307000/firmdig_0307000_20210413_002025.rar",
          "name": "URL-filtering",
          "id": "9999"
        },
        "type": "THREAT",
        "url": {
          "category": "computer-and-internet-info"
        },
        "network": {}
      }
    },
    "network": {
      "community_id": "1:g5ZB/5iC4I+sjLPtLNdrYFX7uuY=",
      "application": "panda-update",
      "transport": "tcp",
      "direction": "inbound"
    },
    "ecs": {
      "version": "1.7.0"
    },
    "related": {
      "hosts": [
        "srvpa5250",
        "corporate.updates.pandasecurity.com"
      ]
    },
    "client": {
      "nat": {},
      "port": 36795,
      "ip": "8.143.72.17"
    },
    "event": {
      "severity": 5,
      "ingested": "2021-04-13T12:42:14.977507565Z",
      "created": "2021-04-13T14:42:12.000+02:00",
      "timezone": "+02:00",
      "kind": "alert",
      "module": "panw",
      "action": "url_filtering",
      "category": [
        "security_threat",
        "intrusion_detection",
        "network"
      ],
      "type": [
        "allowed"
      ],
      "dataset": "panw.panos",
      "outcome": "success"
    },
    "fileset": {
      "name": "panos"
    },
    "url": {
      "path": "/aether/nano/pavsignano/nano_1/av_1/firmdig_0307000/firmdig_0307000_20210413_002025.rar",
      "extension": "rar",
      "original": "corporate.updates.pandasecurity.com/aether/nano/pavsignano/nano_1/av_1/firmdig_0307000/firmdig_0307000_20210413_002025.rar",
      "domain": "corporate.updates.pandasecurity.com"
    },
    "labels": {
      "container_page": true,
      "temporary_match": true
    },
    "input": {
      "type": "syslog"
    },
    "@timestamp": "2021-04-13T14:42:12.000+02:00",
    "service": {
      "type": "panw"
    }
  }
}

Unfortunately packetbet http dataset does not have url.extension so it might be a good idea to add that first with another issue in the Beats repo, so we can use this rule on both datasets?

[willemdh]

Hmmm just had a better look at the results and it seems like the current query captures traffic from internal ip addresses instead of public ip addresses?

event.category:(network OR network_traffic) AND network.protocol:http AND url.path:/.*(rar|ps1)/ AND source.ip:(10.0.0.0\/8 OR 172.16.0.0\/12 OR 192.168.0.0\/16)

The generated signals source.ip are internal addresses, while the rule is for "Downloaded from the Internet"? Can someone verify if my assumption is correct. If so, this is not a rule tuning, but a bug issue.

Also the private ranges seem to be missing some:

10.0.0.0/8 - IPv4 private
127.0.0.0/8 - IPv4 loopback
169.254.0.0/16 - IPv4 link-local
172.16.0.0/12 - IPv4 private
192.168.0.0/16 - IPv4 private
224.0.0.0/4 - IPv4 multicast
::1 - IPv6 loopback
FE80::/10 - IPv6 link-local
FF00::/8 - IPv6 multicast

So the query should more look like:

and  not source.ip:(
    10.0.0.0/8 or
    127.0.0.0/8 or
    169.254.0.0/16 or
    172.16.0.0/12 or
    192.168.0.0/16 or
    224.0.0.0/4 or
    "::1" or
    "FE80::/10" or
    "FF00::/8"
  ) 
and  destination.ip:(
    10.0.0.0/8 or
    172.16.0.0/12 or
    192.168.0.0/16
  )

[willemdh]

@bm11100 could you have a look at this issue? Sth seems not right with this rule no?

[bm11100]

We could also add an or to account for url.path or url.extension to cover both scenarios.

This rule detects when an internal network client sends DNS traffic directly to the Internet.

Looking further, the rule isn't factoring in what the destination ip is, so its possible the destination is an internal IP instead of a public ip and we want to look for the destination ip being a public address.

So what if we just rule out the non-routable destination.ip address spaces listed here?

This should find internal network connections to a publicly routable address space that includes the ps1 and rar extensions.

event.category:(network or network_traffic) and network.protocol:http and
(url.extension:(ps1 or rar) or url.path:(*.ps1 or *.rar)) and
  not destination.ip:(
    10.0.0.0/8 or
    127.0.0.0/8 or
    169.254.0.0/16 or
    172.16.0.0/12 or
    192.0.0.0/24 or
    192.0.0.0/29 or
    192.0.0.8/32 or
    192.0.0.9/32 or
    192.0.0.10/32 or
    192.0.0.170/32 or
    192.0.0.171/32 or
    192.0.2.0/24 or
    192.31.196.0/24 or
    192.52.193.0/24 or
    192.168.0.0/16 or
    192.88.99.0/24 or
    224.0.0.0/4 or
    100.64.0.0/10 or
    192.175.48.0/24 or
    198.18.0.0/15 or
    198.51.100.0/24 or
    203.0.113.0/24 or
    240.0.0.0/4 or
    "::1" or
    "FE80::/10" or
    "FF00::/8"
  ) and
  source.ip:(
    10.0.0.0/8 or
    172.16.0.0/12 or
    192.168.0.0/16
  )

I've pushed the changes in this PR - #1227

[peasead]

Hey @willemdh

First off, thanks for your comments. They are well thought out and relevant.

Efficiency
Regarding the inefficiency, agreed. Having url.extension in Packetbeat would definitely allow this rule to use KQL instead of Lucene.

When this rule was written, leading *'s weren't being considered for KQL rules, which is what led us to use Lucene. As Brent has shown above, we don't have that limitation, so we can clean it up using url.path and make it KQL.

PANOS Data
I think adding PANOS is a fantastic idea. You can certainly recommend a PR by adding the Filebeat index pattern and a comment to the description that the PAN Filebeat module is needed. If you're sending ECS-compliant PAN data into the stack a different way (using Logstash or the like), you'd need to account for that as well.

Of note, when looking at the exported fields for the PAN Filebeat module, I don't see url.extension, so that'd be needed as well.

Source IPs
You're 💯 right, there are more than just the standard RFC1918 IPs. We're actually going through the process of adding those to all the network rules. You beat us to the punch!

Tune Up
I think that what @bm11100 has would work great for url.path and for sources that use url.extension while we work on getting url.extension added to the PAN module and Packetbeat.

What do you think?

[willemdh]

Sounds like a perfect plan, have a nice weekend all

[peasead]

Same to you, @willemdh

For your reference:
https://discuss.elastic.co/t/url-extension-not-populated/274278
elastic/beats#25990

[willemdh]

@peasead Just re-read you comments

Of note, when looking at the exported fields for the PAN Filebeat module, I don't see url.extension, so that'd be needed as well.

And I wanted to add that the by Elastic provided panw dataset definitely has an url.extension field..

[peasead]

Ah, looks like it.

I was going off the Filebeat docs, which may need to be updated...or am I overlooking something?

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html

https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-panw.html

[willemdh]

My guess is the docs probably need some work..

[legoguy1000]

I created elastic/beats#25999 to add the url.extension field for Packetbeat HTTP events. Its ready to merge if anyone wants to test it just to make sure it meets your needs.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[legoguy1000]

Wouldn't it be better to do something like below as to not hardcode IPs that may or not be valid for users?

event.category:(network or network_traffic) and network.protocol:http and
(url.extension:(ps1 or rar) or url.path:(*.ps1 or *.rar)) and network.direction: inbound

This way it takes advantage of theinternal_networks setting that Packetbeat and many filebeat modules have or the other methods that perimeter devices have to detect direction.

[peasead]

@legoguy1000 I think that's a great idea.

There were some issues a while ago with network directions, Zeek, and the associated Filebeat module which is why we leaned away from that approach. That's no longer an issue, so I think that's a solid PR idea.

[legoguy1000]

@peasead ok, I'll submit a PR.

[legoguy1000]

@peasead I started to go through the list and realized a lot of the rules are based on Auditbeat, Winglogbeat, Endpoint events which don't have the internal_networks/network.direction context. so IDK if its the best try and change those to using network.direction: outbound from the current source dest validation against the reserved local IPs. I don't know if there is a good way around that.

[peasead]

Hm, yeah. It looks like the Endpoint uses incoming and outgoing for several protcols, Packetbeat and Auditbeat use ingress and egress.

Let me check on a few things, because incoming and outgoing aren't expected values of network.direction

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[w0rk3r]

@peasead starting from 7.16.0, Endpoint will log network.direction with the ingress/egress value, not incoming/outgoing.
ref #1547 , we can add a similar OR condition ("incoming", "ingress")\("outgoing", "egress")

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.