Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sei: use ECS definition of geo.location #4227

Merged
merged 3 commits into from
Sep 19, 2022
Merged

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Sep 19, 2022

What does this PR do?

This performs the the following replace:

from:

- description: Longitude and latitude\.
  level: core
  (name: .*\.geo\.location)
  type: geo_point

or:

- (name: .*\.geo\.location)
  description: Longitude and latitude\.
  example: '{ "lon": -73\.614830, "lat": 45\.505918 }'
  type: geo_point

or:

- description: Longitude and latitude\.
  example: '{ "lon": -73\.614830, "lat": 45\.505918 }'
  (name: .*\.geo\.location)
  type: geo_point

to:

- external: ecs
  $1

over all packages owned by Security External Integrations, except for deprecated packages.

Changelog and manifest files were updated manually.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 self-assigned this Sep 19, 2022
@efd6 efd6 added Team:Cloud-Monitoring Label for the Cloud Monitoring team 8.5 candidate labels Sep 19, 2022
This performs the the following replace:

from:
	- description: Longitude and latitude\.
	  level: core
	  (name: .*\.geo\.location)
	  type: geo_point
or:
	- (name: .*\.geo\.location)
	  description: Longitude and latitude\.
	  example: '{ "lon": -73\.614830, "lat": 45\.505918 }'
	  type: geo_point
or:
	- description: Longitude and latitude\.
	  example: '{ "lon": -73\.614830, "lat": 45\.505918 }'
	  (name: .*\.geo\.location)
	  type: geo_point
to:
	- external: ecs
	  $1

over all packages owned by Security External Integrations, except for
deprecated packages.

Changelog and manifest files were updated manually.
@elasticmachine
Copy link

elasticmachine commented Sep 19, 2022

🚀 Benchmarks report

Package akamai 👍(1) 💚(0) 💔(0)

Package atlassian_bitbucket 👍(0) 💚(1) 💔(0)

Package atlassian_confluence 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 1872.66 1577.29 -295.37 (-18.73%) 💔

Package atlassian_jira 👍(0) 💚(1) 💔(0)

Package auditd 👍(1) 💚(0) 💔(0)

Package auth0 👍(1) 💚(0) 💔(0)

Package barracuda 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
spamfirewall 37037.04 29411.76 -7625.28 (-25.93%) 💔
waf 37037.04 31250 -5787.04 (-18.52%) 💔

Package bluecoat 👍(0) 💚(1) 💔(0)

Package cef 👍(0) 💚(1) 💔(0)

Package checkpoint 👍(1) 💚(0) 💔(0)

Package cisco_aironet 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2036.66 1754.39 -282.27 (-16.09%) 💔

Package cisco_asa 👍(1) 💚(0) 💔(0)

Package cisco_duo 👍(1) 💚(3) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
offline_enrollment 37037.04 27027.03 -10010.01 (-37.04%) 💔

Package cisco_ftd 👍(1) 💚(0) 💔(0)

Package cisco_ios 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 719.42 615.76 -103.66 (-16.83%) 💔

Package cisco_meraki 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 15151.52 11764.71 -3386.81 (-28.79%) 💔
log 15625 12500 -3125 (-25%) 💔

Package cisco_nexus 👍(1) 💚(0) 💔(0)

Package cisco_secure_endpoint 👍(1) 💚(0) 💔(0)

Package cisco_umbrella 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 13888.89 8264.46 -5624.43 (-68.06%) 💔

Package citrix_waf 👍(1) 💚(0) 💔(0)

Package cloudflare 👍(2) 💚(0) 💔(0)

Package crowdstrike 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
falcon 11904.76 9803.92 -2100.84 (-21.43%) 💔

Package cyberarkpas 👍(1) 💚(0) 💔(0)

Package cylance 👍(1) 💚(0) 💔(0)

Package f5 👍(1) 💚(1) 💔(0)

Package fireeye 👍(1) 💚(0) 💔(0)

Package fortinet_forticlient 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 24390.24 15873.02 -8517.22 (-53.66%) 💔

Package fortinet_fortigate 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1329.79 1059.32 -270.47 (-25.53%) 💔

Package fortinet_fortimail 👍(1) 💚(0) 💔(0)

Package fortinet_fortimanager 👍(0) 💚(1) 💔(0)

Package gcp 👍(4) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
firewall 2000 1730.1 -269.9 (-15.6%) 💔

Package google_workspace 👍(2) 💚(3) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
groups 4184.1 3472.22 -711.88 (-20.5%) 💔

Package hid_bravura_monitor 👍(2) 💚(0) 💔(0)

Package imperva 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
securesphere 37037.04 30303.03 -6734.01 (-22.22%) 💔

Package iptables 👍(1) 💚(0) 💔(0)

Package juniper_junos 👍(0) 💚(1) 💔(0)

Package juniper_netscreen 👍(0) 💚(1) 💔(0)

Package juniper_srx 👍(1) 💚(0) 💔(0)

Package keycloak 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1075.27 887.31 -187.96 (-21.18%) 💔

Package mattermost 👍(1) 💚(0) 💔(0)

Package modsecurity 👍(1) 💚(0) 💔(0)

Package netflow 👍(1) 💚(0) 💔(0)

Package netscout 👍(0) 💚(1) 💔(0)

Package netskope 👍(2) 💚(0) 💔(0)

Package o365 👍(1) 💚(0) 💔(0)

Package okta 👍(1) 💚(0) 💔(0)

Package panw_cortex_xdr 👍(1) 💚(0) 💔(0)

Package pfsense 👍(1) 💚(0) 💔(0)

Package pulse_connect_secure 👍(1) 💚(0) 💔(0)

Package qnap_nas 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2320.19 1964.64 -355.55 (-18.1%) 💔

Package slack 👍(1) 💚(0) 💔(0)

Package sophos 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
utm 22727.27 17241.38 -5485.89 (-31.82%) 💔

Package squid 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 35714.29 27777.78 -7936.51 (-28.57%) 💔

Package suricata 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
eve 525.21 447.03 -78.18 (-17.49%) 💔

Package tomcat 👍(1) 💚(0) 💔(0)

Package zeek 👍(23) 💚(15) 💔(5)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
modbus 23809.52 19607.84 -4201.68 (-21.43%) 💔
ntp 34482.76 27027.03 -7455.73 (-27.59%) 💔
connection 17241.38 10638.3 -6603.08 (-62.07%) 💔
smtp 21739.13 18867.92 -2871.21 (-15.22%) 💔
software 41666.67 30303.03 -11363.64 (-37.5%) 💔

Package zscaler_zpa 👍(3) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
app_connector_status 1669.45 1349.53 -319.92 (-23.71%) 💔
audit 3937.01 2739.73 -1197.28 (-43.7%) 💔

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

elasticmachine commented Sep 19, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-09-19T10:36:43.684+0000

  • Duration: 81 min 59 sec

Test stats 🧪

Test Results
Failed 0
Passed 1708
Skipped 4
Total 1712

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Sep 19, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (124/124) 💚
Files 98.198% (218/222) 👍 0.877
Classes 98.198% (218/222) 👍 0.877
Methods 92.543% (1998/2159) 👍 2.784
Lines 90.909% (40399/44439) 👎 -0.532
Conditionals 100.0% (0/0) 💚

@efd6 efd6 marked this pull request as ready for review September 19, 2022 02:14
@efd6 efd6 requested review from a team as code owners September 19, 2022 02:14
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Copy link
Contributor

@djptek djptek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes look OK and versions match up

for p in `ls packages` ; do cv=`grep -- "- version" packages/${p}/changelog.yml | head -1 | sed -e "s/- //" -e "s/\"//g"` ; mv=`grep -- "^version:" packages/${p}/manifest.yml | sed -e "s/\"//g"` ; if [[ ${cv} = ${mv} ]]; then echo "${p} ok"; else echo "${p} version mismatch ${cv} != ${mv}"; fi ; done

LGTM

Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments, but approved it.
These are the packages ( -> data streams) which seem to be missing the same updates:

atlassian_bitbucket
cloudflare -> audit
google_workspace -> groups, login, saml
hashicorp_vault
keycloak
mattermost
netskope
panw_cortex_xdr
slack
sophos
zscaler_zpa -> app_connector_status
zscaler_zpa -> browser_access

description: Longitude and latitude.
example: '{ "lon": -73.614830, "lat": 45.505918 }'
type: geo_point
external: ecs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's also a source.geo.location field in this file. Does it also need to be changed?

description: Longitude and latitude.
example: '{ "lon": -73.614830, "lat": 45.505918 }'
type: geo_point
external: ecs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are also other fields in this file namely destination.geo.location , source.geo.location, and observer.geo.location

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also in audit.

@@ -100,10 +96,8 @@
name: source.geo.country_iso_code
- external: ecs
name: source.geo.country_name
- description: Longitude and latitude.
level: core
- external: ecs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source.geo.location is duplicated

@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.4.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't have ECS fields of that type. Can be ignored?

Copy link
Member

@endorama endorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gcp package LGTM

@efd6
Copy link
Contributor Author

efd6 commented Sep 19, 2022

  • atlassian_bitbucket
  • cloudflare -> audit
  • google_workspace -> groups, login, saml, user_acounts
  • hashicorp_vault already using external definition
  • keycloak
  • mattermost
  • netskope
  • panw_cortex_xdr
  • slack
  • sophos
  • zscaler_zpa -> app_connector_status
  • zscaler_zpa -> browser_access already using external definition

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.5 candidate enhancement New feature or request Team:Cloud-Monitoring Label for the Cloud Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants