Convert Filebeat system.auth to ECS

[webmat]

Caveats

• Assuming system auth logs only come from local system, not remote systems. So only populating host.hostname and ignoring device.hostname.
• ECS is missing a timezone field, but should have one. Currently leaving to beats.timezone.
• system.auth.ssh.dropped_ip should be at least copied to source.ip.
  • Should we rename it, and mention the dropping of the connection at event.action? Or should we just copy it, and the presence of system.auth.ssh.dropped_ip is what indicates it was dropped?
    • Note that this module is not keeping the original message for ssh events.
• group field set is not part of ECS, at this time. But based on user.id and user.name semantics, I went with their obvious group.* equivalents. It will be fine ;-)
• After performing straight field renames on these auth messages, I wonder if we're missing something. For example the first pattern uses only ECS fields, if it wasn't for one last field that hasn't been renamed (system.auth.ssh.method, line 29), I would never know that its ECS event is a message about SSH. Added to [WIP] Filebeat module issues found during ECS conversion #9208

ECS Fields temporarily defined locally

• host.hostname, added to system/_meta/fields.yml
• group.id, added to system/_meta/fields.yml
• group.name, added to system/_meta/fields.yml

Renames

• system.auth.hostname => host.hostname
• system.auth.pid => process.pid
• system.auth.program => process.name
• system.auth.message => message
• system.auth.useradd.gid => group.id
• system.auth.groupadd.gid => group.id
• system.auth.useradd.uid => user.id
• system.auth.useradd.user => user.name
• system.auth.ssh.event => event.action
• system.auth.ssh.ip => source.ip
• system.auth.ssh.port => source.port
• system.auth.ssh.geoip.* => source.geo.*

TODO

• Changelog
• ECS-migrations.yml
• Create field aliases
• Copy dropped_ip to source_ip via pipeline
• Test failure: "Cannot write to a field alias [system.auth.timestamp]."
• Post-rebase test failure: AssertionError: expected 10 events to compare but got 9.
  • "Cannot write to a field alias [system.auth.groupadd.gid]."

Noticed

• Patterns specific to sshd, sudo, groupadd and useradd. Added to [WIP] Filebeat module issues found during ECS conversion #9208
  • Do not set process.name. They should.
  • Do not set message.
• There is no test for general syslog messages that are not one of: sshd, sudo, groupadd and useradd. Added to [WIP] Filebeat module issues found during ECS conversion #9208
• No longer sure about mapping system.auth.ssh.signature to event.hash.

[webmat]

@ruflin This one is not ready for review. But I’d like to discuss some of the caveats.

[ruflin]

Caveats

• I think host.hostname is in both cases correct. We should probably have a discussion about what we think device is (not in this PR).
• +1 on keeping beat.timezone for now. If I understand it correctly, this is the timezone of the host machine?
• dropped_ip: Sounds like a good case where copy is more accurate then just the alias. As the alias alone would always be there.
• +1 on group
• ssh event: Interesting note but would delay discussion as we have at the moment the information needed.

[webmat]

Finishing up on this PR, I'm no longer sure about mapping system.auth.ssh.signature to event.hash.

Thoughts, @ruflin, @MikePaquette?

Should event.hash be strictly for ensuring integrity of messages in the pipeline (what I understand from the current ECS description), or can it be used to contain the hash or signature an event is describing?

[webmat]

@ruflin Ready for final review.

Prior to merging I'd like opinions on this comment, though #9138 (comment). This signature field is no longer mapped to ECS' event.hash.

Also of note: I thought I would use copy_to for system.auth.ssh.dropped_ip => source.ip, but the copy_to key in the fields.yml doesn't seem to work. Instead I'm doing it via the ingest pipeline.

[ruflin]

@webmat Failures on CI seem to be related.

[ruflin]

Change LGTM but check what happens in the tests.

[webmat]

@ruflin Failure fixed. All tests should be green now. Will merge tomorrow unless something comes up

[ruflin]

@webmat One general comment for ingest pipeline vs copy_to. I think the advantage of using ingest_pipeline is that at one point it can also be applied later again to a raw log line and we it does not depend on a template to be correct.

[webmat]

Merging without waiting on Darwin. Rest of the Jenkins run is green.

[webmat]

GitHub messed up the squash message 🤦‍♂️