[Security Solution] Visual Event Analyzer not available for sysmon events ingested with Elastic Agent

[kowalczyk-p]

Describe the bug:
Visual event analyzer does not work from events ingested from sysmon with Elastic Agent

Kibana/Elasticsearch Stack version:
8.4.3

Server OS version:
Docker

Elastic Agent version:
8.4.3

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):
Visual Events Analyzer

Docs available at https://www.elastic.co/guide/en/security/current/visual-event-analyzer.html states that:

You can only visualize events triggered by hosts configured with the Elastic Defend integration or any sysmon data from winlogbeat.

In KQL, this translates to any event with the agent.type set to either:

endpoint
winlogbeat with event.module set to sysmon

Problem is systmon events ingested with Elastic Agent windows integration have following values:

• event.module : windows
• agent.type : filebeat.

Visual events analyzer works for sysmon events ingested with winlogbeat but does not work for same events ingested with Elastic Agent.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kqualters-elastic]

@kowalczyk-p are you sure these events are coming from winlogbeat + sysmon? The sample fields are from a filebeat event, which won't have the requisite process and parent process information needed to build the visual event analyzer, it's just file info. Do you have a full sample event?

[kowalczyk-p]

@kqualters-elastic events are from sysmon but I am currently in process of migrating from Windows Event Forwarding + winlogbeat to Elastic Agent. Currently every event generated by sysmon is ingested in my Elasticsearch twice:

• once by winlogbeat from event collector - those event have event.module = sysmon and agent.type = winlogbeat
• second by Elastic Agent directly from workstation - whose have event.module = windows and agent.type = filebeat.
  From what I read this is way Elastic Agent works - https://discuss.elastic.co/t/does-the-elastic-agent-contain-winlogbeat/282043.
  I can paste full sample event next week.

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[michaelolo24]

Thanks for opening this issue @kowalczyk-p ! Currently, we explicitly check for a given agent.type & event.module to enable the analyzer. The code for that is here: https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/public/detections/components/alerts_table/timeline_actions/investigate_in_resolver.tsx . With that being said, the use case you've mentioned here is a feasible use case, but we'd need to investigate further to make sure it doesn't enable analyzer in situations where it may not work as expected.

[kowalczyk-p]

Thanks, I hope you will find it easy to solve as the event source (Sysmon) is same and only way of indexing it into Elasticsearch is changed (winlogbeat vs elastic agent).

[kqualters-elastic]

@kowalczyk-p that linked pr should fix this issue, sorry that happened, and even more sorry for the delay in getting it fixed, your issue + the mentions on discuss were the first time we became aware of it, was a breaking change on agent side. The reason that we have to be so restrictive in trying to render the analyzer in the first place, and not just any time an event has a process.entity_id, is because some places in agent use entity_id to be a unique id for the process that is shipping the data, and not as a unique id for the underlying observed processes creating data. I wish this was not the case, but the Ecs docs even mention that the exact meaning of the field is dependent on the data source: https://www.elastic.co/guide/en/ecs/current/ecs-process.html#field-process-entity-id This can result in a basically useless set of information, with all events having the same id, and we want to be sure that we are dealing with a dataset that is using entity_id as expected. Thanks for filing the issue!

[kowalczyk-p]

@kqualters-elastic Thank You! I,m looking forward for version 8.7.

[michaelolo24]

@kowalczyk-p - Just wanted to check in. If you've had a chance to upgrade to 8.7, can you confirm your issue was resolved? Thanks!

[kowalczyk-p]

@michaelolo24 not yet, we have some delay in upgrading our cluster. Currently we are on 8.5 so it will take us some time to reach 8.7.

[PhilippeOberti]

In an effort of triaging/cleaning up old tickets, I'm just checking here @kowalczyk-p if the upgrade was done and you were able to check if this was correctly fixed for you?

[kowalczyk-p]

@PhilippeOberti we are half way there. I expect to be updated to 8.7 in two weeks.

[michaelolo24]

Hey @kowalczyk-p - Just checking back in on this 😄, how did the upgrade go?

[kowalczyk-p]

Upgrade completed today and analyzer is working :) Thank you very much!