[Question]: 'windows.forwarded' and 'windows.sysmon_operational' dataset logs not available under datastreams page for agent deployed

[dikshachauhan-qasource]

Kibana version:
8.0 Snapshot Kibana Cloud environment

Host OS and Browser version:
Windows server 2012, All

Preconditions

1. 8.0 Snapshot Kibana Cloud environment should be available.

BUILD 39668
COMMIT a43d609b094763a61969a00c764d015d3454923f
Artifact link: https://snapshots.elastic.co/8.0.0-95ee58b5/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip

2. Default policy having windows integration should exist.[No change in namespace].
3. Agent should be deployed using above policy.
4. Data streaming is available for the above agent.

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Datastreams Tab.
3. Goto integration dropdowm.
4. Select 'windows' as option.
5. Observe, following dataset logs not available on DataStream page.

• 'windows.forwarded'
• 'windows.powershell_operational'
• 'windows.sysmon_operational'

Observation
On Windows Server 2012
Following dataset logs not available on DataStream page.

• 'windows.forwarded'
• 'windows.powershell_operational'
• 'windows.sysmon_operational'

On Windows 10
Following dataset logs not available on DataStream page.

• 'windows.forwarded'
• 'windows.sysmon_operational'

Queries
We have looked for 'Forwardedevents' and 'sysmon' Events in Event viewer in both OSs and could not find them at location:

Event Viewer(local)
>Application and Services Logs
>Microsoft
>Windows.

So as per our understanding, that could be the reason of non-availability for below datastreams

• 'windows.forwarded'
• 'windows.sysmon_operational'

Query 1: Could you please let us know if it is expected or any action is required to be performed to trigger these events.
So that required data sets gets generated on Data stream page.

Query 2: We have observed that only 'windows.powershel' logs dataset was generated for Windows server 2012 OS.
So, do we need to report defect for 'windows.powershell_operational' or it is working as expected.

Screenshots:

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[fearful-symmetry]

@narph can you look at this? Not sure if this is a bug or just the system not having the logs.

[EricDavisX]

@narph further, anything you want the tem to help re-test - @dikshachauhan-qasource and team can test on more systems or execute certain end-user actions / operations on the host to validate. some of this may be our lack of understanding of how to test it appropriately.

[EricDavisX]

@dikshachauhan-qasource and Manish says they have some research on applications that can help triggre those datasets. Please do post what you know. And, we can install those applications so long as they aren't malware / malicious. :)

[narph]

We have looked for 'Forwardedevents' and 'sysmon' Events in Event viewer in both OSs and could not find them at location:

Event Viewer(local)

Application and Services Logs
Microsoft
Windows.

So as per our understanding, that could be the reason of non-availability for below datastreams

'windows.forwarded'
'windows.sysmon_operational'

Query 1: Could you please let us know if it is expected or any action is required to be performed to trigger these events.
So that required data sets gets generated on Data stream page.

Yes , a few actions are necessary in order to generate events for both Forwarded and sysmon.

The forwarded events log contains only events collected from remote hosts using the Windows Event Collector. This option is only available on operating systems supporting the Windows Event Log API (Microsoft Windows Vista and newer). There is extensive online documentation that goes over the steps of windows log forwarding, if anyone is interested I can try to expand on that and provide more info here.

The sysmon data stream processes event log records from the Sysinternals System Monitor (Sysmon) which is a Windows service and device driver that logs system activity to the event log. Sysmon is not bundled with Windows or Winlogbeat and must be installed independently.

Query 2: We have observed that only 'windows.powershel' logs dataset was generated for Windows server 2012 OS.
So, do we need to report defect for 'windows.powershell_operational' or it is working as expected.

That is interesting, one thing you can quickly check in the Windows Server 2012 is if the event logs are there for Microsoft-Windows-Sysmon/Operational.
You can either do that by opening Event Viewer and checking under Application and Service Logs/Microsoft/Windows/Powershell if Operatonal events are there or just run, for example:

Get-WinEvent -ListLog * | where {$_.RecordCount -gt 0}

in PowerShell and check for for Microsoft-Windows-Sysmon/Operational.
I am sure there is a better cmd but that should list all log names with a record count bigger than 0.
If you notice any events under that log name that means there could be an issue with this windows version, if so I would be happy to try reproducing this issue.

[dikshachauhan-qasource]

Hi @EricDavisX

For sysmon logs, we have researched and found sysmon V13.01 application available on Microsoft Official website. As per our understanding if that application is installed and collecting some logs then we can attempt to validate data for windows.sysmon datasets under Data Streams page.

Further, for 'windows.forwarded' logs dataset, we have gone through this article available on Microsoft Official website. It seems hard to validate. Please have a look and provide your feedback on this.

Thanks
QAS

[narph]

hi @dikshachauhan-qasource , I have replied above.

For sysmon logs, we have researched and found sysmon V13.01 application available on Microsoft Official website. As per our understanding if that application is installed and collecting some logs then we can attempt to validate data for windows.sysmon datasets under Data Streams page.

Yes, we also reference https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon in our Winlogbeat docs, if this is missing in the integration documentation we need to update it.

Further, for 'windows.forwarded' logs dataset, we have gone through this article available on Microsoft Official website. It seems hard to validate. Please have a look and provide your feedback on this.

Not sure if you are still testing on the Windows Server 2012 machine but here is some info and steps here:
https://docs.microsoft.com/en-us/troubleshoot/windows-server/admin-development/configure-eventlog-forwarding-performance
https://petri.com/configure-event-log-forwarding-windows-server-2012-r2

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[dikshachauhan-qasource]

Hi @narph

Today, we have attempted to reproduce the issue on 8.0 snapshot cloud build. However, due to defect #23652, we are blocked.

We will revalidate above observation once reported issue is resolved.

cc @EricDavisX

Thanks
QAS

[EricDavisX]

note- this can be tested on 7.11 latest BC at this point, as the package has been merged to prod. @dikshachauhan-qasource if 7.11 does NOT have the same install problem as 23652 above, please retest there and report back. Thank you.

[amolnater-qasource]

Hi @narph

As per feedback on #551 (comment) we have re-validated this issue on 8.0.0 Snapshot Kibana Cloud environment with agent installed on Windows Server 2012 . Build details are as follows:

Build: 39872
Commit: 0fe7b9e080c67c43aefdb7ea25d5e90a80cb4ade
Artifact link for Windows: https://snapshots.elastic.co/8.0.0-e9bbbb5f/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip

Observations:

• We have found the issue that "windows.powershell_operational" dataset logs are not available under Data Streams tab when "Microsoft-Windows-Powershell/Operational" events are available for Windows Server 2012.
• Hence we have reported a defect for the same- "windows.powershell_operational" not available in Fleet Data Streams tab when relating events are on host Win Server 2012. #570 .
  Further let us know if anything is required for the same.

Query:
@narph Could you please provide us detailed steps to reproduce 'windows.forwarded' and 'windows.sysmon' datasets under Data Streams page .

cc @EricDavisX

Thanks
QAS

[EricDavisX]

Ph and I can throw our brief opinion in, we do not think these are needed for 7.11.

'windows.forwarded' - is this something we want to just skip in manual testing? @narph
'windows.sysmon_operational' - this should be testable, if it is not, we could also skip it.

One thought PH has is that we could list out all of the fields / data_streams and pre-acknowledge which ones are hard to test manually, and will therefore be skipped during manual iterations by the Fleet test team. Further, we could put this as part of the integration package, to make it more clear.

Thoughts?

[dikshachauhan-qasource]

Closing this issue as will be taken care by Onsite Automation team.