Abnormal Security Sentinel New Polling Logic

[rednithin]

New Polling Logic for Abnormal Security Threats and Cases.

Change(s):

• Introduced a new way to poll for Abnormal Security Threats and Cases.

Reason for Change(s):

We were getting complaints from various customers about events not being ingested correctly.

The existing polling approach doesn't account for two things:

• Our systems are eventually consistent systems.
• Race conditions occur for threats events because the lastRemediatedAt timestamp might get updated before the request reaches our service. (network latency)

The new polling approach accounts for both of the above issues and is behind a feature flag. We will be asking some customers to test out the new logic before we switch all of the customers over.

Version Updated:

Should we update this?

Packaging

Ran these commands to package the new changes.

sentinel_test_and_package_output.txt

Testing Completed:

The following are completed

• Wrote new unit tests for the new approach.
• Tested legacy polling logic.
• Tested the new polling logic which is behind a feature flag.

Zip File Used for Testing: https://github.com/Azure/Azure-Sentinel/raw/56150b8fac65730add168e7575816fe6074bd01a/Solutions/AbnormalSecurity/Data%20Connectors/AbnormalSecurityConn.zip

Testing Legacy Method

Tested existing polling method for threats by sending 42 threats with 4 campaigns.

Querying Sentinel Logs

We can see that 3oldpollinglogic 0 two messages are missing.

Sentinel Invocation Logs

Testing cases

Testing New Polling Method

Set ABNORMAL_ENABLE_V2_LOGIC=1 in environment variables

Tested new polling method for threats by sending 75 threats with 4 campaigns.

Querying Sentinel Logs

Sentinel Invocation Logs

Testing cases

Checked that the validations are passing and have addressed any issues that are present:

• This is not applicable?

Guidelines

# Guidance <- remove section before submitting ----------------------------------------------------------------------------------------------------------- ## Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:

Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR. Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process. Providing associated Issues that this resolves also easily connects the reason.

Change(s):

• Updated syntax for XYZ.yaml

Reason for Change(s):

• New schema used for XYZ.yaml
• Resolves ISSUE Add Logstash required version to Readme file #1234

Version updated:

• Yes
• Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly. If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

• Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

• Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

• Guidance for Detection checks
• General contribution guidance
• PR validation troubleshooting

---

[rednithin]

PS: Was able to package it correctly. Can ignore this comment

@v-atulyadav @v-prasadboke what commands should I run to package these changes. I added a new dependency of pydantic.

I used the following to package.

cd Solutions/AbnormalSecurity/Data\ Connectors

python3.8 -m venv .python_packages

pip3 install -r requirements.txt

# In .python_packages/lib folder execute
ln -s python3.8/site-packages site-packages

# source .python_packages
zip -r AbnormalSecurityConn.zip SentinelFunctionsOrchestrator .python_packages/lib/site-packages requirements.txt

But I am getting this error

Is it because the .so file generated is for macos? Should I spin up a linux environment in docker and try again?

[rednithin]

@v-atulyadav @v-prasadboke Have I packaged it correctly? There was a recent upgrade done from Python 3.8 to 3.11

So for packaging of zip file I used the package.sh script that I wrote, where I package only for 3.11. Is that fine?

[rednithin]

@v-prasadboke @v-atulyadav

There was a recent PR by Microsoft that merged an upgrade from Python 3.8 to Python 3.11.

• Wanted to know if all the customers using the AbnormalSecurity DataConnector are on Python 3.11 currently?
  • I have tested the logic using this packaging script (Python 3.11 only) that I wrote.
• If there are customers still using Python 3.8, how do I go about packaging the dependencies in the zip file?
• Also do we need to package for all the Python versions here?