Microsoft Sentinel — Testing detection rules like a Ninja

If you are into DevOps, automation, and security like me, the chance is big that you will love this!

Introduction

Microsoft Sentinel is the SIEM / SOAR solution in Azure used to create alerts, and incidents based on the information ingested in a centralized logging solution called a Log Analytics workspace.

One of the cool things about Microsoft Sentinel is that it’s a very versatile product that can be fully configured to your own wishes when it comes to detection and alerts.

A short while ago I had an interesting discussion with a very large enterprise customer about the possibilities to validate detection rules before they are published to a Microsoft Sentinel production environment.

This customer is using GitHub as their primary repository solution, I was thinking of a solution to create a GitHub Action to validate detection rules before they are committed to the master/main branch.

Detections love Pester

So when thinking about an easy-to-use code testing solution, the first thing that comes to my mind is Pester. But wait, isn’t Pester a testing framework solution for PowerShell modules?

Well yeah, but you can also use it for testing the code quality, and that is exactly what I did.

What I love about Pester is that if you understand PowerShell code, you can also write pester tests, as the language is pretty similar. So the only thing I had to do is write test cases for Microsoft Sentinel detection rules.

Advanced test cases

Most of the test cases added to the solution are pretty straightforward where checks are being done if the correct casing is applied. (yes some properties are case-sensitive) And various tests are executed to validate if the correct value is provided in the entity mappings etc.

The more advanced test cases, which also took most of the development time, are to validate if the provided tactics match the correct techniques and visa-versa from the Mitre Att&ck framework. This was especially difficult because tactics can relate to multiple techniques. But one technique can also cover multiple tactics.

Other tests are using RegEx expressions to validate if a correct time format in ISO8601 duration is configured in the template, but also check if the difference between Query Period and Query Frequency is correct.

A GitHub action was born!

After getting all the logic sorted a GitHub action was born and is now available for free on the GitHub marketplace.

To use this GitHub action in your own repository is very simple.
Check the guide on the Github Marketplace or in my GitHub repository

It is also possible to use the solution as a GitHub workflow. Just clone the repository and add the code to a folder in your repository that also has the detection rules.

Results

The GitHub action works for both detection rules in ARM templates as for the YAML format as provided through the GitHub page of Microsoft Sentinel

When running the action it will go through all detection rules in the provided files path and run all tests against each individual file at the speed of lightning!

Once the test is completed a summary of all failed tests will be shown in the output.

As you can see in the error message above, the test noticed that a technique [1087] was provided that doesn’t belong to the tactic [CommandAndControl] and expects, based on the technique, that the tactic [Discovery] is added to the tactics array in the template file.

Tested against the Microsoft Sentinel Repository

To see how this solution would work in an environment with a very large code base for detection rules, I’ve tested it against the Microsoft Sentinel GitHub which contains over 10,000 detection rules. The result:

In 60 seconds 10220 tests were executed and 121 tests didn’t pass one or more test cases!

Unfortunately, the GitHub Actions summary page has a built-in limit of 10 annotation messages. But luckily when you open the output by clicking on one of the outputs you will be taken to the full test report.

You can also change the verbosity of the GitHub Action to enforce that only the results are shown of tests that didn’t pass. See the documentation.

Happy coding!