Microsoft Sentinel Threat Intelligence APIs The Good, The Bad and the Ugly
In this article I will describe the different API endpoints that are available to send threat intelligence information to Microsoft Sentinel.
To be honest I was quiet a bit frustrated after picking up a question on how to how to ingest threat intelligence information in Sentinel. At first I thought this would be straightforward as everything in Microsoft Azure happens through well documented APIs
No a couple of days later I am writing this article while listening to the Rage Against the Machine song F* you, I wont do what you tell me. How appropriate.
Using the Microsoft Graph API (the good)
The Microsoft Graph API is one of my favorite Microsoft API’s to work with. They are in general well documented and uniform is usage. Claps for the team.
So when looking in to the possibilities on ingesting threat intelligence data into a Microsoft Security product, I wasn’t surprised that there was a Graph API endpoint supporting this.
Endpoints
Like all the Microsoft Graph APIs, the base Uri is https://graph.microsoft.com/beta/
For submitting threat indicators the graph has 2 different endpoints that can be used:
POST /security/tiIndicators
POST /security/tiIndicators/submitTiIndicatorsRequired Permissions
To interact with the Microsoft Graph API, the security principal sending the requests need to have the correct permissions. In most cases these permissions are given to an App Registration, but can also be assigned to a user object. The permissions required for submitting threat indicators are:
#Delegated (work or school account)
ThreatIndicators.ReadWrite.OwnedBy
#Application
ThreatIndicators.ReadWrite.OwnedByThis graph endpoint can be used for both Microsoft Sentinel and Microsoft Defender for Endpoint by providing the correct targetProduct: in the JSON payload.
#Microsoft Sentinel
{
"targetProduct": "Azure Sentinel"
},
...
# Defender for Endpoint
{
"targetProduct": "Microsoft Defender ATP"
},
...Limitations
There are some limitations that are needed to take into account:
- There’s a limit of 15,000 indicators per tenant for MDE
- There’s a limit of 100
tiIndicatorsper request
Payload
The Request body can contain many different properties, but this depends on the targetProduct specified in the payload.
The following example shows only the required properties for a Microsoft Sentinel threat indicator.
{
"action": "alert",
"azureTenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"description": "This is a sample indicator",
"expirationDateTime": "2023-12-31T21:00:00.0000000+00:00",
"targetProduct": "Azure Sentinel",
"threatType": "MaliciousUrl",
"tlpLevel": "green",
"url": "http://31.191.140.228:47935/.i"
}When submitting the data using PowerShell, the API will return the result of the submitted indicator as shown in the image. The id shown at the top looks like a random value, but actually isn’t.
The unique identifier shown in the id field is based on the threatType and the url in this case. When changing any of the other parameters the Id will stay the same, and will only update the existing indicator.
more information about the available parameters can be found here
Pulling my hairs out!
Everything went flawlessly until this far, and the API acted as expected and created a threat indicator for Microsoft Sentinel Right?!
That is what I thought after doing a GET command and the API showed the expected result.
So why are the indicators not visible in my Microsoft Sentinel instance!
After doing some digging and pulling out some more hairs I ended up looking at the data connectors in Microsoft Sentinel. I did enable the Threat Intelligence Upload Indicators API (Preview) data connector as the other connector for threat intelligence is deprecated. It even says in the description that it should not be used anymore.
It ends up that if you are using the Microsoft Graph endpoint for uploading threat indicators, the DEPRECATED data connector needs to be enabled to see the uploaded results.
After enabling the data connector, the previous created indicators are visible.
The Log Analytics Workspace API (the bad)
If the idea was to have a redundant way of submitting threat indicators to Microsoft Sentinel, they have done a good job. The next API in this article is the second in the line of 3 APIs that are available.
I believe it is always good to stay close to the source when it comes to automating things. So when looking into submitting threat indicators to Microsoft Sentinel, it is logical to have an API available at the destination resource Log Analytics.
Disappointment came upon me when interacting with this API. No that it didn’t work at all, but mainly because from a scalability perspective this API doesn’t seem to fit the job. Because the whole idea was to collect threat indicators from open source databases and send them to Microsoft Sentinel, now being able to send a payload containing multiple indicators is really a bummer. This API just doesn’t fit it needs.
Every indicator needs to be sent separately causing a huge delay when trying to sent up to 10,000 indicators.
I won’t spent to much time on this API, but here is how it works.
Endpoints
The API endpoint for submitting the threat indicators is hosted under the resourceId of the Log Analytics workspace. The full Uri looks something like this: {workspaceResourceId}/providers/Microsoft.SecurityInsights/threatIntelligence/main/indicators/{name}?api-version=2023-07-01-preview
# relative path
POST /Microsoft.SecurityInsights/threatIntelligence/main/indicatorsRequired Permissions
To interact with this API the following permissions needs to be assigned to the security principal of App Registration:
# required permissions
Microsoft Sentinel ContributorLimitations
- Only one indicator per request
Payload
The documentation of this API is very sparce and doesn’t clearly describe the parameters that are required in the payload. After some testing I was able to determine the minimum set of parameters.
{
"kind": "indicator",
"properties": {
"source": "Abuse CH",
"pattern": "[url:value = 'https://www.pasteio.com/raw/XqtmHsf45ys']",
"patternType": "url"
}
}Most parameters supported in the payload came directly from the STIX format.
Required data connector
To be able to see the threat indicators in Microsoft Sentinel the following data connector is required: Threat Intelligence Upload Indicators API (Preview)
This API can be useful if only a few threat indicators needs to be submitted to the Microsoft Sentinel workspace. The lack of documentation and missing ability to submit indicators in bulk makes it my least favorite API for this purpose.
Threat Intelligence Upload Indicators API (the ugly)
This is the latest, but not yet greatest, API for uploading threat indicators to Microsoft Sentinel. The good thing is that it is filling the gaps in the previous two APIs. Meaning that this brand new API with ugly Uri can be used to upload indicators in bulk (like the Graph) and correctly displays the source and name of the indicator, (like the Workspace API)
So what is wrong with it?
Well stability in the first place to be point blank. Although the API is still in preview, I would expect it to have proper error handling and predictability.
In the current state it frequently has issues with authenticating correctly using a valid JWT token (bearer token)
Another issue is that there is no clear description on the minimum required permissions to be able to send threat indicators to the API.
update:
I opened a request at Microsoft that was picked-up very quickly by the team and directed me to the required permissions. obviously the user or Service Principal required the Microsoft Sentinel Contributor permissions.
More information can be found on this page.
Is it all that bad?
No, not really, the API also has some very nice stuff that should make you happy.
What I do like about this API is the support for the STIX 2.1 format.
STIX stands for Structured Threat Information Expression and is a standard language for describing cyber threat intelligence in a way that both humans and machines can understand.
STIX describes cyber threats using an extensive set of properties, which include signs of malicious activity (e.g., suspect file hashes, domains, etc.) in addition to contextual information (e.g., adversary tactics, techniques, and procedures)
Endpoints
The endpoint for submitting threat intelligence is not one that is easy to remember. The complete Uri is https://sentinelus.azure-api.net/{workspaceId}/threatintelligence:upload-indicators?api-version=2022-07-01
This is why I defined this API as the ugly
# relative path
POST /threatintelligence:upload-indicators?api-version=2022-07-01Required Permissions
To interact with this API the following permissions needs to be assigned to the security principal or App Registration:
Microsoft Sentinel ContributorLimitations
This API also has some limitation to take into account. These limits are on a per user bases
- 100 indicators per request
- 100 request per minute.
After quickly firing up my calculator this means we are able to send 10,000 indicators per minute. That isn’t to bad in my opinion.
Payload
As explained before, this API supports the STIX format for sending threat indicators, and as we could see in the limitations section it is also possible to send an array of maximum 100 indicators per request.
Because we are dealing with an API this means we need to send the payload in JSON format.
In the first example below we are only sending the required parameters:
{
"sourcesystem": "Feodo Tracker",
"value":[
{
"type": "indicator",
"spec_version": "2.1",
"id": "indicator--f07f2a54-82af-44bb-958f-1d7a0accbcab",
"pattern": "[ipv4-addr:value = '89.101.97.139']",
"pattern_type": "stix",
"valid_from": "2023-09-22T18:29:07.778Z"
}
]
}This second example shows a more advanced payload of a threat indicator in STIX format.
{
"sourcesystem": "Abuse CH",
"value":[
{
"type": "indicator",
"spec_version": "2.1",
"id": "indicator--67e62408-e3de-4783-9480-f595d4fdae52",
"created": "2023-09-23T18:29:07.778Z",
"modified": "2023-09-23T18:29:07.778Z",
"revoked": false,
"labels": [
"VoidRAT",
"exe"
],
"confidence": 100,
"external_references": [
{
"source_name": "URL Hause",
"external_id": "2713825",
"url": "https://urlhaus.abuse.ch/url/2713825/"
}
],
"name": "Malware",
"description": "Remote Access Tool",
"indicator_types": [
"threatstream-severity-low"
],
"pattern": "[url:value = 'https://pasteio.com/raw/xQTmhxjoliab",
"pattern_type": "stix",
"pattern_version": "2.1",
"valid_from": "2023-09-25T08:56:09.778Z"
}
]
}When looking at the documentation the properties created and modified are not required. This makes sense because these values should be provisioned by the API during creation of the indicator. Currently when not providing these properties in the payload an error is shown.
Data Connector
To be able to see the threat indicators in Microsoft Sentinel the following data connector is required: Threat Intelligence Upload Indicators API (Preview)
Summarize
The good
Microsoft has a couple of APIs available for submitting threat intelligence indicators to Microsoft Sentinel and other products. Depending on the use case all three of them are useable.
The Microsoft Graph API is most straight forward API to work with. If you have worked with the Microsoft Graph, you will face little challenges submitting the threat indicators. The API is deprecated for Microsoft Sentinel, so keep this in mind when using this method.
The bad
My least favorite API for submitting threat indicators is the Log Analytics Workspace API. The documentation is very limited and the API itself isn’t really usable when submitting large sets of indicators.
I do like the support of some STIX values as parameters in the payload.
The ugly
My favorite for now is the new API both because it supports STIX 2.1 and its ability to process large arrays of threat indicators. Having a limit of 10,000 indicators per minute should be enough in most cases.
Stability is still an issue to my opinion. Especially because sometimes it shows an authentication error, but when executing the same call again it suddenly works.
The documentation also still needs some work on the missing required parameters. Although I believe that the missing parameters in the documentation are values that should be handled by the API and not submitted by the user.
The API is still in preview but is developing in the right direction.
