Adversaries have started to abuse service principals in Microsoft’s Entra ID identity platform. As was openly discussed in CISA’s investigation of the Microsoft key-signing breach and another breach caused by the threat group called Midnight Blizzard, the use of service principals was a key mechanism for reading emails in victim organization mailboxes.
After crafting valid tokens with newly obtained stolen signing keys, adversaries were able to read emails from their victims’ mailboxes using an unidentified application. The Microsoft breach was reportedly discovered from a customer’s detection analytic internally dubbed the “Big Yellow Taxi.” While the exact detection logic from “Big Yellow Taxi” is not publicly known at this time, we can glean some of its intent from public descriptions. CISA summarized the breach as follows:
“[The detection logic] analyzes data from a log known as
MailItemsAccessed, which tracks access to Microsoft Exchange Online mailboxes. State was able to access theMailItemsAccessedlog to set up these particular Big Yellow Taxi alerts because it had purchased Microsoft’s government agency-focused G5 license that includes enhanced logging capabilities through a product called Microsoft Purview Audit (Premium)….State Department’s SOC designed custom alerting capabilities based on three years of experience dealing with anomalous access to mailboxes.In particular, State curated log events like the
MailItemsAccesseddata to enumerate all applications accessing mailboxes within its infrastructure, and to trigger alerts for any anomalous events. It also designed a rule to detect deviations in mailbox activity by comparing baseline interactions of applications with Exchange Online. These rules provided detailed information about application IDs touching mailboxes, specific application details, and context about particular mailboxes involved, thereby enhancing State’s ability to pinpoint potential issues quickly.”
MailItemsAccessed is a very high volume record since it tracks users reading mail and accessing specific folders in the mailbox they own or have access to. As this is what users do in their mailboxes everyday, it only makes sense that this is such a high-volume record type from the “Premium” Unified Audit Log. Microsoft has since decided to release Premium logs for all customers, which we applaud because these logs are an invaluable resource for investigation and detection.
We’ve discussed a few of the intricacies of the MailItemsAccessed log in a previous blog post, but we recently discovered a few new and interesting tidbits from this log and how it ties back to service principals.
Quick primer on applications and service principals
Service principals can serve multiple purposes. Too many to exhaustively cover in this post, so instead we’ll briefly cover some of the overarching concepts that are crucial to understand during your hunting engagements.
According to Microsoft’s documentation:
“A service principal is the local representation, or application instance, of a global application object in a specific tenant. An application object is used as a template to create a service principal object in every tenant where the application is used. The service principal object defines what the app can actually do in a specific tenant, who can access the app, and what resources the app can access.”
Therefore, application objects may or may not have been created in the tenant in which a corresponding service principal was created. In the case of service principals and application objects living in different tenants, these are typically referred to as “multi-tenant,” as opposed to “single-tenant” ones that live in the same tenant. An application has a lot of specific, customizable properties that define how it behaves. Specifically, we’ll focus on permissions and high-level attributes that can help uniquely identify the application.
If the application object lives in a separate tenant (known as the “home tenant,” or “application’s owner organization ID”), then the object will be retrieved from that home tenant for the instantiation. This blueprint lives in what is called a “manifest file,” which is a JSON configuration file defining all of the properties of the application object.
When the original application object is created in its home tenant, a GUID is automatically created. This GUID value is unique across all Entra ID tenants, and is referred to by many different names—especially in the logs. Here are just a few of the different ways to refer to an application’s GUID:
- Client ID
- Client Application ID
- App ID
- Application ID
Applications can configure two types of permissions, which service principals then inherit during their creation. These types are delegated permissions (commonly referred to as “scopes”) and application permissions (known as “app roles”). At a high level, delegated permissions allow a service principal to take action against a single identity. It’s important to note that delegated permissions cannot supersede an identity’s current privilege level, which prevents privilege escalation. However, by design, application permissions pose more risk since they typically allow for actions to be taken against all users of a tenant.
A delegated permission is one that is exposed by a resource. For example, a common resource would be Microsoft Graph. Resources only can understand the specific scopes that they expose. For example, Microsoft Graph exposes a delegated permission called Mail.Read that allows applications to read a specific user’s mail on that user’s behalf. Alternatively, it also exposes an application permission called Mail.Read, which permits an application to read all users’ mail.
Each of these permissions are defined by an underlying GUID. For example, the Mail.Read scope has the unique value of 570282fd-fa5c-430d-a7fd-fc8dc98a9dca. The Mail.Read app role has a unique value of 810c84a8-4a9e-49e6-bf7d-12d183f40d01.
MailItemsAccessed records
The format of a MailItemsAccessed record can vary depending on the scenario. There are two slightly different logs with corresponding formats for bind and sync events. The big difference between the two is that bind events typically record individual mail items being accessed, whereas sync events are more generalized in their content and make the assumption that all records of an entire folder are accessed and are generally indicative of a desktop application being used.
Although Microsoft’s documentation states that sync events are indicative of a desktop application, we’ve personally seen instances where the macOS version of Microsoft Outlook only performed bind operations–contradicting that statement. We recommend you familiarize yourself with some of the other nuances between the sync and bind events in Microsoft’s documentation.
bind operation
The following log is a sample event in which a service principal accessed mail from Microsoft Graph via application permissions such as Mail.Read. It is worth keeping an eye out for the values of the UserId and the MailboxOwnerUPN fields in this scenario. If another service principal reads mail on your behalf, you’d expect to find evidence of this in the logs, perhaps in the UserId field, but that is not always the case.
{
"CreationTime": "2024-01-01T00:00:00",
"Id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"Operation": "MailItemsAccessed",
"OrganizationId": "XXXXXXXX",
"RecordType": 50,
"ResultStatus": "Succeeded",
"UserKey": "1003111114C7B4B1",
"UserType": 0,
"Version": 1,
"Workload": "Exchange",
"UserId": "test@example.com",
"AppId": "00000003-0000-0000-c000-000000000000",
"ClientAppId": "3aa7b255-50b1-4445-91a9-c25a07779b4d",
"ClientIPAddress": "40.126.23.162",
"ClientInfoString": "Client=REST;Client=RESTSystem;;",
"ExternalAccess": false,
"InternalLogonType": 0,
"LogonType": 0,
"LogonUserSid": "S-1-5-21-XXXXXX",
"MailboxGuid": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"MailboxOwnerSid": "S-1-5-21-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX",
"MailboxOwnerUPN": "test@example.com",
"OperationProperties": [
{
"Name": "MailAccessType",
"Value": "Bind"
},
{
"Name": "IsThrottled",
"Value": "False"
}
],
"OrganizationName": "example.com",
"OriginatingServer": "DM1P220MB0612 (15.20.4200.000)",
"Folders": [
{
"FolderItems": [
{
"ClientRequestId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"Id": "XXXXXXXX",
"InternetMessageId": "XXXXXXXXX@example.com>",
"SizeInBytes": 229740
},
{
"ClientRequestId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"Id": "XXXXXXXX",
"InternetMessageId": "XXXXXXXXX@example.com>",
"SizeInBytes": 126560
}
"Id": "LgACT2ijox64k1qaankQWrr+7AQABk+relByrRYnSvu/GVieFAAAAAAEMAAAB",
"Path": "\\Inbox"
}
]
}
AppId
The AppId field, in this case specifically, is the resource or API in which our client retrieved mail. In this case, that AppId is 00000003-0000-0000-c000-000000000000, which is the global application object GUID for Microsoft Graph.
ClientAppId
The ClientAppId is the service principal’s application ID—not its local tenant object ID. Generally when we see a ClientAppID, it is from a legitimate first-party Microsoft application.
An easy way to collect additional information about an application could be done through the Entra ID portal, or through the Graph API. An example method is using the Microsoft Graph PowerShell SDK as following:
Connect-MgGraph -scopes application.read.all
Get-MgServicePrincipalByAppId -AppId <CLIENT_APP_ID>ClientIPAddress
One of our most interesting discoveries is that the value of the ClientIPAddress field varies depending on other conditions within the log entry.
- If
ClientAppIdis present, theClientIPAddressis always a Microsoft-owned IP address. - If
ClientAppIdis not present, theClientIPAddressbelongs to the user.
According to the schema documentation for ClientIP, this is expected activity but there is no mention of this for the ClientIPAddress field. The documentation mentions when “trusted” applications are observed the IP will be from the source of the request. But, even when an unverified or third-party application is reading mail, the IP address will always be from a Microsoft-owned IP space regardless of where the service principal was being called from.
As stated in the documentation:
“For some services, the value displayed in this property might be the IP address for a trusted application (for example, Office on the web apps) calling into the service on behalf of a user and not the IP address of the device used by person who performed the activity.”
sync operation
Below is a sample record of a Windows desktop Outlook application retrieving mail via a sync access type:
{
"CreationTime": "2024-01-01T00:00:00",
"Id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"Operation": "MailItemsAccessed",
"OrganizationId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"RecordType": 2,
"ResultStatus": "Succeeded",
"UserKey": "1003111114C7B4B1",
"UserType": 0,
"Version": 1,
"Workload": "Exchange",
"ClientIP": "X.X.X.X",
"UserId": "test@example.com",
"AppId": "d3590ed6-52b3-4102-aeff-aad2292ab01c",
"ClientIPAddress": "X.X.X.X",
"ClientInfoString": "Client=MSExchangeRPC",
"ClientProcessName": "OUTLOOK.EXE",
"ClientRequestId": "{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX}",
"ClientVersion": "16.0.425.2024",
"ExternalAccess": false,
"InternalLogonType": 0,
"LogonType": 0,
"LogonUserSid": "S-1-5-21-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX",
"MailboxGuid": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"MailboxOwnerSid": "S-1-5-21-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX",
"MailboxOwnerUPN": "test@example.com",
"OperationProperties": [
{
"Name": "MailAccessType",
"Value": "Sync"
},
{
"Name": "IsThrottled",
"Value": "False"
}
],
"OrganizationName": "example.com",
"OriginatingServer": "DM1PR13MB1111 (15.00.0000.000)",
"SessionId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
"Item": {
"Id": "LgACT1ijOx61k1qaankQWrr+7AQABk+relByrRYnSvu/GVieFAAAAEMAAAB",
"ParentFolder": {
"Id": "LgACT1ijOx61k1qaankQWrr+7AQABk+relByrRYnSvu/GVieFAAAAEMAAAB",
"Name": "Inbox",
"Path": "Not Available"
}
}
}
AppId
In our data sets, when it exists, AppId is almost always d3590ed6-52b3-4102-aeff-aad2292ab01c, indicating an application of Microsoft Office. However, an AppId is non-existent for sync access types events in the vast majority of our records.
ClientAppId
The ClientAppId is never present, which leads us to believe that sync operations via service principals is not a valid way to read mail, but we have not confirmed this.
ClientIP, ClientIPAddress
Notice that there are two IP address fields. The ClientIPAddress field was observed previously, but a new one called ClientIP is now present. We have yet to determine why there may be two addresses here as they appear to always be the same.
Interesting shared fields between bind and sync
RecordType
MailItemsAccessed records may be identified by two RecordType values to help you easily discern between log types:
- 50 (
bind) – Events related to theMailItemsAccessedmailbox auditing action - 2 (
sync) – Events from an Exchange mailbox audit log for actions that are performed on a single item, such as creating or receiving an email message
ClientInfoString
Information about the email client that was used to perform the operation. This field can be useful in determining what API was used to access mail and how it was accessed. In some cases, it may also include the raw user agent from a request when this field includes OWA, WebServices, or OutlookService.
An example of someone reading mail via the EWS API, we would see the following in this field:
client=webservices;microsoft office/16.0 (windows nt 10.0; microsoft outlook 16.0.17425; pro);Based on our data, the possible client values for a MailItemsAccessed record can contain one of the following:
RESTRESTSystemOutlookServiceMSExchangeRPCActiveSyncOWAWebServicesPOP3/IMAP4MSExchangeRPCMessageServiceCoreStoreObjects
OperationProperties
Denotes whether the operation is a sync or bind operation as well as if the log was throttled.
MailboxOwnerUPN
The user principal name (UPN) of the owner of the mailbox that was accessed.
MailboxGuid
The Entra ID object ID of the user who owns the mailbox.
UserKey
An interesting identifier of a user, which denotes the identity’s passport unique ID (PUID). It’s currently unknown why the UserKey is a PUID value instead of the user’s Entra ID object ID. We are currently unable to fully understand how to resolve a user’s PUID back to their original object ID or UPN.
UserId
The user performing the action, or the identity on behalf of which a service principal is behaving.
SessionId
A valuable field that can help correlate multiple different record types together. In addition to bind and sync events, when using SessionId for correlation on Unified Audit Log (UAL) logs, it can also help pull in additional record types such as:
- 1 (
ExchangeAdmin) – Events from the Exchange admin audit log - 3 (
ExchangeItemGroup) – Events from an Exchange mailbox audit log for actions that can be performed on multiple items, such as moving or deleting one or more email messages - 15 (
AzureActiveDirectoryStsLogon) – Secure Token Service (STS) logon events in Microsoft Entra ID
NOTE: For RecordType 15 (AzureActiveDirectoryStsLogon), the SessionId is not as easily accessible, it is under the DeviceProperties object that may need additional parsing. Additionally, session ID may not be present in some logs—almost 50 percent of the time in our data sets. We have not been able to confidently confirm the reasons for this behavior.
Real-world BEC example, with a twist
Up until this point we’ve been looking at example logs simulated in a lab environment. When looking at actual real-world attacks, the logs are quite different from what we expect.
We’ve investigated multiple business email compromise (BEC) cases involving third-party, multi-tenant applications. What this means is that rather than relying on some existing first-party application like Outlook on the web (OWA), or an Outlook desktop client to access a victim’s mailbox, the adversary leverages a third-party one. In order to do so, the new application requires authorization via a consent process, requiring a user to specifically permit access to their data.
We’ve seen two specific applications more often than others in malicious instances:
- PerfectData Software (
ff8d92dc-3d82-41d6-bcbd-b9174d163620) - eM Client (
e9a7fea1-1cc0-4cd9-a31b-9137ca5deedd)
An example of what this “consent” dialog looks like:
When either one of these applications are used, we almost always see a large spike in the number of MailItemsAccessed records from abnormal IP addresses for the victim’s account. This is due to the fact that upon the first initial startup of either application, it needs to pull down a copy of the user’s mailbox for offline viewing, which causes a large spike in activity.
We nearly always see the following pattern of behavior wherein:
It’s worth mentioning again that these applications are legitimate and verified. We have identified that a large majority of eM Client usage we see is legitimate, since it is an actual email client not too different from a desktop version of Microsoft Outlook. Therefore, trying to identify malicious presence of eM Client can be a difficult task if defenders are not wary of usage in their environments. On the other hand, usage of PerfectData Software is extremely rare to see used legitimately, making detection a much easier task against our own customer base.
The twist
When a service principal accesses mail, you can typically identify that with the ClientAppId field. However, during our investigations and manual testing through various APIs, we identified that there are situations in which a new service principal accesses mail and the ClientAppId field does not exist.
Why do these real-world attacks not have a ClientAppId associated in the logs? What we’ve found is that it is much easier to make sense of the logs when a service principal is accessing mail through Microsoft Graph (when the AppId field is 00000003-0000-0000-c000-000000000000). In this case, the ClientAppId field is visible 100 percent of the time and includes our new application ID.
There are several APIs besides Microsoft Graph that commonly access mail, like the popular Exchange Web Services (EWS) via outlook.office365.com, which is what we confirmed eM Client uses to read mail. When an API besides Microsoft Graph is used, it’s a bit unclear at the moment whether the ClientAppID can be a reliable field for detection purposes. For the most part, when an application does not read mail via Microsoft Graph, then the AppId field is the one to monitor.
Confusing as this may be, we urge defenders to maintain a list of what AppId values are present in their logs and alert on any deviations from it. Whether the suspicious application ID lives in either AppId or ClientAppId, it’s a good idea to monitor the values from both fields for more comprehensive detection visibility.
Detection
Detecting this scenario requires baselining the service principals with the ability to access mail that already exists in your environment. The ability to access mail can manifest itself in several different permissions that may be exposed by Microsoft Graph or other applications like Exchange Online.
In an Entra ID tenant, we can detect illegitimate service principals with multiple log sources. This depends on the license level you have and assumes you have the log sources in a queryable state such as a Log Analytics Workspace or a SIEM. On top of the UAL, you may have multiple other log sources that contain the data needed for an analytic. The data sources in which we’ve identified the presence of MailItemsAccessed events include the following:
OfficeActivity– Audit logs for Office 365 tenants collected by Azure SentinelCloudAppEvents– Information about activities in various cloud apps and services covered by Microsoft Cloud App Security
Below is an example KQL query against the OfficeActivity table that collects all application IDs via the AppID and ClientAppId fields observed reading mail. It then performs filtering based on some of the most common first-party applications we’ve seen in our own customer environments. (Therefore, you may need to adjust the query to account for your own environment’s baseline.) The query will then try to resolve the service principal’s name via any matching interactive sign-in logs, as well as any service principal sign-in logs that indicate dangerous application permission usage:
// Establish our allow list of apps commonly reading mail
let allow_list = dynamic(["6326e366-9d6d-4c70-b22a-34c7ea72d73d", "27922004-5251-4030-b22d-91ecd9a37ea4", "d3590ed6-52b3-4102-aeff-aad2292ab01c", "00000002-0000-0ff1-ce00-000000000000", "00000002-0000-0ff1-ce00-000000000000", "13937bba-652e-4c46-b222-3003f4d1ff97", "00000007-0000-0000-c000-000000000000", "47629505-c2b6-4a80-adb1-9b3a3d233b7b", "91ca2ca5-3b3e-41dd-ab65-809fa3dffffa", "dfe74da8-9279-44ec-8fb2-2aed9e1c73d0", "00b41c95-dab0-4487-9791-b9d2c32c80f2", "a3883eba-fbe9-48bd-9ed3-dca3e0e84250", "82d8ab62-be52-a567-14ea-1616c4ee06c4"]);
// Look for all AppId's reading mail
let Apps = OfficeActivity
| where (RecordType == 2 or RecordType == 50)
| extend app = AppId;
// Look for all ClientAppid's reading mail
let ClientApps = OfficeActivity
| where (RecordType == 2 or RecordType == 50)
| extend app = ClientAppId;
// Clear out any common first party apps, typically seen as false positives or normal behavior
// Additional App ID's can be found for validation outside the allow_list which is not 100% complete
// https://learn.microsoft.com/en-us/troubleshoot/azure/entra/entra-id/governance/verify-first-party-apps-sign-in
// https://github.com/emilyvanputten/Microsoft-Owned-Enterprise-Applications/blob/main/Microsoft%20Owned%20Enterprise%20Applications%20Overview.md
// https://github.com/merill/microsoft-info/blob/main/customdata/OtherMicrosoftApps.csv
let new_apps_reading_mail = union ClientApps, Apps
| where app !in (allow_list)
| distinct app;
// Attempt to resolve the Application Name from Signin logs
// Resolve against ResourceDisplayName if the App Id reading mail may be a first party app now in allow list
let resolved_app_names = SigninLogs
| where ResultType == 0 and ( AppId in (new_apps_reading_mail) or ResourceId in (new_apps_reading_mail) )
| extend app_name = iif(AppId in (new_apps_reading_mail), strcat(AppDisplayName, " -> ", AppId), "n/a")
| extend resource_app = iif(ResourceId in (new_apps_reading_mail), strcat(ResourceDisplayName, " -> " , ResourceId), "n/a")
| where app_name != "n/a" or resource_app != "n/a";
let service_principal_sign_ins = AADServicePrincipalSignInLogs
| where ResultType == 0 and ( AppId in (new_apps_reading_mail) )
| extend app_name = iif(AppId in (new_apps_reading_mail), strcat(ServicePrincipalName, " -> ", AppId, " SERVICE PRINCIPAL SIGN IN DIRECTLY"), "n/a")
| where app_name != "n/a";
union service_principal_sign_ins, resolved_app_names
| distinct app_name
Conclusion
If you find a nefarious service principal with this hunting methodology, the next step would generally be to investigate when it was created, what user account provided consent, and what permissions are associated with it. This may be a difficult topic to grasp if you are new to investigating this sort of attack. Luckily, Microsoft provides some great resources that take a deeper dive into defensive and reactive situations to help guide you, which we highly recommend you check out before such an incident occurs.
- Compromised and malicious applications investigation
- Microsoft Entra security operations guide for applications
- Compromised and malicious applications investigation
Even though MailItemsAccessed records may pose a challenge due to their high volume, we believe their benefits far surpass any drawbacks. As we’ve shown, they can be a good source for detecting malicious activity and also reveal what adversaries may access during a BEC attack.