MITRE Engenuity conducted its first ever ATT&CK Evaluations for Managed Services this summer. ATT&CK Evals, as they’re commonly called, have been around for a few years, having conducted three enterprise evaluations against various vendors’ endpoint detection and response (EDR) products and one industrial control system (ICS) evaluation testing ICS security vendors.
In past enterprise evaluations, each vendor relied on its own EDR sensor to gather telemetry and generate alerts. The managed services test evaluated 15 managed detection and response (MDR) vendors and managed security service providers (MSSP) on their products’ ability to operate a detection service on either their own proprietary tooling or by relying in part or entirely on third-party tooling.
We were proud to take part in this evaluation alongside some of the best security products in the business. There’s certainly some nuance in the ways that different vendors presented what they observed and detected during the evaluation, but the good news for the whole industry is this: everyone seems to have successfully detected a simulation of this adversary.
In this blog, we’re going to empty our notebook and walk you through what we detected, why, and how we responded to MITRE Engenuity’s emulation activity. We’ll try to cover everything, but there was a lot going on all at once.
Background on the test
We’re going to spend most of this blog explaining how our intel analysts, detection engineers, threat researchers, and incident handlers reacted to the emulation, and showing you what this looked like from the perspective of the customer, in this case MITRE. However, before we get to the fun stuff, we need to take some time to set the stage.
What did MITRE do?
Generally speaking, the Managed Services ATT&CK eval was a closed book test for which we instrumented a specially configured environment (more on this in a moment) for the MITRE Engenuity team. The MITRE folks then ran a 10-step adversary emulation, testing our ability to observe, detect, and respond to the threats they were emulating. This test was fundamentally different from MITRE’s previous Enterprise and ICS Evaluations because MITRE did not announce the adversary they would emulate nor the techniques they would leverage in advance.
MITRE Engenuity was basically a Red Canary customer for one week. They ran a red team engagement against Red Canary in an environment that was intentionally configured to allow the emulation to run unencumbered. The point, after all, was to test detection, not prevention.
In this case, the MITRE Engenuity Team chose to emulate a threat actor commonly known as OilRig. OilRig is a so-called advanced persistent threat (APT), thought to be working in the interest (or at the behest) of the Iranian government. The group is well known for a series of attacks targeting public, private, and civil society organizations in Saudi Arabia, but OilRig remains active and has been accused of targeting a wide array of organizations in Gulf States and more broadly around the Middle East.
In other words, our friends at MITRE pretended to be our foes at OilRig.
Due to the test’s requirements, all prevention technology and features provided by our EDR partners were turned off. Red Canary’s Automate feature was enabled and set to report the remediation actions it would have taken based on threats published during the evaluation.
So… how’d we do?!
This blog post makes no effort to qualitatively assess Red Canary or anyone else’s performance in MITRE’s Managed Services Evaluation. Instead, we want to show you everything we observed and detected, independent from MITRE’s assessment of our performance. Our goal is to tell the whole story of MITRE’s emulation like we would for any other adversary, as if we’re writing a post-incident report. In order to do that, though, we need to provide some quantitative information about our performance in the test.
The evaluation consisted of 10 steps, each of which included multiple substeps. In all, we detected 82.9 percent of the substeps. We could nitpick endlessly about what counted as a detection, why certain substeps might not warrant detecting at all, and whether all of the substeps should’ve been weighted equally, but that’s really beside the point. In the spirit of transparency, we’re going to highlight the things we missed right alongside the things we detected. While most of the misses were discovery techniques that might not warrant a detection under normal circumstances, we admittedly and undeniably missed one important substep.
If you’re interested in learning how to interpret the results of the test, you’ll want to read our executive briefing.
Dissecting MITRE’s Managed Service Evaluations
We’ll start with a high-level overview of everything that happened—mapped to MITRE’s official 10-step evaluation—before examining everything we observed and detected in depth.
Click on each step to navigate to a more detailed description of what we saw for each event.
Step 1: A user downloaded a malicious document on a workstation called
10.1.0.5), establishing command and control (C2) and persistence.
Step 2: The adversary performed various discovery commands for situational awareness and to enable follow-on targeting on
Step 3: The adversary conducted credential theft on
theblock and exfiltrated those credentials to an adversary-controlled C2.
Step 4: The adversary downloaded a web shell to
theblock and copied the webshell to an Exchange server,
10.1.0.6), leveraging stolen credentials from step 3.
Step 5: The adversary leveraged the webshell on
waterfalls to execute various discovery commands for situational awareness.
Step 6: The adversary accessed the web shell implanted on
waterfalls to drop and execute credential theft tooling and exfiltrate stolen credentials back to adversary-controlled C2.
Step 7: The adversary moved laterally via port forwarding on
theblock to access
waterfalls via RDP.
Step 8: The adversary performed a pass-the-hash attack to gain a shell in the context of another user, a SQL administrator, on the
waterfalls Exchange server, then used this access to move laterally to a SQL server,
Step 9: The adversary discovered and collected SQL server database backups and exfiltrated them directly over the Exchange Web Services API.
Step 10: The adversary removed any file indicators on
The long version
NOTE: In the following sections, we attempt to describe MITRE’s emulation activity in chronological order from the time we observed each behavior. This analysis mostly aligns with MITRE’s 10 steps (summarized above).
As is the case with most bad things on the internet, it all started with phishing…
On July 13 at 12:10 UTC, a user with the account name “Gosta” logged into a workstation called
theblock.boom.box, clicked a link, and downloaded a ZIP file named
In an actual customer environment, this attack likely ends right here. Microsoft Defender for Endpoint (MDE) recognized that this document was malicious and generated an alert. The following screenshot shows part of the threat timeline that our detection engineers built as they analyzed this event:
Side note: Preventive controls and missed detections
If MDE didn’t block the document outright, then Red Canary Automate would have triggered a playbook to notify the customer, ban the hash associated with the malicious doc, and isolate the endpoint. However, as we noted earlier, the point of this exercise was to test detection capabilities across an entire campaign, and therefore we disabled many preventive and response controls that would’ve otherwise acted on this threat.
MITRE did not give us credit for detecting the Phishing technique in this step. When we communicate information about threats to our customers, we only communicate information about which we are confident. In this case, we hypothesized it was likely a phishing email, but we didn’t have direct evidence so we didn’t speculate in the alert we sent to MITRE. In a real intrusion, we’d work with the customer to gather information necessary to determine without question the initial access vector and respond accordingly.
Execution, Discovery, Defense Evasion
Within minutes of the initial login (12:11 UTC), Gosta had unzipped the file and opened a Microsoft Word document titled
Visual Basic, Obfuscated Files or Information
Gosta closed the document at 12:12 UTC, which initiated a VBA macro. We extracted, decoded, and analyzed the macro, which initiated a barrage of activity that we will detail in the following paragraphs.
Side note: Missing Discovery and Defense Evasion activities
MITRE did not give us credit for detecting mostly discovery-related substeps (and one defense evasion substep) emanating from the macro. We certainly missed some substeps and didn’t effectively communicate about others that we observed in the macro code (which was included in our detection timeline but omitted from this blog due to its length). In the real world, however, making an effort to detect discovery actions at scale will only mire your security team in noise. Catching and acting on the malicious macro is the most important thing.
Ingress Tool Transfer, Masquerading
Importantly, the macro retrieved a remote access trojan (RAT) called
b.doc that was actually a binary named
Side note: Early attribution clues emerge
At this point, analysts on our Intelligence Team assessed that MITRE was plausibly emulating OilRig for the following reasons:
- OilRig has reportedly leveraged a binary called
SystemFailureReporter.exein past campaigns
- The ZIP file is downloaded from
shirinfarhad[.]com, an allusion to a 12th century Persian poem that’s popular in Iran
- We knew that MITRE had emulated well-known APT groups in previous evaluations
While we suspected MITRE might be emulating OilRig, it was not yet confirmed at this point.
The macro also created a configuration file called
update.xml and a scheduled task that would launch the
SystemFailureReporter.exe binary every five minutes once the user Gosta had been idle for 10 minutes.
SystemFailureReporter.exe first ran at 14:05 UTC in the user context of Gosta, who was listed in the enterprise admins security group.
Side note: Automated response playbooks
At 12:48 UTC, roughly 30 minutes after Gosta downloaded a suspicious document and executed the malicious VBA macros stored in it, Red Canary Automate began triggering notification playbooks based on indicators associated with the malicious VBA macro described above. Due to the requirements of the evaluation, these playbooks were intentionally configured only to inform the customer (in this case MITRE) that we had detected a threat. Under normal conditions, these playbooks also would have isolated the endpoint
theblock.boom.box and banned any relevant hashes, including
SystemFailureReporter.exe, long before the critical follow-on activity we’re about to describe. In all likelihood, this would have seriously hindered or entirely stymied the adversary.
Command and Control
Web Protocols, Symmetric Cryptography
At 14:05 UTC,
SystemFailureReporter.exe establishes a connection to the command and control server at
192.168.0.4 over the HTTP protocol on port 443.
Side note: Missing substep!
We failed to report to MITRE that
SystemFailureReporter.exe also made an XOR-encrypted connection to that same IP address.
Too many techniques to list
SystemFailureReporter.exe initiated a sequence of discovery actions beginning at 16:05 UTC and ending at 16:10 UTC. The activity included the following commands:
This command displays the domain and user currently logged on to this host.
This command displays the name of this host.
This command displays the full configuration for all network adapters on this host.
This command displays the user account information for this host’s primary domain.
This command displays the group information for this host’s primary domain.
This command enumerates all members of the Domain Admins group; members of this group have admin rights over the entire domain.
This command enumerates all members of the Exchange Trusted Subsystem group; members of this group have full control over the domain root.
This command checks the policy settings for this host’s primary domain.
This command shows the current user’s account information.
This command enumerates all members of the local administrators group; members of this group have administrative rights for this endpoint.
This command displays all the active TCP and UDP connections on which the host is listening.
This command displays all of the currently running processes.
This command displays the information for currently running services.
This command displays the info for the user Gosta from the domain controller on the current domain.
This command displays the info for the SQL Admins group from the domain controller on the current domain.
This command queries DNS to obtain the network info for the host
The last command,
nslookup WATERFALLS, is a harbinger of what’s to come, but before we get into the next phase of this attack, we need a brief sidebar.
Side note: Detecting discovery actions
We observed the execution of
SystemFailureReporter.exe less than two hours (14:05 UTC) after Gosta opened the malicious document that initiated this incident. Another two hours passed before we started to observe the aggressive discovery actions detailed above (16:05 UTC).
GGMS Overview.doc had slipped past the initial preventive controls and managed to execute its hidden VBA macro, Red Canary would have detected the above discovery actions and initiated response controls to isolate the affected endpoint right then—less than four hours after Gosta logged in and well before any credential theft, lateral movement, and exfiltration activity.
Command and Control
Ingress Tool Transfer
At 18:03 UTC,
SystemFailureReporter.exe ran via a scheduled task and wrote another binary to disk,
Windows Credential Manager
Upon analysis, our intelligence and research teams determined that the
b.exe binary was emulating a known malware strain called VALUEVAULT, all but confirming our suspicion that MITRE was in fact emulating OilRig.
Side note: Identifying OilRig!
FireEye first reported on VALUEVAULT while investigating social media phishing campaigns in 2019, and the malware is now widely associated with OilRig. VALUEVAULT is known to extract credentials from the Windows Vault and create a SQLITE database file in the
AppData\Roaming directory named
fsociety.dat to store the stolen credentials.
This is precisely what happened in the evaluation:
b.exe dumped the windows credentials stored on
theblock.boom.box into a SQLite3 database called
At this point, our Intelligence Team was almost certain that MITRE was emulating OilRig. In a real-world scenario, there would remain some doubt about this attribution because any number of adversaries could have leveraged VALUEVAULT, written credentials to “fsociety.dat, and mimicked other indicators. However, our assessment was made easier by knowing that MITRE would be emulating a well known adversary.
Exfiltration Over C2 Channel
At this point, the adversary exfiltrated
theblock.boom.box to their C2 server (
192.168.0.4) via an HTTP post request.
Side note: We missed this!
This is probably our most glaring omission. We simply did not observe the adversary passing the
fsociety.dat file back to their C2 infrastructure, which was an important part of the intrusion. Many C2 frameworks offer adversaries the ability manipulate packet structure and effectively hide the contents of their communications during exfiltration. We were able to identify requests that likely contained the exfiltration, but complete visibility into network telemetry is a known short-coming of most EDR products. As we do with any misses during testing, we’re considering how we could improve our detection coverage to catch similar behavior in the future. Further, we support the ingestion of Network Security telemetry and alerts from products like firewalls, network detection and response, and DNS records, which would have proven helpful in fully observing this substep.
Command and Control, Lateral Movement
Ingress Tool Transfer, Lateral Tool Transfer
At 19:33 UTC,
contact.aspx to disk. Shortly thereafter (19:43 UTC), it copied that
contact.aspx to another endpoint (technically an Exchange server)
waterfalls.boom.box, which is the same endpoint upon which the adversary performed a DNS lookup earlier as part of their discovery.
This web shell provided the adversary the ability to write, download, and delete files from disk, as well as execute binaries that are present on the infected endpoint. The previous image illustrated telemetry from
theblock.boom.box copying an Active Server Page Extended (
.aspx) file to
waterfalls.boom.box. The following image shows the telemetry associated with that action from the perspective of
Hidden Files and Directories, File Deletion
Meanwhile, back on
theblock.boom.box at 19:48 UTC, the adversary deleted
contact.aspx from the endpoint and leveraged the
attrib +h command to hide the
.aspx file from the typical folder view and from ordinary directory listings on
On July 14 at 17:30 UTC, the adversary then leveraged the web shell to launch the Command Processor (
cmd.exe), which ran as SYSTEM (this has important implications when we get to the Mimikatz activity later). It proceeded to run multiple commands consistent with post-exploitation, which we will highlight in the coming sections. The following graphic shows
w3wp.exe spawning the command shell:
Multiple Discovery techniques
In the space of a few minutes, the adversary used the newly spawned command shell to gather information about system network configuration, enumerate a list of active network connections, and see who had system access on the host via the
whoami commands. The following image shows telemetry associated with the
Command and Control, Credential Access
Ingress Tool Transfer
Leveraging the web shell, the adversary then wrote a binary named
m64.exe to disk at 18:33 UTC on
waterfalls.boom.box. This binary was in fact a renamed version of the credential-stealing tool Mimikatz. As we noted earlier, the command shell that spawned Mimikatz ran with system-level privileges, which granted Mimikatz the privilege level required to perform the actions described in the next section.
Immediately thereafter, the adversary leveraged Mimikatz to open a handle into the Local Security Authority Subsystem Service (
lsass.exe) and used the
sekurlsa::logonPasswords command to extract credentials from it, writing them to a file called
01.txt, which potentially contained hashes for the following users:
Exfiltration Over C2 Channel
The web shell then ran its
HandlePOSTFileDownloadServer() function to download files off disk, very likely in an effort to exfiltrate credentials from
At 18:37, the adversary ran commands intended to delete
Side note: Detecting Mimikatz
We’ve developed detection analytics for different aspects of Mimikatz, at least three of which are on display in the above image. One option involves looking for processes that obtain a handle into
lsass.exe. Alternatively, you can look for module loads of dynamic link libraries like
vaultcli.dll, which you can see in the following image:
A third option is to detect specific command-line parameters that are associated with Mimikatz. In this case, those commands were:
Command and Control
Ingress Tool Transfer
theblock.boom.box on July 15 at 12:06 UTC,
SystemFailureReporter.exe wrote a copy of
plink.exe, the PuTTY command-line interface used for SSH tunneling.
At 12:11 UTC, the adversary leveraged
plink.exe to create an SSH tunnel and make RDP connections to their adversary-controlled machine, attempting to gain direct RDP access on
Lateral Movement, Defense Evasion
Remote Desktop Protocol, Domain Accounts
At 12:13 UTC, the adversary leveraged the stolen Gosta credentials to open an RDP session on
waterfalls.boom.box using the established SSH tunnel via
Persistence, Command and Control
Web Shell, Ingress Tool Transfer
Starting at 12:15 UTC, the adversary leveraged the web shell to write several files to disk on
ps.exe, a renamed version of the Sysinternals tool
mom64.exe, another renamed version of Mimikatz
nt.dat, an email-based C2 tool
The following images show the adversary writing each of these tools to disk.
Side note: Detecting renamed instances of PsExec
As we’ve written about extensively, you can often detect renamed tools by looking for the execution of binaries where the binary’s presented filename is different from the internal filename listed in its binary metadata. In this case, the file was called
ps.exe, but the file’s internal binary name was “PsExec.”
Defense Evasion, Execution, Lateral Movement
Pass the Hash
The adversary then abused Mimikatzʼs pass-the-hash functionality and a previously extracted NTLM hash to switch the user’s context to the user “tous” at 12:18 UTC. Importantly, “Tous” is a member of the SQL Admins group, which the adversary discovered in a previous step.
Windows Command Shell
mom64.exe (aka Mimikatz) opened an interactive command shell, as shown in the following image:
Lateral Tool Transfer, SMB/Windows Admin Shares, Service Execution
At 12:20 UTC, the adversary—now operating in the context of the user “tous”—leveraged PsExec to execute commands on a new host, a SQL server called
endofroad1.boom.box, before moving
nt.dat to the new host.
File and Directory Discovery
Next, the adversary performed discovery of backup files on the SQL server
Side note: We missed this!
We failed to include these actions in our timeline or report them to the team at MITRE. As with all misses, we’re considering how we could better detect similar behaviors in the future.
Collection, Defense Evasion
Local Data Staging, Match Legitimate Name or Location
At 12:23 UTC, the adversary used PsExec to spawn a command shell and create a folder named “Vmware” and a file named
An analysis of
vmware.exe revealed that the binary was actually a renamed instance of the email C2 tool from earlier,
nt.dat. The adversary moved that
nt.dat file into the Vmware folder and renamed it.
Collection, Exfiltration, Defense Evasion
Data From Local System, Data Transfer Size Limits, Exfiltration Over Unencrypted Non-C2 Protocol, Obfuscated Files or Information
At 12:26 UTC, the adversary leveraged
nt.dat (masquerading as
vmware.exe) to exfiltrate a steganographic
.bmp file containing the contents of a SQL database backup. Using credentials for the user Gosta, the adversary sent an email via Microsoft Exchangeʼs API and then removed their custom tooling from disk. In the process, the adversary resized some of the data to evade detection.
Last but certainly not least, the adversary cleaned up their tracks by removing
vmware.exe and the Vmware folder from
endofroad1.boom.box at 12:26 UTC. At 12:31 the adversary removed the following files from
Side note: We missed a pair of removals
We got partial credit on this step for reporting half of the removed files to MITRE. We failed to explicitly report the deletion of the VMware directory and the deletion of
The MITRE ATT&CK Evaluations for Managed Services is a great resource for evaluating different MDR providers and determining how they would help protect your organization in a real-world intrusion. All the participants should be commended for being transparent enough to take part in the evaluation, and the majority will likely be proud of their performance.
One of Red Canary’s core values is authenticity, which is why we were so excited to hear there would be an ATT&CK Evaluation for Managed Services. We welcome readers of the evaluation to dig into the data we provided and ask any questions about how this would translate to their security program.
When evaluating the test results, please remember that a stack ranking of technique coverage is not the best way to interpret the results. Instead, we recommended focusing on the security outcomes that each vendor was able to achieve. The best resource available to discern this information (other than this document, of course) is the archive of reports, emails, and other content that MITRE made available for each participant.