MITRE Engenuity conducted its first ever ATT&CK Evaluations for Managed Services this summer. ATT&CK Evals, as they’re commonly called, have been around for a few years, having conducted three enterprise evaluations against various vendors’ endpoint detection and response (EDR) products and one industrial control system (ICS) evaluation testing ICS security vendors.
In past enterprise evaluations, each vendor relied on its own EDR sensor to gather telemetry and generate alerts. The managed services test evaluated 15 managed detection and response (MDR) vendors and managed security service providers (MSSP) on their products’ ability to operate a detection service on either their own proprietary tooling or by relying in part or entirely on third-party tooling.
We were proud to take part in this evaluation alongside some of the best security products in the business. There’s certainly some nuance in the ways that different vendors presented what they observed and detected during the evaluation, but the good news for the whole industry is this: everyone seems to have successfully detected a simulation of this adversary.
In this blog, we’re going to empty our notebook and walk you through what we detected, why, and how we responded to MITRE Engenuity’s emulation activity. We’ll try to cover everything, but there was a lot going on all at once.
Background on the test
We’re going to spend most of this blog explaining how our intel analysts, detection engineers, threat researchers, and incident handlers reacted to the emulation, and showing you what this looked like from the perspective of the customer, in this case MITRE. However, before we get to the fun stuff, we need to take some time to set the stage.
What did MITRE do?
Generally speaking, the Managed Services ATT&CK eval was a closed book test for which we instrumented a specially configured environment (more on this in a moment) for the MITRE Engenuity team. The MITRE folks then ran a 10-step adversary emulation, testing our ability to observe, detect, and respond to the threats they were emulating. This test was fundamentally different from MITRE’s previous Enterprise and ICS Evaluations because MITRE did not announce the adversary they would emulate nor the techniques they would leverage in advance.
MITRE Engenuity was basically a Red Canary customer for one week. They ran a red team engagement against Red Canary in an environment that was intentionally configured to allow the emulation to run unencumbered. The point, after all, was to test detection, not prevention.
In this case, the MITRE Engenuity Team chose to emulate a threat actor commonly known as OilRig. OilRig is a so-called advanced persistent threat (APT), thought to be working in the interest (or at the behest) of the Iranian government. The group is well known for a series of attacks targeting public, private, and civil society organizations in Saudi Arabia, but OilRig remains active and has been accused of targeting a wide array of organizations in Gulf States and more broadly around the Middle East.
In other words, our friends at MITRE pretended to be our foes at OilRig.
Due to the test’s requirements, all prevention technology and features provided by our EDR partners were turned off. Red Canary’s Automate feature was enabled and set to report the remediation actions it would have taken based on threats published during the evaluation.
So… how’d we do?!
This blog post makes no effort to qualitatively assess Red Canary or anyone else’s performance in MITRE’s Managed Services Evaluation. Instead, we want to show you everything we observed and detected, independent from MITRE’s assessment of our performance. Our goal is to tell the whole story of MITRE’s emulation like we would for any other adversary, as if we’re writing a post-incident report. In order to do that, though, we need to provide some quantitative information about our performance in the test.
The evaluation consisted of 10 steps, each of which included multiple substeps. In all, we detected 82.9 percent of the substeps. We could nitpick endlessly about what counted as a detection, why certain substeps might not warrant detecting at all, and whether all of the substeps should’ve been weighted equally, but that’s really beside the point. In the spirit of transparency, we’re going to highlight the things we missed right alongside the things we detected. While most of the misses were discovery techniques that might not warrant a detection under normal circumstances, we admittedly and undeniably missed one important substep.
If you’re interested in learning how to interpret the results of the test, you’ll want to read our executive briefing.
Dissecting MITRE’s Managed Service Evaluations
We’ll start with a high-level overview of everything that happened—mapped to MITRE’s official 10-step evaluation—before examining everything we observed and detected in depth.
TL:DR
Click on each step to navigate to a more detailed description of what we saw for each event.
Step 1: A user downloaded a malicious document on a workstation called theblock
(10.1.0.5
), establishing command and control (C2) and persistence.
Step 2: The adversary performed various discovery commands for situational awareness and to enable follow-on targeting on theblock
.
Step 3: The adversary conducted credential theft on theblock
and exfiltrated those credentials to an adversary-controlled C2.
Step 4: The adversary downloaded a web shell to theblock
and copied the webshell to an Exchange server, waterfalls
(10.1.0.6
), leveraging stolen credentials from step 3.
Step 5: The adversary leveraged the webshell on waterfalls
to execute various discovery commands for situational awareness.
Step 6: The adversary accessed the web shell implanted on waterfalls
to drop and execute credential theft tooling and exfiltrate stolen credentials back to adversary-controlled C2.
Step 7: The adversary moved laterally via port forwarding on theblock
to access waterfalls
via RDP.
Step 8: The adversary performed a pass-the-hash attack to gain a shell in the context of another user, a SQL administrator, on the waterfalls
Exchange server, then used this access to move laterally to a SQL server, endofroad1
.
Step 9: The adversary discovered and collected SQL server database backups and exfiltrated them directly over the Exchange Web Services API.
Step 10: The adversary removed any file indicators on endofroad1
.
The long version
NOTE: In the following sections, we attempt to describe MITRE’s emulation activity in chronological order from the time we observed each behavior. This analysis mostly aligns with MITRE’s 10 steps (summarized above).
Step 1
Initial Access
As is the case with most bad things on the internet, it all started with phishing…
Spearphishing Link
On July 13 at 12:10 UTC, a user with the account name “Gosta” logged into a workstation called theblock.boom.box
, clicked a link, and downloaded a ZIP file named marketing_material.zip
from shirinfarhad[.]com
.
In an actual customer environment, this attack likely ends right here. Microsoft Defender for Endpoint (MDE) recognized that this document was malicious and generated an alert. The following screenshot shows part of the threat timeline that our detection engineers built as they analyzed this event:
Side note: Preventive controls and missed detections
If MDE didn’t block the document outright, then Red Canary Automate would have triggered a playbook to notify the customer, ban the hash associated with the malicious doc, and isolate the endpoint. However, as we noted earlier, the point of this exercise was to test detection capabilities across an entire campaign, and therefore we disabled many preventive and response controls that would’ve otherwise acted on this threat.
MITRE did not give us credit for detecting the Phishing technique in this step. When we communicate information about threats to our customers, we only communicate information about which we are confident. In this case, we hypothesized it was likely a phishing email, but we didn’t have direct evidence so we didn’t speculate in the alert we sent to MITRE. In a real intrusion, we’d work with the customer to gather information necessary to determine without question the initial access vector and respond accordingly.
Execution, Discovery, Defense Evasion
Malicious File
Within minutes of the initial login (12:11 UTC), Gosta had unzipped the file and opened a Microsoft Word document titled GGMS Overview.doc
.
Visual Basic, Obfuscated Files or Information
Gosta closed the document at 12:12 UTC, which initiated a VBA macro. We extracted, decoded, and analyzed the macro, which initiated a barrage of activity that we will detail in the following paragraphs.
Side note: Missing Discovery and Defense Evasion activities
MITRE did not give us credit for detecting mostly discovery-related substeps (and one defense evasion substep) emanating from the macro. We certainly missed some substeps and didn’t effectively communicate about others that we observed in the macro code (which was included in our detection timeline but omitted from this blog due to its length). In the real world, however, making an effort to detect discovery actions at scale will only mire your security team in noise. Catching and acting on the malicious macro is the most important thing.
Ingress Tool Transfer, Masquerading
Importantly, the macro retrieved a remote access trojan (RAT) called b.doc
that was actually a binary named SystemFailureReporter.exe
.
Side note: Early attribution clues emerge
At this point, analysts on our Intelligence Team assessed that MITRE was plausibly emulating OilRig for the following reasons:
- OilRig has reportedly leveraged a binary called
SystemFailureReporter.exe
in past campaigns - The ZIP file is downloaded from
shirinfarhad[.]com
, an allusion to a 12th century Persian poem that’s popular in Iran - We knew that MITRE had emulated well-known APT groups in previous evaluations
While we suspected MITRE might be emulating OilRig, it was not yet confirmed at this point.
Persistence
Scheduled Task
The macro also created a configuration file called update.xml
and a scheduled task that would launch the SystemFailureReporter.exe
binary every five minutes once the user Gosta had been idle for 10 minutes. SystemFailureReporter.exe
first ran at 14:05 UTC in the user context of Gosta, who was listed in the enterprise admins security group.
Side note: Automated response playbooks
At 12:48 UTC, roughly 30 minutes after Gosta downloaded a suspicious document and executed the malicious VBA macros stored in it, Red Canary Automate began triggering notification playbooks based on indicators associated with the malicious VBA macro described above. Due to the requirements of the evaluation, these playbooks were intentionally configured only to inform the customer (in this case MITRE) that we had detected a threat. Under normal conditions, these playbooks also would have isolated the endpoint theblock.boom.box
and banned any relevant hashes, including SystemFailureReporter.exe
, long before the critical follow-on activity we’re about to describe. In all likelihood, this would have seriously hindered or entirely stymied the adversary.
Command and Control
Web Protocols, Symmetric Cryptography
At 14:05 UTC, SystemFailureReporter.exe
establishes a connection to the command and control server at 192.168.0.4
over the HTTP protocol on port 443.
Side note: Missing substep!
We failed to report to MITRE that SystemFailureReporter.exe
also made an XOR-encrypted connection to that same IP address.
Step 2
Discovery
Too many techniques to list
SystemFailureReporter.exe
initiated a sequence of discovery actions beginning at 16:05 UTC and ending at 16:10 UTC. The activity included the following commands:
Command | Function |
---|---|
Command:
| Function : This command displays the domain and user currently logged on to this host. |
Command:
| Function : This command displays the name of this host. |
Command:
| Function : This command displays the full configuration for all network adapters on this host. |
Command:
| Function : This command displays the user account information for this host’s primary domain. |
Command:
| Function : This command displays the group information for this host’s primary domain. |
Command:
| Function : This command enumerates all members of the Domain Admins group; members of this group have admin rights over the entire domain. |
Command:
| Function : This command enumerates all members of the Exchange Trusted Subsystem group; members of this group have full control over the domain root. |
Command:
| Function : This command checks the policy settings for this host’s primary domain. |
Command:
| Function : This command shows the current user’s account information. |
Command:
| Function : This command enumerates all members of the local administrators group; members of this group have administrative rights for this endpoint. |
Command:
| Function : This command displays all the active TCP and UDP connections on which the host is listening. |
Command:
| Function : This command displays all of the currently running processes. |
Command:
| Function : This command displays the information for currently running services. |
Command:
| Function : This command displays the info for the user Gosta from the domain controller on the current domain. |
Command:
| Function : This command displays the info for the SQL Admins group from the domain controller on the current domain. |
Command:
| Function : This command queries DNS to obtain the network info for the host |
The last command, nslookup WATERFALLS
, is a harbinger of what’s to come, but before we get into the next phase of this attack, we need a brief sidebar.
Side note: Detecting discovery actions
We observed the execution of SystemFailureReporter.exe
less than two hours (14:05 UTC) after Gosta opened the malicious document that initiated this incident. Another two hours passed before we started to observe the aggressive discovery actions detailed above (16:05 UTC).
If GGMS Overview.doc
had slipped past the initial preventive controls and managed to execute its hidden VBA macro, Red Canary would have detected the above discovery actions and initiated response controls to isolate the affected endpoint right then—less than four hours after Gosta logged in and well before any credential theft, lateral movement, and exfiltration activity.
Step 3
Command and Control
Ingress Tool Transfer
At 18:03 UTC, SystemFailureReporter.exe
ran via a scheduled task and wrote another binary to disk, b.exe
.
Credential Access
Windows Credential Manager
Upon analysis, our intelligence and research teams determined that the b.exe
binary was emulating a known malware strain called VALUEVAULT, all but confirming our suspicion that MITRE was in fact emulating OilRig.
Side note: Identifying OilRig!
FireEye first reported on VALUEVAULT while investigating social media phishing campaigns in 2019, and the malware is now widely associated with OilRig. VALUEVAULT is known to extract credentials from the Windows Vault and create a SQLITE database file in the AppData\Roaming
directory named fsociety.dat
to store the stolen credentials.
This is precisely what happened in the evaluation: b.exe
dumped the windows credentials stored on theblock.boom.box
into a SQLite3 database called fsociety.dat
.
At this point, our Intelligence Team was almost certain that MITRE was emulating OilRig. In a real-world scenario, there would remain some doubt about this attribution because any number of adversaries could have leveraged VALUEVAULT, written credentials to “fsociety.dat, and mimicked other indicators. However, our assessment was made easier by knowing that MITRE would be emulating a well known adversary.
Exfiltration
Exfiltration Over C2 Channel
At this point, the adversary exfiltrated fsociety.dat
from theblock.boom.box
to their C2 server (192.168.0.4
) via an HTTP post request.
Side note: We missed this!
This is probably our most glaring omission. We simply did not observe the adversary passing the fsociety.dat
file back to their C2 infrastructure, which was an important part of the intrusion. Many C2 frameworks offer adversaries the ability manipulate packet structure and effectively hide the contents of their communications during exfiltration. We were able to identify requests that likely contained the exfiltration, but complete visibility into network telemetry is a known short-coming of most EDR products. As we do with any misses during testing, we’re considering how we could improve our detection coverage to catch similar behavior in the future. Further, we support the ingestion of Network Security telemetry and alerts from products like firewalls, network detection and response, and DNS records, which would have proven helpful in fully observing this substep.
Step 4
Command and Control, Lateral Movement
Ingress Tool Transfer, Lateral Tool Transfer
At 19:33 UTC, SystemFailureReporter.exe
wrote contact.aspx
to disk. Shortly thereafter (19:43 UTC), it copied that contact.aspx
to another endpoint (technically an Exchange server) waterfalls.boom.box
, which is the same endpoint upon which the adversary performed a DNS lookup earlier as part of their discovery.
Persistence
Web Shell
This web shell provided the adversary the ability to write, download, and delete files from disk, as well as execute binaries that are present on the infected endpoint. The previous image illustrated telemetry from theblock.boom.box
copying an Active Server Page Extended (.aspx
) file to waterfalls.boom.box
. The following image shows the telemetry associated with that action from the perspective of waterfalls.boom.box
:
Defense Evasion
Hidden Files and Directories, File Deletion
Meanwhile, back on theblock.boom.box
at 19:48 UTC, the adversary deleted contact.aspx
from the endpoint and leveraged the attrib +h
command to hide the .aspx
file from the typical folder view and from ordinary directory listings on waterfalls.boom.box
.
Step 5
Persistence
Web Shell
On July 14 at 17:30 UTC, the adversary then leveraged the web shell to launch the Command Processor (cmd.exe
), which ran as SYSTEM (this has important implications when we get to the Mimikatz activity later). It proceeded to run multiple commands consistent with post-exploitation, which we will highlight in the coming sections. The following graphic shows w3wp.exe
spawning the command shell:
Discovery
Multiple Discovery techniques
In the space of a few minutes, the adversary used the newly spawned command shell to gather information about system network configuration, enumerate a list of active network connections, and see who had system access on the host via the ipconfig
, netstat
, and whoami
commands. The following image shows telemetry associated with the ipconfig
and netstat
commands:
Step 6
Command and Control, Credential Access
Ingress Tool Transfer
Leveraging the web shell, the adversary then wrote a binary named m64.exe
to disk at 18:33 UTC on waterfalls.boom.box
. This binary was in fact a renamed version of the credential-stealing tool Mimikatz. As we noted earlier, the command shell that spawned Mimikatz ran with system-level privileges, which granted Mimikatz the privilege level required to perform the actions described in the next section.
LSASS Memory
Immediately thereafter, the adversary leveraged Mimikatz to open a handle into the Local Security Authority Subsystem Service (lsass.exe
) and used the sekurlsa::logonPasswords
command to extract credentials from it, writing them to a file called 01.txt
, which potentially contained hashes for the following users:
gosta
tous
Mariam
vendor_domain_admin
evals_domain_admin
Exfiltration Over C2 Channel
The web shell then ran its HandlePOSTFileDownloadServer()
function to download files off disk, very likely in an effort to exfiltrate credentials from 01.txt
.
Defense Evasion
File Deletion
At 18:37, the adversary ran commands intended to delete m64.exe
and 01.txt
from waterfalls.boom.box
.
Side note: Detecting Mimikatz
We’ve developed detection analytics for different aspects of Mimikatz, at least three of which are on display in the above image. One option involves looking for processes that obtain a handle into lsass.exe
. Alternatively, you can look for module loads of dynamic link libraries like samlib.dll
or vaultcli.dll
, which you can see in the following image:
A third option is to detect specific command-line parameters that are associated with Mimikatz. In this case, those commands were: privilege::debug
and sekurlsa::logonpasswords
.
Step 7
Command and Control
Ingress Tool Transfer
Back on theblock.boom.box
on July 15 at 12:06 UTC, SystemFailureReporter.exe
wrote a copy of plink.exe
, the PuTTY command-line interface used for SSH tunneling.
Protocol Tunneling
At 12:11 UTC, the adversary leveraged plink.exe
to create an SSH tunnel and make RDP connections to their adversary-controlled machine, attempting to gain direct RDP access on waterfalls.boom.box
.
Lateral Movement, Defense Evasion
Remote Desktop Protocol, Domain Accounts
At 12:13 UTC, the adversary leveraged the stolen Gosta credentials to open an RDP session on waterfalls.boom.box
using the established SSH tunnel via plink.exe
.
Step 8
Persistence, Command and Control
Web Shell, Ingress Tool Transfer
Starting at 12:15 UTC, the adversary leveraged the web shell to write several files to disk on waterfalls.boom.box
:
ps.exe
, a renamed version of the Sysinternals toolmom64.exe
, another renamed version of Mimikatznt.dat
, an email-based C2 tool
The following images show the adversary writing each of these tools to disk.
PsExec
Nt.dat
Mimikatz
Side note: Detecting renamed instances of PsExec
As we’ve written about extensively, you can often detect renamed tools by looking for the execution of binaries where the binary’s presented filename is different from the internal filename listed in its binary metadata. In this case, the file was called ps.exe
, but the file’s internal binary name was “PsExec.”
Defense Evasion, Execution, Lateral Movement
Pass the Hash
The adversary then abused Mimikatzʼs pass-the-hash functionality and a previously extracted NTLM hash to switch the user’s context to the user “tous” at 12:18 UTC. Importantly, “Tous” is a member of the SQL Admins group, which the adversary discovered in a previous step.
Windows Command Shell
Seconds later, mom64.exe
(aka Mimikatz) opened an interactive command shell, as shown in the following image:
Lateral Tool Transfer, SMB/Windows Admin Shares, Service Execution
At 12:20 UTC, the adversary—now operating in the context of the user “tous”—leveraged PsExec to execute commands on a new host, a SQL server called endofroad1.boom.box
, before moving nt.dat
to the new host.
Step 9
Discovery
File and Directory Discovery
Next, the adversary performed discovery of backup files on the SQL server endofroad1.boom.box
.
Side note: We missed this!
We failed to include these actions in our timeline or report them to the team at MITRE. As with all misses, we’re considering how we could better detect similar behaviors in the future.
Step 10
Collection, Defense Evasion
Local Data Staging, Match Legitimate Name or Location
At 12:23 UTC, the adversary used PsExec to spawn a command shell and create a folder named “Vmware” and a file named vmware.exe
on endofroad1.boom.box
.
An analysis of vmware.exe
revealed that the binary was actually a renamed instance of the email C2 tool from earlier, nt.dat
. The adversary moved that nt.dat
file into the Vmware folder and renamed it.
Collection, Exfiltration, Defense Evasion
Data From Local System, Data Transfer Size Limits, Exfiltration Over Unencrypted Non-C2 Protocol, Obfuscated Files or Information
At 12:26 UTC, the adversary leveraged nt.dat
(masquerading as vmware.exe
) to exfiltrate a steganographic .bmp
file containing the contents of a SQL database backup. Using credentials for the user Gosta, the adversary sent an email via Microsoft Exchangeʼs API and then removed their custom tooling from disk. In the process, the adversary resized some of the data to evade detection.
Defense Evasion
Indicator Removal
Last but certainly not least, the adversary cleaned up their tracks by removing vmware.exe
and the Vmware folder from endofroad1.boom.box
at 12:26 UTC. At 12:31 the adversary removed the following files from waterfalls.boom.box
:
mom64.exe
nt.dat
ps.exe
b.exe
fsociety.dat
plink.exe
update.xml
Side note: We missed a pair of removals
We got partial credit on this step for reporting half of the removed files to MITRE. We failed to explicitly report the deletion of the VMware directory and the deletion of mom64.exe
, nt.dat
, and ps.exe
.
Conclusion
The MITRE ATT&CK Evaluations for Managed Services is a great resource for evaluating different MDR providers and determining how they would help protect your organization in a real-world intrusion. All the participants should be commended for being transparent enough to take part in the evaluation, and the majority will likely be proud of their performance.
One of Red Canary’s core values is authenticity, which is why we were so excited to hear there would be an ATT&CK Evaluation for Managed Services. We welcome readers of the evaluation to dig into the data we provided and ask any questions about how this would translate to their security program.
When evaluating the test results, please remember that a stack ranking of technique coverage is not the best way to interpret the results. Instead, we recommended focusing on the security outcomes that each vendor was able to achieve. The best resource available to discern this information (other than this document, of course) is the archive of reports, emails, and other content that MITRE made available for each participant.