A lesson I learned early in my career is that technology professionals often inherit older problems. This is especially true of administrators responsible for network services and security because they inherit the biggest snowball of problems: an enterprise network. Networks often grow in ways that make them harder to secure and maintain as they age, and admins often implement new security controls only to find that something evil is already in their environment.
Red Canary’s detection engineers are no stranger to this concept. Because of our growing relationship with incident response partners, Red Canary is commonly deployed in environments where an incident is already in progress. Despite not being around to observe the initial compromise, our Cyber Incident Response Team (CIRT) can provide a holistic view of adversary tactics to accelerate remediation.
In this article, I’ll cover a scenario where we joined the hunt with one of our IR partners to stop a compromise in progress. I’ll also share detection strategies for spotting the adversarial tactics used in this incident, as well as prevention strategies that go beyond the well-worn (but still appropriate!) advice of patch management.
First Signs of Evil
When Red Canary is deployed through an IR partner, an incident handler typically briefs the detection engineering team to provide background information about the incoming incident. In this case, Red Canary’s awesome Eric Groce let us know we’d soon be jumping in to provide investigative support for an organization whose network was unusable due to malicious activity. The issues had severely impacted the organization’s business and we’d be seeing a flood of events. Shortly after, we saw the first events come through, starting with persistence mechanisms that had already been placed on devices by adversaries.
Our first sign of evil was an application whitelisting bypass technique made popular years ago by our own Casey Smith, Director of Applied Research. In this case, a Windows Scheduled Task ran, executing an instance of Regsvr32.exe to download and execute code. (This is often referred to as the squiblydoo attack. Casey has written about detecting for this attack in this Atomic Red Team post.)
The downloaded payload claimed to be a PHP script file, but turned out to be a malicious payload that further caused the Windows Command Processor (cmd.exe) to spawn PowerShell. PowerShell then executed a Base64-encoded command to download and execute a second payload: antivirus.ps1.
From here, additional PowerShell executed to retrieve code previously stored as a property of a Windows Management Instrumentation (WMI) event subscription. This is a slight departure from what we usually see in “fileless” malware execution, which commonly uses Windows Registry keys to store PowerShell payloads. The use of WMI class properties is more rare and harder for defenders to inspect and remediate.
At this point we could already see that the adversaries had established a decent amount of persistence across the organization’s endpoints. We received events from over 100 of their hosts to this effect showing Scheduled Task execution and the retrieval of WMI class properties via PowerShell.
The Mining Begins
Our next indication of evil came from a host on which WMI was used to spawn PowerShell. We can make this conclusion based on the parent of PowerShell: processes spawned via WMI have a parent of wmiprvse.exe. The PowerShell command was encoded using Base64 again, and we could see a complex script after decoding it.
The adversary had created a script that would check for new versions of their tools and attempt to download a new payload when the queried version number changed. Later in the script we also observed the adversary querying the endpoints’ active network connections, and we soon found out why. For every process with an established network connection, the malicious PowerShell script executed an instance of findstr.exe. This built-in Windows utility was used to search files for strings of text specified by the user. In this case, our adversary used it to search for a string related to cryptomining: “cryptonight.”
This odd behavior made sense once we compared it to tradecraft we’ve seen in instances of malicious cryptomining. Adversaries that deploy cryptominers as payloads typically follow a pattern:
- Download mining tools to the endpoint.
- Check the endpoint to see if miners are already running.
- End any existing miner processes.
- Start the adversary’s miner and carry on.
In this instance, findstr.exe was executed against each of the binaries running as network-connected processes. If the string “cryptonight” was contained in any of those processes, the malicious script would end that process and begin carrying out cryptomining activities.
The data observed in Carbon Black (Cb) Response didn’t show a specific binary associated with these activities other than PowerShell, so we hypothesized that the miner was loaded in a way that was invisible to Cb Response. We later confirmed this by obtaining the malicious PowerShell script and discovering it used a version of Invoke-ReflectivePEInjection (from PowerSploit) to load an XMRig miner binary into PowerShell. This binary was encoded using Base64 and stored within the script, which later extracted it dynamically at runtime. By using reflective injection, the script loaded the miner binary in a way that avoided Windows API calls like the LoadLibrary call. This technique hides data from tools that rely on Windows API calls to know which binaries and code libraries have been loaded.
Essentially, this single script contained everything it needed to deploy a miner at runtime and did so in a manner invisible to most tools that don’t incorporate some form of memory forensic analysis to inspect PowerShell’s memory space instead of trusting the API.
Time for Lateral Movement
While mining, the PowerShell script also began scanning the local network for hosts with port 445 open for Server Message Block (SMB) services. Astute readers of our blog will notice we’ve seen this pattern before. In fact, the world saw this pattern in 2017 during the WannaCry ransomware campaign and in the months following, when adversaries creatively merged WannaCry tactics and exploits with mining payloads to distribute WannaMine malware.
Alongside the mining activities, we began to observe signs of credential theft. PowerShell loaded two dynamic-link libraries (DLLs) commonly observed during credential theft attacks: samlib.dll and vaultcli.dll. Additionally, we also spotted PowerShell reading the memory space of the Windows Local Security Authority process (lsass.exe).
These behaviors closely mimic what we observe when adversaries use Mimikatz to steal or abuse password hashes. When pieced together in this context, we hypothesized that a tool like Mimikatz had been loaded into PowerShell to gather legitimate credentials, which were then used to authenticate to other endpoints and spread a cryptomining payload. This tactic clearly worked because we identified similar activity on 200 hosts within the environment.
The Possible Patient Zero
Out of the 200+ detections, only one endpoint exhibited a significant departure in the pattern of adversarial behavior. On this host, we observed odd commands spawning from a Java process. An instance of the Windows Command Processor (cmd.exe) was used to write an encoded PowerShell command into a VBScript for execution, naming the script javaupdato.vbs.
Looking back to the Java parent of this process, it appeared the command had spawned from a Java-based web server hosting an application named “WebHelpDesk.” Judging from the context provided about the server, we were able to discern its role as a Solarwinds Web Help Desk server that had been publicly accessible from the Internet. Within raw telemetry data, we dove into the Java parent and found evidence of malicious contact with the web server. An adversary had uploaded a malicious Java class file and caused the execution of cmd.exe in a manner similar to web shell activity. In turn, cmd.exe wrote a malicious script and executed it.
PowerShell: an Adversary’s Tool of Choice
Hopefully you’ve noticed a trend by now: this adversary loves to use PowerShell. In this case, the malicious VBScript spawned PowerShell to download a slightly different payload than seen on other hosts. We later found out the payload contained an exploit to escalate privileges on the web server. The exploits weren’t needed because the adversary had unwittingly gained access to run processes as the local SYSTEM account of the Web Help Desk server. When the web server was configured, it had been set to run in the security context of the highest administrator on the system!
At some point, the adversary realized this and used another tactic to trigger the download and execution of the mining/lateral movement payload seen on other hosts. This time, they used an instance of wmic.exe, feeding the process a malicious stylesheet to trigger the later download in PowerShell.
Detection Strategies: How to Spot These Techniques
Without the proper telemetry, this organization’s defenders didn’t have the data to detect these techniques until their network ground to a halt. Proper detection requires telemetry in the form of process execution monitoring. Defenders can achieve this by deploying an Endpoint Detection and Response (EDR) sensor such as Carbon Black Response or CrowdStrike Falcon. Another possibility is the usage of Sysmon from Sysinternals to create Windows Event Logs for process execution. The SMB network scanning component of this incident could be observed by using telemetry from Intrusion Detection Systems, Netflow, or other network controls.
Once telemetry sensors are in place, follow the detection strategies below to find and stop the various techniques.
Regsvr32.exe
The easiest tactic to detect from this incident is the one involving Regsvr32.exe. The tactic shown, known as the Squiblydoo attack, is fairly easily detected by identifying instances of Regsvr32.exe with web addresses and the string “scrobj.dll” in the command line.
WMI or Scheduled Tasks
Monitoring for processes spawned via WMI or Scheduled Tasks can be achieved by looking for child processes of wmiprvse.exe (WMI) or taskeng.exe (Scheduled Tasks). This detection strategy can be extremely noisy depending on the size and configuration of your organization, so some tuning will be needed to reduce false positives and emphasize the malicious alerts.
PowerShell
The best detection strategies combine process execution monitoring with Windows Event Log controls. If you upgrade all the Windows clients on your network to use PowerShell v5 instead of older versions, you can benefit from additional security and logging configurations. An excellent place to start is this resource about PowerShell logging from Black Hills Information Security. Once logs are generated that contain PowerShell commands executed, you can create detection capabilities on individual behaviors involved in malicious scripts.
Malicious Web Server Exploitation
Malicious web server exploitation can be observed by identifying suspicious processes spawning from a web server. As with WMI and Scheduled Task monitoring, this strategy can be noisy depending on the configuration of your systems. For higher-fidelity alerts, you can tune detection capabilities down to monitoring cmd.exe or other shells spawning from web servers in the account of that web server process. Most web servers are amenable to executing as an account that is highly monitored and audited, so take advantage of it!
Testing Your Detection Coverage
Want to test some of these tactics to see if your controls observe them? That’s why we created Atomic Red Team, an open source collection of small, highly portable tests mapped to the corresponding techniques in the MITRE ATT&CK framework.
Check out the following Atomic Red Team tests to see if your existing controls can detect them:
- Regsvr32.exe
- Scheduled Task
- PowerShell
- WMI Event Subscription Persistence
- Credential Dumping (Think Mimikatz)
What About Prevention Tactics?
If you’ve been thinking that parts of this incident could have been prevented with patch management, you’re right. We’ve written a number of articles addressing the importance of patch management and other IT fundamentals, and most security professionals are no stranger to this advice. Patch management is a good start, but for this post I want to dig into other prevention tactics: network segmentation and web server security.
Network Segmentation
A key defense against rapidly spread infections like WannaCry and WannaMine is to segment your network in a manner that limits communications to approved channels between segments. If SMB scanning shouldn’t be allowed from your marketing office to the rest of the network, disallow it and monitor it with network controls!
Network segmentation with proper access control lists can prevent a host from spreading evil undeterred into the rest of an enterprise network. When properly segmented, a rapidly spreading infection will be limited to a single segment of a network rather than spreading throughout the organization as it might in a flat network with no barriers. This helps defenders greatly as it keeps an incident partitioned away from other working systems.
That said, this case with a Solarwinds Web Help Desk server represents a special circumstance. Solarwinds software and other IT management tools are often placed in a central area of a network where they can access systems across all network segments. This placement presents a challenge as it allows one host to become a potential vector to compromise all segments of a network. Defenders should make sure these systems are not publicly accessible via the Internet. If there are instances where you need IT management tools to be available outside your organization’s network, consider using virtual private network (VPN) solutions to provide secure, authenticated access so adversaries can’t scan and compromise these systems as easily.
Web Server Security
The secure configuration of web servers merits discussion as web applications comprise a great amount of modern enterprise software. Additionally, many of these web applications are designed for public access, increasing the risk involved. For adversaries to take advantage of web applications for remote code execution, an exploit or web shell functionality is used to take advantage of user privileges given to a web server process. Preventing the exploit of web shell functionality usually involves the testing and patching of web servers and applications, but the user privilege portion can be managed by clever administrators with an understanding of their server needs. Apache and NGINX on Linux web servers can be easily configured to use unprivileged accounts for web applications, limiting the amount of damage an exploit can inflict.
On the Windows side, this is a bit more difficult as configuration of a web server user can require more specialized site or service configuration than on Linux. The general rule of thumb should be to use an account that possesses no administrator privileges on the web server host or any other system on the network. Once you’ve selected an unprivileged account, configure controls to monitor actions executed by that account to keep an eye on your web server.
We’ve given the idea of privileged accounts a thorough lashing here, so I want to mention why it’s especially bad to run a web server as the Windows local SYSTEM account. This account has the highest level of privilege in Windows, above that of local administrator accounts used by IT personnel to manage systems. While this account is unique per computer and doesn’t typically have privileges on other hosts, adversaries with SYSTEM access can perform any action they want on a single host under the account. This includes the possibility of stealing credentials cached on a system that can be used to move laterally across the network. In theory, the exploitation of a service running as SYSTEM could lead adversaries to obtain cached credentials for a domain administrator (that can move laterally) and potentially lock legitimate users out of a system to prevent remediation.
Be conscious of web servers you’ve made available to the Internet, and harden their configurations with the least amount of privilege needed. Additionally, if a web server provides access to IT management tools, keep it away from public access at all costs. Secure external access to those management resources by requiring the use of a VPN with authentication.
Key Takeaways
We hope that taking a deep dive into this incident helps defenders understand why it’s important to take proactive steps to improve detection capabilities and prevention. And if you need help improving detection coverage, Red Canary is here!
Related Articles
The Trainman’s Guide to overlooked entry points in Microsoft Azure