After detection: teaming up to shut down a web server attack
When an adversary used encoded PowerShell to attack a web server belonging to a DFDR Consulting client, Red Canary’s detection engineering team spotted a few things that looked awry. But the story didn’t end there. A DFDR digital forensics consultant and a Red Canary detection engineer share their sides of the joint investigation.
Digital Forensics Consultant, DFDR Consulting
Detection Engineer, Red Canary
Digital forensics and incident response professionals live in a world where the only thing known for certain is that the next phone call or email can unleash a torrent of uncertainty. The morning of July 29 was no different. After getting out of bed and hopping on my recently procured Peloton spin bike for a bit, I did what so many in the COVID age are also doing: I went to the grocery store on a random day at a random time. While in the checkout line, my phone began to vibrate in my pocket. I noticed a missed call and voicemail and several new text and email messages. That was when it hit me: it was going to be one of THOSE days.
Thankfully, because my consulting firm (DFDR) partners with Red Canary to detect threats on our clients’ endpoints and automate various response actions, I was confident that things could only escalate so far.
After taking a moment to review the communications, it became apparent that this was not a typical malware alert. The alert timeline clearly illustrated that the threat actors had managed to escape a web server and were actively executing commands against the host machine to compromise and exfiltrate user credentials.
Endpoint detection and response (EDR) wasn’t exactly built for detecting web server compromises. To catch these, we typically have to rely on the the adversary performing some type of post-exploitation behavior on the host since EDR does not provide visibility into web applications. Because every web service functions differently, being able to detect anomalous web service behavior effectively at scale is a daunting task. What may look suspicious on one server in one environment may be completely normal on another server in a different environment.
In this particular instance, one of DFDR’s customers hosted an exchange server that contained a remote code execution (RCE) enabling vulnerability, which the adversary exploited to dump the memory space of, and extract credentials from the Local Security Authority Security Service (lsass.exe).
My first order of business was to contact our client’s IT team and update them on the status of the alert. Based on the host name, it was clear that the compromised web server was actually a Microsoft Exchange server. During the initial call, the IT team was instructed to temporarily suspend external access to Outlook Web Agent (OWA) and other external authentication points such as the VPN. The affected server was also temporarily placed into isolation mode (a feature of Carbon Black) until the firewall rules could be updated.
After gaining confidence that the immediate threat had been held at bay, we set off to understand the true impacts of the incident. Typically, the “Forensics” playbook found within Automate provides a wealth of information about the system and areas where malware likes to hide persistence mechanisms. With this alert, the IIS logs were going to hold the needed answers because Carbon Black does not have visibility into the inner workings of IIS.
A Live Response session preserved the IIS logs on the host and transferred them to DFDR for review. Thankfully, there were approximately 60 days of IIS logs available for review in this instance, clocking in at close to 60GB of data. Heading back to the alert timeline in the Red Canary portal, we were able to determine that the threat actors attempted to modify the permissions of a specific user. This proved to be incredibly useful, as it gave us a place to start the log review.
A review of the IIS logs associated with the affected user revealed a number of suspicious entries with “VIEWSTATEENCRYPTED” contained within the GET request. Drawing upon years of experience within the industry, we immediately recognized this as…something to be Googled. After ingesting some great content and cross-referencing other data points and indicators of compromise, we felt confident that this incident was attributable to an authenticated remote code execution flaw in Exchange server that Microsoft patched back in February (CVE-2020-0688).
Since Red Canary relies heavily on telemetry streamed from endpoint agents, we don’t have good visibility into IIS logs. Despite that blindspot, these kinds of compromises consistently generate a predictable variety of endpoint telemetry, so let’s walk through some of the detection logic that was helpful in surfacing this threat and discuss the malicious behaviors associated with them.
It’s never a surprise when an adversary leverages PowerShell for malicious intent, as its capabilities are endless. Sadly, many organizations still struggle with detecting some of the easy wins, which is why adversaries continue to utilize it. Even when we feel we are making strides at having complete coverage for PowerShell, research pops up leaving us to quickly verify our coverage.
Having previously done system administration, I can count on zero hands the number of times I’ve ever needed or wanted to encode any PowerShell commands/usage. There are always edge cases, but system administrators tend to want to work smarter, not harder, and that’s why simply looking for variations of encodedcommand within your environment can be effective at detecting evil.
Detecting on all encodedcommand variations is easier said than done, but even if you do not have complete coverage, having some detection logic that looks for a PowerShell command with the -enc string will make an immediate impact within your environment. There are so many PowerShell exploitation frameworks that leverage that shortened usage of -encodedcommand that the shortened -enc string alone has almost become an indicator of compromise by itself due to it being the default by most frameworks.
In this case, the IIS worker process spawned PowerShell, which might be suspicious on its own in certain environments (which we’ll discuss below), and PowerShell then ran an encoded command. Although we wouldn’t have determined this activity was malicious purely based off of the usage of -encodedcommand, had the adversary chose to not utilize encoding, then this activity would have been more difficult to alert on at scale without the additional detection logic that follows.
Another indicator within this activity that helped us quickly identify this threat was the usage of rundll32 leveraging comsvcs.dll. Comsvcs.dll has a minidumpw function that can be abused as a memory dumping technique and has been popping up more frequently as of late. Looking for instances of rundll32.exe loading comsvcs.dll to leverage the minidump function and/or rundll32.exe with a command line containing the string comsvcs.dll has been an effective way for us to identify this activity. For this technique to be successful, debug privileges are needed; in this instance, the exploit performed by the adversary provided them with requisite system-level privileges.
Injection into lsass
Since adversaries can readily change up command-line parameters, command line-based detection methods are a bit brittle. To enhance these kinds of detection analytics, it’s a good idea to look out for any indications of processes injecting into lsass.exe. With an appropriate amount of tuning, this “behavior” can be very effective at not only catching the above comsvcs.dll activity, but identifying any custom code that is utilized to perform various types of credential dumping, like you might expect to see from Mimikatz or other similar tools.
IIS spawning PowerShell
Microsoft Exchange isn’t what you typically think of for a web application, but because it does depend on IIS, the telemetry generated after a compromise will look very similar. For any third-party vendor, monitoring activity where IIS is spawning PowerShell alone doesn’t provide a lot of detection value because there are many administrative applications that we see on a daily basis that leverage PowerShell in similar ways, resulting in a lot of additional noise that can be detrimental to any security team. Additionally, this activity alone isn’t malicious by any means, but it can be helpful as supporting detection logic if you understand how your web services are expected to behave within your environment.
Now that we know the adversary leveraged an exploit to access the Exchange server and performed a bunch of actions that have been thoroughly documented by Red Canary, it’s time to examine those IIS logs.
Fortunately, in this instance, the compromised user account was tied to someone who was no longer employed by the victim organization. This meant that there was only a single interaction between this user account and IIS in the previous two months, and it occurred 48 hours prior, most likely to confirm that the credentials were valid in anticipation of future activities. The logs were also reviewed for entries containing “VIEWSTATEENCRYPTED” from other users; there were none.
With all business endeavors, even criminal ones, time is money—and this incident was no different. Shortly after initial access on the morning of July 29, the threat actors began submitting encrypted payloads via RCE. This proved to be their ultimate downfall as it caught the attention of Red Canary (see Shane’s section above). Working in tandem with Red Canary, DFDR Consulting was able to take this incident from initial access to resolution in the span of approximately four hours.
DFDR then provided a list of all IP addresses associated with suspicious IIS log entries to Red Canary. Red Canary cross-referenced this list against all the client’s telemetry data, confirming there was no ingress or egress traffic at times outside of this specific alert or from other endpoints.
We used Microsoft’s Log Parser Studio to analyze the IIS logs for valuable artifacts. An initial search for the affected username in the IIS logs returned a number of entries. The “VIEWSTATEENCRYPTED” entries are linked to commands the attacker passed back to the IIS server. The highlighted row lines up with the timestamp associated with the PowerShell command that initially caught Red Canary’s attention. This occurred a mere 16 seconds after the first attempt to interact with the host server.
What you can do now
Should you run into any similar activity on your environment, Shane and Steven recommend the following potential remediations:
User deprovisioning: User accounts weren’t disabled when an employee left the organization. A process was being used to change passwords, to what was likely a standardized string, when an employee left. A compromised user account was necessary for the threat actors to log into OWA and leverage the RCE. Remember: Every pair of active usernames and passwords in an environment is a key waiting to be found.
Patch management: The remote monitoring management (RMM) responsible for Windows Updates indicated that everything was fully patched. It turns out the RMM had a blind spot: Cumulative Updates. Threat actors would have been unable to submit encrypted commands to the server had Exchange been fully patched. Trust but verify is wisdom to live by.
Two-Factor Authentication: A number of deficiencies were identified in the rollout of two-factor authentication (2FA). It had only been deployed to the VPN and was not applied in a uniform manner. Deploy it uniformly to all external authentication points and consider its deployment internally as well.
Visibility: Your security team needs to have proper visibility into their environment if they want to have any chance of catching this activity
Detection coverage: Security teams should strive to use their visibility to develop detection coverage across activities that are likely to occur in their environment. In this case, that activity includes encoded PowerShell commands, suspicious interactions with lsass, and more. For additional detection guidance, check out Red Canary’s Threat Detection Report and Threat Detection blog series.
Privacy & Cookies Policy
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.